EVOLUTION-MANAGER
Edit File: analysis.html
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <title>analysis module — Whoosh 2.7.4 documentation</title> <link rel="stylesheet" href="../_static/default.css" type="text/css" /> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" /> <script type="text/javascript"> var DOCUMENTATION_OPTIONS = { URL_ROOT: '../', VERSION: '2.7.4', COLLAPSE_INDEX: false, FILE_SUFFIX: '.html', HAS_SOURCE: true }; </script> <script type="text/javascript" src="../_static/jquery.js"></script> <script type="text/javascript" src="../_static/underscore.js"></script> <script type="text/javascript" src="../_static/doctools.js"></script> <link rel="top" title="Whoosh 2.7.4 documentation" href="../index.html" /> <link rel="up" title="Whoosh API" href="api.html" /> <link rel="next" title="codec.base module" href="codec/base.html" /> <link rel="prev" title="Whoosh API" href="api.html" /> </head> <body> <div class="related"> <h3>Navigation</h3> <ul> <li class="right" style="margin-right: 10px"> <a href="../genindex.html" title="General Index" accesskey="I">index</a></li> <li class="right" > <a href="../py-modindex.html" title="Python Module Index" >modules</a> |</li> <li class="right" > <a href="codec/base.html" title="codec.base module" accesskey="N">next</a> |</li> <li class="right" > <a href="api.html" title="Whoosh API" accesskey="P">previous</a> |</li> <li><a href="../index.html">Whoosh 2.7.4 documentation</a> »</li> <li><a href="api.html" accesskey="U">Whoosh API</a> »</li> </ul> </div> <div class="document"> <div class="documentwrapper"> <div class="bodywrapper"> <div class="body"> <div class="section" id="module-whoosh.analysis"> <span id="analysis-module"></span><h1><tt class="docutils literal"><span class="pre">analysis</span></tt> module<a class="headerlink" href="#module-whoosh.analysis" title="Permalink to this headline">¶</a></h1> <p>Classes and functions for turning a piece of text into an indexable stream of “tokens” (usually equivalent to words). There are three general classes involved in analysis:</p> <ul> <li><p class="first">Tokenizers are always at the start of the text processing pipeline. They take a string and yield Token objects (actually, the same token object over and over, for performance reasons) corresponding to the tokens (words) in the text.</p> <p>Every tokenizer is a callable that takes a string and returns an iterator of tokens.</p> </li> <li><p class="first">Filters take the tokens from the tokenizer and perform various transformations on them. For example, the LowercaseFilter converts all tokens to lowercase, which is usually necessary when indexing regular English text.</p> <p>Every filter is a callable that takes a token generator and returns a token generator.</p> </li> <li><p class="first">Analyzers are convenience functions/classes that “package up” a tokenizer and zero or more filters into a single unit. For example, the StandardAnalyzer combines a RegexTokenizer, LowercaseFilter, and StopFilter.</p> <p>Every analyzer is a callable that takes a string and returns a token iterator. (So Tokenizers can be used as Analyzers if you don’t need any filtering).</p> </li> </ul> <p>You can compose tokenizers and filters together using the <tt class="docutils literal"><span class="pre">|</span></tt> character:</p> <div class="highlight-python"><div class="highlight"><pre><span class="n">my_analyzer</span> <span class="o">=</span> <span class="n">RegexTokenizer</span><span class="p">()</span> <span class="o">|</span> <span class="n">LowercaseFilter</span><span class="p">()</span> <span class="o">|</span> <span class="n">StopFilter</span><span class="p">()</span> </pre></div> </div> <p>The first item must be a tokenizer and the rest must be filters (you can’t put a filter first or a tokenizer after the first item).</p> <div class="section" id="analyzers"> <h2>Analyzers<a class="headerlink" href="#analyzers" title="Permalink to this headline">¶</a></h2> <dl class="function"> <dt id="whoosh.analysis.IDAnalyzer"> <tt class="descclassname">whoosh.analysis.</tt><tt class="descname">IDAnalyzer</tt><big>(</big><em>lowercase=False</em><big>)</big><a class="headerlink" href="#whoosh.analysis.IDAnalyzer" title="Permalink to this definition">¶</a></dt> <dd><p>Deprecated, just use an IDTokenizer directly, with a LowercaseFilter if desired.</p> </dd></dl> <dl class="function"> <dt id="whoosh.analysis.KeywordAnalyzer"> <tt class="descclassname">whoosh.analysis.</tt><tt class="descname">KeywordAnalyzer</tt><big>(</big><em>lowercase=False</em>, <em>commas=False</em><big>)</big><a class="headerlink" href="#whoosh.analysis.KeywordAnalyzer" title="Permalink to this definition">¶</a></dt> <dd><p>Parses whitespace- or comma-separated tokens.</p> <div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">ana</span> <span class="o">=</span> <span class="n">KeywordAnalyzer</span><span class="p">()</span> <span class="gp">>>> </span><span class="p">[</span><span class="n">token</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">ana</span><span class="p">(</span><span class="s">"Hello there, this is a TEST"</span><span class="p">)]</span> <span class="go">["Hello", "there,", "this", "is", "a", "TEST"]</span> </pre></div> </div> <table class="docutils field-list" frame="void" rules="none"> <col class="field-name" /> <col class="field-body" /> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> <li><strong>lowercase</strong> – whether to lowercase the tokens.</li> <li><strong>commas</strong> – if True, items are separated by commas rather than whitespace.</li> </ul> </td> </tr> </tbody> </table> </dd></dl> <dl class="function"> <dt id="whoosh.analysis.RegexAnalyzer"> <tt class="descclassname">whoosh.analysis.</tt><tt class="descname">RegexAnalyzer</tt><big>(</big><em>expression='\\w+(\\.?\\w+)*'</em>, <em>gaps=False</em><big>)</big><a class="headerlink" href="#whoosh.analysis.RegexAnalyzer" title="Permalink to this definition">¶</a></dt> <dd><p>Deprecated, just use a RegexTokenizer directly.</p> </dd></dl> <dl class="function"> <dt id="whoosh.analysis.SimpleAnalyzer"> <tt class="descclassname">whoosh.analysis.</tt><tt class="descname">SimpleAnalyzer</tt><big>(</big><em>expression=<_sre.SRE_Pattern object at 0x7fffb5fd2230></em>, <em>gaps=False</em><big>)</big><a class="headerlink" href="#whoosh.analysis.SimpleAnalyzer" title="Permalink to this definition">¶</a></dt> <dd><p>Composes a RegexTokenizer with a LowercaseFilter.</p> <div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">ana</span> <span class="o">=</span> <span class="n">SimpleAnalyzer</span><span class="p">()</span> <span class="gp">>>> </span><span class="p">[</span><span class="n">token</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">ana</span><span class="p">(</span><span class="s">"Hello there, this is a TEST"</span><span class="p">)]</span> <span class="go">["hello", "there", "this", "is", "a", "test"]</span> </pre></div> </div> <table class="docutils field-list" frame="void" rules="none"> <col class="field-name" /> <col class="field-body" /> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> <li><strong>expression</strong> – The regular expression pattern to use to extract tokens.</li> <li><strong>gaps</strong> – If True, the tokenizer <em>splits</em> on the expression, rather than matching on the expression.</li> </ul> </td> </tr> </tbody> </table> </dd></dl> <dl class="function"> <dt id="whoosh.analysis.StandardAnalyzer"> <tt class="descclassname">whoosh.analysis.</tt><tt class="descname">StandardAnalyzer</tt><big>(</big><em>expression=<_sre.SRE_Pattern object at 0x7fffb5fd2230></em>, <em>stoplist=frozenset(['and'</em>, <em>'is'</em>, <em>'it'</em>, <em>'an'</em>, <em>'as'</em>, <em>'at'</em>, <em>'have'</em>, <em>'in'</em>, <em>'yet'</em>, <em>'if'</em>, <em>'from'</em>, <em>'for'</em>, <em>'when'</em>, <em>'by'</em>, <em>'to'</em>, <em>'you'</em>, <em>'be'</em>, <em>'we'</em>, <em>'that'</em>, <em>'may'</em>, <em>'not'</em>, <em>'with'</em>, <em>'tbd'</em>, <em>'a'</em>, <em>'on'</em>, <em>'your'</em>, <em>'this'</em>, <em>'of'</em>, <em>'us'</em>, <em>'will'</em>, <em>'can'</em>, <em>'the'</em>, <em>'or'</em>, <em>'are'])</em>, <em>minsize=2</em>, <em>maxsize=None</em>, <em>gaps=False</em><big>)</big><a class="headerlink" href="#whoosh.analysis.StandardAnalyzer" title="Permalink to this definition">¶</a></dt> <dd><p>Composes a RegexTokenizer with a LowercaseFilter and optional StopFilter.</p> <div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">ana</span> <span class="o">=</span> <span class="n">StandardAnalyzer</span><span class="p">()</span> <span class="gp">>>> </span><span class="p">[</span><span class="n">token</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">ana</span><span class="p">(</span><span class="s">"Testing is testing and testing"</span><span class="p">)]</span> <span class="go">["testing", "testing", "testing"]</span> </pre></div> </div> <table class="docutils field-list" frame="void" rules="none"> <col class="field-name" /> <col class="field-body" /> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> <li><strong>expression</strong> – The regular expression pattern to use to extract tokens.</li> <li><strong>stoplist</strong> – A list of stop words. Set this to None to disable the stop word filter.</li> <li><strong>minsize</strong> – Words smaller than this are removed from the stream.</li> <li><strong>maxsize</strong> – Words longer that this are removed from the stream.</li> <li><strong>gaps</strong> – If True, the tokenizer <em>splits</em> on the expression, rather than matching on the expression.</li> </ul> </td> </tr> </tbody> </table> </dd></dl> <dl class="function"> <dt id="whoosh.analysis.StemmingAnalyzer"> <tt class="descclassname">whoosh.analysis.</tt><tt class="descname">StemmingAnalyzer</tt><big>(</big><em>expression=<_sre.SRE_Pattern object at 0x7fffb5fd2230></em>, <em>stoplist=frozenset(['and'</em>, <em>'is'</em>, <em>'it'</em>, <em>'an'</em>, <em>'as'</em>, <em>'at'</em>, <em>'have'</em>, <em>'in'</em>, <em>'yet'</em>, <em>'if'</em>, <em>'from'</em>, <em>'for'</em>, <em>'when'</em>, <em>'by'</em>, <em>'to'</em>, <em>'you'</em>, <em>'be'</em>, <em>'we'</em>, <em>'that'</em>, <em>'may'</em>, <em>'not'</em>, <em>'with'</em>, <em>'tbd'</em>, <em>'a'</em>, <em>'on'</em>, <em>'your'</em>, <em>'this'</em>, <em>'of'</em>, <em>'us'</em>, <em>'will'</em>, <em>'can'</em>, <em>'the'</em>, <em>'or'</em>, <em>'are'])</em>, <em>minsize=2</em>, <em>maxsize=None</em>, <em>gaps=False</em>, <em>stemfn=<function stem at 0x7fffb5f50de8></em>, <em>ignore=None</em>, <em>cachesize=50000</em><big>)</big><a class="headerlink" href="#whoosh.analysis.StemmingAnalyzer" title="Permalink to this definition">¶</a></dt> <dd><p>Composes a RegexTokenizer with a lower case filter, an optional stop filter, and a stemming filter.</p> <div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">ana</span> <span class="o">=</span> <span class="n">StemmingAnalyzer</span><span class="p">()</span> <span class="gp">>>> </span><span class="p">[</span><span class="n">token</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">ana</span><span class="p">(</span><span class="s">"Testing is testing and testing"</span><span class="p">)]</span> <span class="go">["test", "test", "test"]</span> </pre></div> </div> <table class="docutils field-list" frame="void" rules="none"> <col class="field-name" /> <col class="field-body" /> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> <li><strong>expression</strong> – The regular expression pattern to use to extract tokens.</li> <li><strong>stoplist</strong> – A list of stop words. Set this to None to disable the stop word filter.</li> <li><strong>minsize</strong> – Words smaller than this are removed from the stream.</li> <li><strong>maxsize</strong> – Words longer that this are removed from the stream.</li> <li><strong>gaps</strong> – If True, the tokenizer <em>splits</em> on the expression, rather than matching on the expression.</li> <li><strong>ignore</strong> – a set of words to not stem.</li> <li><strong>cachesize</strong> – the maximum number of stemmed words to cache. The larger this number, the faster stemming will be but the more memory it will use. Use None for no cache, or -1 for an unbounded cache.</li> </ul> </td> </tr> </tbody> </table> </dd></dl> <dl class="function"> <dt id="whoosh.analysis.FancyAnalyzer"> <tt class="descclassname">whoosh.analysis.</tt><tt class="descname">FancyAnalyzer</tt><big>(</big><em>expression='\\s+'</em>, <em>stoplist=frozenset(['and'</em>, <em>'is'</em>, <em>'it'</em>, <em>'an'</em>, <em>'as'</em>, <em>'at'</em>, <em>'have'</em>, <em>'in'</em>, <em>'yet'</em>, <em>'if'</em>, <em>'from'</em>, <em>'for'</em>, <em>'when'</em>, <em>'by'</em>, <em>'to'</em>, <em>'you'</em>, <em>'be'</em>, <em>'we'</em>, <em>'that'</em>, <em>'may'</em>, <em>'not'</em>, <em>'with'</em>, <em>'tbd'</em>, <em>'a'</em>, <em>'on'</em>, <em>'your'</em>, <em>'this'</em>, <em>'of'</em>, <em>'us'</em>, <em>'will'</em>, <em>'can'</em>, <em>'the'</em>, <em>'or'</em>, <em>'are'])</em>, <em>minsize=2</em>, <em>maxsize=None</em>, <em>gaps=True</em>, <em>splitwords=True</em>, <em>splitnums=True</em>, <em>mergewords=False</em>, <em>mergenums=False</em><big>)</big><a class="headerlink" href="#whoosh.analysis.FancyAnalyzer" title="Permalink to this definition">¶</a></dt> <dd><p>Composes a RegexTokenizer with an IntraWordFilter, LowercaseFilter, and StopFilter.</p> <div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">ana</span> <span class="o">=</span> <span class="n">FancyAnalyzer</span><span class="p">()</span> <span class="gp">>>> </span><span class="p">[</span><span class="n">token</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">ana</span><span class="p">(</span><span class="s">"Should I call getInt or get_real?"</span><span class="p">)]</span> <span class="go">["should", "call", "getInt", "get", "int", "get_real", "get", "real"]</span> </pre></div> </div> <table class="docutils field-list" frame="void" rules="none"> <col class="field-name" /> <col class="field-body" /> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> <li><strong>expression</strong> – The regular expression pattern to use to extract tokens.</li> <li><strong>stoplist</strong> – A list of stop words. Set this to None to disable the stop word filter.</li> <li><strong>minsize</strong> – Words smaller than this are removed from the stream.</li> <li><strong>maxsize</strong> – Words longer that this are removed from the stream.</li> <li><strong>gaps</strong> – If True, the tokenizer <em>splits</em> on the expression, rather than matching on the expression.</li> </ul> </td> </tr> </tbody> </table> </dd></dl> <dl class="function"> <dt id="whoosh.analysis.NgramAnalyzer"> <tt class="descclassname">whoosh.analysis.</tt><tt class="descname">NgramAnalyzer</tt><big>(</big><em>minsize</em>, <em>maxsize=None</em><big>)</big><a class="headerlink" href="#whoosh.analysis.NgramAnalyzer" title="Permalink to this definition">¶</a></dt> <dd><p>Composes an NgramTokenizer and a LowercaseFilter.</p> <div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">ana</span> <span class="o">=</span> <span class="n">NgramAnalyzer</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span> <span class="gp">>>> </span><span class="p">[</span><span class="n">token</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">ana</span><span class="p">(</span><span class="s">"hi there"</span><span class="p">)]</span> <span class="go">["hi t", "i th", " the", "ther", "here"]</span> </pre></div> </div> </dd></dl> <dl class="function"> <dt id="whoosh.analysis.NgramWordAnalyzer"> <tt class="descclassname">whoosh.analysis.</tt><tt class="descname">NgramWordAnalyzer</tt><big>(</big><em>minsize</em>, <em>maxsize=None</em>, <em>tokenizer=None</em>, <em>at=None</em><big>)</big><a class="headerlink" href="#whoosh.analysis.NgramWordAnalyzer" title="Permalink to this definition">¶</a></dt> <dd></dd></dl> <dl class="function"> <dt id="whoosh.analysis.LanguageAnalyzer"> <tt class="descclassname">whoosh.analysis.</tt><tt class="descname">LanguageAnalyzer</tt><big>(</big><em>lang</em>, <em>expression=<_sre.SRE_Pattern object at 0x7fffb5fd2230></em>, <em>gaps=False</em>, <em>cachesize=50000</em><big>)</big><a class="headerlink" href="#whoosh.analysis.LanguageAnalyzer" title="Permalink to this definition">¶</a></dt> <dd><p>Configures a simple analyzer for the given language, with a LowercaseFilter, StopFilter, and StemFilter.</p> <div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">ana</span> <span class="o">=</span> <span class="n">LanguageAnalyzer</span><span class="p">(</span><span class="s">"es"</span><span class="p">)</span> <span class="gp">>>> </span><span class="p">[</span><span class="n">token</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">ana</span><span class="p">(</span><span class="s">"Por el mar corren las liebres"</span><span class="p">)]</span> <span class="go">['mar', 'corr', 'liebr']</span> </pre></div> </div> <p>The list of available languages is in <cite>whoosh.lang.languages</cite>. You can use <tt class="xref py py-func docutils literal"><span class="pre">whoosh.lang.has_stemmer()</span></tt> and <tt class="xref py py-func docutils literal"><span class="pre">whoosh.lang.has_stopwords()</span></tt> to check if a given language has a stemming function and/or stop word list available.</p> <table class="docutils field-list" frame="void" rules="none"> <col class="field-name" /> <col class="field-body" /> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> <li><strong>expression</strong> – The regular expression pattern to use to extract tokens.</li> <li><strong>gaps</strong> – If True, the tokenizer <em>splits</em> on the expression, rather than matching on the expression.</li> <li><strong>cachesize</strong> – the maximum number of stemmed words to cache. The larger this number, the faster stemming will be but the more memory it will use.</li> </ul> </td> </tr> </tbody> </table> </dd></dl> </div> <div class="section" id="tokenizers"> <h2>Tokenizers<a class="headerlink" href="#tokenizers" title="Permalink to this headline">¶</a></h2> <dl class="class"> <dt id="whoosh.analysis.IDTokenizer"> <em class="property">class </em><tt class="descclassname">whoosh.analysis.</tt><tt class="descname">IDTokenizer</tt><a class="headerlink" href="#whoosh.analysis.IDTokenizer" title="Permalink to this definition">¶</a></dt> <dd><p>Yields the entire input string as a single token. For use in indexed but untokenized fields, such as a document’s path.</p> <div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">idt</span> <span class="o">=</span> <span class="n">IDTokenizer</span><span class="p">()</span> <span class="gp">>>> </span><span class="p">[</span><span class="n">token</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">idt</span><span class="p">(</span><span class="s">"/a/b 123 alpha"</span><span class="p">)]</span> <span class="go">["/a/b 123 alpha"]</span> </pre></div> </div> </dd></dl> <dl class="class"> <dt id="whoosh.analysis.RegexTokenizer"> <em class="property">class </em><tt class="descclassname">whoosh.analysis.</tt><tt class="descname">RegexTokenizer</tt><big>(</big><em>expression=<_sre.SRE_Pattern object at 0x7fffb5fd2230></em>, <em>gaps=False</em><big>)</big><a class="headerlink" href="#whoosh.analysis.RegexTokenizer" title="Permalink to this definition">¶</a></dt> <dd><p>Uses a regular expression to extract tokens from text.</p> <div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">rex</span> <span class="o">=</span> <span class="n">RegexTokenizer</span><span class="p">()</span> <span class="gp">>>> </span><span class="p">[</span><span class="n">token</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">rex</span><span class="p">(</span><span class="n">u</span><span class="p">(</span><span class="s">"hi there 3.141 big-time under_score"</span><span class="p">))]</span> <span class="go">["hi", "there", "3.141", "big", "time", "under_score"]</span> </pre></div> </div> <table class="docutils field-list" frame="void" rules="none"> <col class="field-name" /> <col class="field-body" /> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> <li><strong>expression</strong> – A regular expression object or string. Each match of the expression equals a token. Group 0 (the entire matched text) is used as the text of the token. If you require more complicated handling of the expression match, simply write your own tokenizer.</li> <li><strong>gaps</strong> – If True, the tokenizer <em>splits</em> on the expression, rather than matching on the expression.</li> </ul> </td> </tr> </tbody> </table> </dd></dl> <dl class="class"> <dt id="whoosh.analysis.CharsetTokenizer"> <em class="property">class </em><tt class="descclassname">whoosh.analysis.</tt><tt class="descname">CharsetTokenizer</tt><big>(</big><em>charmap</em><big>)</big><a class="headerlink" href="#whoosh.analysis.CharsetTokenizer" title="Permalink to this definition">¶</a></dt> <dd><p>Tokenizes and translates text according to a character mapping object. Characters that map to None are considered token break characters. For all other characters the map is used to translate the character. This is useful for case and accent folding.</p> <p>This tokenizer loops character-by-character and so will likely be much slower than <a class="reference internal" href="#whoosh.analysis.RegexTokenizer" title="whoosh.analysis.RegexTokenizer"><tt class="xref py py-class docutils literal"><span class="pre">RegexTokenizer</span></tt></a>.</p> <p>One way to get a character mapping object is to convert a Sphinx charset table file using <a class="reference internal" href="support/charset.html#whoosh.support.charset.charset_table_to_dict" title="whoosh.support.charset.charset_table_to_dict"><tt class="xref py py-func docutils literal"><span class="pre">whoosh.support.charset.charset_table_to_dict()</span></tt></a>.</p> <div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">whoosh.support.charset</span> <span class="kn">import</span> <span class="n">charset_table_to_dict</span> <span class="gp">>>> </span><span class="kn">from</span> <span class="nn">whoosh.support.charset</span> <span class="kn">import</span> <span class="n">default_charset</span> <span class="gp">>>> </span><span class="n">charmap</span> <span class="o">=</span> <span class="n">charset_table_to_dict</span><span class="p">(</span><span class="n">default_charset</span><span class="p">)</span> <span class="gp">>>> </span><span class="n">chtokenizer</span> <span class="o">=</span> <span class="n">CharsetTokenizer</span><span class="p">(</span><span class="n">charmap</span><span class="p">)</span> <span class="gp">>>> </span><span class="p">[</span><span class="n">t</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="n">chtokenizer</span><span class="p">(</span><span class="s">u'Stra</span><span class="se">\xdf</span><span class="s">e ABC'</span><span class="p">)]</span> <span class="go">[u'strase', u'abc']</span> </pre></div> </div> <p>The Sphinx charset table format is described at <a class="reference external" href="http://www.sphinxsearch.com/docs/current.html#conf-charset-table">http://www.sphinxsearch.com/docs/current.html#conf-charset-table</a>.</p> <table class="docutils field-list" frame="void" rules="none"> <col class="field-name" /> <col class="field-body" /> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><strong>charmap</strong> – a mapping from integer character numbers to unicode characters, as used by the unicode.translate() method.</td> </tr> </tbody> </table> </dd></dl> <dl class="function"> <dt id="whoosh.analysis.SpaceSeparatedTokenizer"> <tt class="descclassname">whoosh.analysis.</tt><tt class="descname">SpaceSeparatedTokenizer</tt><big>(</big><big>)</big><a class="headerlink" href="#whoosh.analysis.SpaceSeparatedTokenizer" title="Permalink to this definition">¶</a></dt> <dd><p>Returns a RegexTokenizer that splits tokens by whitespace.</p> <div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">sst</span> <span class="o">=</span> <span class="n">SpaceSeparatedTokenizer</span><span class="p">()</span> <span class="gp">>>> </span><span class="p">[</span><span class="n">token</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">sst</span><span class="p">(</span><span class="s">"hi there big-time, what's up"</span><span class="p">)]</span> <span class="go">["hi", "there", "big-time,", "what's", "up"]</span> </pre></div> </div> </dd></dl> <dl class="function"> <dt id="whoosh.analysis.CommaSeparatedTokenizer"> <tt class="descclassname">whoosh.analysis.</tt><tt class="descname">CommaSeparatedTokenizer</tt><big>(</big><big>)</big><a class="headerlink" href="#whoosh.analysis.CommaSeparatedTokenizer" title="Permalink to this definition">¶</a></dt> <dd><p>Splits tokens by commas.</p> <p>Note that the tokenizer calls unicode.strip() on each match of the regular expression.</p> <div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">cst</span> <span class="o">=</span> <span class="n">CommaSeparatedTokenizer</span><span class="p">()</span> <span class="gp">>>> </span><span class="p">[</span><span class="n">token</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">cst</span><span class="p">(</span><span class="s">"hi there, what's , up"</span><span class="p">)]</span> <span class="go">["hi there", "what's", "up"]</span> </pre></div> </div> </dd></dl> <dl class="class"> <dt id="whoosh.analysis.NgramTokenizer"> <em class="property">class </em><tt class="descclassname">whoosh.analysis.</tt><tt class="descname">NgramTokenizer</tt><big>(</big><em>minsize</em>, <em>maxsize=None</em><big>)</big><a class="headerlink" href="#whoosh.analysis.NgramTokenizer" title="Permalink to this definition">¶</a></dt> <dd><p>Splits input text into N-grams instead of words.</p> <div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">ngt</span> <span class="o">=</span> <span class="n">NgramTokenizer</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span> <span class="gp">>>> </span><span class="p">[</span><span class="n">token</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">ngt</span><span class="p">(</span><span class="s">"hi there"</span><span class="p">)]</span> <span class="go">["hi t", "i th", " the", "ther", "here"]</span> </pre></div> </div> <p>Note that this tokenizer does NOT use a regular expression to extract words, so the grams emitted by it will contain whitespace, punctuation, etc. You may want to massage the input or add a custom filter to this tokenizer’s output.</p> <p>Alternatively, if you only want sub-word grams without whitespace, you could combine a RegexTokenizer with NgramFilter instead.</p> <table class="docutils field-list" frame="void" rules="none"> <col class="field-name" /> <col class="field-body" /> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> <li><strong>minsize</strong> – The minimum size of the N-grams.</li> <li><strong>maxsize</strong> – The maximum size of the N-grams. If you omit this parameter, maxsize == minsize.</li> </ul> </td> </tr> </tbody> </table> </dd></dl> <dl class="class"> <dt id="whoosh.analysis.PathTokenizer"> <em class="property">class </em><tt class="descclassname">whoosh.analysis.</tt><tt class="descname">PathTokenizer</tt><big>(</big><em>expression='[^/]+'</em><big>)</big><a class="headerlink" href="#whoosh.analysis.PathTokenizer" title="Permalink to this definition">¶</a></dt> <dd><p>A simple tokenizer that given a string <tt class="docutils literal"><span class="pre">"/a/b/c"</span></tt> yields tokens <tt class="docutils literal"><span class="pre">["/a",</span> <span class="pre">"/a/b",</span> <span class="pre">"/a/b/c"]</span></tt>.</p> </dd></dl> </div> <div class="section" id="filters"> <h2>Filters<a class="headerlink" href="#filters" title="Permalink to this headline">¶</a></h2> <dl class="class"> <dt id="whoosh.analysis.PassFilter"> <em class="property">class </em><tt class="descclassname">whoosh.analysis.</tt><tt class="descname">PassFilter</tt><a class="headerlink" href="#whoosh.analysis.PassFilter" title="Permalink to this definition">¶</a></dt> <dd><p>An identity filter: passes the tokens through untouched.</p> </dd></dl> <dl class="class"> <dt id="whoosh.analysis.LoggingFilter"> <em class="property">class </em><tt class="descclassname">whoosh.analysis.</tt><tt class="descname">LoggingFilter</tt><big>(</big><em>logger=None</em><big>)</big><a class="headerlink" href="#whoosh.analysis.LoggingFilter" title="Permalink to this definition">¶</a></dt> <dd><p>Prints the contents of every filter that passes through as a debug log entry.</p> <table class="docutils field-list" frame="void" rules="none"> <col class="field-name" /> <col class="field-body" /> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><strong>target</strong> – the logger to use. If omitted, the “whoosh.analysis” logger is used.</td> </tr> </tbody> </table> </dd></dl> <dl class="class"> <dt id="whoosh.analysis.MultiFilter"> <em class="property">class </em><tt class="descclassname">whoosh.analysis.</tt><tt class="descname">MultiFilter</tt><big>(</big><em>**kwargs</em><big>)</big><a class="headerlink" href="#whoosh.analysis.MultiFilter" title="Permalink to this definition">¶</a></dt> <dd><p>Chooses one of two or more sub-filters based on the ‘mode’ attribute of the token stream.</p> <p>Use keyword arguments to associate mode attribute values with instantiated filters.</p> <div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">iwf_for_index</span> <span class="o">=</span> <span class="n">IntraWordFilter</span><span class="p">(</span><span class="n">mergewords</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">mergenums</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span> <span class="gp">>>> </span><span class="n">iwf_for_query</span> <span class="o">=</span> <span class="n">IntraWordFilter</span><span class="p">(</span><span class="n">mergewords</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">mergenums</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span> <span class="gp">>>> </span><span class="n">mf</span> <span class="o">=</span> <span class="n">MultiFilter</span><span class="p">(</span><span class="n">index</span><span class="o">=</span><span class="n">iwf_for_index</span><span class="p">,</span> <span class="n">query</span><span class="o">=</span><span class="n">iwf_for_query</span><span class="p">)</span> </pre></div> </div> <p>This class expects that the value of the mode attribute is consistent among all tokens in a token stream.</p> </dd></dl> <dl class="class"> <dt id="whoosh.analysis.TeeFilter"> <em class="property">class </em><tt class="descclassname">whoosh.analysis.</tt><tt class="descname">TeeFilter</tt><big>(</big><em>*filters</em><big>)</big><a class="headerlink" href="#whoosh.analysis.TeeFilter" title="Permalink to this definition">¶</a></dt> <dd><p>Interleaves the results of two or more filters (or filter chains).</p> <p>NOTE: because it needs to create copies of each token for each sub-filter, this filter is quite slow.</p> <div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">target</span> <span class="o">=</span> <span class="s">"ALFA BRAVO CHARLIE"</span> <span class="gp">>>> </span><span class="c"># In one branch, we'll lower-case the tokens</span> <span class="gp">>>> </span><span class="n">f1</span> <span class="o">=</span> <span class="n">LowercaseFilter</span><span class="p">()</span> <span class="gp">>>> </span><span class="c"># In the other branch, we'll reverse the tokens</span> <span class="gp">>>> </span><span class="n">f2</span> <span class="o">=</span> <span class="n">ReverseTextFilter</span><span class="p">()</span> <span class="gp">>>> </span><span class="n">ana</span> <span class="o">=</span> <span class="n">RegexTokenizer</span><span class="p">(</span><span class="s">r"\S+"</span><span class="p">)</span> <span class="o">|</span> <span class="n">TeeFilter</span><span class="p">(</span><span class="n">f1</span><span class="p">,</span> <span class="n">f2</span><span class="p">)</span> <span class="gp">>>> </span><span class="p">[</span><span class="n">token</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">ana</span><span class="p">(</span><span class="n">target</span><span class="p">)]</span> <span class="go">["alfa", "AFLA", "bravo", "OVARB", "charlie", "EILRAHC"]</span> </pre></div> </div> <p>To combine the incoming token stream with the output of a filter chain, use <tt class="docutils literal"><span class="pre">TeeFilter</span></tt> and make one of the filters a <a class="reference internal" href="#whoosh.analysis.PassFilter" title="whoosh.analysis.PassFilter"><tt class="xref py py-class docutils literal"><span class="pre">PassFilter</span></tt></a>.</p> <div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">f1</span> <span class="o">=</span> <span class="n">PassFilter</span><span class="p">()</span> <span class="gp">>>> </span><span class="n">f2</span> <span class="o">=</span> <span class="n">BiWordFilter</span><span class="p">()</span> <span class="gp">>>> </span><span class="n">ana</span> <span class="o">=</span> <span class="n">RegexTokenizer</span><span class="p">(</span><span class="s">r"\S+"</span><span class="p">)</span> <span class="o">|</span> <span class="n">TeeFilter</span><span class="p">(</span><span class="n">f1</span><span class="p">,</span> <span class="n">f2</span><span class="p">)</span> <span class="o">|</span> <span class="n">LowercaseFilter</span><span class="p">()</span> <span class="gp">>>> </span><span class="p">[</span><span class="n">token</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">ana</span><span class="p">(</span><span class="n">target</span><span class="p">)]</span> <span class="go">["alfa", "alfa-bravo", "bravo", "bravo-charlie", "charlie"]</span> </pre></div> </div> </dd></dl> <dl class="class"> <dt id="whoosh.analysis.ReverseTextFilter"> <em class="property">class </em><tt class="descclassname">whoosh.analysis.</tt><tt class="descname">ReverseTextFilter</tt><a class="headerlink" href="#whoosh.analysis.ReverseTextFilter" title="Permalink to this definition">¶</a></dt> <dd><p>Reverses the text of each token.</p> <div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">ana</span> <span class="o">=</span> <span class="n">RegexTokenizer</span><span class="p">()</span> <span class="o">|</span> <span class="n">ReverseTextFilter</span><span class="p">()</span> <span class="gp">>>> </span><span class="p">[</span><span class="n">token</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">ana</span><span class="p">(</span><span class="s">"hello there"</span><span class="p">)]</span> <span class="go">["olleh", "ereht"]</span> </pre></div> </div> </dd></dl> <dl class="class"> <dt id="whoosh.analysis.LowercaseFilter"> <em class="property">class </em><tt class="descclassname">whoosh.analysis.</tt><tt class="descname">LowercaseFilter</tt><a class="headerlink" href="#whoosh.analysis.LowercaseFilter" title="Permalink to this definition">¶</a></dt> <dd><p>Uses unicode.lower() to lowercase token text.</p> <div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">rext</span> <span class="o">=</span> <span class="n">RegexTokenizer</span><span class="p">()</span> <span class="gp">>>> </span><span class="n">stream</span> <span class="o">=</span> <span class="n">rext</span><span class="p">(</span><span class="s">"This is a TEST"</span><span class="p">)</span> <span class="gp">>>> </span><span class="p">[</span><span class="n">token</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">LowercaseFilter</span><span class="p">(</span><span class="n">stream</span><span class="p">)]</span> <span class="go">["this", "is", "a", "test"]</span> </pre></div> </div> </dd></dl> <dl class="class"> <dt id="whoosh.analysis.StripFilter"> <em class="property">class </em><tt class="descclassname">whoosh.analysis.</tt><tt class="descname">StripFilter</tt><a class="headerlink" href="#whoosh.analysis.StripFilter" title="Permalink to this definition">¶</a></dt> <dd><p>Calls unicode.strip() on the token text.</p> </dd></dl> <dl class="class"> <dt id="whoosh.analysis.StopFilter"> <em class="property">class </em><tt class="descclassname">whoosh.analysis.</tt><tt class="descname">StopFilter</tt><big>(</big><em>stoplist=frozenset(['and'</em>, <em>'is'</em>, <em>'it'</em>, <em>'an'</em>, <em>'as'</em>, <em>'at'</em>, <em>'have'</em>, <em>'in'</em>, <em>'yet'</em>, <em>'if'</em>, <em>'from'</em>, <em>'for'</em>, <em>'when'</em>, <em>'by'</em>, <em>'to'</em>, <em>'you'</em>, <em>'be'</em>, <em>'we'</em>, <em>'that'</em>, <em>'may'</em>, <em>'not'</em>, <em>'with'</em>, <em>'tbd'</em>, <em>'a'</em>, <em>'on'</em>, <em>'your'</em>, <em>'this'</em>, <em>'of'</em>, <em>'us'</em>, <em>'will'</em>, <em>'can'</em>, <em>'the'</em>, <em>'or'</em>, <em>'are'])</em>, <em>minsize=2</em>, <em>maxsize=None</em>, <em>renumber=True</em>, <em>lang=None</em><big>)</big><a class="headerlink" href="#whoosh.analysis.StopFilter" title="Permalink to this definition">¶</a></dt> <dd><p>Marks “stop” words (words too common to index) in the stream (and by default removes them).</p> <p>Make sure you precede this filter with a <a class="reference internal" href="#whoosh.analysis.LowercaseFilter" title="whoosh.analysis.LowercaseFilter"><tt class="xref py py-class docutils literal"><span class="pre">LowercaseFilter</span></tt></a>.</p> <div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">stopper</span> <span class="o">=</span> <span class="n">RegexTokenizer</span><span class="p">()</span> <span class="o">|</span> <span class="n">StopFilter</span><span class="p">()</span> <span class="gp">>>> </span><span class="p">[</span><span class="n">token</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">stopper</span><span class="p">(</span><span class="s">u"this is a test"</span><span class="p">)]</span> <span class="go">["test"]</span> <span class="gp">>>> </span><span class="n">es_stopper</span> <span class="o">=</span> <span class="n">RegexTokenizer</span><span class="p">()</span> <span class="o">|</span> <span class="n">StopFilter</span><span class="p">(</span><span class="n">lang</span><span class="o">=</span><span class="s">"es"</span><span class="p">)</span> <span class="gp">>>> </span><span class="p">[</span><span class="n">token</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">es_stopper</span><span class="p">(</span><span class="s">u"el lapiz es en la mesa"</span><span class="p">)]</span> <span class="go">["lapiz", "mesa"]</span> </pre></div> </div> <p>The list of available languages is in <cite>whoosh.lang.languages</cite>. You can use <tt class="xref py py-func docutils literal"><span class="pre">whoosh.lang.has_stopwords()</span></tt> to check if a given language has a stop word list available.</p> <table class="docutils field-list" frame="void" rules="none"> <col class="field-name" /> <col class="field-body" /> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> <li><strong>stoplist</strong> – A collection of words to remove from the stream. This is converted to a frozenset. The default is a list of common English stop words.</li> <li><strong>minsize</strong> – The minimum length of token texts. Tokens with text smaller than this will be stopped. The default is 2.</li> <li><strong>maxsize</strong> – The maximum length of token texts. Tokens with text larger than this will be stopped. Use None to allow any length.</li> <li><strong>renumber</strong> – Change the ‘pos’ attribute of unstopped tokens to reflect their position with the stopped words removed.</li> <li><strong>lang</strong> – Automatically get a list of stop words for the given language</li> </ul> </td> </tr> </tbody> </table> </dd></dl> <dl class="class"> <dt id="whoosh.analysis.StemFilter"> <em class="property">class </em><tt class="descclassname">whoosh.analysis.</tt><tt class="descname">StemFilter</tt><big>(</big><em>stemfn=<function stem at 0x7fffb5f50de8></em>, <em>lang=None</em>, <em>ignore=None</em>, <em>cachesize=50000</em><big>)</big><a class="headerlink" href="#whoosh.analysis.StemFilter" title="Permalink to this definition">¶</a></dt> <dd><p>Stems (removes suffixes from) the text of tokens using the Porter stemming algorithm. Stemming attempts to reduce multiple forms of the same root word (for example, “rendering”, “renders”, “rendered”, etc.) to a single word in the index.</p> <div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">stemmer</span> <span class="o">=</span> <span class="n">RegexTokenizer</span><span class="p">()</span> <span class="o">|</span> <span class="n">StemFilter</span><span class="p">()</span> <span class="gp">>>> </span><span class="p">[</span><span class="n">token</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">stemmer</span><span class="p">(</span><span class="s">"fundamentally willows"</span><span class="p">)]</span> <span class="go">["fundament", "willow"]</span> </pre></div> </div> <p>You can pass your own stemming function to the StemFilter. The default is the Porter stemming algorithm for English.</p> <div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">stemfilter</span> <span class="o">=</span> <span class="n">StemFilter</span><span class="p">(</span><span class="n">stem_function</span><span class="p">)</span> </pre></div> </div> <p>You can also use one of the Snowball stemming functions by passing the <cite>lang</cite> keyword argument.</p> <div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">stemfilter</span> <span class="o">=</span> <span class="n">StemFilter</span><span class="p">(</span><span class="n">lang</span><span class="o">=</span><span class="s">"ru"</span><span class="p">)</span> </pre></div> </div> <p>The list of available languages is in <cite>whoosh.lang.languages</cite>. You can use <tt class="xref py py-func docutils literal"><span class="pre">whoosh.lang.has_stemmer()</span></tt> to check if a given language has a stemming function available.</p> <p>By default, this class wraps an LRU cache around the stemming function. The <tt class="docutils literal"><span class="pre">cachesize</span></tt> keyword argument sets the size of the cache. To make the cache unbounded (the class caches every input), use <tt class="docutils literal"><span class="pre">cachesize=-1</span></tt>. To disable caching, use <tt class="docutils literal"><span class="pre">cachesize=None</span></tt>.</p> <p>If you compile and install the py-stemmer library, the <tt class="xref py py-class docutils literal"><span class="pre">PyStemmerFilter</span></tt> provides slightly easier access to the language stemmers in that library.</p> <table class="docutils field-list" frame="void" rules="none"> <col class="field-name" /> <col class="field-body" /> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> <li><strong>stemfn</strong> – the function to use for stemming.</li> <li><strong>lang</strong> – if not None, overrides the stemfn with a language stemmer from the <tt class="docutils literal"><span class="pre">whoosh.lang.snowball</span></tt> package.</li> <li><strong>ignore</strong> – a set/list of words that should not be stemmed. This is converted into a frozenset. If you omit this argument, all tokens are stemmed.</li> <li><strong>cachesize</strong> – the maximum number of words to cache. Use <tt class="docutils literal"><span class="pre">-1</span></tt> for an unbounded cache, or <tt class="docutils literal"><span class="pre">None</span></tt> for no caching.</li> </ul> </td> </tr> </tbody> </table> </dd></dl> <dl class="class"> <dt id="whoosh.analysis.CharsetFilter"> <em class="property">class </em><tt class="descclassname">whoosh.analysis.</tt><tt class="descname">CharsetFilter</tt><big>(</big><em>charmap</em><big>)</big><a class="headerlink" href="#whoosh.analysis.CharsetFilter" title="Permalink to this definition">¶</a></dt> <dd><p>Translates the text of tokens by calling unicode.translate() using the supplied character mapping object. This is useful for case and accent folding.</p> <p>The <tt class="docutils literal"><span class="pre">whoosh.support.charset</span></tt> module has a useful map for accent folding.</p> <div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">whoosh.support.charset</span> <span class="kn">import</span> <span class="n">accent_map</span> <span class="gp">>>> </span><span class="n">retokenizer</span> <span class="o">=</span> <span class="n">RegexTokenizer</span><span class="p">()</span> <span class="gp">>>> </span><span class="n">chfilter</span> <span class="o">=</span> <span class="n">CharsetFilter</span><span class="p">(</span><span class="n">accent_map</span><span class="p">)</span> <span class="gp">>>> </span><span class="p">[</span><span class="n">t</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="n">chfilter</span><span class="p">(</span><span class="n">retokenizer</span><span class="p">(</span><span class="s">u'café'</span><span class="p">))]</span> <span class="go">[u'cafe']</span> </pre></div> </div> <p>Another way to get a character mapping object is to convert a Sphinx charset table file using <a class="reference internal" href="support/charset.html#whoosh.support.charset.charset_table_to_dict" title="whoosh.support.charset.charset_table_to_dict"><tt class="xref py py-func docutils literal"><span class="pre">whoosh.support.charset.charset_table_to_dict()</span></tt></a>.</p> <div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">whoosh.support.charset</span> <span class="kn">import</span> <span class="n">charset_table_to_dict</span> <span class="gp">>>> </span><span class="kn">from</span> <span class="nn">whoosh.support.charset</span> <span class="kn">import</span> <span class="n">default_charset</span> <span class="gp">>>> </span><span class="n">retokenizer</span> <span class="o">=</span> <span class="n">RegexTokenizer</span><span class="p">()</span> <span class="gp">>>> </span><span class="n">charmap</span> <span class="o">=</span> <span class="n">charset_table_to_dict</span><span class="p">(</span><span class="n">default_charset</span><span class="p">)</span> <span class="gp">>>> </span><span class="n">chfilter</span> <span class="o">=</span> <span class="n">CharsetFilter</span><span class="p">(</span><span class="n">charmap</span><span class="p">)</span> <span class="gp">>>> </span><span class="p">[</span><span class="n">t</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="n">chfilter</span><span class="p">(</span><span class="n">retokenizer</span><span class="p">(</span><span class="s">u'Stra</span><span class="se">\xdf</span><span class="s">e'</span><span class="p">))]</span> <span class="go">[u'strase']</span> </pre></div> </div> <p>The Sphinx charset table format is described at <a class="reference external" href="http://www.sphinxsearch.com/docs/current.html#conf-charset-table">http://www.sphinxsearch.com/docs/current.html#conf-charset-table</a>.</p> <table class="docutils field-list" frame="void" rules="none"> <col class="field-name" /> <col class="field-body" /> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><strong>charmap</strong> – a dictionary mapping from integer character numbers to unicode characters, as required by the unicode.translate() method.</td> </tr> </tbody> </table> </dd></dl> <dl class="class"> <dt id="whoosh.analysis.NgramFilter"> <em class="property">class </em><tt class="descclassname">whoosh.analysis.</tt><tt class="descname">NgramFilter</tt><big>(</big><em>minsize</em>, <em>maxsize=None</em>, <em>at=None</em><big>)</big><a class="headerlink" href="#whoosh.analysis.NgramFilter" title="Permalink to this definition">¶</a></dt> <dd><p>Splits token text into N-grams.</p> <div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">rext</span> <span class="o">=</span> <span class="n">RegexTokenizer</span><span class="p">()</span> <span class="gp">>>> </span><span class="n">stream</span> <span class="o">=</span> <span class="n">rext</span><span class="p">(</span><span class="s">"hello there"</span><span class="p">)</span> <span class="gp">>>> </span><span class="n">ngf</span> <span class="o">=</span> <span class="n">NgramFilter</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span> <span class="gp">>>> </span><span class="p">[</span><span class="n">token</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">ngf</span><span class="p">(</span><span class="n">stream</span><span class="p">)]</span> <span class="go">["hell", "ello", "ther", "here"]</span> </pre></div> </div> <table class="docutils field-list" frame="void" rules="none"> <col class="field-name" /> <col class="field-body" /> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> <li><strong>minsize</strong> – The minimum size of the N-grams.</li> <li><strong>maxsize</strong> – The maximum size of the N-grams. If you omit this parameter, maxsize == minsize.</li> <li><strong>at</strong> – If ‘start’, only take N-grams from the start of each word. if ‘end’, only take N-grams from the end of each word. Otherwise, take all N-grams from the word (the default).</li> </ul> </td> </tr> </tbody> </table> </dd></dl> <dl class="class"> <dt id="whoosh.analysis.IntraWordFilter"> <em class="property">class </em><tt class="descclassname">whoosh.analysis.</tt><tt class="descname">IntraWordFilter</tt><big>(</big><em>delims=u'-_'"()!@#$%^&*[]{}<>\|;:</em>, <em>./?`~=+'</em>, <em>splitwords=True</em>, <em>splitnums=True</em>, <em>mergewords=False</em>, <em>mergenums=False</em><big>)</big><a class="headerlink" href="#whoosh.analysis.IntraWordFilter" title="Permalink to this definition">¶</a></dt> <dd><p>Splits words into subwords and performs optional transformations on subword groups. This filter is funtionally based on yonik’s WordDelimiterFilter in Solr, but shares no code with it.</p> <ul class="simple"> <li>Split on intra-word delimiters, e.g. <cite>Wi-Fi</cite> -> <cite>Wi</cite>, <cite>Fi</cite>.</li> <li>When splitwords=True, split on case transitions, e.g. <cite>PowerShot</cite> -> <cite>Power</cite>, <cite>Shot</cite>.</li> <li>When splitnums=True, split on letter-number transitions, e.g. <cite>SD500</cite> -> <cite>SD</cite>, <cite>500</cite>.</li> <li>Leading and trailing delimiter characters are ignored.</li> <li>Trailing possesive “‘s” removed from subwords, e.g. <cite>O’Neil’s</cite> -> <cite>O</cite>, <cite>Neil</cite>.</li> </ul> <p>The mergewords and mergenums arguments turn on merging of subwords.</p> <p>When the merge arguments are false, subwords are not merged.</p> <ul class="simple"> <li><cite>PowerShot</cite> -> <cite>0</cite>:<cite>Power</cite>, <cite>1</cite>:<cite>Shot</cite> (where <cite>0</cite> and <cite>1</cite> are token positions).</li> </ul> <p>When one or both of the merge arguments are true, consecutive runs of alphabetic and/or numeric subwords are merged into an additional token with the same position as the last sub-word.</p> <ul class="simple"> <li><cite>PowerShot</cite> -> <cite>0</cite>:<cite>Power</cite>, <cite>1</cite>:<cite>Shot</cite>, <cite>1</cite>:<cite>PowerShot</cite></li> <li><cite>A’s+B’s&C’s</cite> -> <cite>0</cite>:<cite>A</cite>, <cite>1</cite>:<cite>B</cite>, <cite>2</cite>:<cite>C</cite>, <cite>2</cite>:<cite>ABC</cite></li> <li><cite>Super-Duper-XL500-42-AutoCoder!</cite> -> <cite>0</cite>:<cite>Super</cite>, <cite>1</cite>:<cite>Duper</cite>, <cite>2</cite>:<cite>XL</cite>, <cite>2</cite>:<cite>SuperDuperXL</cite>, <cite>3</cite>:<cite>500</cite>, <cite>4</cite>:<cite>42</cite>, <cite>4</cite>:<cite>50042</cite>, <cite>5</cite>:<cite>Auto</cite>, <cite>6</cite>:<cite>Coder</cite>, <cite>6</cite>:<cite>AutoCoder</cite></li> </ul> <p>When using this filter you should use a tokenizer that only splits on whitespace, so the tokenizer does not remove intra-word delimiters before this filter can see them, and put this filter before any use of LowercaseFilter.</p> <div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">rt</span> <span class="o">=</span> <span class="n">RegexTokenizer</span><span class="p">(</span><span class="s">r"\S+"</span><span class="p">)</span> <span class="gp">>>> </span><span class="n">iwf</span> <span class="o">=</span> <span class="n">IntraWordFilter</span><span class="p">()</span> <span class="gp">>>> </span><span class="n">lcf</span> <span class="o">=</span> <span class="n">LowercaseFilter</span><span class="p">()</span> <span class="gp">>>> </span><span class="n">analyzer</span> <span class="o">=</span> <span class="n">rt</span> <span class="o">|</span> <span class="n">iwf</span> <span class="o">|</span> <span class="n">lcf</span> </pre></div> </div> <p>One use for this filter is to help match different written representations of a concept. For example, if the source text contained <cite>wi-fi</cite>, you probably want <cite>wifi</cite>, <cite>WiFi</cite>, <cite>wi-fi</cite>, etc. to match. One way of doing this is to specify mergewords=True and/or mergenums=True in the analyzer used for indexing, and mergewords=False / mergenums=False in the analyzer used for querying.</p> <div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">iwf_i</span> <span class="o">=</span> <span class="n">IntraWordFilter</span><span class="p">(</span><span class="n">mergewords</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">mergenums</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> <span class="gp">>>> </span><span class="n">iwf_q</span> <span class="o">=</span> <span class="n">IntraWordFilter</span><span class="p">(</span><span class="n">mergewords</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">mergenums</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span> <span class="gp">>>> </span><span class="n">iwf</span> <span class="o">=</span> <span class="n">MultiFilter</span><span class="p">(</span><span class="n">index</span><span class="o">=</span><span class="n">iwf_i</span><span class="p">,</span> <span class="n">query</span><span class="o">=</span><span class="n">iwf_q</span><span class="p">)</span> <span class="gp">>>> </span><span class="n">analyzer</span> <span class="o">=</span> <span class="n">RegexTokenizer</span><span class="p">(</span><span class="s">r"\S+"</span><span class="p">)</span> <span class="o">|</span> <span class="n">iwf</span> <span class="o">|</span> <span class="n">LowercaseFilter</span><span class="p">()</span> </pre></div> </div> <p>(See <a class="reference internal" href="#whoosh.analysis.MultiFilter" title="whoosh.analysis.MultiFilter"><tt class="xref py py-class docutils literal"><span class="pre">MultiFilter</span></tt></a>.)</p> <table class="docutils field-list" frame="void" rules="none"> <col class="field-name" /> <col class="field-body" /> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> <li><strong>delims</strong> – a string of delimiter characters.</li> <li><strong>splitwords</strong> – if True, split at case transitions, e.g. <cite>PowerShot</cite> -> <cite>Power</cite>, <cite>Shot</cite></li> <li><strong>splitnums</strong> – if True, split at letter-number transitions, e.g. <cite>SD500</cite> -> <cite>SD</cite>, <cite>500</cite></li> <li><strong>mergewords</strong> – merge consecutive runs of alphabetic subwords into an additional token with the same position as the last subword.</li> <li><strong>mergenums</strong> – merge consecutive runs of numeric subwords into an additional token with the same position as the last subword.</li> </ul> </td> </tr> </tbody> </table> </dd></dl> <dl class="class"> <dt id="whoosh.analysis.CompoundWordFilter"> <em class="property">class </em><tt class="descclassname">whoosh.analysis.</tt><tt class="descname">CompoundWordFilter</tt><big>(</big><em>wordset</em>, <em>keep_compound=True</em><big>)</big><a class="headerlink" href="#whoosh.analysis.CompoundWordFilter" title="Permalink to this definition">¶</a></dt> <dd><p>Given a set of words (or any object with a <tt class="docutils literal"><span class="pre">__contains__</span></tt> method), break any tokens in the stream that are composites of words in the word set into their individual parts.</p> <p>Given the correct set of words, this filter can break apart run-together words and trademarks (e.g. “turbosquid”, “applescript”). It can also be useful for agglutinative languages such as German.</p> <p>The <tt class="docutils literal"><span class="pre">keep_compound</span></tt> argument lets you decide whether to keep the compound word in the token stream along with the word segments.</p> <div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">cwf</span> <span class="o">=</span> <span class="n">CompoundWordFilter</span><span class="p">(</span><span class="n">wordset</span><span class="p">,</span> <span class="n">keep_compound</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> <span class="gp">>>> </span><span class="n">analyzer</span> <span class="o">=</span> <span class="n">RegexTokenizer</span><span class="p">(</span><span class="s">r"\S+"</span><span class="p">)</span> <span class="o">|</span> <span class="n">cwf</span> <span class="gp">>>> </span><span class="p">[</span><span class="n">t</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="n">analyzer</span><span class="p">(</span><span class="s">"I do not like greeneggs and ham"</span><span class="p">)</span> <span class="go">["I", "do", "not", "like", "greeneggs", "green", "eggs", "and", "ham"]</span> <span class="gp">>>> </span><span class="n">cwf</span><span class="o">.</span><span class="n">keep_compound</span> <span class="o">=</span> <span class="bp">False</span> <span class="gp">>>> </span><span class="p">[</span><span class="n">t</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="n">analyzer</span><span class="p">(</span><span class="s">"I do not like greeneggs and ham"</span><span class="p">)</span> <span class="go">["I", "do", "not", "like", "green", "eggs", "and", "ham"]</span> </pre></div> </div> <table class="docutils field-list" frame="void" rules="none"> <col class="field-name" /> <col class="field-body" /> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> <li><strong>wordset</strong> – an object with a <tt class="docutils literal"><span class="pre">__contains__</span></tt> method, such as a set, containing strings to look for inside the tokens.</li> <li><strong>keep_compound</strong> – if True (the default), the original compound token will be retained in the stream before the subwords.</li> </ul> </td> </tr> </tbody> </table> </dd></dl> <dl class="class"> <dt id="whoosh.analysis.BiWordFilter"> <em class="property">class </em><tt class="descclassname">whoosh.analysis.</tt><tt class="descname">BiWordFilter</tt><big>(</big><em>sep='-'</em><big>)</big><a class="headerlink" href="#whoosh.analysis.BiWordFilter" title="Permalink to this definition">¶</a></dt> <dd><p>Merges adjacent tokens into “bi-word” tokens, so that for example:</p> <div class="highlight-python"><div class="highlight"><pre><span class="s">"the"</span><span class="p">,</span> <span class="s">"sign"</span><span class="p">,</span> <span class="s">"of"</span><span class="p">,</span> <span class="s">"four"</span> </pre></div> </div> <p>becomes:</p> <div class="highlight-python"><div class="highlight"><pre><span class="s">"the-sign"</span><span class="p">,</span> <span class="s">"sign-of"</span><span class="p">,</span> <span class="s">"of-four"</span> </pre></div> </div> <p>This can be used to create fields for pseudo-phrase searching, where if all the terms match the document probably contains the phrase, but the searching is faster than actually doing a phrase search on individual word terms.</p> <p>The <tt class="docutils literal"><span class="pre">BiWordFilter</span></tt> is much faster than using the otherwise equivalent <tt class="docutils literal"><span class="pre">ShingleFilter(2)</span></tt>.</p> </dd></dl> <dl class="class"> <dt id="whoosh.analysis.ShingleFilter"> <em class="property">class </em><tt class="descclassname">whoosh.analysis.</tt><tt class="descname">ShingleFilter</tt><big>(</big><em>size=2</em>, <em>sep='-'</em><big>)</big><a class="headerlink" href="#whoosh.analysis.ShingleFilter" title="Permalink to this definition">¶</a></dt> <dd><p>Merges a certain number of adjacent tokens into multi-word tokens, so that for example:</p> <div class="highlight-python"><div class="highlight"><pre><span class="s">"better"</span><span class="p">,</span> <span class="s">"a"</span><span class="p">,</span> <span class="s">"witty"</span><span class="p">,</span> <span class="s">"fool"</span><span class="p">,</span> <span class="s">"than"</span><span class="p">,</span> <span class="s">"a"</span><span class="p">,</span> <span class="s">"foolish"</span><span class="p">,</span> <span class="s">"wit"</span> </pre></div> </div> <p>with <tt class="docutils literal"><span class="pre">ShingleFilter(3,</span> <span class="pre">'</span> <span class="pre">')</span></tt> becomes:</p> <div class="highlight-python"><div class="highlight"><pre><span class="s">'better a witty'</span><span class="p">,</span> <span class="s">'a witty fool'</span><span class="p">,</span> <span class="s">'witty fool than'</span><span class="p">,</span> <span class="s">'fool than a'</span><span class="p">,</span> <span class="s">'than a foolish'</span><span class="p">,</span> <span class="s">'a foolish wit'</span> </pre></div> </div> <p>This can be used to create fields for pseudo-phrase searching, where if all the terms match the document probably contains the phrase, but the searching is faster than actually doing a phrase search on individual word terms.</p> <p>If you’re using two-word shingles, you should use the functionally equivalent <tt class="docutils literal"><span class="pre">BiWordFilter</span></tt> instead because it’s faster than <tt class="docutils literal"><span class="pre">ShingleFilter</span></tt>.</p> </dd></dl> <dl class="class"> <dt id="whoosh.analysis.DelimitedAttributeFilter"> <em class="property">class </em><tt class="descclassname">whoosh.analysis.</tt><tt class="descname">DelimitedAttributeFilter</tt><big>(</big><em>delimiter='^'</em>, <em>attribute='boost'</em>, <em>default=1.0</em>, <em>type=<type 'float'></em><big>)</big><a class="headerlink" href="#whoosh.analysis.DelimitedAttributeFilter" title="Permalink to this definition">¶</a></dt> <dd><p>Looks for delimiter characters in the text of each token and stores the data after the delimiter in a named attribute on the token.</p> <p>The defaults are set up to use the <tt class="docutils literal"><span class="pre">^</span></tt> character as a delimiter and store the value after the <tt class="docutils literal"><span class="pre">^</span></tt> as the boost for the token.</p> <div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">daf</span> <span class="o">=</span> <span class="n">DelimitedAttributeFilter</span><span class="p">(</span><span class="n">delimiter</span><span class="o">=</span><span class="s">"^"</span><span class="p">,</span> <span class="n">attribute</span><span class="o">=</span><span class="s">"boost"</span><span class="p">)</span> <span class="gp">>>> </span><span class="n">ana</span> <span class="o">=</span> <span class="n">RegexTokenizer</span><span class="p">(</span><span class="s">"</span><span class="se">\\</span><span class="s">S+"</span><span class="p">)</span> <span class="o">|</span> <span class="n">DelimitedAttributeFilter</span><span class="p">()</span> <span class="gp">>>> </span><span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="n">ana</span><span class="p">(</span><span class="n">u</span><span class="p">(</span><span class="s">"image render^2 file^0.5"</span><span class="p">))</span> <span class="gp">... </span> <span class="k">print</span><span class="p">(</span><span class="s">"</span><span class="si">%r</span><span class="s"> </span><span class="si">%f</span><span class="s">"</span> <span class="o">%</span> <span class="p">(</span><span class="n">t</span><span class="o">.</span><span class="n">text</span><span class="p">,</span> <span class="n">t</span><span class="o">.</span><span class="n">boost</span><span class="p">))</span> <span class="go">'image' 1.0</span> <span class="go">'render' 2.0</span> <span class="go">'file' 0.5</span> </pre></div> </div> <p>Note that you need to make sure your tokenizer includes the delimiter and data as part of the token!</p> <table class="docutils field-list" frame="void" rules="none"> <col class="field-name" /> <col class="field-body" /> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> <li><strong>delimiter</strong> – a string that, when present in a token’s text, separates the actual text from the “data” payload.</li> <li><strong>attribute</strong> – the name of the attribute in which to store the data on the token.</li> <li><strong>default</strong> – the value to use for the attribute for tokens that don’t have delimited data.</li> <li><strong>type</strong> – the type of the data, for example <tt class="docutils literal"><span class="pre">str</span></tt> or <tt class="docutils literal"><span class="pre">float</span></tt>. This is used to convert the string value of the data before storing it in the attribute.</li> </ul> </td> </tr> </tbody> </table> </dd></dl> <dl class="class"> <dt id="whoosh.analysis.DoubleMetaphoneFilter"> <em class="property">class </em><tt class="descclassname">whoosh.analysis.</tt><tt class="descname">DoubleMetaphoneFilter</tt><big>(</big><em>primary_boost=1.0</em>, <em>secondary_boost=0.5</em>, <em>combine=False</em><big>)</big><a class="headerlink" href="#whoosh.analysis.DoubleMetaphoneFilter" title="Permalink to this definition">¶</a></dt> <dd><p>Transforms the text of the tokens using Lawrence Philips’s Double Metaphone algorithm. This algorithm attempts to encode words in such a way that similar-sounding words reduce to the same code. This may be useful for fields containing the names of people and places, and other uses where tolerance of spelling differences is desireable.</p> <table class="docutils field-list" frame="void" rules="none"> <col class="field-name" /> <col class="field-body" /> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> <li><strong>primary_boost</strong> – the boost to apply to the token containing the primary code.</li> <li><strong>secondary_boost</strong> – the boost to apply to the token containing the secondary code, if any.</li> <li><strong>combine</strong> – if True, the original unencoded tokens are kept in the stream, preceding the encoded tokens.</li> </ul> </td> </tr> </tbody> </table> </dd></dl> <dl class="class"> <dt id="whoosh.analysis.SubstitutionFilter"> <em class="property">class </em><tt class="descclassname">whoosh.analysis.</tt><tt class="descname">SubstitutionFilter</tt><big>(</big><em>pattern</em>, <em>replacement</em><big>)</big><a class="headerlink" href="#whoosh.analysis.SubstitutionFilter" title="Permalink to this definition">¶</a></dt> <dd><p>Performs a regular expression substitution on the token text.</p> <p>This is especially useful for removing text from tokens, for example hyphens:</p> <div class="highlight-python"><div class="highlight"><pre><span class="n">ana</span> <span class="o">=</span> <span class="n">RegexTokenizer</span><span class="p">(</span><span class="s">r"\S+"</span><span class="p">)</span> <span class="o">|</span> <span class="n">SubstitutionFilter</span><span class="p">(</span><span class="s">"-"</span><span class="p">,</span> <span class="s">""</span><span class="p">)</span> </pre></div> </div> <p>Because it has the full power of the re.sub() method behind it, this filter can perform some fairly complex transformations. For example, to take tokens like <tt class="docutils literal"><span class="pre">'a=b',</span> <span class="pre">'c=d',</span> <span class="pre">'e=f'</span></tt> and change them to <tt class="docutils literal"><span class="pre">'b=a',</span> <span class="pre">'d=c',</span> <span class="pre">'f=e'</span></tt>:</p> <div class="highlight-python"><div class="highlight"><pre><span class="c"># Analyzer that swaps the text on either side of an equal sign</span> <span class="n">rt</span> <span class="o">=</span> <span class="n">RegexTokenizer</span><span class="p">(</span><span class="s">r"\S+"</span><span class="p">)</span> <span class="n">sf</span> <span class="o">=</span> <span class="n">SubstitutionFilter</span><span class="p">(</span><span class="s">"([^/]*)/(./*)"</span><span class="p">,</span> <span class="s">r"\2/\1"</span><span class="p">)</span> <span class="n">ana</span> <span class="o">=</span> <span class="n">rt</span> <span class="o">|</span> <span class="n">sf</span> </pre></div> </div> <table class="docutils field-list" frame="void" rules="none"> <col class="field-name" /> <col class="field-body" /> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> <li><strong>pattern</strong> – a pattern string or compiled regular expression object describing the text to replace.</li> <li><strong>replacement</strong> – the substitution text.</li> </ul> </td> </tr> </tbody> </table> </dd></dl> </div> <div class="section" id="token-classes-and-functions"> <h2>Token classes and functions<a class="headerlink" href="#token-classes-and-functions" title="Permalink to this headline">¶</a></h2> <dl class="class"> <dt id="whoosh.analysis.Token"> <em class="property">class </em><tt class="descclassname">whoosh.analysis.</tt><tt class="descname">Token</tt><big>(</big><em>positions=False</em>, <em>chars=False</em>, <em>removestops=True</em>, <em>mode=''</em>, <em>**kwargs</em><big>)</big><a class="headerlink" href="#whoosh.analysis.Token" title="Permalink to this definition">¶</a></dt> <dd><p>Represents a “token” (usually a word) extracted from the source text being indexed.</p> <p>See “Advanced analysis” in the user guide for more information.</p> <p>Because object instantiation in Python is slow, tokenizers should create ONE SINGLE Token object and YIELD IT OVER AND OVER, changing the attributes each time.</p> <p>This trick means that consumers of tokens (i.e. filters) must never try to hold onto the token object between loop iterations, or convert the token generator into a list. Instead, save the attributes between iterations, not the object:</p> <div class="highlight-python"><div class="highlight"><pre><span class="k">def</span> <span class="nf">RemoveDuplicatesFilter</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">stream</span><span class="p">):</span> <span class="c"># Removes duplicate words.</span> <span class="n">lasttext</span> <span class="o">=</span> <span class="bp">None</span> <span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">stream</span><span class="p">:</span> <span class="c"># Only yield the token if its text doesn't</span> <span class="c"># match the previous token.</span> <span class="k">if</span> <span class="n">lasttext</span> <span class="o">!=</span> <span class="n">token</span><span class="o">.</span><span class="n">text</span><span class="p">:</span> <span class="k">yield</span> <span class="n">token</span> <span class="n">lasttext</span> <span class="o">=</span> <span class="n">token</span><span class="o">.</span><span class="n">text</span> </pre></div> </div> <p>...or, call token.copy() to get a copy of the token object.</p> <table class="docutils field-list" frame="void" rules="none"> <col class="field-name" /> <col class="field-body" /> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> <li><strong>positions</strong> – Whether tokens should have the token position in the ‘pos’ attribute.</li> <li><strong>chars</strong> – Whether tokens should have character offsets in the ‘startchar’ and ‘endchar’ attributes.</li> <li><strong>removestops</strong> – whether to remove stop words from the stream (if the tokens pass through a stop filter).</li> <li><strong>mode</strong> – contains a string describing the purpose for which the analyzer is being called, i.e. ‘index’ or ‘query’.</li> </ul> </td> </tr> </tbody> </table> </dd></dl> <dl class="function"> <dt id="whoosh.analysis.unstopped"> <tt class="descclassname">whoosh.analysis.</tt><tt class="descname">unstopped</tt><big>(</big><em>tokenstream</em><big>)</big><a class="headerlink" href="#whoosh.analysis.unstopped" title="Permalink to this definition">¶</a></dt> <dd><p>Removes tokens from a token stream where token.stopped = True.</p> </dd></dl> </div> </div> </div> </div> </div> <div class="sphinxsidebar"> <div class="sphinxsidebarwrapper"> <h3><a href="../index.html">Table Of Contents</a></h3> <ul> <li><a class="reference internal" href="#"><tt class="docutils literal"><span class="pre">analysis</span></tt> module</a><ul> <li><a class="reference internal" href="#analyzers">Analyzers</a></li> <li><a class="reference internal" href="#tokenizers">Tokenizers</a></li> <li><a class="reference internal" href="#filters">Filters</a></li> <li><a class="reference internal" href="#token-classes-and-functions">Token classes and functions</a></li> </ul> </li> </ul> <h4>Previous topic</h4> <p class="topless"><a href="api.html" title="previous chapter">Whoosh API</a></p> <h4>Next topic</h4> <p class="topless"><a href="codec/base.html" title="next chapter"><tt class="docutils literal"><span class="pre">codec.base</span></tt> module</a></p> <h3>This Page</h3> <ul class="this-page-menu"> <li><a href="../_sources/api/analysis.txt" rel="nofollow">Show Source</a></li> </ul> <div id="searchbox" style="display: none"> <h3>Quick search</h3> <form class="search" action="../search.html" method="get"> <input type="text" name="q" /> <input type="submit" value="Go" /> <input type="hidden" name="check_keywords" value="yes" /> <input type="hidden" name="area" value="default" /> </form> <p class="searchtip" style="font-size: 90%"> Enter search terms or a module, class or function name. </p> </div> <script type="text/javascript">$('#searchbox').show(0);</script> </div> </div> <div class="clearer"></div> </div> <div class="related"> <h3>Navigation</h3> <ul> <li class="right" style="margin-right: 10px"> <a href="../genindex.html" title="General Index" >index</a></li> <li class="right" > <a href="../py-modindex.html" title="Python Module Index" >modules</a> |</li> <li class="right" > <a href="codec/base.html" title="codec.base module" >next</a> |</li> <li class="right" > <a href="api.html" title="Whoosh API" >previous</a> |</li> <li><a href="../index.html">Whoosh 2.7.4 documentation</a> »</li> <li><a href="api.html" >Whoosh API</a> »</li> </ul> </div> <div class="footer"> © Copyright 2007-2012 Matt Chaput. Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 1.1.3. </div> </body> </html>