EVOLUTION-MANAGER

Edit File: facets.html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">
 <head>
 <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
 
 <title>Sorting and faceting &mdash; Whoosh 2.7.4 documentation</title>
 
 <link rel="stylesheet" href="_static/default.css" type="text/css" />
 <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
 
 <script type="text/javascript">
 var DOCUMENTATION_OPTIONS = {
 URL_ROOT: '',
 VERSION: '2.7.4',
 COLLAPSE_INDEX: false,
 FILE_SUFFIX: '.html',
 HAS_SOURCE: true
 };
 </script>
 <script type="text/javascript" src="_static/jquery.js"></script>
 <script type="text/javascript" src="_static/underscore.js"></script>
 <script type="text/javascript" src="_static/doctools.js"></script>
 <link rel="top" title="Whoosh 2.7.4 documentation" href="index.html" />
 <link rel="next" title="How to create highlighted search result excerpts" href="highlight.html" />
 <link rel="prev" title="Indexing and searching N-grams" href="ngrams.html" /> 
 </head>
 <body>
 <div class="related">
 <h3>Navigation</h3>
 <ul>
 <li class="right" style="margin-right: 10px">
 <a href="genindex.html" title="General Index"
 accesskey="I">index</a></li>
 <li class="right" >
 <a href="py-modindex.html" title="Python Module Index"
 >modules</a> |</li>
 <li class="right" >
 <a href="highlight.html" title="How to create highlighted search result excerpts"
 accesskey="N">next</a> |</li>
 <li class="right" >
 <a href="ngrams.html" title="Indexing and searching N-grams"
 accesskey="P">previous</a> |</li>
 <li><a href="index.html">Whoosh 2.7.4 documentation</a> &raquo;</li> 
 </ul>
 </div>

<div class="document">
 <div class="documentwrapper">
 <div class="bodywrapper">
 <div class="body">
 
 <div class="section" id="sorting-and-faceting">
<h1>Sorting and faceting<a class="headerlink" href="#sorting-and-faceting" title="Permalink to this headline">¶</a></h1>
<div class="admonition note">
Note
The API for sorting and faceting changed in Whoosh 3.0.
</div>
<div class="section" id="overview">
<h2>Overview<a class="headerlink" href="#overview" title="Permalink to this headline">¶</a></h2>
Sorting and faceting search results in Whoosh is based on facets. Each
facet associates a value with each document in the search results, allowing you
to sort by the keys or use them to group the documents. Whoosh includes a variety
of facet types you can use for sorting and grouping (see below).
</div>
<div class="section" id="sorting">
<h2>Sorting<a class="headerlink" href="#sorting" title="Permalink to this headline">¶</a></h2>
By default, the results of a search are sorted with the highest-scoring
documents first. You can use the <tt class="docutils literal">sortedby</tt> keyword argument to order the
results by some other criteria instead, such as the value of a field.
<div class="section" id="making-fields-sortable">
<h3>Making fields sortable<a class="headerlink" href="#making-fields-sortable" title="Permalink to this headline">¶</a></h3>
In order to sort on a field, you should create the field using the
<tt class="docutils literal">sortable=True</tt> keyword argument:
<div class="highlight-python"><div class="highlight"><pre>schema = fields.Schema(title=fields.TEXT(sortable=True),
 content=fields.TEXT,
 modified=fields.DATETIME(sortable=True)
 )
</pre></div>
</div>
It&#8217;s possible to sort on a field that doesn&#8217;t have <tt class="docutils literal">sortable=True</tt>, but this
requires Whoosh to load the unique terms in the field into memory. Using
<tt class="docutils literal">sortable</tt> is much more efficient.
</div>
<div class="section" id="about-column-types">
<h3>About column types<a class="headerlink" href="#about-column-types" title="Permalink to this headline">¶</a></h3>
When you create a field using <tt class="docutils literal">sortable=True</tt>, you are telling Whoosh to store
per-document values for that field in a column. A column object specifies the
format to use to store the per-document values on disk.
The <a class="reference internal" href="api/columns.html#module-whoosh.columns" title="whoosh.columns"><tt class="xref py py-mod docutils literal">whoosh.columns</tt></a> module contains several different column object
implementations. Each field type specifies a reasonable default column type (for
example, the default for text fields is <a class="reference internal" href="api/columns.html#whoosh.columns.VarBytesColumn" title="whoosh.columns.VarBytesColumn"><tt class="xref py py-class docutils literal">whoosh.columns.VarBytesColumn</tt></a>,
the default for numeric fields is <a class="reference internal" href="api/columns.html#whoosh.columns.NumericColumn" title="whoosh.columns.NumericColumn"><tt class="xref py py-class docutils literal">whoosh.columns.NumericColumn</tt></a>).
However, if you want maximum efficiency you may want to use a different column
type for a field.
For example, if all document values in a field are a fixed length, you can use a
<a class="reference internal" href="api/columns.html#whoosh.columns.FixedBytesColumn" title="whoosh.columns.FixedBytesColumn"><tt class="xref py py-class docutils literal">whoosh.columns.FixedBytesColumn</tt></a>. If you have a field where many
documents share a relatively small number of possible values (an example might
be a &#8220;category&#8221; field, or &#8220;month&#8221; or other enumeration type fields), you might
want to use <a class="reference internal" href="api/columns.html#whoosh.columns.RefBytesColumn" title="whoosh.columns.RefBytesColumn"><tt class="xref py py-class docutils literal">whoosh.columns.RefBytesColumn</tt></a> (which can handle both
variable and fixed-length values). There are column types for storing
per-document bit values, structs, pickled objects, and compressed byte values.
To specify a custom column object for a field, pass it as the <tt class="docutils literal">sortable</tt>
keyword argument instead of <tt class="docutils literal">True</tt>:
<div class="highlight-python"><pre>from whoosh import columns, fields

category_col = columns.RefBytesColumn()
schema = fields.Schema(title=fields.TEXT(sortable=True),
 category=fields.KEYWORD(sortable=category_col)</pre>
</div>
</div>
<div class="section" id="using-a-column-field-for-custom-sort-keys">
<h3>Using a COLUMN field for custom sort keys<a class="headerlink" href="#using-a-column-field-for-custom-sort-keys" title="Permalink to this headline">¶</a></h3>
When you add a document with a sortable field, Whoosh uses the value you pass
for the field as the sortable value. For example, if &#8220;title&#8221; is a sortable
field, and you add this document:
<div class="highlight-python"><div class="highlight"><pre>writer.add_document(title=&quot;Mr. Palomar&quot;)
</pre></div>
</div>
...then <tt class="docutils literal">Mr. Palomar</tt> is stored in the field column as the sorting key for the
document.
This is usually good, but sometimes you need to &#8220;massage&#8221; the sortable key so
it&#8217;s different from the value the user searches and/or sees in the interface.
For example, if you allow the user to sort by title, you might want to use
different values for the visible title and the value used for sorting:
<div class="highlight-python"><div class="highlight"><pre># Visible title
title = &quot;The Unbearable Lightness of Being&quot;

# Sortable title: converted to lowercase (to prevent different ordering
# depending on uppercase/lowercase), with initial article moved to the end
sort_title = &quot;unbearable lightness of being, the&quot;
</pre></div>
</div>
The best way to do this is to use an additional field just for sorting. You can
use the <tt class="xref py py-class docutils literal">whoosh.fields.COLUMN</tt> field type to create a field that is not
indexed or stored, it only holds per-document column values:
<div class="highlight-python"><div class="highlight"><pre>schema = fields.Schema(title=fields.TEXT(stored=True),
 sort_title=fields.COLUMN(columns.VarBytesColumn())
 )
</pre></div>
</div>
The single argument to the <tt class="xref py py-class docutils literal">whoosh.fields.COLUMN</tt> initializer is a
<tt class="xref py py-class docutils literal">whoosh.columns.ColumnType</tt> object. You can use any of the various
column types in the <a class="reference internal" href="api/columns.html#module-whoosh.columns" title="whoosh.columns"><tt class="xref py py-mod docutils literal">whoosh.columns</tt></a> module.
As another example, say you are indexing documents that have a custom sorting
order associated with each document, such as a &#8220;priority&#8221; number:
<div class="highlight-python"><pre>name=Big Wheel
price=100
priority=1

name=Toss Across
price=40
priority=3

name=Slinky
price=25
priority=2
...</pre>
</div>
You can use a column field with a numeric column object to hold the &#8220;priority&#8221;
and use it for sorting:
<div class="highlight-python"><pre>schema = fields.Schema(name=fields.TEXT(stored=True),
 price=fields.NUMERIC(stored=True),
 priority=fields.COLUMN(columns.NumericColumn("i"),
 )</pre>
</div>
(Note that <tt class="xref py py-class docutils literal">columns.NumericColumn</tt> takes a type code character like the
codes used by Python&#8217;s <tt class="docutils literal">struct</tt> and <tt class="docutils literal">array</tt> modules.)
</div>
<div class="section" id="making-existing-fields-sortable">
<h3>Making existing fields sortable<a class="headerlink" href="#making-existing-fields-sortable" title="Permalink to this headline">¶</a></h3>
If you have an existing index from before the <tt class="docutils literal">sortable</tt> argument was added
in Whoosh 3.0, or you didn&#8217;t think you needed a field to be sortable but now
you find that you need to sort it, you can add &#8220;sortability&#8221; to an existing
index using the <tt class="xref py py-func docutils literal">whoosh.sorting.add_sortable()</tt> utility function:
<div class="highlight-python"><div class="highlight"><pre>from whoosh import columns, fields, index, sorting

# Say we have an existing index with this schema
schema = fields.Schema(title=fields.TEXT,
 price=fields.NUMERIC)

# To use add_sortable, first open a writer for the index
ix = index.open_dir(&quot;indexdir&quot;)
with ix.writer() as w:
 # Add sortable=True to the &quot;price&quot; field using field terms as the
 # sortable values
 sorting.add_sortable(w, &quot;price&quot;, sorting.FieldFacet(&quot;price&quot;))

# Add sortable=True to the &quot;title&quot; field using the
 # stored field values as the sortable value
 sorting.add_sortable(w, &quot;title&quot;, sorting.StoredFieldFacet(&quot;title&quot;))
</pre></div>
</div>
You can specify a custom column type when you call <tt class="docutils literal">add_sortable</tt> using the
<tt class="docutils literal">column</tt> keyword argument:
<div class="highlight-python"><div class="highlight"><pre>add_sortable(w, &quot;chapter&quot;, sorting.FieldFacet(&quot;chapter&quot;),
 column=columns.RefBytesColumn())
</pre></div>
</div>
See the documentation for <tt class="xref py py-func docutils literal">add_sortable()</tt> for more
information.
</div>
<div class="section" id="sorting-search-results">
<h3>Sorting search results<a class="headerlink" href="#sorting-search-results" title="Permalink to this headline">¶</a></h3>
When you tell Whoosh to sort by a field (or fields), it uses the per-document
values in the field&#8217;s column as sorting keys for the documents.
Normally search results are sorted by descending relevance score. You can tell
Whoosh to use a different ordering by passing the <tt class="docutils literal">sortedby</tt> keyword argument
to the <a class="reference internal" href="api/searching.html#whoosh.searching.Searcher.search" title="whoosh.searching.Searcher.search"><tt class="xref py py-meth docutils literal">search()</tt></a> method:
<div class="highlight-python"><div class="highlight"><pre>from whoosh import fields, index, qparser

schema = fields.Schema(title=fields.TEXT(stored=True),
 price=fields.NUMERIC(sortable=True))
ix = index.create_in(&quot;indexdir&quot;, schema)

with ix.writer() as w:
 w.add_document(title=&quot;Big Deal&quot;, price=20)
 w.add_document(title=&quot;Mr. Big&quot;, price=10)
 w.add_document(title=&quot;Big Top&quot;, price=15)

with ix.searcher() as s:
 qp = qparser.QueryParser(&quot;big&quot;, ix.schema)
 q = qp.parse(user_query_string)

# Sort search results from lowest to highest price
 results = s.search(q, sortedby=&quot;price&quot;)
 for hit in results:
 print(hit[&quot;title&quot;])
</pre></div>
</div>
You can use any of the following objects as <tt class="docutils literal">sortedby</tt> values:
<dl class="docutils">
<dt>A <tt class="docutils literal">FacetType</tt> object</dt>
<dd>Uses this object to sort the documents. See below for the available facet
types.</dd>
<dt>A field name string</dt>
<dd>Converts the field name into a <tt class="docutils literal">FieldFacet</tt> (see below) and uses it to
sort the documents.</dd>
<dt>A list of <tt class="docutils literal">FacetType</tt> objects and/or field name strings</dt>
<dd>Bundles the facets together into a <tt class="docutils literal">MultiFacet</tt> so you can sort by
multiple keys. Note that this shortcut does not allow you to reverse
the sort direction of individual facets. To do that, you need to construct
the <tt class="docutils literal">MultiFacet</tt> object yourself.</dd>
</dl>
<div class="admonition note">
Note
You can use the <tt class="docutils literal">reverse=True</tt> keyword argument to the
<tt class="docutils literal">Searcher.search()</tt> method to reverse the overall sort direction. This
is more efficient than reversing each individual facet.
</div>
</div>
<div class="section" id="examples">
<h3>Examples<a class="headerlink" href="#examples" title="Permalink to this headline">¶</a></h3>
Sort by the value of the size field:
<div class="highlight-python"><div class="highlight"><pre>results = searcher.search(myquery, sortedby=&quot;size&quot;)
</pre></div>
</div>
Sort by the reverse (highest-to-lowest) order of the &#8220;price&#8221; field:
<div class="highlight-python"><div class="highlight"><pre>facet = sorting.FieldFacet(&quot;price&quot;, reverse=True)
results = searcher.search(myquery, sortedby=facet)
</pre></div>
</div>
Sort by ascending size and then descending price:
<div class="highlight-python"><div class="highlight"><pre>mf = sorting.MultiFacet()
mf.add_field(&quot;size&quot;)
mf.add_field(&quot;price&quot;, reverse=True)
results = searcher.search(myquery, sortedby=mf)

# or...
sizes = sorting.FieldFacet(&quot;size&quot;)
prices = sorting.FieldFacet(&quot;price&quot;, reverse=True)
results = searcher.search(myquery, sortedby=[sizes, prices])
</pre></div>
</div>
Sort by the &#8220;category&#8221; field, then by the document&#8217;s score:
<div class="highlight-python"><div class="highlight"><pre>cats = sorting.FieldFacet(&quot;category&quot;)
scores = sorting.ScoreFacet()
results = searcher.search(myquery, sortedby=[cats, scores])
</pre></div>
</div>
</div>
<div class="section" id="accessing-column-values">
<h3>Accessing column values<a class="headerlink" href="#accessing-column-values" title="Permalink to this headline">¶</a></h3>
Per-document column values are available in <a class="reference internal" href="api/searching.html#whoosh.searching.Hit" title="whoosh.searching.Hit"><tt class="xref py py-class docutils literal">Hit</tt></a>
objects just like stored field values:
<div class="highlight-python"><div class="highlight"><pre>schema = fields.Schema(title=fields.TEXT(stored=True),
 price=fields.NUMERIC(sortable=True))

...

colreader = s.reader().column_reader(&quot;price&quot;)
 for docnum in reader.all_doc_ids():
 print(colreader[docnum])
</pre></div>
</div>
</div>
</div>
<div class="section" id="grouping">
<h2>Grouping<a class="headerlink" href="#grouping" title="Permalink to this headline">¶</a></h2>
It is often very useful to present &#8220;faceted&#8221; search results to the user.
Faceting is dynamic grouping of search results into categories. The
categories let users view a slice of the total results based on the categories
they&#8217;re interested in.
For example, if you are programming a shopping website, you might want to
display categories with the search results such as the manufacturers and price
ranges.
<table border="1" class="docutils">
<colgroup>
<col width="54%" />
<col width="46%" />
</colgroup>
<tbody valign="top">
<tr class="row-odd"><td>Manufacturer</td>
<td>Price</td>
</tr>
<tr class="row-even"><td>Apple (5)</td>
<td>$0 - $100 (2)</td>
</tr>
<tr class="row-odd"><td>Sanyo (1)</td>
<td>$101 - $500 (10)</td>
</tr>
<tr class="row-even"><td>Sony (2)</td>
<td>$501 - $1000 (1)</td>
</tr>
<tr class="row-odd"><td>Toshiba (5)</td>
<td>&nbsp;</td>
</tr>
</tbody>
</table>
You can let your users click the different facet values to only show results
in the given categories.
Another useful UI pattern is to show, say, the top 5 results for different
types of found documents, and let the user click to see more results from a
category they&#8217;re interested in, similarly to how the Spotlight quick results
work on Mac OS X.
<div class="section" id="the-groupedby-keyword-argument">
<h3>The <tt class="docutils literal">groupedby</tt> keyword argument<a class="headerlink" href="#the-groupedby-keyword-argument" title="Permalink to this headline">¶</a></h3>
You can use the following objects as <tt class="docutils literal">groupedby</tt> values:
<dl class="docutils">
<dt>A <tt class="docutils literal">FacetType</tt> object</dt>
<dd>Uses this object to group the documents. See below for the available facet
types.</dd>
<dt>A field name string</dt>
<dd>Converts the field name into a <tt class="docutils literal">FieldFacet</tt> (see below) and uses it to
sort the documents. The name of the field is used as the facet name.</dd>
<dt>A list or tuple of field name strings</dt>
<dd>Sets up multiple field grouping criteria.</dd>
<dt>A dictionary mapping facet names to <tt class="docutils literal">FacetType</tt> objects</dt>
<dd>Sets up multiple grouping criteria.</dd>
<dt>A <tt class="docutils literal">Facets</tt> object</dt>
<dd>This object is a lot like using a dictionary, but has some convenience
methods to make setting up multiple groupings a little easier.</dd>
</dl>
</div>
<div class="section" id="id1">
<h3>Examples<a class="headerlink" href="#id1" title="Permalink to this headline">¶</a></h3>
Group by the value of the &#8220;category&#8221; field:
<div class="highlight-python"><div class="highlight"><pre>results = searcher.search(myquery, groupedby=&quot;category&quot;)
</pre></div>
</div>
Group by the value of the &#8220;category&#8221; field and also by the value of the &#8220;tags&#8221;
field and a date range:
<div class="highlight-python"><div class="highlight"><pre>cats = sorting.FieldFacet(&quot;category&quot;)
tags = sorting.FieldFacet(&quot;tags&quot;, allow_overlap=True)
results = searcher.search(myquery, groupedby={&quot;category&quot;: cats, &quot;tags&quot;: tags})

# ...or, using a Facets object has a little less duplication
facets = sorting.Facets()
facets.add_field(&quot;category&quot;)
facets.add_field(&quot;tags&quot;, allow_overlap=True)
results = searcher.search(myquery, groupedby=facets)
</pre></div>
</div>
To group results by the intersected values of multiple fields, use a
<tt class="docutils literal">MultiFacet</tt> object (see below). For example, if you have two fields named
<tt class="docutils literal">tag</tt> and <tt class="docutils literal">size</tt>, you could group the results by all combinations of the
<tt class="docutils literal">tag</tt> and <tt class="docutils literal">size</tt> field, such as <tt class="docutils literal">('tag1', 'small')</tt>,
<tt class="docutils literal">('tag2', 'small')</tt>, <tt class="docutils literal">('tag1', 'medium')</tt>, and so on:
<div class="highlight-python"><div class="highlight"><pre># Generate a grouping from the combination of the &quot;tag&quot; and &quot;size&quot; fields
mf = MultiFacet(&quot;tag&quot;, &quot;size&quot;)
results = searcher.search(myquery, groupedby={&quot;tag/size&quot;: mf})
</pre></div>
</div>
</div>
<div class="section" id="getting-the-faceted-groups">
<h3>Getting the faceted groups<a class="headerlink" href="#getting-the-faceted-groups" title="Permalink to this headline">¶</a></h3>
The <tt class="docutils literal">Results.groups(&quot;facetname&quot;)</tt> method returns a dictionary mapping
category names to lists of document IDs:
<div class="highlight-python"><div class="highlight"><pre>myfacets = sorting.Facets().add_field(&quot;size&quot;).add_field(&quot;tag&quot;)
results = mysearcher.search(myquery, groupedby=myfacets)
results.groups(&quot;size&quot;)
# {&quot;small&quot;: [8, 5, 1, 2, 4], &quot;medium&quot;: [3, 0, 6], &quot;large&quot;: [7, 9]}
</pre></div>
</div>
If there is only one facet, you can just use <tt class="docutils literal">Results.groups()</tt> with no
argument to access its groups:
<div class="highlight-python"><div class="highlight"><pre>results = mysearcher.search(myquery, groupedby=myfunctionfacet)
results.groups()
</pre></div>
</div>
By default, the values in the dictionary returned by <tt class="docutils literal">groups()</tt> are lists of
document numbers in the same relative order as in the results. You can use the
<tt class="docutils literal">Searcher</tt> object&#8217;s <tt class="docutils literal">stored_fields()</tt> method to take a document number and
return the document&#8217;s stored fields as a dictionary:
<div class="highlight-python"><div class="highlight"><pre>for category_name in categories:
 print &quot;Top 5 documents in the %s category&quot; % category_name
 doclist = categories[category_name]
 for docnum, score in doclist[:5]:
 print &quot; &quot;, searcher.stored_fields(docnum)
 if len(doclist) &gt; 5:
 print &quot; (%s more)&quot; % (len(doclist) - 5)
</pre></div>
</div>
If you want different information about the groups, for example just the count
of documents in each group, or you don&#8217;t need the groups to be ordered, you can
specify a <a class="reference internal" href="api/sorting.html#whoosh.sorting.FacetMap" title="whoosh.sorting.FacetMap"><tt class="xref py py-class docutils literal">whoosh.sorting.FacetMap</tt></a> type or instance with the
<tt class="docutils literal">maptype</tt> keyword argument when creating the <tt class="docutils literal">FacetType</tt>:
<div class="highlight-python"><div class="highlight"><pre># This is the same as the default
myfacet = FieldFacet(&quot;size&quot;, maptype=sorting.OrderedList)
results = mysearcher.search(myquery, groupedby=myfacet)
results.groups()
# {&quot;small&quot;: [8, 5, 1, 2, 4], &quot;medium&quot;: [3, 0, 6], &quot;large&quot;: [7, 9]}

# Don&#39;t sort the groups to match the order of documents in the results
# (faster)
myfacet = FieldFacet(&quot;size&quot;, maptype=sorting.UnorderedList)
results = mysearcher.search(myquery, groupedby=myfacet)
results.groups()
# {&quot;small&quot;: [1, 2, 4, 5, 8], &quot;medium&quot;: [0, 3, 6], &quot;large&quot;: [7, 9]}

# Only count the documents in each group
myfacet = FieldFacet(&quot;size&quot;, maptype=sorting.Count)
results = mysearcher.search(myquery, groupedby=myfacet)
results.groups()
# {&quot;small&quot;: 5, &quot;medium&quot;: 3, &quot;large&quot;: 2}

# Only remember the &quot;best&quot; document in each group
myfacet = FieldFacet(&quot;size&quot;, maptype=sorting.Best)
results = mysearcher.search(myquery, groupedby=myfacet)
results.groups()
# {&quot;small&quot;: 8, &quot;medium&quot;: 3, &quot;large&quot;: 7}
</pre></div>
</div>
Alternatively you can specify a <tt class="docutils literal">maptype</tt> argument in the
<tt class="docutils literal">Searcher.search()</tt> method call which applies to all facets:
<div class="highlight-python"><div class="highlight"><pre>results = mysearcher.search(myquery, groupedby=[&quot;size&quot;, &quot;tag&quot;],
 maptype=sorting.Count)
</pre></div>
</div>
(You can override this overall <tt class="docutils literal">maptype</tt> argument on individual facets by
specifying the <tt class="docutils literal">maptype</tt> argument for them as well.)
</div>
</div>
<div class="section" id="facet-types">
<h2>Facet types<a class="headerlink" href="#facet-types" title="Permalink to this headline">¶</a></h2>
<div class="section" id="fieldfacet">
<h3>FieldFacet<a class="headerlink" href="#fieldfacet" title="Permalink to this headline">¶</a></h3>
This is the most common facet type. It sorts or groups based on the
value in a certain field in each document. This generally works best
(or at all) if each document has only one term in the field (e.g. an ID
field):
<div class="highlight-python"><div class="highlight"><pre># Sort search results by the value of the &quot;path&quot; field
facet = sorting.FieldFacet(&quot;path&quot;)
results = searcher.search(myquery, sortedby=facet)

# Group search results by the value of the &quot;parent&quot; field
facet = sorting.FieldFacet(&quot;parent&quot;)
results = searcher.search(myquery, groupedby=facet)
parent_groups = results.groups(&quot;parent&quot;)
</pre></div>
</div>
By default, <tt class="docutils literal">FieldFacet</tt> only supports non-overlapping grouping, where a
document cannot belong to multiple facets at the same time (each document will
be sorted into one category arbitrarily.) To get overlapping groups with
multi-valued fields, use the <tt class="docutils literal">allow_overlap=True</tt> keyword argument:
<div class="highlight-python"><div class="highlight"><pre>facet = sorting.FieldFacet(fieldname, allow_overlap=True)
</pre></div>
</div>
This supports overlapping group membership where documents have more than one
term in a field (e.g. KEYWORD fields). If you don&#8217;t need overlapping, don&#8217;t
use <tt class="docutils literal">allow_overlap</tt> because it&#8217;s much slower and uses more memory (see
the secion on <tt class="docutils literal">allow_overlap</tt> below).
</div>
<div class="section" id="queryfacet">
<h3>QueryFacet<a class="headerlink" href="#queryfacet" title="Permalink to this headline">¶</a></h3>
You can set up categories defined by arbitrary queries. For example, you can
group names using prefix queries:
<div class="highlight-python"><div class="highlight"><pre># Use queries to define each category
# (Here I&#39;ll assume &quot;price&quot; is a NUMERIC field, so I&#39;ll use
# NumericRange)
qdict = {}
qdict[&quot;A-D&quot;] = query.TermRange(&quot;name&quot;, &quot;a&quot;, &quot;d&quot;)
qdict[&quot;E-H&quot;] = query.TermRange(&quot;name&quot;, &quot;e&quot;, &quot;h&quot;)
qdict[&quot;I-L&quot;] = query.TermRange(&quot;name&quot;, &quot;i&quot;, &quot;l&quot;)
# ...

qfacet = sorting.QueryFacet(qdict)
r = searcher.search(myquery, groupedby={&quot;firstltr&quot;: qfacet})
</pre></div>
</div>
By default, <tt class="docutils literal">QueryFacet</tt> only supports non-overlapping grouping, where a
document cannot belong to multiple facets at the same time (each document will
be sorted into one category arbitrarily). To get overlapping groups with
multi-valued fields, use the <tt class="docutils literal">allow_overlap=True</tt> keyword argument:
<div class="highlight-python"><div class="highlight"><pre>facet = sorting.QueryFacet(querydict, allow_overlap=True)
</pre></div>
</div>
</div>
<div class="section" id="rangefacet">
<h3>RangeFacet<a class="headerlink" href="#rangefacet" title="Permalink to this headline">¶</a></h3>
The <tt class="docutils literal">RangeFacet</tt> is for NUMERIC field types. It divides a range of possible
values into groups. For example, to group documents based on price into
buckets $100 &#8220;wide&#8221;:
<div class="highlight-python"><div class="highlight"><pre>pricefacet = sorting.RangeFacet(&quot;price&quot;, 0, 1000, 100)
</pre></div>
</div>
The first argument is the name of the field. The next two arguments are the
full range to be divided. Value outside this range (in this example, values
below 0 and above 1000) will be sorted into the &#8220;missing&#8221; (None) group. The
fourth argument is the &#8220;gap size&#8221;, the size of the divisions in the range.
The &#8220;gap&#8221; can be a list instead of a single value. In that case, the values in
the list will be used to set the size of the initial divisions, with the last
value in the list being the size for all subsequent divisions. For example:
<div class="highlight-python"><div class="highlight"><pre>pricefacet = sorting.RangeFacet(&quot;price&quot;, 0, 1000, [5, 10, 35, 50])
</pre></div>
</div>
...will set up divisions of 0-5, 5-15, 15-50, 50-100, and then use 50 as the
size for all subsequent divisions (i.e. 100-150, 150-200, and so on).
The <tt class="docutils literal">hardend</tt> keyword argument controls whether the last division is clamped
to the end of the range or allowed to go past the end of the range. For
example, this:
<div class="highlight-python"><div class="highlight"><pre>facet = sorting.RangeFacet(&quot;num&quot;, 0, 10, 4, hardend=False)
</pre></div>
</div>
...gives divisions 0-4, 4-8, and 8-12, while this:
<div class="highlight-python"><div class="highlight"><pre>facet = sorting.RangeFacet(&quot;num&quot;, 0, 10, 4, hardend=True)
</pre></div>
</div>
...gives divisions 0-4, 4-8, and 8-10. (The default is <tt class="docutils literal">hardend=False</tt>.)
<div class="admonition note">
Note
The ranges/buckets are always inclusive at the start and exclusive
at the end.
</div>
</div>
<div class="section" id="daterangefacet">
<h3>DateRangeFacet<a class="headerlink" href="#daterangefacet" title="Permalink to this headline">¶</a></h3>
This is like <tt class="docutils literal">RangeFacet</tt> but for DATETIME fields. The start and end values
must be <tt class="docutils literal">datetime.datetime</tt> objects, and the gap(s) is/are
<tt class="docutils literal">datetime.timedelta</tt> objects.
For example:
<div class="highlight-python"><div class="highlight"><pre>from datetime import datetime, timedelta

start = datetime(2000, 1, 1)
end = datetime.now()
gap = timedelta(days=365)
bdayfacet = sorting.DateRangeFacet(&quot;birthday&quot;, start, end, gap)
</pre></div>
</div>
As with <tt class="docutils literal">RangeFacet</tt>, you can use a list of gaps and the <tt class="docutils literal">hardend</tt> keyword
argument.
</div>
<div class="section" id="scorefacet">
<h3>ScoreFacet<a class="headerlink" href="#scorefacet" title="Permalink to this headline">¶</a></h3>
This facet is sometimes useful for sorting.
For example, to sort by the &#8220;category&#8221; field, then for documents with the same
category, sort by the document&#8217;s score:
<div class="highlight-python"><div class="highlight"><pre>cats = sorting.FieldFacet(&quot;category&quot;)
scores = sorting.ScoreFacet()
results = searcher.search(myquery, sortedby=[cats, scores])
</pre></div>
</div>
The <tt class="docutils literal">ScoreFacet</tt> always sorts higher scores before lower scores.
<div class="admonition note">
Note
While using <tt class="docutils literal">sortedby=ScoreFacet()</tt> should give the same results as using
the default scored ordering (<tt class="docutils literal">sortedby=None</tt>), using the facet will be
slower because Whoosh automatically turns off many optimizations when
sorting.
</div>
</div>
<div class="section" id="functionfacet">
<h3>FunctionFacet<a class="headerlink" href="#functionfacet" title="Permalink to this headline">¶</a></h3>
This facet lets you pass a custom function to compute the sorting/grouping key
for documents. (Using this facet type may be easier than subclassing FacetType
and Categorizer to set up some custom behavior.)
The function will be called with the index searcher and index document ID as
arguments. For example, if you have an index with term vectors:
<div class="highlight-python"><div class="highlight"><pre>schema = fields.Schema(id=fields.STORED,
 text=fields.TEXT(stored=True, vector=True))
ix = RamStorage().create_index(schema)
</pre></div>
</div>
...you could use a function to sort documents higher the closer they are to
having equal occurances of two terms:
<div class="highlight-python"><div class="highlight"><pre>def fn(searcher, docnum):
 v = dict(searcher.vector_as(&quot;frequency&quot;, docnum, &quot;text&quot;))
 # Sort documents that have equal number of &quot;alfa&quot; and &quot;bravo&quot; first
 return 0 - (1.0 / (abs(v.get(&quot;alfa&quot;, 0) - v.get(&quot;bravo&quot;, 0)) + 1.0))

facet = sorting.FunctionFacet(fn)
results = searcher.search(myquery, sortedby=facet)
</pre></div>
</div>
</div>
<div class="section" id="storedfieldfacet">
<h3>StoredFieldFacet<a class="headerlink" href="#storedfieldfacet" title="Permalink to this headline">¶</a></h3>
This facet lets you use stored field values as the sorting/grouping key for
documents. This is usually slower than using an indexed field, but when using
<tt class="docutils literal">allow_overlap</tt> it can actually be faster for large indexes just because it
avoids the overhead of reading posting lists.
<a class="reference internal" href="api/sorting.html#whoosh.sorting.StoredFieldFacet" title="whoosh.sorting.StoredFieldFacet"><tt class="xref py py-class docutils literal">StoredFieldFacet</tt></a> supports <tt class="docutils literal">allow_overlap</tt> by
splitting the stored value into separate keys. By default it calls the value&#8217;s
<tt class="docutils literal">split()</tt> method (since most stored values are strings), but you can supply
a custom split function. See the section on <tt class="docutils literal">allow_overlap</tt> below.
</div>
</div>
<div class="section" id="multifacet">
<h2>MultiFacet<a class="headerlink" href="#multifacet" title="Permalink to this headline">¶</a></h2>
This facet type returns a composite of the keys returned by two or more
sub-facets, allowing you to sort/group by the intersected values of multiple
facets.
<tt class="docutils literal">MultiFacet</tt> has methods for adding facets:
<div class="highlight-python"><div class="highlight"><pre>myfacet = sorting.RangeFacet(0, 1000, 10)

mf = sorting.MultiFacet()
mf.add_field(&quot;category&quot;)
mf.add_field(&quot;price&quot;, reverse=True)
mf.add_facet(myfacet)
mf.add_score()
</pre></div>
</div>
You can also pass a list of field names and/or <tt class="docutils literal">FacetType</tt> objects to the
initializer:
<div class="highlight-python"><div class="highlight"><pre>prices = sorting.FieldFacet(&quot;price&quot;, reverse=True)
scores = sorting.ScoreFacet()
mf = sorting.MultiFacet(&quot;category&quot;, prices, myfacet, scores)
</pre></div>
</div>
</div>
<div class="section" id="missing-values">
<h2>Missing values<a class="headerlink" href="#missing-values" title="Permalink to this headline">¶</a></h2>
<ul class="simple">
<li>When sorting, documents without any terms in a given field, or whatever else
constitutes &#8220;missing&#8221; for different facet types, will always sort to the end.</li>
<li>When grouping, &#8220;missing&#8221; documents will appear in a group with the
key <tt class="docutils literal">None</tt>.</li>
</ul>
</div>
<div class="section" id="using-overlapping-groups">
<h2>Using overlapping groups<a class="headerlink" href="#using-overlapping-groups" title="Permalink to this headline">¶</a></h2>
The common supported workflow for grouping and sorting is where the given field
has one value for document, for example a <tt class="docutils literal">path</tt> field containing the file
path of the original document. By default, facets are set up to support this
single-value approach.
Of course, there are situations where you want documents to be sorted into
multiple groups based on a field with multiple terms per document. The most
common example would be a <tt class="docutils literal">tags</tt> field. The <tt class="docutils literal">allow_overlap</tt> keyword
argument to the <a class="reference internal" href="api/sorting.html#whoosh.sorting.FieldFacet" title="whoosh.sorting.FieldFacet"><tt class="xref py py-class docutils literal">FieldFacet</tt></a>,
<a class="reference internal" href="api/sorting.html#whoosh.sorting.QueryFacet" title="whoosh.sorting.QueryFacet"><tt class="xref py py-class docutils literal">QueryFacet</tt></a>, and
<a class="reference internal" href="api/sorting.html#whoosh.sorting.StoredFieldFacet" title="whoosh.sorting.StoredFieldFacet"><tt class="xref py py-class docutils literal">StoredFieldFacet</tt></a> allows this multi-value approach.
However, there is an important caveat: using <tt class="docutils literal">allow_overlap=True</tt> is slower
than the default, potentially much slower for very large result sets. This is
because Whoosh must read every posting of every term in the field to
create a temporary &#8220;forward index&#8221; mapping documents to terms.
If a field is indexed with term vectors, <tt class="docutils literal">FieldFacet</tt> will use them to
speed up <tt class="docutils literal">allow_overlap</tt> faceting for small result sets, but for large result
sets, where Whoosh has to open the vector list for every matched document, this
can still be very slow.
For very large indexes and result sets, if a field is stored, you can get
faster overlapped faceting using <a class="reference internal" href="api/sorting.html#whoosh.sorting.StoredFieldFacet" title="whoosh.sorting.StoredFieldFacet"><tt class="xref py py-class docutils literal">StoredFieldFacet</tt></a>
instead of <tt class="docutils literal">FieldFacet</tt>. While reading stored values is usually slower than
using the index, in this case avoiding the overhead of opening large numbers of
posting readers can make it worthwhile.
<tt class="docutils literal">StoredFieldFacet</tt> supports <tt class="docutils literal">allow_overlap</tt> by loading the stored value for
the given field and splitting it into multiple values. The default is to call
the value&#8217;s <tt class="docutils literal">split()</tt> method.
For example, if you&#8217;ve stored the <tt class="docutils literal">tags</tt> field as a string like
<tt class="docutils literal">&quot;tag1 tag2 tag3&quot;</tt>:
<div class="highlight-python"><div class="highlight"><pre>schema = fields.Schema(name=fields.TEXT(stored=True),
 tags=fields.KEYWORD(stored=True))
ix = index.create_in(&quot;indexdir&quot;)
with ix.writer() as w:
 w.add_document(name=&quot;A Midsummer Night&#39;s Dream&quot;, tags=&quot;comedy fairies&quot;)
 w.add_document(name=&quot;Hamlet&quot;, tags=&quot;tragedy denmark&quot;)
 # etc.
</pre></div>
</div>
...Then you can use a <tt class="docutils literal">StoredFieldFacet</tt> like this:
<div class="highlight-python"><div class="highlight"><pre>ix = index.open_dir(&quot;indexdir&quot;)
with ix.searcher() as s:
 sff = sorting.StoredFieldFacet(&quot;tags&quot;, allow_overlap=True)
 results = s.search(myquery, groupedby={&quot;tags&quot;: sff})
</pre></div>
</div>
For stored Python objects other than strings, you can supply a split function
(using the <tt class="docutils literal">split_fn</tt> keyword argument to <tt class="docutils literal">StoredFieldFacet</tt>). The function
should accept a single argument (the stored value) and return a list or tuple
of grouping keys.
</div>
<div class="section" id="using-a-custom-sort-order">
<h2>Using a custom sort order<a class="headerlink" href="#using-a-custom-sort-order" title="Permalink to this headline">¶</a></h2>
It is sometimes useful to have a custom sort order per-search. For example,
different languages use different sort orders. If you have a function to return
the sorting order you want for a given field value, such as an implementation of
the Unicode Collation Algorithm (UCA), you can customize the sort order
for the user&#8217;s language.
The <tt class="xref py py-class docutils literal">whoosh.sorting.TranslateFacet</tt> lets you apply a function to the
value of another facet. This lets you &#8220;translate&#8221; a field value into an
arbitrary sort key, such as with UCA:
<div class="highlight-python"><div class="highlight"><pre>from pyuca import Collator

# The Collator object has a sort_key() method which takes a unicode
# string and returns a sort key
c = Collator(&quot;allkeys.txt&quot;)

# Make a facet object for the field you want to sort on
nf = sorting.FieldFacet(&quot;name&quot;)

# Wrap the facet in a TranslateFacet with the translation function
# (the Collator object&#39;s sort_key method)
tf = sorting.TranslateFacet(facet, c.sort_key)

# Use the facet to sort the search results
results = searcher.search(myquery, sortedby=tf)
</pre></div>
</div>
(You can pass multiple &#8220;wrapped&#8221; facets to the <tt class="docutils literal">TranslateFacet</tt>, and it will
call the function with the values of the facets as multiple arguments.)
The TranslateFacet can also be very useful with numeric fields to sort on the
output of some formula:
<div class="highlight-python"><div class="highlight"><pre># Sort based on the average of two numeric fields
def average(a, b):
 return (a + b) / 2.0

# Create two facets for the fields and pass them with the function to
# TranslateFacet
af = sorting.FieldFacet(&quot;age&quot;)
wf = sorting.FieldFacet(&quot;weight&quot;)
facet = sorting.TranslateFacet(average, af, wf)

results = searcher.search(myquery. sortedby=facet)
</pre></div>
</div>
Remember that you can still sort by multiple facets. For example, you could sort
by a numeric value transformed by a quantizing function first, and then if that
is equal sort by the value of another field:
<div class="highlight-python"><div class="highlight"><pre># Sort by a quantized size first, then by name
tf = sorting.TranslateFacet(quantize, sorting.FieldFacet(&quot;size&quot;))
results = searcher.search(myquery, sortedby=[tf, &quot;name&quot;])
</pre></div>
</div>
</div>
<div class="section" id="expert-writing-your-own-facet">
<h2>Expert: writing your own facet<a class="headerlink" href="#expert-writing-your-own-facet" title="Permalink to this headline">¶</a></h2>
TBD.
</div>
</div>

</div>
 </div>
 </div>
 <div class="sphinxsidebar">
 <div class="sphinxsidebarwrapper">
 <h3><a href="index.html">Table Of Contents</a></h3>
 <ul>
<li><a class="reference internal" href="#">Sorting and faceting</a><ul>
<li><a class="reference internal" href="#overview">Overview</a></li>
<li><a class="reference internal" href="#sorting">Sorting</a><ul>
<li><a class="reference internal" href="#making-fields-sortable">Making fields sortable</a></li>
<li><a class="reference internal" href="#about-column-types">About column types</a></li>
<li><a class="reference internal" href="#using-a-column-field-for-custom-sort-keys">Using a COLUMN field for custom sort keys</a></li>
<li><a class="reference internal" href="#making-existing-fields-sortable">Making existing fields sortable</a></li>
<li><a class="reference internal" href="#sorting-search-results">Sorting search results</a></li>
<li><a class="reference internal" href="#examples">Examples</a></li>
<li><a class="reference internal" href="#accessing-column-values">Accessing column values</a></li>
</ul>
</li>
<li><a class="reference internal" href="#grouping">Grouping</a><ul>
<li><a class="reference internal" href="#the-groupedby-keyword-argument">The <tt class="docutils literal">groupedby</tt> keyword argument</a></li>
<li><a class="reference internal" href="#id1">Examples</a></li>
<li><a class="reference internal" href="#getting-the-faceted-groups">Getting the faceted groups</a></li>
</ul>
</li>
<li><a class="reference internal" href="#facet-types">Facet types</a><ul>
<li><a class="reference internal" href="#fieldfacet">FieldFacet</a></li>
<li><a class="reference internal" href="#queryfacet">QueryFacet</a></li>
<li><a class="reference internal" href="#rangefacet">RangeFacet</a></li>
<li><a class="reference internal" href="#daterangefacet">DateRangeFacet</a></li>
<li><a class="reference internal" href="#scorefacet">ScoreFacet</a></li>
<li><a class="reference internal" href="#functionfacet">FunctionFacet</a></li>
<li><a class="reference internal" href="#storedfieldfacet">StoredFieldFacet</a></li>
</ul>
</li>
<li><a class="reference internal" href="#multifacet">MultiFacet</a></li>
<li><a class="reference internal" href="#missing-values">Missing values</a></li>
<li><a class="reference internal" href="#using-overlapping-groups">Using overlapping groups</a></li>
<li><a class="reference internal" href="#using-a-custom-sort-order">Using a custom sort order</a></li>
<li><a class="reference internal" href="#expert-writing-your-own-facet">Expert: writing your own facet</a></li>
</ul>
</li>
</ul>

<h4>Previous topic</h4>
 <a href="ngrams.html"
 title="previous chapter">Indexing and searching N-grams</a>
 <h4>Next topic</h4>
 <a href="highlight.html"
 title="next chapter">How to create highlighted search result excerpts</a>
 <h3>This Page</h3>
 <ul class="this-page-menu">
 <li><a href="_sources/facets.txt"
 rel="nofollow">Show Source</a></li>
 </ul>
<div id="searchbox" style="display: none">
 <h3>Quick search</h3>
 <form class="search" action="search.html" method="get">
 <input type="text" name="q" />
 <input type="submit" value="Go" />
 <input type="hidden" name="check_keywords" value="yes" />
 <input type="hidden" name="area" value="default" />
 </form>
 
 Enter search terms or a module, class or function name.
 
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
 </div>
 </div>
 <div class="clearer"></div>
 </div>
 <div class="related">
 <h3>Navigation</h3>
 <ul>
 <li class="right" style="margin-right: 10px">
 <a href="genindex.html" title="General Index"
 >index</a></li>
 <li class="right" >
 <a href="py-modindex.html" title="Python Module Index"
 >modules</a> |</li>
 <li class="right" >
 <a href="highlight.html" title="How to create highlighted search result excerpts"
 >next</a> |</li>
 <li class="right" >
 <a href="ngrams.html" title="Indexing and searching N-grams"
 >previous</a> |</li>
 <li><a href="index.html">Whoosh 2.7.4 documentation</a> &raquo;</li> 
 </ul>
 </div>
 <div class="footer">
 &copy; Copyright 2007-2012 Matt Chaput.
 Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 1.1.3.
 </div>
 </body>
</html>