EVOLUTION-MANAGER

Edit File: tests.html

<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="Docutils 0.8.1: http://docutils.sourceforge.net/" />
<title>Tests</title>
<style type="text/css">

/*
:Author: David Goodger (goodger@python.org)
:Id: $Id: html4css1.css 7056 2011-06-17 10:50:48Z milde $
:Copyright: This stylesheet has been placed in the public domain.

Default cascading style sheet for the HTML output of Docutils.

See http://docutils.sf.net/docs/howto/html-stylesheets.html for how to
customize this style sheet.
*/

/* used to remove borders from tables and images */
.borderless, table.borderless td, table.borderless th {
  border: 0 }

table.borderless td, table.borderless th {
  /* Override padding for "table.docutils td" with "! important".
     The right padding separates the table cells. */
  padding: 0 0.5em 0 0 ! important }

.first {
  /* Override more specific margin styles with "! important". */
  margin-top: 0 ! important }

.last, .with-subtitle {
  margin-bottom: 0 ! important }

.hidden {
  display: none }

a.toc-backref {
  text-decoration: none ;
  color: black }

blockquote.epigraph {
  margin: 2em 5em ; }

dl.docutils dd {
  margin-bottom: 0.5em }

object[type="image/svg+xml"], object[type="application/x-shockwave-flash"] {
  overflow: hidden;
}

/* Uncomment (and remove this text!) to get bold-faced definition list terms
dl.docutils dt {
  font-weight: bold }
*/

div.abstract {
  margin: 2em 5em }

div.abstract p.topic-title {
  font-weight: bold ;
  text-align: center }

div.admonition, div.attention, div.caution, div.danger, div.error,
div.hint, div.important, div.note, div.tip, div.warning {
  margin: 2em ;
  border: medium outset ;
  padding: 1em }

div.admonition p.admonition-title, div.hint p.admonition-title,
div.important p.admonition-title, div.note p.admonition-title,
div.tip p.admonition-title {
  font-weight: bold ;
  font-family: sans-serif }

div.attention p.admonition-title, div.caution p.admonition-title,
div.danger p.admonition-title, div.error p.admonition-title,
div.warning p.admonition-title {
  color: red ;
  font-weight: bold ;
  font-family: sans-serif }

/* Uncomment (and remove this text!) to get reduced vertical space in
   compound paragraphs.
div.compound .compound-first, div.compound .compound-middle {
  margin-bottom: 0.5em }

div.compound .compound-last, div.compound .compound-middle {
  margin-top: 0.5em }
*/

div.dedication {
  margin: 2em 5em ;
  text-align: center ;
  font-style: italic }

div.dedication p.topic-title {
  font-weight: bold ;
  font-style: normal }

div.figure {
  margin-left: 2em ;
  margin-right: 2em }

div.footer, div.header {
  clear: both;
  font-size: smaller }

div.line-block {
  display: block ;
  margin-top: 1em ;
  margin-bottom: 1em }

div.line-block div.line-block {
  margin-top: 0 ;
  margin-bottom: 0 ;
  margin-left: 1.5em }

div.sidebar {
  margin: 0 0 0.5em 1em ;
  border: medium outset ;
  padding: 1em ;
  background-color: #ffffee ;
  width: 40% ;
  float: right ;
  clear: right }

div.sidebar p.rubric {
  font-family: sans-serif ;
  font-size: medium }

div.system-messages {
  margin: 5em }

div.system-messages h1 {
  color: red }

div.system-message {
  border: medium outset ;
  padding: 1em }

div.system-message p.system-message-title {
  color: red ;
  font-weight: bold }

div.topic {
  margin: 2em }

h1.section-subtitle, h2.section-subtitle, h3.section-subtitle,
h4.section-subtitle, h5.section-subtitle, h6.section-subtitle {
  margin-top: 0.4em }

h1.title {
  text-align: center }

h2.subtitle {
  text-align: center }

hr.docutils {
  width: 75% }

img.align-left, .figure.align-left, object.align-left {
  clear: left ;
  float: left ;
  margin-right: 1em }

img.align-right, .figure.align-right, object.align-right {
  clear: right ;
  float: right ;
  margin-left: 1em }

img.align-center, .figure.align-center, object.align-center {
  display: block;
  margin-left: auto;
  margin-right: auto;
}

.align-left {
  text-align: left }

.align-center {
  clear: both ;
  text-align: center }

.align-right {
  text-align: right }

/* reset inner alignment in figures */
div.align-right {
  text-align: inherit }

/* div.align-center * { */
/*   text-align: left } */

ol.simple, ul.simple {
  margin-bottom: 1em }

ol.arabic {
  list-style: decimal }

ol.loweralpha {
  list-style: lower-alpha }

ol.upperalpha {
  list-style: upper-alpha }

ol.lowerroman {
  list-style: lower-roman }

ol.upperroman {
  list-style: upper-roman }

p.attribution {
  text-align: right ;
  margin-left: 50% }

p.caption {
  font-style: italic }

p.credits {
  font-style: italic ;
  font-size: smaller }

p.label {
  white-space: nowrap }

p.rubric {
  font-weight: bold ;
  font-size: larger ;
  color: maroon ;
  text-align: center }

p.sidebar-title {
  font-family: sans-serif ;
  font-weight: bold ;
  font-size: larger }

p.sidebar-subtitle {
  font-family: sans-serif ;
  font-weight: bold }

p.topic-title {
  font-weight: bold }

pre.address {
  margin-bottom: 0 ;
  margin-top: 0 ;
  font: inherit }

pre.literal-block, pre.doctest-block, pre.math {
  margin-left: 2em ;
  margin-right: 2em }

span.classifier {
  font-family: sans-serif ;
  font-style: oblique }

span.classifier-delimiter {
  font-family: sans-serif ;
  font-weight: bold }

span.interpreted {
  font-family: sans-serif }

span.option {
  white-space: nowrap }

span.pre {
  white-space: pre }

span.problematic {
  color: red }

span.section-subtitle {
  /* font-size relative to parent (h1..h6 element) */
  font-size: 80% }

table.citation {
  border-left: solid 1px gray;
  margin-left: 1px }

table.docinfo {
  margin: 2em 4em }

table.docutils {
  margin-top: 0.5em ;
  margin-bottom: 0.5em }

table.footnote {
  border-left: solid 1px black;
  margin-left: 1px }

table.docutils td, table.docutils th,
table.docinfo td, table.docinfo th {
  padding-left: 0.5em ;
  padding-right: 0.5em ;
  vertical-align: top }

table.docutils th.field-name, table.docinfo th.docinfo-name {
  font-weight: bold ;
  text-align: left ;
  white-space: nowrap ;
  padding-left: 0 }

h1 tt.docutils, h2 tt.docutils, h3 tt.docutils,
h4 tt.docutils, h5 tt.docutils, h6 tt.docutils {
  font-size: 100% }

ul.auto-toc {
  list-style-type: none }

</style>
</head>
<body>
<div class="document" id="tests">
<h1 class="title">Tests</h1>

<div class="contents topic" id="contents">
<p class="topic-title first">Contents</p>
<ul class="simple">
<li><a class="reference internal" href="#a-brief-guide-to-running-tests" id="id1">A Brief Guide to Running Tests</a></li>
<li><a class="reference internal" href="#a-brief-guide-to-writing-tests" id="id2">A Brief Guide to Writing Tests</a></li>
</ul>
</div>
<div class="section" id="a-brief-guide-to-running-tests">
<span id="running"></span><h1><a class="toc-backref" href="#id1">A Brief Guide to Running Tests</a></h1>
<p>After a successful &quot;<tt class="docutils literal">make</tt>&quot;, try &quot;<tt class="docutils literal">make check</tt>&quot;.</p>
<p>It's possible to run test cases individually, and get verbose output
when one fails, etc. For more information, see the &quot;Running test
programs&quot; section of HACKING.</p>
</div>
<div class="section" id="a-brief-guide-to-writing-tests">
<span id="writing"></span><h1><a class="toc-backref" href="#id2">A Brief Guide to Writing Tests</a></h1>
<p>Test programs live in <tt class="docutils literal">tests/</tt>. They mostly use a standard test
harness, in <tt class="docutils literal">tests/harness/</tt>, which wraps each test, reports results,
and generally packages things up nicely. The test harness counts how
many testcases pass/fail/skip, catches signals and unhandled exceptions,
and so forth. It can also also check for memory leaks and accesses to
uninitialised values by making use of valgrind, for platforms which
valgrind supports (configure automatically enables use of valgrind if a
suitably recent version is detected).</p>
<p>A typical test program has three parts: the tests themselves (at the
top), a table of tests (at the bottom), and a tiny main which sets the
test harness in motion. It uses the table to figure out what the tests
are called, and what function to call to run them.</p>
<p>The most important test system for most people will be <tt class="docutils literal">apitest</tt>. This
also uses the test harness, but has several tables of tests to be run
depending what facilities each backend supports. A lot of the work is
done by macros and helper functions, which may make it hard to work out
quite what is going on, but make life easier once you've grasped what's
going on. The <tt class="docutils literal">main()</tt> function and other bits are in <tt class="docutils literal">apitest.cc</tt>,
and tests themselves are in various other C++ files starting api_. Each
one of these has its own tables for various different groups of tests
(eg: <tt class="docutils literal">api_db.cc</tt>, which performs tests on the API that require a
database backend, has basic tests, a few specialised groups that only
contain one or two tests, tests that require a writable database, tests
that require a local database, and finally tests that require a remote
database).</p>
<p>To add a new api test, figure out what the test will be dependent on and
put it in the appropriate place (eg: if adding a test for a bug that
occurs while writing to a database, you want a writable database, so you
add a test to <tt class="docutils literal">api_db.cc</tt> and reference it in the <tt class="docutils literal">writabledb_tests</tt>
table).</p>
<p>Currently, there's <tt class="docutils literal">api_nodb.cc</tt> (no db required, largely testing
query construction and boundary conditions), <tt class="docutils literal">api_posdb.cc</tt> (db with
positional information required) and <tt class="docutils literal">api_db.cc</tt> (everything else,
with lots of subgroups of tests). It's easiest to base a test on an
existing one.</p>
<p>You'll notice in <tt class="docutils literal">apitest.cc</tt> that it runs all appropriate test groups
against each backend that is being built. The backends are inmemory,
multi, brass, chert, flint, remoteprog and remotetcp. If you need to
create a new test group with different requirements to any current ones,
put it in the appropriate api_ file (or create a new one, and add it
into Makefile.am) and remember to add the group to all pertinent
backends in <tt class="docutils literal">apitest.cc</tt>.</p>
<p>Incidentally, when fixing bugs, it's often better to write the test
before fixing the bug. Firstly, it's easier to assure yourself that the
bug is (a) genuine, and (b) fixed, because you see the test go from fail
to pass (though sometimes you don't get the testcase quite right, so
this isn't doesn't always work as well as it should). Secondly you're
more likely to write the test carefully, because once you've fixed
something there's often a feeling that you should commit it for the good
of the world, which tends to distract you.</p>
<p>The framework is done for you, so you don't need to worry about that
much. You are responsible for doing two things:</p>
<ol class="arabic simple">
<li>writing a minimal test or tests for the feature</li>
<li>adding that test to the list of tests to be run</li>
</ol>
<p>Adding the test is simple. There's a test_desc array in each file that
comprises a set of tests (I'll come to that in a minute), and you just
add another entry. The entry is an array consisting of a name for the
test and a pointer to the function that is the test. Easy. The procedure
is even simpler for apitest tests - there you just use DEFINE_TESTCASE
to define your new testcase, and a script picks it up and makes sure it
is run.</p>
<p>Look at the bottom of <tt class="docutils literal">tests/stemtest.cc</tt> for the test_desc array.
Now look up about 20 lines to where the test functions are defined. You
need to write a function like these which will return true or false
depending on whether it failed or not.</p>
<p>In addition, there are a bunch of macros to help you perform standards
testing tasks. Things like TEST_EQUAL are all in
<tt class="docutils literal">tests/harness/testsuite.h</tt>. They're pretty simple to use.</p>
</div>
</div>
</body>
</html>