From: jurko.gospodnetic_at_[hidden]
Date: 2008-05-01 13:08:04
Author: jurko
Date: 2008-05-01 13:08:04 EDT (Thu, 01 May 2008)
New Revision: 44983
URL: http://svn.boost.org/trac/boost/changeset/44983
Log:
Updated the Boost Build test system documentation. Now all the command line options are described. Several function descriptions updated.
Text files modified:
trunk/tools/build/v2/test/test_system.html | 330 ++++++++++++++++++++++++---------------
1 files changed, 201 insertions(+), 129 deletions(-)
Modified: trunk/tools/build/v2/test/test_system.html
==============================================================================
--- trunk/tools/build/v2/test/test_system.html (original)
+++ trunk/tools/build/v2/test/test_system.html 2008-05-01 13:08:04 EDT (Thu, 01 May 2008)
@@ -12,14 +12,14 @@
<style type="text/css">
hr { color: black }
p.revision { text-align: right; font-style: italic }
- pre.code { margin-left: 2em }
- pre.example { margin-left: 2em; border: solid black thin }
+ pre.code { margin-left: 2em }
+ pre.example { margin-left: 2em; border: solid black thin }
pre.output { margin-left: 2em }
img.banner { border: 0; float: left }
h1 { text-align: right }
br.clear { clear: left }
div.attention { color: red }
-
+
</style>
</head>
@@ -34,6 +34,12 @@
<dl class="page-index">
<dt>Introduction for users</dt>
+ <dd>
+ <dl class="page-index">
+ <dt>Command line options</dt>
+ </dl>
+ </dd>
+
<dt>Introduction for developers</dt>
<dd>
@@ -41,8 +47,8 @@
<dt><a href="#sec-intro-changing">Changing the working
directory</a></dt>
- <dt><a href="#sec-intro-examining">Examining the working directory
- and changing it</a></dt>
+ <dt><a href="#sec-intro-examining">Examining the working directory and
+ changing it</a></dt>
<dt>Test result</dt>
</dl>
@@ -87,14 +93,13 @@
<h2><a name="sec-intro">Introduction for users</a></h2>
- <p>The testing system for Boost.Build is a small set of Python modules
- and scripts for automatically testing user-obversable behaviour. It uses
- components from testing systems of <a href=
- "http://www.scons.org">Scons</a> and <a href=
- "http://subversion.tigris.org">Subversion</a>, together with some
- additional functionality.</p>
+ <p>The testing system for Boost.Build is a small set of Python modules and
+ scripts for automatically testing user-obversable behaviour. It uses
+ components from testing systems of Scons
+ and Subversion, together with
+ some additional functionality.</p>
- <p>To run the tests you'd need:</p>
+ <p>To run the tests you need to:</p>
<ol>
<li>Get the source tree of Boost.Build (located at <tt>tools/build</tt>
@@ -115,24 +120,46 @@
for testing.</li>
</ol>
- <p>When all is done, you can run the tests with</p>
+ <p>When all is set, you can run all the tests using the <tt>test_all.py</tt>
+ script or you can run a specific test by starting its Python script
+ directly.</p>
+
+ <p>Examples:</p>
+
<pre class="code">
python test_all.py
+python generators_test.py
</pre>
- <p>which will use the default toolset, or you can specify toolset on the
- command line, for example:</p>
+ <p>If everything is OK, you will see a list of passed tests. Otherwise, a
+ failure will be reported.</p>
+
+ <h3><a name="sec-command-line-options">Command line options</a></h3>
+
+ <p>Test scripts will use the toolset you configured to be the default or
+ you can specify a specific one on the command line:</p>
+
<pre class="code">
python test_all.py borland
+python generators_test.py msvc-7.1
</pre>
- <p>If everything's OK, you'll see a list of passed tests. Otherwise, a
- failure will be reported.</p>
+ <p>Other test script flags you can specify on the command line are:</p>
- <p>It is possible to run a specific test, for example:</p>
-<pre class="code">
-python generators_test.py
-</pre>
+ <ul>
+ <li><tt>--default-bjam</tt> -- By default the test system will use the
+ Boost Jam executable found built in its default development build
+ location. This option makes it use the default one available on your
+ system, i.e. the one found in the system path.</li>
+
+ <li><tt>--preserve</tt> -- In case of a failed test its working
+ directory will be copied to the "failed_test" directory under the
+ current directory.</li>
+
+ <li><tt>--verbose</tt> -- Makes the test system and the run build system
+ display additional output. Note though that this may cause tests that
+ check the build system output to fail.</li>
+ </ul>
<h2><a name="sec-developers">Introduction for developers</a></h2>
@@ -144,28 +171,28 @@
<li>For an interpreted language like Jam, without any static checks,
testing is simply the only sefeguard we can have.</li>
- <li>Good tests allow to change internal design much more safely, and we
- didn't nailed everything down yet.</li>
+ <li>Good tests allow us to change internal design much more safely, and we
+ have not gotten everything nailed down yet.</li>
</ul>
<p>Adding a new test is simple:</p>
<ol>
<li>Go to <tt>$boost_build_root/test/test_all.py</tt> and add new test
- name to the list at the end of file. Suppose the test name is
- "hello".</li>
+ name to the list at the end of the file. Suppose the test name is "hello".
+ </li>
- <li>Add a new python module, in this example "hello.py", to do actual
+ <li>Add a new python module, in this example "hello.py", to do the actual
testing.</li>
</ol>
<p>The module, in general will perform these basic actions:</p>
<ol>
- <li>Setting the initial working directory state</li>
+ <li>Set up the initial working directory state</li>
<li>
- Running the build system and checking:
+ Run the build system and check the results:
<ol>
<li>generated output,</li>
@@ -176,10 +203,10 @@
</ol>
</li>
- <li>Adding, removing or touching files, or changing their content and
- then repeating the previous step, until satisfied.</li>
+ <li>Add, remove or touch files or change their content and then repeat
+ the previous step until satisfied.</li>
- <li>Cleaning up</li>
+ <li>Clean up</li>
</ol>
<p>The "hello.py" module might contain:</p>
@@ -204,12 +231,12 @@
t.run_build_system()
-# First, create a list of three pathnames
+# First, create a list of three pathnames.
file_list = List("bin/$toolset/debug/") * List("hello.exe hello.obj")
# Second, assert that those files were added as result of the last build system invocation.
t.expect_addition(file_list)
-# Invoke the build system once again
+# Invoke the build system once again.
t.run_build_system("clean")
# Check if the files added previously were removed.
t.expect_removal(file_list)
@@ -221,18 +248,18 @@
<p>The <tt>test</tt> directory contains a file "template.py" which can be
used as a start for your own tests.</p>
- <p>Overview of the most important methods of class
- <tt>TestBoostBuild</tt> follows.</p>
+ <p>Overview of the most important methods of class <tt>Tester</tt> follows.
+ </p>
<h3><a name="sec-intro-changing">Changing the working directory</a></h3>
- <p>The class <tt>TestBoostBuild</tt> creates a temporary directory in its
+ <p>The class <tt>Tester</tt> creates a temporary directory in its
constructor and changes to that directory. It can be modified by calling
these methods:</p>
<ul>
- <li><tt>set_tree</tt> -- sets the content of the working directory to
- be equal to the content of the specified directory. This method is
+ <li><tt>set_tree</tt> -- sets the content of the working directory to be
+ equal to the content of the specified directory. This method is
preferrable when directory tree for testing is large.</li>
<li><tt>write</tt> -- sets the content of file in a working directory.
@@ -245,28 +272,27 @@
<h3><a name="sec-intro-examining">Examining the working directory and
changing it</a></h3>
- <p>The method <tt>read</tt>, inherited from the <tt>TestCmd</tt> class,
- can be used to read any file in the working directory and check its
- content. <tt>TestBoostBuild</tt> adds another method for tracking
- changes. Whenever build system is run (via <tt>run_build_system</tt>),
- the state of working dir before and after running is recorded. In
- addition, difference between the two states -- i.e. lists of files that
- were added, removed, modified or touched -- is stored in two member
- variables, <tt>tree_difference</tt> and
- <tt>unexpected_difference</tt>.</p>
-
- <p>After than, the test author may specify that some change is expected,
- for example, by calling <tt>expect_addition("foo")</tt>. This call will
- check if the file was indeed added, and if so, will remove its name from
- the list of added files in <tt>unexpected_difference</tt>. Likewise, it's
- possible to specify that some changes are not interesting, for example a
- call <tt>ignore("*.obj")</tt> will just remove every files with ".obj"
+ <p>The method <tt>read</tt>, inherited from the <tt>TestCmd</tt> class, can
+ be used to read any file in the working directory and check its content.
+ <tt>Tester</tt> adds another method for tracking changes. Whenever the build
+ system is run (using <a href="#method-run_build_system"><tt>run_build_system
+ </tt></a>), the working dir state before and after running is recorded. In
+ addition, difference between the two states -- i.e. lists of files that were
+ added, removed, modified or touched -- are stored in two member variables -
+ <tt>tree_difference</tt> and <tt>unexpected_difference</tt>.</p>
+
+ <p>After than, the test author may specify that some change is expected, for
+ example, by calling <tt>expect_addition("foo")</tt>. This call will check if
+ the file was indeed added, and if so, will remove its name from the list of
+ added files in <tt>unexpected_difference</tt>. Likewise, it is possible to
+ specify that some changes are not interesting, for example a call to
+ <tt>ignore("*.obj")</tt> will just remove every file with the ".obj"
extension from <tt>unexpected_difference</tt>.</p>
<p>When test has finished with expectations and ignoring, the member
- <tt>unexpected_difference</tt> will contain the list of all changes not
- yet accounted for. It is possible to assure that this list is empty by
- calling <tt>expect_nothing_more</tt> member function.</p>
+ <tt>unexpected_difference</tt> will contain the list of all changes not yet
+ accounted for. It is possible to assure that this list is empty by calling
+ the <tt>expect_nothing_more</tt> member function.</p>
<h3><a name="sec-intro-results">Test result</a></h3>
@@ -277,24 +303,74 @@
<tt>pass_test</tt> and <tt>fail_test</tt> are used to explicitly give the
test outcome.</p>
- <p>Typically, after test termination, the working directory is erased. To
- debug a failed test, you should add "--preserve" option when invoking
- test. On failure, the working directory will be copied to "failed_test"
- directory in the current dir.</p>
+ <p>Typically, after test termination, the working directory is erased. See
+ the "--preserve" command line option
+ for information on how to preserve the working directory content for failed
+ tests for debugging purposes.</p>
<h2 id="sec-reference">Reference documentation</h2>
<p>The test system is composed of class <tt>Tester</tt>, derived form
- <tt>TestCmd.TestCmd</tt>, and helper class <tt>List</tt>. The methods of
- <tt>Tester</tt>, and the class <tt>List</tt> are described below.</p>
+ <tt>TestCmd.TestCmd</tt>, and helper class <tt>List</tt>. <tt>Tester</tt>
+ and <tt>List</tt> methods are described below.</p>
+
+ <p>The documentation frequently refers to <tt>filename</tt>. In all cases,
+ files are specified in unix style: a sequence of components, separated by
+ "/". This is true on all platforms. In some contexts a list of files is
+ allowed. In those cases any object with a sequence interface is allowed.</p>
+
+ <h3><a name="method-__init__">Method <tt>__init__(self, arguments="",
+ executable="bjam", match=TestCmd.match_exact, boost_build_path=None,
+ translate_suffixes=True, pass_toolset=True, use_test_config=True,
+ ignore_toolset_requirements=True, workdir="", **keywords)</tt></a></h3>
- <p>The documentation frequently refer to filename. In all cases, files
- are specified in unix style: a sequence of components, separated by "/".
- This is true on all platforms. In some contexts, a list of files is
- allowed. In that case any object with sequence interface is allowed.</p>
+ <p><b>Optional arguments:</b></p>
- <h3><a name="method-__init__">Method <tt>__init__(self, workdir='',
- arguments='', executable='bjam')</tt></a></h3>
+ <ul>
+ <li><tt>arguments</tt>
+ - Arguments passed to the run executable.</li>
+ <li><tt>executable</tt>
+ - Name of the executable to invoke.</li>
+ <li><tt>match</tt>
+ - Function to use for compating actual and expected file contents.
+ </li>
+ <li><tt>boost_build_path</tt>
+ - Boost build path to be passed to the run executable.</li>
+ <li><tt>translate_suffixes</tt>
+ - Whether to update suffixes on the the file names passed from the
+ test script so they match those actually created by the current
+ toolset. For example, static library files are specified by using
+ the .lib suffix but when the 'gcc' toolset is used it actually
+ creates them using the .a suffix.</li>
+ <li><tt>pass_toolset</tt>
+ - Whether the test system should pass the specified toolset to the
+ run executable.</li>
+ <li><tt>use_test_config</tt>
+ - Whether the test system should tell the run executable to read in
+ the test_config.jam configuration file.</li>
+ <li><tt>ignore_toolset_requirements</tt>
+ - Whether the test system should tell the run executable to ignore
+ toolset requirements.</li>
+ <li><tt>workdir</tt>
+ - indicates an absolute directory where the test will be run from.
+ </li>
+ </ul>
+
+ <p><b>Optional arguments inherited from the base class:</b></p>
+
+ <ul>
+ <li><tt>description</tt>
+ - Test description string displayed in case of a failed test.</li>
+ <li><tt>subdir</tt>
+ - List of subdirectories to automatically create under the working
+ directory. Each subdirectory needs to be specified separately
+ parent coming before its child.</li>
+ <li><tt>verbose</tt>
+ - Flag that may be used to enable more verbose test system output.
+ Note that it does not also enable more verbose build system output
+ like the <a href="#sec-command-line-options">"--verbose" command
+ line option</a> does.</li>
+ </ul>
<p><b>Effects:</b></p>
@@ -302,20 +378,23 @@
<li>Remembers the current working directory in member
<tt>original_workdir</tt>.</li>
- <li>Determines the location of executable (<code>bjam</code> by
+ <li>Determines the location of the executable (<code>bjam</code> by
default) and build system files, assuming that the current directory is
<tt>tools/build/test</tt>. Formulates jam invocation command, which
- will include explicit setting for <tt>BOOST_BUILD_PATH</tt> variable
+ will include explicit setting for the <tt>BOOST_BUILD_PATH</tt> variable
and arguments passed to this methods, if any. This command will be used
- by subsequent invocation of <a href=
- "#method-run_build_system"><tt>run_build_system</tt></a>. Finally,
- initializes the base class.</li>
+ by subsequent invocation of <a href="#method-run_build_system"><tt>
+ run_build_system</tt></a>. Finally, initializes the base class.</li>
- <li>Changes current working dir to the temporary working directory
- created by the base constructor.</li>
+ <li>Changes the current working directory to the temporary working
+ directory created by the base constructor.</li>
- <li>If you want to run a test in a existing directory, pass it to
+ <li>If you want to run a test in an existing directory, pass it as
<tt>workdir</tt>.</li>
+
+ <li> Most parameters passed to this constructor function may be overruled
+ for each specific test system run (using <a href=
+ "#method-run_build_system"><tt> run_build_system</tt></a> parameters).
</ol>
<h3><a name="method-set_tree">Method <tt>set_tree(self,
@@ -353,7 +432,8 @@
relative paths.</p>
<h3><a name="method-run_build_system">Method <tt>run_build_system(self,
- subdir='', extra_args='', stdout=None, stderr='', status=0,
+ extra_args="", subdir="", stdout=None, stderr="", status=0, match=None,
+ pass_toolset=None, use_test_config=None, ignore_toolset_requirements=None,
**kw)</tt></a></h3>
<p><b>Effects:</b></p>
@@ -396,104 +476,95 @@
<p>Read the specified file and returns it content. Raises an exception is
the file is absent.</p>
- <h3><a name="method-read_and_strip">Method <tt>read_and_strip(self,
- name)</tt></a></h3>
+ <h3><a name="method-read_and_strip">Method <tt>read_and_strip(self, name)
+ </tt></a></h3>
<p><b>Effects:</b></p>
- <p>Read the specified file and returns it content, after removing
- trailing whitespace from every line. Raises an exception is the file is
- absent.</p>
+ <p>Read the specified file and returns it content, after removing trailing
+ whitespace from every line. Raises an exception is the file is absent.</p>
<p><b>Rationale:</b></p>
- <p>Althought this method is questionable, there are a lot of cases when
- jam or shells it uses insert spaces. It seems that introducing this
- method is much simpler than dealing with all those cases.</p>
-
- <h3><a name="methods-expectations">Methods for declaring
- expectations</a></h3>
-
- <p>Accordingly to the number of changes kinds that are detected, there
- are four methods that specify that test author expects a specific change
- to occur. They check <tt>self.unexpected_difference</tt>, and if the
- change is present there, it is removed. Otherwise, test fails.</p>
+ <p>Althought this method is questionable, there are a lot of cases when jam
+ or shells it uses insert spaces. It seems that introducing this method is
+ much simpler than dealing with all those cases.</p>
+
+ <h3><a name="methods-expectations">Methods for declaring expectations</a>
+ </h3>
+
+ <p>Accordingly to the number of changes kinds that are detected, there are
+ four methods that specify that test author expects a specific change to
+ occur. They check <tt>self.unexpected_difference</tt>, and if the change is
+ present there, it is removed. Otherwise, test fails.</p>
<p>Each method accepts a list of names. Those names use <tt>/</tt> path
- separator on all systems. Additionaly, the test system translates
- suffixes appropriately. For the test to be portable, suffixes should use
- Windows convention: <tt>exe</tt> for executables, <tt>dll</tt> for
- dynamic libraries and <tt>lib</tt> for static libraries. Lastly, the
- string "$toolset" in file names is replaced by the name of tested
- toolset.</p>
+ separator on all systems. Additionaly, the test system translates suffixes
+ appropriately. For the test to be portable, suffixes should use Windows
+ convention: <tt>exe</tt> for executables, <tt>dll</tt> for dynamic libraries
+ and <tt>lib</tt> for static libraries. Lastly, the string "$toolset" in file
+ names is replaced by the name of tested toolset.</p>
<p><b>Note:</b> The <tt>List</tt> helper class might be useful to create
lists of names.</p>
- <p><b>Note:</b> The file content can be examined using
+ <p><b>Note:</b> The file content can be examined using the
<tt>TestCmd.read</tt> function.</p>
<p>The members are:</p>
<ul>
<li>expect_addition</li>
-
<li>expect_removal</li>
-
<li>expect_modification</li>
-
<li>expect_nothing</li>
</ul>
<p>Note that <tt>expect_modification</tt> is used to check that a either
- file content or timestamp has changed. The rationale is that some
- compilers change content even if sources does not change, and it's easier
- to have a method which checks for both content and time changes.</p>
+ file content or timestamp has changed. The rationale is that some compilers
+ change content even if sources does not change, and it's easier to have a
+ method which checks for both content and time changes.</p>
- <p>There's also a member <tt>expect_nothing_more</tt>, which checks that
- all the changes are either expected or ignored, in other words that
+ <p>There's also a member <tt>expect_nothing_more</tt>, which checks that all
+ the changes are either expected or ignored, in other words that
<tt>unexpected_difference</tt> is empty by now.</p>
<p>Lastly, there's a method to compare file content with expected
content:</p>
- <tt>expect_content(self, name, content, exact=0)</tt>
+ <tt>expect_content(self, name, content, exact=0)</tt>
- <p>The method fails the test if the content of file identified by 'name'
- is different from 'content'. If 'exact' is true, the file content is used
+ <p>The method fails the test if the content of file identified by 'name' is
+ different from 'content'. If 'exact' is true, the file content is used
as-is, otherwise, two transformations are applied:</p>
<ul>
<li>The <tt>read_and_strip</tt> method is used to read the file, which
removes trailing whitespace</li>
- <li>Each backslash in the file content is converted to forward
- slash.</li>
+ <li>Each backslash in the file content is converted to forward slash.</li>
</ul>
<h3><a name="methods-ignoring">Methods for ignoring changes</a></h3>
<p>There are five methods which ignore changes made to the working tree.
- They silently remove elements from <tt>self.unexpected_difference</tt>,
- and don't generate error if element is not found. They accept shell style
+ They silently remove elements from <tt>self.unexpected_difference</tt>, and
+ don't generate error if element is not found. They accept shell style
wildcard.</p>
<p>The following methods correspond to four kinds of changes:</p>
<ul>
<li>ignore_addition(self, wildcard)</li>
-
<li>ignore_removal(self, wildcard)</li>
-
<li>ignore_modification(self, wildcard)</li>
-
<li>ignore_touch(self, wilcard)</li>
</ul>
- <p>The method <tt>ignore(self, wildcard)</tt> ignores all the changes
- made to files that match a wildcard.</p>
+ <p>The method <tt>ignore(self, wildcard)</tt> ignores all the changes made
+ to files that match a wildcard.</p>
- <h3><a name="methods-result">Methods for explicitly specifying
- results</a></h3>
+ <h3><a name="methods-result">Methods for explicitly specifying results</a>
+ </h3>
<h4>Method <tt>pass_test(self, condition=1)</tt></h4>
@@ -507,7 +578,7 @@
true.</p>
<h3><a name="class-list">Helper class <tt>List</tt></a></h3>
- The class has sequence interface and two additional methods.
+ The class has sequence interface and two additional methods.
<h4>Method <tt>__init__(self, string)</tt></h4>
@@ -529,8 +600,8 @@
<pre>
l = "a b" * List("c d")
for e in l:
- print e
-
+ print e
+
</pre>
<p>will output</p>
@@ -539,13 +610,14 @@
ad
bc
bd
-
+
</pre>
<hr>
- <p class="revision">Last modified: Mar 11, 2005</p>
+ <p class="revision">Last modified: May 01, 2008</p>
- <p>© Copyright Vladimir Prus 2002, 2003, 2004, 2005.
+ <p>© Copyright Vladimir Prus 2002, 2003, 2004, 2005.<br>
+ © Copyright Jurko Gospodnetic 2008.<br>
Distributed under the Boost Software License, Version 1.0.
(See accompanying file LICENSE_1_0.txt or http://www.boost.org/LICENSE_1_0.txt)</p>
</body>