Subject: Re: [boost] What Should we do About Boost.Test?
From: John Maddock (boost.regex_at_[hidden])
Date: 2012-09-17 13:02:45
> I don't know what to do about this. Because of the lack of redundancy
> (i.e. tests and documentation), it's hard to tell whether this library
> is correct or even to define what "correct" should mean. It seems like,
> as long as the code is incompletely / incorrectly documented and tested,
> it's just someone's personal coding project that we happen to keep
> shipping with Boost, and not really a library for general use. This
> situation reflects poorly on Boost as a whole and the fact that it
> centers around a _testing_ library, which is concerned with
> robustness... well, let's just say that the irony isn't lost on me.
While we're on the subject, I confess I thought about writing something in
this area, but hey, too much already! :-)
Here's my wish list though:
* Lightweight, header only if possible.
* Clear separation between components (execution monitor, from unit test
call framework, from testing macros). Ideally each would be a separate mini
library if that's possible, with the executable linking against just what it
needs and no more.
* Thread safe from the start, testing from multiple threads should be a
no-brainer.
* Easy debugging: if I step into a test case in the debugger the first thing
I should see is *my code*. As it is I have to step in and out of dozens of
Boost.Test functions before I get to my code. This one really annoys me.
* Rapid execution of each test case, a BOOST_CHECK(no-op) should be as near
to a no-op as possible. I was unable to use Boost.Test for a lot of the
Math lib tests for this reason - looping over thousands of tests was simply
impractical from a time point of view (maybe this has improved since then, I
haven't checked).
* Exemplary error messages when things fail - Boost.Test has improved in
this area, but IMO not enough.
* An easy way to tell if the last test has failed, and/or an easy way to
print auxiliary information when the last test has failed. This is
primarily for testing in loops, when iterating over tabulated test data.
* Relatively simple C++ code, with no advanced/poorly supported compiler
features used. This is one library that should be usable anywhere and
everywhere.
* Ultra stable code. Exempting bug fixes, I'd like to see a testing library
almost never change, or only change after very careful consideration, for
example if a new C++ language feature requires special testing support.
And what I don't want:
* Breaking changes: Boost authors have absolutely no time to track breaking
changes in their dependencies, since a successful testing library would be
used universally by all of Boost, this is particularly important for this
library.
* No feature creep. Keep it small, focused, quick to compile. If new
features are added they should be separate (i.e. only pay for what you use).
John.