From: Beman Dawes (bdawes_at_[hidden])
Date: 2002-08-12 13:26:40
At 11:58 AM 8/12/2002, Daniel Frey wrote:
>...
>
>Maybe we should separate tests to pure regression tests that check the
>existence / correctness of a feature and benchmark-tests that give
>information about the efficiency / speed of some feature. During
>development, you run the regression tests first and only if they pass,
>you may want to run the benchmark tests. The latter would not report
>results as booleans (pass/fail), but it will have timings, object
>counters, size of objects (to see the overhead of a library), etc.
Interesting. I already have a request in to the Jam Boost folks for
version 2 to supply timings, but had not given any thought to space and
size issues.
Space and size is probably not something bjam can report directly. Some
subsidiary process is likely to be needed to capture that information. Does
anyone have any ideas about how that might work?
After so many people have put so much effort into the bjam regression
testing system, and now that the compiler vendors are taking regular notice
of the results we post, I've come to feel we should definitely report more
than just pass/fail.
We probably won't do much more right away, since we need to finish the
current transition to bjam based tests, and version 2 of Boost.Build isn't
ready anyhow. But we do need to start thinking about future directions for
testing. Better factoring and other organizational issues are part of it,
but time and space testing should also be given attention, IMO.
--Beman