From: Beman Dawes (bdawes_at_[hidden])
Date: 2004-02-12 12:09:17
At 08:30 AM 2/12/2004, David Abrahams wrote:
>Beman Dawes <bdawes_at_[hidden]> writes:
>
>> Anyhow, I think your point about multiple reporting is a good one. The
>> volume of tests is just too high. Fewer, more comprehensive, tests
>> would be easier to monitor. Also fewer compilers. Do we really need to
>> test every version of GCC and VC++ for the last four years?
>
>Yes, IMO, if people want to support those compilers, we do need to
>test them.
It comes down to resources. I suppose if people really care about older
compilers, they will be willing to contribute testing.
>
>> If our testing was more focused, we could cycle the tests more often
>> too.
>
>I'm really unconvinced that we need smaller tests on fewer compilers.
I don't think smaller tests is a good idea. I'm asking for "more
comprehensive" tests.
Say a library now covers 100 test cases spread out among five test
programs. I'd like to see those five programs refactored into one program,
still covering the 100 test cases. Or even adding more test cases. A single
program would cut the overhead, including the human overhead.
It would be easier to know if any particular tests was a resource problem
if there was a bjam option to time the build and run aspects of the tests.
The change Jeremy made recently to one or two of the graph tests noticeably
sped testing, but it was only happenstance that I realized they were
problems.
>Human monitoring is just too error-prone. Why risk it? Why not have
>comprehensive tests with automated notifications when something
>breaks? It seems to me that less testing can only result in fewer
>things working, and coupled with human monitoring it's just going to
>make things worse, not better.
Great! Let's give it a try. Who has to do what to start?
--Beman