From: Doug Gregor (dgregor_at_[hidden])
Date: 2005-04-01 16:01:54
On Mar 11, 2005, at 10:08 PM, Misha Bergal wrote:
> Douglas Gregor <doug.gregor_at_[hidden]> writes:
>
>> To improve the reproducibility of results and make testing more
>> predictable, we might want to have the regression scripts always check
>> out using a given date/time tag, e.g., 12:00am EST each night. That
>> way, all of the tests for the day will be on the same codfe. If it
>> helps fix other problems with regression testing, great!
>
> Doug, I want to put in writing some things which I believe to be
> important to be stated explicitly. I believe that at the end, you are
> the release manager for 1.33.0. As such, you have a great influence on
> what the testing group goals are - you are our "main customer".
I've been a rather silent customer, but time is again (temporarily) my
friend :)
> You just need to tell us what you consider to be your main problems
> and their priorities.
>
> For example,
>
> * Developers/Release manager need to see what changes caused this
> particular test to fail.
I think getting more immediate feedback about new failures would go
along way in helping is determine which changes cause failures. The GCC
developers have a wonderful system that complains very loudly when
someone checks in broken code. When new regressions are found, it:
(1) Determines what code has changed since the last-known-good version
(2) Determines who made changes to the repository since then
(3) E-mails everyone that made changes, giving them a summary of the
new failures and a link to the log file. Alternatively, we could spam
the developer list or a regression-testing list (in decreasing order of
effectiveness).
(4) Keeps e-mailing everyone until the problem is fixed.
The last one isn't really necessary, but having something that does the
first three would be *great*. It might even be nice to know when we fix
a regression (it can sometimes be hard to tell give the compiler status
tables).
> * Release manager should be able to set what revision of codebase is
> getting tested and get all results for that particular revision
This is extremely important. Being able to easily change all of the
regression testing to a branch means that we might be able to move to a
more lightweight release management process. If it's easy for the
release manager to switch over testing, stability the branch in a week
or two, then release, we'll get releases out without so much fuss.
> * When developer checks something in - she should see the results of
> that checkin ASAP. Something like telling the testing system, test
> me my library on all toolsets and return be the results, quick!
This takes more hardware than we currently have access to, so it's not
very high priority to me.
Doug