From: Douglas Gregor (gregod_at_[hidden])
Date: 2002-01-21 16:26:29
On Monday 21 January 2002 03:40 pm, you wrote:
> At 11:36 AM 1/21/2002, Douglas Gregor wrote:
> > The use of a Dart dashboard would meet or exceed Boost's regression
> >testing needs.
>
> I'm interested in Dart dashboard, but don't see how Boost could give up
> control of our regression testing to a third-party.
I'm not quite sure what you mean by "control" with respect to regression
testing. The Dart server is run at Kitware, but the server does not perform
any regression testing itself; it merely collects regression testing data
submitted by the Dart clients. If by "control" you mean the ability to
maintain the server, I think that is largely irrelevant in this case. Kitware
runs Dart dashboards for several of their own projects on the same server
that the Boost dashboard will be available. They can't afford to have their
dashboards down, so problems will be fixed as fast as possible. In the worst
case, Dart is open source software and the Boost dashboard could be moved
elsewhere.
The actual regression testing is done by the Dart client. Anyone can setup a
Dart client and submit regression tests. At the moment, Kitware runs one
nightly regression test, and I run three others on different platforms. Dart
clients will perform a "cvs update" every night, so all nightly build entries
posted on the dashboard for a specific day will be for the same version of
the Boost source (this is a _huge_ problem for me now, with the current
regression testing system; I look at the tests, exclaim "I broke WHAT?", and
then realize that the tests are two months old).
I should just ask: what do you mean by "control"? I see no problem with the
server side of a Dart-based system, and Dart clients are in the hands of
those that have the hardware to run regression tests on.
> While it may not be obvious to all Boost members, what tends to torment
> Boost moderators (or at least me) are the functions where someone else has
> control of our resources. YahooGroups. SourceForge.
I understand this completely, but I think regression testing is a slightly
different problem. It's _impossible_ to control all of the resources for
regression testing unless Boost had an infinite number of platform/compiler
combinations in someone's basement.
> Toolchains that aren't covered by the Boost distribution are also a
> problem. I'm setting up a new machine right now. To run various Boost
> procedures, I need to install five C++ compilers, Cygwin, Python, WinCvs,
> and be able to cope with Windows/bat, Python, and bash scripts. People
> would like change logs. The only scripts I can find are in Perl. So add
> that to the list. I'm not at all happy to even think about anything else.
This was brought up earlier: Dart requires Tcl, so performing all regression
testing with Dart would require everyone to install both Dart and Tcl
locally. The solution is to split regression testing into a front-end and a
back-end: the front-end is just a common method of specifying regression
tests in Jam, and the back-end does the actual work. Two backends would be
supported:
1) A Dart client backend that is used for anything submitted to the Dart
dashboard.
2) A "local testing" backend that works like status/regression.cpp. This
would generate the HTML files to which we are accustomed, and would not
require Tcl or Dart.
Brad's posted some prototypes for the front-end, and I believe he's also
started converting the Jamfiles we created into a Dart-based back-end.
The result: only those running tests to be submitted to the Dashboard need to
have Tcl and Dart. Nobody needs to have five C++ compilers, because each
build variant can be tested and submitted separately.
Doug