Subject: Re: [boost] Enhancing regression tests summary
From: Adam Wulkiewicz (adam.wulkiewicz_at_[hidden])
Date: 2014-10-31 15:29:20
Beman Dawes wrote:
> On Fri, Oct 31, 2014 at 11:31 AM, Rene Rivera <grafikrobot_at_[hidden]> wrote:
>
>> On Fri, Oct 31, 2014 at 10:18 AM, Robert Ramey <ramey_at_[hidden]> wrote:
>>
>>> Adam Wulkiewicz wrote
>>>> Hi,
>>>>
>>>> Some time ago I was writing about regression summaries enhancements,
>>>> that I'd be nice to see more meaningful list of errors, etc. I asked
>>>> questions, even prepared a PR for regression, to no avail.
>>>>
>>>> So I developed a simple program downloading summaries, test logs and
>>>> saving the enhanced result.
>>>> You may check it out here:
>> https://github.com/awulkiew/summary-enhancer
>>> Very nice - but rather than downloading and enhancing the results,
>> wouldn't
>>> it
>>> be simpler to just enhance the current program. That is clone the the
>>> current results display tool, incorporate the enhancements, and then
>>> merge the changes into the current tool?
>>>
>> +1
>>
> +1
>
Of course it would be simpler. In fact I proposed this some time ago
(http://boost.2283326.n4.nabble.com/testing-Proposal-regression-tests-results-display-upgrade-td4662953.html).
But in practice there wasn't enough feedback/help.
At that Point I had some questions about the way how summary pages are
generated. Which tools are used exactly, what parameters are passed etc.
I found C++ code of a tool generating summary pages, modified it and
tested on locally generated XMLs (by the run.py). I prepared a PR
(https://github.com/boostorg/boost/pull/25) adding simple enhancements
to check if my locally-tested program will work with the official setup.
I have no access to the machine on which the pages are generated or even
to the actual files sent by the runners so can't do any real testing.
Even though I wasn't sure if this C++ generator is the one that had been
used to generate the official summary pages, I wanted to have something
that actually works. In fact I suspect that they're generated by the
XSLT. It's because there are some attributes on summary pages that
couldn't be generated by the C++ tool. On the other hand they can be
found in the XSLT files.
Furthermore, I suspect that summary pages for develop and master
branches may be generated somehow differently because for some libraries
the pages for master are missing (e.g. Geometry, Spirit) but this is a
wild guess.
So finally I decided to implement something myself, something that would
be useful for me and anyone interested but didn't require anyone's
involvement. In the same time I wanted to develop a proof of concept
showing what I have in mind. So if you have any ideas how to move this
forward and add this functionality to the officially used tool, I'm open
to suggestion.
Regards,
Adam