From: Edward Diener (eldiener_at_[hidden])
Date: 2020-04-27 16:52:24
On 4/26/2020 12:07 PM, Andrey Semashev via Boost wrote:
> On 2020-04-26 18:17, Edward Diener via Boost wrote:
>> Sometimes CI testing fails because of a bug in a compiler. It is
>> possible to designate in a jam file that a particular test should not
>> be done for a particular compiler version so that CI testing succeeds,
>> and therefore a PR can be more easily merged because the CI tests
>> "succeed". But then the fact that the compiler has a bug is masked by
>> such a change to the jam file. Is there a solution to such a problem,
>> or maybe this is really not a problem at all ? What do others think
>> about this ?
>
> If the bug affects the library, and you still intend to support it, it
> follows that you have to add a workaround for the bug in the library.
> The failing test indicates that users of this compiler have an actual
> problem.
So if there is no workaround, should the feature for the particular
compiler/version be disabled ? Should the test "succeed" for the feature
for the particular compiler/version by either being disabled in Boost
Build or by "passing" because the feature is disabled for the
compiler/version ?
>
> If you don't intend to support this compile, just remove it from the CI.
> In the old build matrix this can't be done, except to disable it in the
> test/Jamfile.
I do not think the entire compiler/version should be removed from
testing if just a very small part of the library does not work because
of a bug. What you are saying is that the tests are meant to succeed
rather than show a valid bug in some compiler/version. The opposite
viewpoint is that a failing test is a good indication of a
compiler/version bug and therefore a failing test is not an indication
of a failing library. For the purposes of Github changes, particularly
PRs, with CI tests, it does seem as if the tests should succeed, which
would back your viewpoint.