From: Aleksey Gurtovoy (agurtovoy_at_[hidden])
Date: 2002-07-18 22:57:04
David Abrahams wrote:
> Yes, of course. The big problem is how to deal with the
> expected failures.
>
> I want to know exactly which sub-checks in a test are expected to
> fail/succeed on a given compiler.
>
> In the overall results, I want to see:
>
> FAIL if any of the expected successes fails
> pass if it passes all tests.
> PASS* if it meets expectations exactly, but if there are expected
> failure
> PASS? if all expected successes and some expected failures pass.
>
> I also want a link to a list of expected failures and at the
> end of a link somewhere.
>
> Thoughts?
How about something like this:
// some trait to test
template< typename T1 > struct identity
{
typedef T1 type;
};
IS_SAME(false,identity<int>::type,int) // no failures expected; compilation
error ("FAIL") if the check fails
IS_SAME(TT_MSVC < 1300,identity<char*>::type,char*) // expected failure on
MSVC 6.5 and below
If the test is compiled under MSVC and the last line fails, the test will
not _link_ with an error like this:
test.obj : error LNK2019: unresolved external symbol "public: static void
__cdecl tt_test<1,struct void_,39>::EXPECTED_FAILURE(void)"
where "39" is the line number, which in your classification would correspond
to "PASS*". If it unexpectedly passes under MSVC, then you will also get a
link error, but a slightly different one:
test.obj : error LNK2019: unresolved external symbol "public: static void
__cdecl tt_test<1,struct void_,39>::UNEXPECTED_PASS(void)"
which would mean "PASS?". If it fails on another compiler, it's still a
compilation error ("FAIL").
Of course, we would need a special support somewhere in the chain to
transform those link errors into a test status.
Just a thought.
Aleksey