Subject: Re: [boost] [build] Test marked as compile-fail is now reported as a failure.
From: Adam Wulkiewicz (adam.wulkiewicz_at_[hidden])
Date: 2015-01-12 19:36:39
Hi Vladimir,
Vladimir Prus wrote:
> I am fairly sure that dark green vs green depends only on
> status/explicit-failures-markup.xml,
> and not on the type of the test. In the absence of any such markup:
>
> - The regular compile test tries to compile a source file, and is
> deemed a pass if compilation
> succeeds
> - The compile-fail test tries to compile a source file, and is deemed
> a pass if compilation
> fails
>
> Pass is green, fail is yellow.
Still AFAIU compile-fail should work as expected even if the test wasn't
marked as failing in the xml file.
>
>> Yes, the cause may be located elsewhere, in Geometry, Regression,
>> Test, etc.
>>
>> For a clean run I'm now unable to build the tests due to some undefined
>> references but the problem is most likely located elsewhere.
>
> I think you first need to determine whether the actual compilation of
> the source file results in
> success (i.e. object file is produced) or failure (i.e. errors from
> compiler). For a compile-fail
> test, the compilation is expected to result in failure. If it does
> not, it's a problem in test itself
> or geometry library. If compilation results in failure, but regression
> matrix shows it as failed test,
> then something is wrong down the pipeline.
>
Yes, you're right. The test is compiling, and the object file is
created. But to detect this I was forced to compile it manually.
The message produced by b2 is slightly missleading. On my machine it
looks like this:
gcc.compile.c++
..\..\bin.v2\libs\geometry\index\test\rtree\rtree_values_invalid.test\gcc-mingw-4.9.1\debug\rtree_values_invalid.o
"g++" -ftemplate-depth-128 -O0 -fno-inline -Wall -g -std=c++11
-DBOOST_ALL_NO_LIB=1 -DBOOST_CHRONO_DYN_LINK=1
-DBOOST_CHRONO_THREAD_DISABLED -DBOOST_SYSTEM_DYN_LINK=1
-DBOOST_SYSTEM_NO_DEPRECATED -DBOOST_TIMER_DYN_LINK=1 -I"..\.."
-I"..\..\boost\geometry\extensions\contrib\ttmath" -I"index\test"
-I"test" -c -o
"..\..\bin.v2\libs\geometry\index\test\rtree\rtree_values_invalid.test\gcc-mingw-4.9.1\debug\rtree_values_invalid.o"
"index\test\rtree\rtree_values_invalid.cpp"
...failed gcc.compile.c++
..\..\bin.v2\libs\geometry\index\test\rtree\rtree_values_invalid.test\gcc-mingw-4.9.1\debug\rtree_values_invalid.o...
...removing
..\..\bin.v2\libs\geometry\index\test\rtree\rtree_values_invalid.test\gcc-mingw-4.9.1\debug\rtree_values_invalid.o
So the test fails to compile. The compiler output is not printed, but
not in all cases. The last line doesn't really mean anything, it might
as well be some cleanup just in case.
If the test fails to compile (failure not expected) the compiler output
is printed, bjam's output is nearly the same. Just the last line
"...removing ..." is not displayed.
On the other hand when the compilation failure is expected and the test
really fails to compile bjam displays nice (failed-as-expected) instead
of "...failed gcc.compile.c++".
Would it be possible to display slightly different message when a test
expected to fail didn't fail, e.g. (passed-unexpectedly),
(unexpected-pass) or something similar?
At least not "...failed gcc.compile.c++" because the compilation didn't
really fail.
Regards,
Adam