I copied the junit logger code from titan-core and adapted it to write a text file like
/tmp/tmp.I59eae315e/msc-tester/summary-5523.log
-----------------------------------------------
pass MSC_Tests.TC_cr_before_reset
FAIL MSC_Tests.TC_lu_imsi_reject
FAIL MSC_Tests.TC_lu_imsi_timeout_gsup
In the presence of an expected-verdicts.txt file (a copy of a log output like above), the output becomes
/tmp/tmp.I59eae315e/msc-tester/summary-5523.log
-----------------------------------------------
pass MSC_Tests.TC_cr_before_reset
xfail MSC_Tests.TC_lu_imsi_reject
xfail MSC_Tests.TC_lu_imsi_timeout_gsup
In result, we can do something like
if grep -v '^\(pass\|xfail\)' summary*.log; then
echo UNEXPECTED RESULTS
exit 1
fi
Still need to figure out some details:
- compilation: so far I need to compile the summary logger separately with an adapted Makefile from titan-core, to produce an .so file that is accepted and doesn't segfault.
- ideally make it so that the expected-results.txt can sit in osmo-ttcn3-hacks/foo/ and is used for both dockerized tests and manual tests automagically.
- if some expected-to-fail tests currently start passing, do we still mark the jenkins job as failed? I think we have to remind ourselves to update the expected results somehow.
Jenkins could also keep a separate expected-results file in the workspace and only ensure that we never get worse, and update the file automatically if it gets better.
I'm not actually sure that the logger plugin is the best way to solve this. An external tool that compares, for example, junit xml files would cut short all the linking issues, and we would get a shell return code from it marking failure or success without needing grep magic... The biggest hindrance is that it would be nice not to add huge dependencies like python to the docker images; also the junit xml files don't provide as fine-grained verdicts as ttcn3 outputs (INCONC, ERROR, FAIL all amount to a failure)
