osmo-ttcn3-hacks: provide a way to summarize test results in a human readable way, and to define currently expected test results
a) our jenkins builds are continuously failing. We know that certain tests currently fail, so we could mark them expected-to-fail, and mark jenkins job runs as failed only if more failures are added.
b) daily hacking: it is hard to figure out what results the last jenkins run yielded, and it's hard to quickly read the junit xml to compare it to a jenkins result XML. This information should best fall out at the end of a test run, be it in jenkins or manual.
#3 Updated by neels over 1 year ago
I copied the junit logger code from titan-core and adapted it to write a text file like
/tmp/tmp.I59eae315e/msc-tester/summary-5523.log ----------------------------------------------- pass MSC_Tests.TC_cr_before_reset FAIL MSC_Tests.TC_lu_imsi_reject FAIL MSC_Tests.TC_lu_imsi_timeout_gsup
In the presence of an expected-verdicts.txt file (a copy of a log output like above), the output becomes
/tmp/tmp.I59eae315e/msc-tester/summary-5523.log ----------------------------------------------- pass MSC_Tests.TC_cr_before_reset xfail MSC_Tests.TC_lu_imsi_reject xfail MSC_Tests.TC_lu_imsi_timeout_gsup
In result, we can do something like
if grep -v '^\(pass\|xfail\)' summary*.log; then echo UNEXPECTED RESULTS exit 1 fi
Still need to figure out some details:
- compilation: so far I need to compile the summary logger separately with an adapted Makefile from titan-core, to produce an .so file that is accepted and doesn't segfault.
- ideally make it so that the expected-results.txt can sit in osmo-ttcn3-hacks/foo/ and is used for both dockerized tests and manual tests automagically.
- if some expected-to-fail tests currently start passing, do we still mark the jenkins job as failed? I think we have to remind ourselves to update the expected results somehow.
Jenkins could also keep a separate expected-results file in the workspace and only ensure that we never get worse, and update the file automatically if it gets better.
I'm not actually sure that the logger plugin is the best way to solve this. An external tool that compares, for example, junit xml files would cut short all the linking issues, and we would get a shell return code from it marking failure or success without needing grep magic... The biggest hindrance is that it would be nice not to add huge dependencies like python to the docker images; also the junit xml files don't provide as fine-grained verdicts as ttcn3 outputs (INCONC, ERROR, FAIL all amount to a failure)
#6 Updated by neels over 1 year ago
I've got a bash script that parses the junit xml now, which is much less trouble: no compilation, no added dependencies, easy to call on arbitrary paths:
Already added the expected results from jenkins runs and integrated in start-testsuite.sh, should be useful for both jenkins runs and manual invocation, now.
On jenkins, we might want to add --allow-xpass (allow unexpected pass of a test), though that is debatable.
If we fixed a test, we should immediately update the expected results. Otherwise the entire scheme becomes futile.
If we wanted to allow xpass, we'd do in docker-playground:
--- a/jenkins-common.sh +++ b/jenkins-common.sh @@ -39,3 +39,5 @@ mkdir $VOL_SUITE_DIR rm -rf $WORKSPACE/logs || /bin/true mkdir -p $WORKSPACE/logs + +export OSMO_TTCN3_COMPARE_ARGS="--allow-xpass"