Cellular Network Infrastructure

I copied the junit logger code from titan-core and adapted it to write a text file like

/tmp/tmp.I59eae315e/msc-tester/summary-5523.log
-----------------------------------------------
pass  MSC_Tests.TC_cr_before_reset
FAIL  MSC_Tests.TC_lu_imsi_reject
FAIL  MSC_Tests.TC_lu_imsi_timeout_gsup

In the presence of an expected-verdicts.txt file (a copy of a log output like above), the output becomes

/tmp/tmp.I59eae315e/msc-tester/summary-5523.log
-----------------------------------------------
pass  MSC_Tests.TC_cr_before_reset
xfail MSC_Tests.TC_lu_imsi_reject
xfail MSC_Tests.TC_lu_imsi_timeout_gsup

In result, we can do something like

if grep -v '^\(pass\|xfail\)' summary*.log; then
  echo UNEXPECTED RESULTS
  exit 1
fi

Still need to figure out some details:

• compilation: so far I need to compile the summary logger separately with an adapted Makefile from titan-core, to produce an .so file that is accepted and doesn't segfault.
• ideally make it so that the expected-results.txt can sit in osmo-ttcn3-hacks/foo/ and is used for both dockerized tests and manual tests automagically.
• if some expected-to-fail tests currently start passing, do we still mark the jenkins job as failed? I think we have to remind ourselves to update the expected results somehow.
  Jenkins could also keep a separate expected-results file in the workspace and only ensure that we never get worse, and update the file automatically if it gets better.

I'm not actually sure that the logger plugin is the best way to solve this. An external tool that compares, for example, junit xml files would cut short all the linking issues, and we would get a shell return code from it marking failure or success without needing grep magic... The biggest hindrance is that it would be nice not to add huge dependencies like python to the docker images; also the junit xml files don't provide as fine-grained verdicts as ttcn3 outputs (INCONC, ERROR, FAIL all amount to a failure)

created a duplicate by accident. Let's go on there. #3136

I've got a bash script that parses the junit xml now, which is much less trouble: no compilation, no added dependencies, easy to call on arbitrary paths:
https://gerrit.osmocom.org/7646

Already added the expected results from jenkins runs and integrated in start-testsuite.sh, should be useful for both jenkins runs and manual invocation, now.
On jenkins, we might want to add --allow-xpass (allow unexpected pass of a test), though that is debatable.
If we fixed a test, we should immediately update the expected results. Otherwise the entire scheme becomes futile.

If we wanted to allow xpass, we'd do in docker-playground:

--- a/jenkins-common.sh
+++ b/jenkins-common.sh
@@ -39,3 +39,5 @@ mkdir $VOL_SUITE_DIR

 rm -rf $WORKSPACE/logs || /bin/true
 mkdir -p $WORKSPACE/logs
+
+export OSMO_TTCN3_COMPARE_ARGS="--allow-xpass" 

merged