Project

General

Profile

Feature #3136

osmo-ttcn3-hacks: provide a way to summarize test results in a human readable way, and to define currently expected test results

Added by neels over 1 year ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
Start date:
04/05/2018
Due date:
% Done:

100%

Spec Reference:

Description

Use cases:

a) our jenkins builds are continuously failing. We know that certain tests currently fail, so we could mark them expected-to-fail, and mark jenkins job runs as failed only if more failures are added.

b) daily hacking: it is hard to figure out what results the last jenkins run yielded, and it's hard to quickly read the junit xml to compare it to a jenkins result XML. This information should best fall out at the end of a test run, be it in jenkins or manual.


Related issues

Is duplicate of Cellular Network Infrastructure - Feature #3089: osmo-ttcn3-hacks: plainly show changes in test results / show expected-to-fail testsClosed03/20/2018

History

#1 Updated by neels over 1 year ago

  • Status changed from New to In Progress

#2 Updated by neels over 1 year ago

  • Is duplicate of Feature #3089: osmo-ttcn3-hacks: plainly show changes in test results / show expected-to-fail tests added

#3 Updated by neels over 1 year ago

I copied the junit logger code from titan-core and adapted it to write a text file like

/tmp/tmp.I59eae315e/msc-tester/summary-5523.log
-----------------------------------------------
pass  MSC_Tests.TC_cr_before_reset
FAIL  MSC_Tests.TC_lu_imsi_reject
FAIL  MSC_Tests.TC_lu_imsi_timeout_gsup

In the presence of an expected-verdicts.txt file (a copy of a log output like above), the output becomes

/tmp/tmp.I59eae315e/msc-tester/summary-5523.log
-----------------------------------------------
pass  MSC_Tests.TC_cr_before_reset
xfail MSC_Tests.TC_lu_imsi_reject
xfail MSC_Tests.TC_lu_imsi_timeout_gsup

In result, we can do something like

if grep -v '^\(pass\|xfail\)' summary*.log; then
  echo UNEXPECTED RESULTS
  exit 1
fi

Still need to figure out some details:

  • compilation: so far I need to compile the summary logger separately with an adapted Makefile from titan-core, to produce an .so file that is accepted and doesn't segfault.
  • ideally make it so that the expected-results.txt can sit in osmo-ttcn3-hacks/foo/ and is used for both dockerized tests and manual tests automagically.
  • if some expected-to-fail tests currently start passing, do we still mark the jenkins job as failed? I think we have to remind ourselves to update the expected results somehow.
    Jenkins could also keep a separate expected-results file in the workspace and only ensure that we never get worse, and update the file automatically if it gets better.

I'm not actually sure that the logger plugin is the best way to solve this. An external tool that compares, for example, junit xml files would cut short all the linking issues, and we would get a shell return code from it marking failure or success without needing grep magic... The biggest hindrance is that it would be nice not to add huge dependencies like python to the docker images; also the junit xml files don't provide as fine-grained verdicts as ttcn3 outputs (INCONC, ERROR, FAIL all amount to a failure)

#4 Updated by neels over 1 year ago

  • Status changed from In Progress to Closed

created a duplicate by accident. Let's go on there. #3136

#5 Updated by neels over 1 year ago

  • Status changed from Closed to In Progress

#6 Updated by neels over 1 year ago

I've got a bash script that parses the junit xml now, which is much less trouble: no compilation, no added dependencies, easy to call on arbitrary paths:
https://gerrit.osmocom.org/7646

Already added the expected results from jenkins runs and integrated in start-testsuite.sh, should be useful for both jenkins runs and manual invocation, now.
On jenkins, we might want to add --allow-xpass (allow unexpected pass of a test), though that is debatable.
If we fixed a test, we should immediately update the expected results. Otherwise the entire scheme becomes futile.

If we wanted to allow xpass, we'd do in docker-playground:

--- a/jenkins-common.sh
+++ b/jenkins-common.sh
@@ -39,3 +39,5 @@ mkdir $VOL_SUITE_DIR

 rm -rf $WORKSPACE/logs || /bin/true
 mkdir -p $WORKSPACE/logs
+
+export OSMO_TTCN3_COMPARE_ARGS="--allow-xpass" 

#7 Updated by neels over 1 year ago

  • % Done changed from 0 to 90

#8 Updated by neels over 1 year ago

  • Status changed from In Progress to Resolved
  • % Done changed from 90 to 100

merged

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)