osmo-bsc test suite regression since build 694
- from build 209 in https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bsc-test-sccplite-latest/test_results_analyzer/
- from build 463 in https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bsc-test-sccplite/test_results_analyzer/
- from build 301 in https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bsc-test-latest/test_results_analyzer/
- from build 694 in https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bsc-test/test_results_analyzer/
These may or may not be caused by the same root cause, but I don't really remember any other event where we had such massive regressions for many consecutive days all over all BSC test suites. This needs to be investigated and fixed - and we also need to figure out how related patches ended up untested in the code tree in the first place.
There are three types of errors:a) 5 builds ago (2019-08-14)
- failing tests:
- failing tests:
- failing tests:
- all from TC_bssap_rsld_does_not_cause_bssmap_reset to TC_chan_rel_rll_rel_ind
- ... (many more!)
Regarding c): I've reproduced it locally, and found that reverting commit "RAN_Adapter: invert check for RAN_ops == omit" in osmo-ttcn3-hacks.git fixes it: https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/15233
Regarding b): I'm assuming that we don't care if handover related tests pass with sccplite, so this can be ignored. It seems that the tests were passing before, although they should not have been passing, and now they are correctly failing.
Regarding a): "5 builds ago" is when new TC_ho_neighbor_config_* tests were introduced, and related osmo-bsc config changes were made (see https://gerrit.osmocom.org/q/topic:ho ). Note that the sccplite related tests are not failing, so this might be a config related issue (the sccplite tests have their own osmo-bsc.cfg, and that one was not modified).
So only a) is remaining, I'm on it :)
Fix for TC_oml_unknown_unit_id: https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/15254
Now that the > 60 failing tests we had yesterday are resolved, it appears that we have one more failing test since yesterday: TC_paging_resp_unsol
This should be the last one, after which everything is back to like it was before this regression wave.
I'm looking into resolving this one too.
- Status changed from In Progress to Resolved
- % Done changed from 80 to 100
Maybe it would have been better to wait with merging the osmo-ttcn3-hacks patch until the osmo-bsc patch was merged?
we also need to figure out how related patches ended up untested in the code tree in the first place.
The patch that caused the most fall out is this library change:
It looks good, but since we don't automatically run the TTCN3 tests in gerrit verification, it was not clear that this breaks the BSC testsuite. From reading the commit message, it seems like it was probably manually tested against SGSN tests, and testing manually against all testsuites does not seem feasible.
I understand that it is not feasible right now to run the TTCN3 tests in the gerrit verification job either (due to the long execution times, especially if there is a library change), but hopefully we can get there eventually? I would be happy to work on this if it seems realistic and we can fit it in at some point in the future.
All related patches to fix the regressions are merged, setting this issue to resolved.