Project

General

Profile

Bug #4156

osmo-bsc test suite regression since build 694

Added by laforge about 1 month ago. Updated about 1 month ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
-
Start date:
08/19/2019
Due date:
% Done:

100%

Spec Reference:

Description

We see logs of regressions

These may or may not be caused by the same root cause, but I don't really remember any other event where we had such massive regressions for many consecutive days all over all BSC test suites. This needs to be investigated and fixed - and we also need to figure out how related patches ended up untested in the code tree in the first place.

History

#1 Updated by osmith about 1 month ago

  • Status changed from New to In Progress

#2 Updated by osmith about 1 month ago

There are three types of errors:

a) 5 builds ago (2019-08-14)
  • affected:
    • ttcn3-bsc-test-latest
    • ttcn3-bsc-test
  • failing tests:
    • TC_ctrl
    • TC_oml_unknown_unit_id
b) 5 builds ago (2019-08-14)
  • affected:
    • ttcn3-bsc-test-sccplite
  • failing tests:
    • TC_ho_int
    • TC_ho_out_fail_*
    • TC_ho_out_of_this_bsc
c) 1 build ago (2019-08-19)
  • affected:
    • ttcn3-bsc-test-sccplite
    • ttcn3-bsc-test-sccplite-latest
    • ttcn3-bsc-test-latest
    • ttcn3-bsc-test
  • failing tests:
    • TC_assignment_csd
    • TC_assignment_ctm
    • all from TC_bssap_rsld_does_not_cause_bssmap_reset to TC_chan_rel_rll_rel_ind
    • ... (many more!)

Regarding c): I've reproduced it locally, and found that reverting commit "RAN_Adapter: invert check for RAN_ops == omit" in osmo-ttcn3-hacks.git fixes it: https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/15233

Regarding b): I'm assuming that we don't care if handover related tests pass with sccplite, so this can be ignored. It seems that the tests were passing before, although they should not have been passing, and now they are correctly failing.

Regarding a): "5 builds ago" is when new TC_ho_neighbor_config_* tests were introduced, and related osmo-bsc config changes were made (see https://gerrit.osmocom.org/q/topic:ho ). Note that the sccplite related tests are not failing, so this might be a config related issue (the sccplite tests have their own osmo-bsc.cfg, and that one was not modified).

So only a) is remaining, I'm on it :)

#4 Updated by osmith about 1 month ago

  • % Done changed from 0 to 80

The TC_oml_unknown_unit_id failure is caused by changes in docker-playground.git, it passed locally when I checked out a ~3 weeks old commit ef8b48516e0d5215a92dfe6de32983a450f4ed42.

#5 Updated by osmith about 1 month ago

#6 Updated by osmith about 1 month ago

Now that the > 60 failing tests we had yesterday are resolved, it appears that we have one more failing test since yesterday: TC_paging_resp_unsol

This should be the last one, after which everything is back to like it was before this regression wave.

I'm looking into resolving this one too.

#7 Updated by osmith about 1 month ago

  • Status changed from In Progress to Resolved
  • % Done changed from 80 to 100
With git bisect I found that the TC_paging_resp_unsol failure is expected: the test-expectation has been changed by dexter, but the patch that adjusts osmo-bsc behavior was not merged yet:

Maybe it would have been better to wait with merging the osmo-ttcn3-hacks patch until the osmo-bsc patch was merged?

laforge wrote:

we also need to figure out how related patches ended up untested in the code tree in the first place.

The patch that caused the most fall out is this library change:
https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/15157

It looks good, but since we don't automatically run the TTCN3 tests in gerrit verification, it was not clear that this breaks the BSC testsuite. From reading the commit message, it seems like it was probably manually tested against SGSN tests, and testing manually against all testsuites does not seem feasible.

I understand that it is not feasible right now to run the TTCN3 tests in the gerrit verification job either (due to the long execution times, especially if there is a library change), but hopefully we can get there eventually? I would be happy to work on this if it seems realistic and we can fit it in at some point in the future.

All related patches to fix the regressions are merged, setting this issue to resolved.
https://gerrit.osmocom.org/q/topic:ttcn3-fix-bsc

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)