Cellular Network Infrastructure » Radio Access Network » OsmoBTS

A number of them are "VTY timeout for prompt" which doesn't sount good.

Looking at the junit xml at https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bts-test/983/console it seems TC_sabm_incorrect_c is the last correctly executed test case, and afterwards all further tests (starting with BTS_Tests:hopping.TC_chan_act_stress) all tests fail with either "Timeout waiting for RSL bring up" or "VTY Timeout for prompt"

Interestingly enough, I could not reproduce those failures on my machine in Docker:

pass BTS_Tests_SMSCB.TC_cbch_load_idle_no_cbch
pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_1block
pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_2block
pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_3block
pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_4block
pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_multi
pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_schedule
pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_default_only
pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_default_and_normal
pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_default_then_null
pass BTS_Tests_SMSCB.TC_cbc_sdcch4_load_idle
pass BTS_Tests_SMSCB.TC_cbc_sdcch4_load_overload
pass BTS_Tests_SMSCB.TC_etws_p1ro
pass BTS_Tests_SMSCB.TC_etws_p1ro_end
pass BTS_Tests_SMSCB.TC_etws_pcu
NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_1block
NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_2block
NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_3block
NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_4block
NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_multi
NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_schedule
NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_default_only
NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_default_and_normal
NEW: PASS BTS_Tests_SMSCB.TC_cbc_sdcch8_load_idle
NEW: PASS BTS_Tests_SMSCB.TC_cbc_sdcch8_load_overload

I also tried to run BTS_Tests_SMSCB.control exclusively in Jenkins [1], all SMSCB test cases passed.
The reason why they fail when ttcn3-bts-test is started by timer remains a mystery.

[1] https://jenkins.osmocom.org/jenkins/view/TTCN3-centos/job/TTCN3-centos-bts-test/88/

Apparently, this is not a regression. I think the problem was that before [1] jenkins.sh did not append ':hopping' to both BTS_Tests_{SMSCB,LAPDm} groups (sorry, my fault), so the hopping test results got mixed somehow with non-hopping ones in junit-*.xml. Since [1] has been merged, non-hopping SMSCB test cases passing two days on Jenkins as before [2][3][4].

[1] https://gerrit.osmocom.org/c/docker-playground/+/19637
[2] https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bts-test/991/
[3] https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bts-test-latest/677/
[4] https://jenkins.osmocom.org/jenkins/view/TTCN3-centos/job/TTCN3-centos-bts-test/89/

We should still investigate why the hopping test cases fail and fix them. Hijacking this ticket, setting priority to 'Low' because I am pretty sure that it's not a problem in the IUT, but rather in the test suite itself (e.g. BTS_Tests_SMSCB.f_smscb_setup() does not consider hopping parameters).

setting priority to normal. It doesn't matter whether it's a bug in osmo-bts or in the test suite. We cannot prove it ither way unless we fix it. As neels is currently working on a sysmocom-custmer ticket that aims to validate SMSCB is fully functional, I'm sure at least he (and the unnamed sysmocom customer) would greatly appreciate if the tests passed. Particularly as they are using frequency hopping.

i'm so far only working on SMSCB in OsmoBSC, if I should continue in OsmoBTS then let me know ... !

i'm so far only working on SMSCB in OsmoBSC, if I should continue in OsmoBTS then let me know ... !

Since I am more familiar with the related code in ttcn3-bts-test, I'll take care of it. Thanks anyway!

Please see:

https://gerrit.osmocom.org/c/docker-playground/+/20098 ttcn3-bts-test/fh: do not configure CBCH slots statically

and

https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/20100 BTS_Tests: simplify MA length calculation in f_resolve_fh_params()
https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/20101 BTS_Tests: group hopping parameters into a separate record
https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/20102 BTS_Tests_SMSCB: handle frequency hopping parameters

Here are the results from a test run on my machine:

pass BTS_Tests_SMSCB.TC_cbch_load_idle_no_cbch
pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_1block
pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_2block
pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_3block
pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_4block
pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_multi
pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_schedule
pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_default_only
pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_default_and_normal
pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_default_then_null
pass BTS_Tests_SMSCB.TC_cbc_sdcch4_load_idle
pass BTS_Tests_SMSCB.TC_cbc_sdcch4_load_overload
pass BTS_Tests_SMSCB.TC_etws_p1ro
pass BTS_Tests_SMSCB.TC_etws_p1ro_end
pass BTS_Tests_SMSCB.TC_etws_pcu
NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_1block
NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_2block
NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_3block
NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_4block
NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_multi
NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_schedule
NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_default_only
NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_default_and_normal
NEW: PASS BTS_Tests_SMSCB.TC_cbc_sdcch8_load_idle
NEW: PASS BTS_Tests_SMSCB.TC_cbc_sdcch8_load_overload

All changes have been merged.