Project

General

Profile

Actions

Support #3863

open

setup testing of osmo-bts-oc2g on real hardware with ttcn3 and osmocon/osmocom-bb

Added by dexter almost 5 years ago. Updated over 4 years ago.

Status:
Stalled
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
Start date:
03/26/2019
Due date:
% Done:

40%

Spec Reference:

Description

In order to be able to debug problems with automatic interop testing a local instance of an osmocom-bb phone and BTS is required. TRXCONT/FAKETRX are exchanged with a real BTS/PHONE.


Files

BTS_Tests.cfg BTS_Tests.cfg 4.41 KB dexter, 04/23/2019 01:33 PM
BTS_Tests.default BTS_Tests.default 661 Bytes dexter, 04/23/2019 01:33 PM
osmo-bsc.cfg osmo-bsc.cfg 4.7 KB dexter, 04/23/2019 01:33 PM
junit-xml-3840.log junit-xml-3840.log 15.8 KB dexter, 04/23/2019 02:32 PM
Actions #1

Updated by dexter almost 5 years ago

  • Status changed from New to In Progress

At the moment I am having problems to set this up. I have configured osmo-bsc so that it tells the BTS the RSL IP where TTCN3 listens. Also the VTY is reachable for TTCN3. However, I still get:

MTC@lobotron: Test case TC_si_sched_2bis finished. Verdict: fail reason: Could not connect IPA socket from "" port 0 to "127.0.0.1" port 4238; check your configuration

Actions #2

Updated by dexter almost 5 years ago

Actions #3

Updated by dexter almost 5 years ago

  • % Done changed from 0 to 10

I came a bit further with my setup. However, at the moment I still have a problem:

12:42:09.567042 5 IPA_Emulation.ttcnpp:242 entering f__IPL4__PROVIDER__connect: :0 -> 127.0.0.1:4238 / TCP
12:42:09.568008 5 IPA_Emulation.ttcnpp:245 setverdict(fail): none -> fail reason: "Could not connect IPA socket from "" port 0 to "127.0.0.1" port 4238; check your configuration", new component reason: "Could not connect IPA socket from "" port 0 to "127.0.0.1" port 4238; check your configuration" 
12:42:09.568074 5 IPA_Emulation.ttcnpp:247 Stopping MTC. The current test case will be terminated.
12:42:09.568302 mtc Osmocom_CTRL_Adapter.ttcn:37 Stop was requested from MC.
12:42:09.568343 mtc Osmocom_CTRL_Adapter.ttcn:37 Stopping test component execution.
12:42:09.568775 mtc BTS_Tests.ttcn:580 Test case TC_chan_act_react was stopped.

I was not aware that there is also a project osmo-pcap (ctrl interface port 4238) and that we apparently use it in our tests. Probably to get pcap traces from the inner workings of the BTS. This makes sense but I think for my experimental setup this is not needed. I wonder how I can disable this feature but what really confuses me is that when I run the tests locally osmo-pcap seems not to be required. I don't even have it installed on my workstation.

Actions #4

Updated by laforge almost 5 years ago

osmo-pcap is not required for any of this. Please coordinate with pespin
on how to run the TTCN3 tests with OsmocomBB, as this is already started+executed
by osmo-gsm-tester.

Actions #5

Updated by pespin almost 5 years ago

Tests for bts-oc2g are already being run in osmo-gsm-tester+ttcn3 with osmocom-bb setup.

I didn't check results for bts-oc2g yet though, but they can be found there:
https://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_ttcn3/test_results_analyzer/

Actions #6

Updated by dexter almost 5 years ago

I managed to get the overall setup running. The osmocom-bb phone and the BTS are communicating with each other over the air. There was some race condition I have fixed by adding a delay after the BTS is started up. Apparently it takes a second or two until the BTS is running stable. After that synchronization and RACH request work reliably and a lot more tests do pass now.

Next step should be to reproduce the results I have locally on the tester.

Actions #7

Updated by dexter almost 5 years ago

  • % Done changed from 10 to 20
Actions #8

Updated by dexter almost 5 years ago

I am currently investigating why the the ttcn3 tests on the tester fail. I think it makes sense to focus on ttcn3_bts_tests:sysmo first. In the logs I see an error which I have seen before while trying to get the tests run locally:

HC@27b61be607b6: Warning: Option `FileMask' was given more than once in section [LOGGING] of the configuration file.
HC@27b61be607b6: Warning: Option `FileMask' was given more than once in section [LOGGING] of the configuration file.
HC@27b61be607b6: Error while setting parameter field 'BTS_Tests.mp_bb_trxc_ip' to '"127.0.0.1"': Module parameter cannot be set, because no parameter with  name 'mp_bb_trxc_ip' exists in module 'BTS_Tests'. (Note: field names and array indexes are not supported in the Load Test Runtime).
HC@27b61be607b6: Error while setting parameter field 'BTS_Tests.mp_bb_trxc_port' to '- 1': Module parameter cannot be set, because no parameter with  name 'mp_bb_trxc_port' exists in module 'BTS_Tests'. (Note: field names and array indexes are not supported in the Load Test Runtime).
Error: There were errors during configuring HCs.

I think this BTS_Tests.mp_bb_trxc_port is just wrong. Attached one finds a set of config files that works for me locally.

There are also a lot of components that are not needed for the test. I think osmo-msc, osmo-mgw, osmo-hlr, and osmo-stp should be removed. The only purpose osmo-bsc is serving here is to bootstrap the BTS via OML.

Actions #9

Updated by dexter almost 5 years ago

Here is a test result that reflects the current state:

pass BTS_Tests.TC_chan_act_stress
pass BTS_Tests.TC_chan_act_react
pass BTS_Tests.TC_chan_deact_not_active
pass BTS_Tests.TC_chan_act_wrong_nr
pass BTS_Tests.TC_deact_sacch
pass->FAIL BTS_Tests.TC_sacch_filling
pass BTS_Tests.TC_sacch_info_mod
pass BTS_Tests.TC_sacch_multi
pass->FAIL BTS_Tests.TC_sacch_multi_chg
pass->FAIL BTS_Tests.TC_rach_content
pass->FAIL BTS_Tests.TC_rach_count
pass->FAIL BTS_Tests.TC_rach_max_ta
pass->FAIL BTS_Tests.TC_meas_res_sign_tchf
xfail BTS_Tests.TC_meas_res_sign_tchh
xfail BTS_Tests.TC_meas_res_sign_sdcch4
xfail BTS_Tests.TC_meas_res_sign_sdcch8
xfail BTS_Tests.TC_meas_res_sign_tchh_toa256
pass->FAIL BTS_Tests.TC_rsl_ms_pwr_ctrl
pass BTS_Tests.TC_conn_fail_crit
pass BTS_Tests.TC_paging_imsi_80percent
pass->FAIL BTS_Tests.TC_paging_tmsi_80percent
pass->FAIL BTS_Tests.TC_paging_imsi_200percent
pass->FAIL BTS_Tests.TC_paging_tmsi_200percent
pass BTS_Tests.TC_rsl_protocol_error
pass BTS_Tests.TC_rsl_mand_ie_error
pass BTS_Tests.TC_rsl_ie_content_error
pass BTS_Tests.TC_si_sched_default
pass BTS_Tests.TC_si_sched_1
pass BTS_Tests.TC_si_sched_2bis
pass BTS_Tests.TC_si_sched_2ter
pass BTS_Tests.TC_si_sched_2ter_2bis
pass BTS_Tests.TC_si_sched_2quater
pass BTS_Tests.TC_si_sched_13
pass->FAIL BTS_Tests.TC_si_sched_13_2bis_2ter_2quater
pass BTS_Tests.TC_ipa_dlcx_not_active
pass BTS_Tests.TC_ipa_crcx_twice_not_active
pass BTS_Tests.TC_ipa_crcx_mdcx_dlcx_not_active
pass BTS_Tests.TC_ipa_crcx_mdcx_mdcx_dlcx_not_active
pass BTS_Tests.TC_ipa_crcx_sdcch_not_active
skipped BTS_Tests.TC_pcu_act_req
skipped BTS_Tests.TC_pcu_act_req_wrong_ts
skipped BTS_Tests.TC_pcu_act_req_wrong_bts
skipped BTS_Tests.TC_pcu_act_req_wrong_trx
skipped BTS_Tests.TC_pcu_deact_req
skipped BTS_Tests.TC_pcu_deact_req_wrong_ts
skipped BTS_Tests.TC_pcu_ver_si13
skipped BTS_Tests.TC_pcu_data_req_wrong_bts
skipped BTS_Tests.TC_pcu_data_req_wrong_trx
skipped BTS_Tests.TC_pcu_data_req_wrong_ts
skipped BTS_Tests.TC_pcu_data_req_ts_inactive
skipped BTS_Tests.TC_pcu_data_req_pdtch
skipped BTS_Tests.TC_pcu_data_req_ptcch
skipped BTS_Tests.TC_pcu_data_req_agch
skipped BTS_Tests.TC_pcu_data_req_imm_ass_pch
skipped BTS_Tests.TC_pcu_rach_content
skipped BTS_Tests.TC_pcu_paging_from_rsl
pass->FAIL BTS_Tests.TC_dyn_osmo_pdch_act_deact
pass BTS_Tests.TC_dyn_osmo_pdch_unsol_deact
pass->FAIL BTS_Tests.TC_dyn_osmo_pdch_double_act
pass BTS_Tests.TC_dyn_osmo_pdch_tchf_act
pass BTS_Tests.TC_dyn_osmo_pdch_tchh_act
skipped BTS_Tests.TC_dyn_ipa_pdch_act_deact
pass BTS_Tests.TC_dyn_ipa_pdch_tchf_act
pass BTS_Tests.TC_dyn_ipa_pdch_tchf_act_pdch_act_nack
skipped BTS_Tests.TC_dyn_ipa_pdch_act_tchf_act_nack
pass BTS_Tests.TC_rll_est_ind
pass BTS_Tests.TC_rll_est_req_DCCH_3
pass BTS_Tests.TC_rll_est_req_ACCH_3
pass BTS_Tests.TC_rll_rel_ind_DCCH_0
pass BTS_Tests.TC_rll_rel_ind_DCCH_3
pass BTS_Tests.TC_rll_rel_ind_ACCH_0
pass BTS_Tests.TC_rll_rel_ind_ACCH_3
pass BTS_Tests.TC_rll_rel_req
pass BTS_Tests.TC_rll_unit_data_req_DCCH
pass BTS_Tests.TC_rll_unit_data_req_ACCH
pass BTS_Tests.TC_rll_unit_data_ind_DCCH
pass BTS_Tests.TC_rll_unit_data_ind_ACCH
pass BTS_Tests.TC_chan_act_a51
pass->FAIL BTS_Tests.TC_chan_act_a52
pass->FAIL BTS_Tests.TC_chan_act_a53
pass BTS_Tests.TC_encr_cmd_a51
pass->FAIL BTS_Tests.TC_encr_cmd_a52
pass->FAIL BTS_Tests.TC_encr_cmd_a53
pass BTS_Tests.TC_lapdm_selftest
pass BTS_Tests.TC_tch_sign_l2_fill_frame
xfail BTS_Tests.TC_tch_sign_l2_fill_frame_dtxd
pass BTS_Tests.TC_chopped_ipa_ping
pass BTS_Tests.TC_chopped_ipa_payload
--------------------
47 pass
5 xfail
19 skipped
17 FAIL
Actions #10

Updated by laforge almost 5 years ago

Some feedback below:

On Tue, Apr 23, 2019 at 02:34:43PM +0000, dexter [REDMINE] wrote:

pass->FAIL BTS_Tests.TC_sacch_filling
pass->FAIL BTS_Tests.TC_sacch_multi_chg
pass->FAIL BTS_Tests.TC_rach_content

for those above there is no reason this should fail in a physical setup.
I guess there's a bug in our test, or one in OsmoBTS.

pass->FAIL BTS_Tests.TC_rach_count
pass->FAIL BTS_Tests.TC_rach_max_ta
pass->FAIL BTS_Tests.TC_meas_res_sign_tchf
xfail BTS_Tests.TC_meas_res_sign_tchh
xfail BTS_Tests.TC_meas_res_sign_sdcch4
xfail BTS_Tests.TC_meas_res_sign_sdcch8
xfail BTS_Tests.TC_meas_res_sign_tchh_toa256

Those above are more or less expected to fail (some even already xfail).
However, with a coaxial wiring + attenuator setup and/or some larger
tolerances, they should be passing reliably even in a physical setup

pass->FAIL BTS_Tests.TC_rsl_ms_pwr_ctrl

pass->FAIL BTS_Tests.TC_paging_tmsi_80percent
pass->FAIL BTS_Tests.TC_paging_imsi_200percent
pass->FAIL BTS_Tests.TC_paging_tmsi_200percent

Those should also be able to make "pass". The number of paging request
that we can process in a virtual BTS and a real hardware BTS is
identical, as b

pass->FAIL BTS_Tests.TC_si_sched_13_2bis_2ter_2quater

this one should definitely pass

pass->FAIL BTS_Tests.TC_dyn_osmo_pdch_act_deact
pass->FAIL BTS_Tests.TC_dyn_osmo_pdch_double_act

those should pass, too.

pass->FAIL BTS_Tests.TC_chan_act_a52

I'm not sure if the hardware supports A5/2. I think it does.

pass->FAIL BTS_Tests.TC_chan_act_a53
pass->FAIL BTS_Tests.TC_encr_cmd_a52
pass->FAIL BTS_Tests.TC_encr_cmd_a53

A5/3 is definitely supported, so those should also pass. Either a bug
in the test or in the BTS.

Actions #11

Updated by pespin almost 5 years ago

Hi Philipp,

As far as I know the parameters where fine at least until recently, since osmo-gsm-tester+ttcn3 was using an old version of ttcn3-bts-test docker container. Cause of running an old version is basically that I didn't find time until recently to add a jenkins job (sysmocom jenkins job "ttcn3-bts-test_docker-registry") to build and push a new version of ttcn3-bts-test nightly. So it could be that since it is now running newer versions, then some stuff in TTCn3 files changes and we need to update osmo-gsm-tester conf templates to adapt to those changes.

BTS_Tests.mp_bb_trxc_port being -1 was done on purpose for sysmobts & oc2g, since that disables (at least it used to) use of some parts of the test which are not needed, only when running osmo-bts-trx (which the regular nightly TTCN3 test do).

Regarding your comment "There are also a lot of components that are not needed for the test. I think osmo-msc, osmo-mgw, osmo-hlr, and osmo-stp should be removed. The only purpose osmo-bsc is serving here is to bootstrap the BTS via OML.":
I agree actually only osmo-bsc is used, but anyway osmo-gsm-tester test needs to create all those in order to generate proper config files. Looking at it, I think we could be able to simply instantiate the objects and don't run the processes by avoid calling ".start()" on them in ttcn3_bts_tests.py, but anyway this specifically is not very urgent.

Regarding osmo-gsm-tester+ttcn3 tests, you can find most stuff used under this directory: https://git.osmocom.org/osmo-gsm-tester/tree/ttcn3

Specially interesting:
https://git.osmocom.org/osmo-gsm-tester/tree/ttcn3/suites/ttcn3_bts_tests/ttcn3_bts_tests.py <-- test run by osmo-gsm-tester (each time for a different BTS type) in job osmo-gsm-tester_ttcn3.
https://git.osmocom.org/osmo-gsm-tester/tree/ttcn3/suites/ttcn3_bts_tests/scripts/run_ttcn3_docker.sh <-- script run by ttcn3_bts_tests.py to set up and run the docker TTCN3 container
https://git.osmocom.org/osmo-gsm-tester/tree/ttcn3/suites/ttcn3_bts_tests/scripts/BTS_Tests.cfg.tmpl <-- Template from where osmo-gsm-tester (ttcn3_bts_tests.py) generates the BTS_Tests.cfg config file to use on the TTCN3 docker container.

Can you please submit a patch against osmo-gsm-tester updating the config files/templates as needed so I can review? Or at least paste a diff here of the changes you required.

Actions #12

Updated by dexter almost 5 years ago

To me it looks like if BTS_Tests.mp_bb_trxc_port were renamed to BTS_Tests.mp_bts_trxc_port. I have submitted a patch. Once that is merged we should get past the error:

https://gerrit.osmocom.org/#/c/osmo-gsm-tester/+/13771 BTS_Tests.cfg.tmpl: rename BTS_Tests.mp_bb_trxc_port

Actions #13

Updated by dexter almost 5 years ago

We now get past the error, however there is a similar conflict now:

https://gerrit.osmocom.org/#/c/osmo-gsm-tester/+/13772 BTS_Tests.cfg.tmpl rename BTS_Tests.mp_bb_trxc_ip

Actions #14

Updated by dexter almost 5 years ago

  • % Done changed from 20 to 40

The test results look much better now. For oc2g we even get results in the test results analyzer. I have checked the artifacts, the testsuites for the other tests run too, at least to some point. The test for sysmobts gets aborted with SIGINT while the testsuite for osmo-trx seems to complete. I wonder why the results do not appear in the test results analyzer.

https://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_ttcn3/test_results_analyzer/

Actions #15

Updated by dexter almost 5 years ago

The results for trx and oc2g are there. The visualization is just a bit irritating. The results for sysmobts are missing. This makes sense since the testsuite abortetd with SIGINT while the others did complete. Presumably we just need to find out why the sysmobts test gets killed (timeout?) and once it completes the xml will be there and the the test results analyzer can display it.

Actions #16

Updated by laforge over 4 years ago

  • Priority changed from Normal to Urgent

ping?

Actions #17

Updated by laforge over 4 years ago

  • Status changed from In Progress to Stalled
  • Assignee deleted (dexter)
  • Priority changed from Urgent to Normal
Actions

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)