Support #3863
opensetup testing of osmo-bts-oc2g on real hardware with ttcn3 and osmocon/osmocom-bb
Added by dexter about 5 years ago. Updated over 4 years ago.
40%
Description
In order to be able to debug problems with automatic interop testing a local instance of an osmocom-bb phone and BTS is required. TRXCONT/FAKETRX are exchanged with a real BTS/PHONE.
Files
| BTS_Tests.cfg | BTS_Tests.cfg | 4.41 KB | dexter, 04/23/2019 01:33 PM | ||
| BTS_Tests.default | BTS_Tests.default | 661 Bytes | dexter, 04/23/2019 01:33 PM | ||
| osmo-bsc.cfg | osmo-bsc.cfg | 4.7 KB | dexter, 04/23/2019 01:33 PM | ||
| junit-xml-3840.log | junit-xml-3840.log | 15.8 KB | dexter, 04/23/2019 02:32 PM |
Updated by dexter about 5 years ago
- Status changed from New to In Progress
At the moment I am having problems to set this up. I have configured osmo-bsc so that it tells the BTS the RSL IP where TTCN3 listens. Also the VTY is reachable for TTCN3. However, I still get:
MTC@lobotron: Test case TC_si_sched_2bis finished. Verdict: fail reason: Could not connect IPA socket from "" port 0 to "127.0.0.1" port 4238; check your configuration
Updated by dexter about 5 years ago
- % Done changed from 0 to 10
I came a bit further with my setup. However, at the moment I still have a problem:
12:42:09.567042 5 IPA_Emulation.ttcnpp:242 entering f__IPL4__PROVIDER__connect: :0 -> 127.0.0.1:4238 / TCP 12:42:09.568008 5 IPA_Emulation.ttcnpp:245 setverdict(fail): none -> fail reason: "Could not connect IPA socket from "" port 0 to "127.0.0.1" port 4238; check your configuration", new component reason: "Could not connect IPA socket from "" port 0 to "127.0.0.1" port 4238; check your configuration" 12:42:09.568074 5 IPA_Emulation.ttcnpp:247 Stopping MTC. The current test case will be terminated. 12:42:09.568302 mtc Osmocom_CTRL_Adapter.ttcn:37 Stop was requested from MC. 12:42:09.568343 mtc Osmocom_CTRL_Adapter.ttcn:37 Stopping test component execution. 12:42:09.568775 mtc BTS_Tests.ttcn:580 Test case TC_chan_act_react was stopped.
I was not aware that there is also a project osmo-pcap (ctrl interface port 4238) and that we apparently use it in our tests. Probably to get pcap traces from the inner workings of the BTS. This makes sense but I think for my experimental setup this is not needed. I wonder how I can disable this feature but what really confuses me is that when I run the tests locally osmo-pcap seems not to be required. I don't even have it installed on my workstation.
Updated by laforge about 5 years ago
osmo-pcap is not required for any of this. Please coordinate with pespin
on how to run the TTCN3 tests with OsmocomBB, as this is already started+executed
by osmo-gsm-tester.
Updated by pespin about 5 years ago
Tests for bts-oc2g are already being run in osmo-gsm-tester+ttcn3 with osmocom-bb setup.
I didn't check results for bts-oc2g yet though, but they can be found there:
https://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_ttcn3/test_results_analyzer/
Updated by dexter almost 5 years ago
I managed to get the overall setup running. The osmocom-bb phone and the BTS are communicating with each other over the air. There was some race condition I have fixed by adding a delay after the BTS is started up. Apparently it takes a second or two until the BTS is running stable. After that synchronization and RACH request work reliably and a lot more tests do pass now.
Next step should be to reproduce the results I have locally on the tester.
Updated by dexter almost 5 years ago
- File BTS_Tests.cfg BTS_Tests.cfg added
- File BTS_Tests.default BTS_Tests.default added
- File osmo-bsc.cfg osmo-bsc.cfg added
I am currently investigating why the the ttcn3 tests on the tester fail. I think it makes sense to focus on ttcn3_bts_tests:sysmo first. In the logs I see an error which I have seen before while trying to get the tests run locally:
HC@27b61be607b6: Warning: Option `FileMask' was given more than once in section [LOGGING] of the configuration file. HC@27b61be607b6: Warning: Option `FileMask' was given more than once in section [LOGGING] of the configuration file. HC@27b61be607b6: Error while setting parameter field 'BTS_Tests.mp_bb_trxc_ip' to '"127.0.0.1"': Module parameter cannot be set, because no parameter with name 'mp_bb_trxc_ip' exists in module 'BTS_Tests'. (Note: field names and array indexes are not supported in the Load Test Runtime). HC@27b61be607b6: Error while setting parameter field 'BTS_Tests.mp_bb_trxc_port' to '- 1': Module parameter cannot be set, because no parameter with name 'mp_bb_trxc_port' exists in module 'BTS_Tests'. (Note: field names and array indexes are not supported in the Load Test Runtime). Error: There were errors during configuring HCs.
I think this BTS_Tests.mp_bb_trxc_port is just wrong. Attached one finds a set of config files that works for me locally.
There are also a lot of components that are not needed for the test. I think osmo-msc, osmo-mgw, osmo-hlr, and osmo-stp should be removed. The only purpose osmo-bsc is serving here is to bootstrap the BTS via OML.
Updated by dexter almost 5 years ago
- File junit-xml-3840.log junit-xml-3840.log added
Here is a test result that reflects the current state:
pass BTS_Tests.TC_chan_act_stress pass BTS_Tests.TC_chan_act_react pass BTS_Tests.TC_chan_deact_not_active pass BTS_Tests.TC_chan_act_wrong_nr pass BTS_Tests.TC_deact_sacch pass->FAIL BTS_Tests.TC_sacch_filling pass BTS_Tests.TC_sacch_info_mod pass BTS_Tests.TC_sacch_multi pass->FAIL BTS_Tests.TC_sacch_multi_chg pass->FAIL BTS_Tests.TC_rach_content pass->FAIL BTS_Tests.TC_rach_count pass->FAIL BTS_Tests.TC_rach_max_ta pass->FAIL BTS_Tests.TC_meas_res_sign_tchf xfail BTS_Tests.TC_meas_res_sign_tchh xfail BTS_Tests.TC_meas_res_sign_sdcch4 xfail BTS_Tests.TC_meas_res_sign_sdcch8 xfail BTS_Tests.TC_meas_res_sign_tchh_toa256 pass->FAIL BTS_Tests.TC_rsl_ms_pwr_ctrl pass BTS_Tests.TC_conn_fail_crit pass BTS_Tests.TC_paging_imsi_80percent pass->FAIL BTS_Tests.TC_paging_tmsi_80percent pass->FAIL BTS_Tests.TC_paging_imsi_200percent pass->FAIL BTS_Tests.TC_paging_tmsi_200percent pass BTS_Tests.TC_rsl_protocol_error pass BTS_Tests.TC_rsl_mand_ie_error pass BTS_Tests.TC_rsl_ie_content_error pass BTS_Tests.TC_si_sched_default pass BTS_Tests.TC_si_sched_1 pass BTS_Tests.TC_si_sched_2bis pass BTS_Tests.TC_si_sched_2ter pass BTS_Tests.TC_si_sched_2ter_2bis pass BTS_Tests.TC_si_sched_2quater pass BTS_Tests.TC_si_sched_13 pass->FAIL BTS_Tests.TC_si_sched_13_2bis_2ter_2quater pass BTS_Tests.TC_ipa_dlcx_not_active pass BTS_Tests.TC_ipa_crcx_twice_not_active pass BTS_Tests.TC_ipa_crcx_mdcx_dlcx_not_active pass BTS_Tests.TC_ipa_crcx_mdcx_mdcx_dlcx_not_active pass BTS_Tests.TC_ipa_crcx_sdcch_not_active skipped BTS_Tests.TC_pcu_act_req skipped BTS_Tests.TC_pcu_act_req_wrong_ts skipped BTS_Tests.TC_pcu_act_req_wrong_bts skipped BTS_Tests.TC_pcu_act_req_wrong_trx skipped BTS_Tests.TC_pcu_deact_req skipped BTS_Tests.TC_pcu_deact_req_wrong_ts skipped BTS_Tests.TC_pcu_ver_si13 skipped BTS_Tests.TC_pcu_data_req_wrong_bts skipped BTS_Tests.TC_pcu_data_req_wrong_trx skipped BTS_Tests.TC_pcu_data_req_wrong_ts skipped BTS_Tests.TC_pcu_data_req_ts_inactive skipped BTS_Tests.TC_pcu_data_req_pdtch skipped BTS_Tests.TC_pcu_data_req_ptcch skipped BTS_Tests.TC_pcu_data_req_agch skipped BTS_Tests.TC_pcu_data_req_imm_ass_pch skipped BTS_Tests.TC_pcu_rach_content skipped BTS_Tests.TC_pcu_paging_from_rsl pass->FAIL BTS_Tests.TC_dyn_osmo_pdch_act_deact pass BTS_Tests.TC_dyn_osmo_pdch_unsol_deact pass->FAIL BTS_Tests.TC_dyn_osmo_pdch_double_act pass BTS_Tests.TC_dyn_osmo_pdch_tchf_act pass BTS_Tests.TC_dyn_osmo_pdch_tchh_act skipped BTS_Tests.TC_dyn_ipa_pdch_act_deact pass BTS_Tests.TC_dyn_ipa_pdch_tchf_act pass BTS_Tests.TC_dyn_ipa_pdch_tchf_act_pdch_act_nack skipped BTS_Tests.TC_dyn_ipa_pdch_act_tchf_act_nack pass BTS_Tests.TC_rll_est_ind pass BTS_Tests.TC_rll_est_req_DCCH_3 pass BTS_Tests.TC_rll_est_req_ACCH_3 pass BTS_Tests.TC_rll_rel_ind_DCCH_0 pass BTS_Tests.TC_rll_rel_ind_DCCH_3 pass BTS_Tests.TC_rll_rel_ind_ACCH_0 pass BTS_Tests.TC_rll_rel_ind_ACCH_3 pass BTS_Tests.TC_rll_rel_req pass BTS_Tests.TC_rll_unit_data_req_DCCH pass BTS_Tests.TC_rll_unit_data_req_ACCH pass BTS_Tests.TC_rll_unit_data_ind_DCCH pass BTS_Tests.TC_rll_unit_data_ind_ACCH pass BTS_Tests.TC_chan_act_a51 pass->FAIL BTS_Tests.TC_chan_act_a52 pass->FAIL BTS_Tests.TC_chan_act_a53 pass BTS_Tests.TC_encr_cmd_a51 pass->FAIL BTS_Tests.TC_encr_cmd_a52 pass->FAIL BTS_Tests.TC_encr_cmd_a53 pass BTS_Tests.TC_lapdm_selftest pass BTS_Tests.TC_tch_sign_l2_fill_frame xfail BTS_Tests.TC_tch_sign_l2_fill_frame_dtxd pass BTS_Tests.TC_chopped_ipa_ping pass BTS_Tests.TC_chopped_ipa_payload -------------------- 47 pass 5 xfail 19 skipped 17 FAIL
Updated by laforge almost 5 years ago
Some feedback below:
On Tue, Apr 23, 2019 at 02:34:43PM +0000, dexter [REDMINE] wrote:
pass->FAIL BTS_Tests.TC_sacch_filling
pass->FAIL BTS_Tests.TC_sacch_multi_chg
pass->FAIL BTS_Tests.TC_rach_content
for those above there is no reason this should fail in a physical setup.
I guess there's a bug in our test, or one in OsmoBTS.
pass->FAIL BTS_Tests.TC_rach_count
pass->FAIL BTS_Tests.TC_rach_max_ta
pass->FAIL BTS_Tests.TC_meas_res_sign_tchf
xfail BTS_Tests.TC_meas_res_sign_tchh
xfail BTS_Tests.TC_meas_res_sign_sdcch4
xfail BTS_Tests.TC_meas_res_sign_sdcch8
xfail BTS_Tests.TC_meas_res_sign_tchh_toa256
Those above are more or less expected to fail (some even already xfail).
However, with a coaxial wiring + attenuator setup and/or some larger
tolerances, they should be passing reliably even in a physical setup
pass->FAIL BTS_Tests.TC_rsl_ms_pwr_ctrl
pass->FAIL BTS_Tests.TC_paging_tmsi_80percent
pass->FAIL BTS_Tests.TC_paging_imsi_200percent
pass->FAIL BTS_Tests.TC_paging_tmsi_200percent
Those should also be able to make "pass". The number of paging request
that we can process in a virtual BTS and a real hardware BTS is
identical, as b
pass->FAIL BTS_Tests.TC_si_sched_13_2bis_2ter_2quater
this one should definitely pass
pass->FAIL BTS_Tests.TC_dyn_osmo_pdch_act_deact
pass->FAIL BTS_Tests.TC_dyn_osmo_pdch_double_act
those should pass, too.
pass->FAIL BTS_Tests.TC_chan_act_a52
I'm not sure if the hardware supports A5/2. I think it does.
pass->FAIL BTS_Tests.TC_chan_act_a53
pass->FAIL BTS_Tests.TC_encr_cmd_a52
pass->FAIL BTS_Tests.TC_encr_cmd_a53
A5/3 is definitely supported, so those should also pass. Either a bug
in the test or in the BTS.
Updated by pespin almost 5 years ago
Hi Philipp,
As far as I know the parameters where fine at least until recently, since osmo-gsm-tester+ttcn3 was using an old version of ttcn3-bts-test docker container. Cause of running an old version is basically that I didn't find time until recently to add a jenkins job (sysmocom jenkins job "ttcn3-bts-test_docker-registry") to build and push a new version of ttcn3-bts-test nightly. So it could be that since it is now running newer versions, then some stuff in TTCn3 files changes and we need to update osmo-gsm-tester conf templates to adapt to those changes.
BTS_Tests.mp_bb_trxc_port being -1 was done on purpose for sysmobts & oc2g, since that disables (at least it used to) use of some parts of the test which are not needed, only when running osmo-bts-trx (which the regular nightly TTCN3 test do).
Regarding your comment "There are also a lot of components that are not needed for the test. I think osmo-msc, osmo-mgw, osmo-hlr, and osmo-stp should be removed. The only purpose osmo-bsc is serving here is to bootstrap the BTS via OML.":
I agree actually only osmo-bsc is used, but anyway osmo-gsm-tester test needs to create all those in order to generate proper config files. Looking at it, I think we could be able to simply instantiate the objects and don't run the processes by avoid calling ".start()" on them in ttcn3_bts_tests.py, but anyway this specifically is not very urgent.
Regarding osmo-gsm-tester+ttcn3 tests, you can find most stuff used under this directory: https://git.osmocom.org/osmo-gsm-tester/tree/ttcn3
Specially interesting:
https://git.osmocom.org/osmo-gsm-tester/tree/ttcn3/suites/ttcn3_bts_tests/ttcn3_bts_tests.py <-- test run by osmo-gsm-tester (each time for a different BTS type) in job osmo-gsm-tester_ttcn3.
https://git.osmocom.org/osmo-gsm-tester/tree/ttcn3/suites/ttcn3_bts_tests/scripts/run_ttcn3_docker.sh <-- script run by ttcn3_bts_tests.py to set up and run the docker TTCN3 container
https://git.osmocom.org/osmo-gsm-tester/tree/ttcn3/suites/ttcn3_bts_tests/scripts/BTS_Tests.cfg.tmpl <-- Template from where osmo-gsm-tester (ttcn3_bts_tests.py) generates the BTS_Tests.cfg config file to use on the TTCN3 docker container.
Can you please submit a patch against osmo-gsm-tester updating the config files/templates as needed so I can review? Or at least paste a diff here of the changes you required.
Updated by dexter almost 5 years ago
To me it looks like if BTS_Tests.mp_bb_trxc_port were renamed to BTS_Tests.mp_bts_trxc_port. I have submitted a patch. Once that is merged we should get past the error:
https://gerrit.osmocom.org/#/c/osmo-gsm-tester/+/13771 BTS_Tests.cfg.tmpl: rename BTS_Tests.mp_bb_trxc_port
Updated by dexter almost 5 years ago
We now get past the error, however there is a similar conflict now:
https://gerrit.osmocom.org/#/c/osmo-gsm-tester/+/13772 BTS_Tests.cfg.tmpl rename BTS_Tests.mp_bb_trxc_ip
Updated by dexter almost 5 years ago
- % Done changed from 20 to 40
The test results look much better now. For oc2g we even get results in the test results analyzer. I have checked the artifacts, the testsuites for the other tests run too, at least to some point. The test for sysmobts gets aborted with SIGINT while the testsuite for osmo-trx seems to complete. I wonder why the results do not appear in the test results analyzer.
Updated by dexter almost 5 years ago
The results for trx and oc2g are there. The visualization is just a bit irritating. The results for sysmobts are missing. This makes sense since the testsuite abortetd with SIGINT while the others did complete. Presumably we just need to find out why the sysmobts test gets killed (timeout?) and once it completes the xml will be there and the the test results analyzer can display it.
Updated by laforge over 4 years ago
- Status changed from In Progress to Stalled
- Assignee deleted (
dexter) - Priority changed from Urgent to Normal
