Bug #6411
openttcn3: execute testsuites with a more realistic link performance
0%
Description
This idea by laforge originates from #6375: we should try running at least some of our BTS / BSC tests over an Abis link with simulated latency and non-zero packet loss. I did some experiments using tc-netem, running the whole ttcn3-bts-test with a simulated delay: #6375#note-14. Let's continue discussing this here.
Updated by fixeria about 1 month ago
This patch:
https://gerrit.osmocom.org/c/docker-playground/+/36372 ttcn3-*-test: allow running tests with artificial link delay
makes simulating the link delay as easy as passing an additional environment variable:
$ export DOCKER_ARGS="--cap-add=NET_ADMIN -e 'OSMO_TTCN3_NETEM_DELAY=30'"
$ cd ttcn3-bsc-test
$ ./jenkins.sh
Somehow, the --cap-add=NET_ADMIN (needed for the tc-netem) breaks tcpdump/dumpcap:
dumpcap: The file to which the capture would be saved ("/data/BTS_Tests.TC_est_dchan.pcap") could not be opened: Permission denied.
This needs to be investigated and fixed/worked around before we merge the patch.
Updated by fixeria about 1 month ago
I gave a few testsuites a spin with OSMO_TTCN3_NETEM_DELAY=30.
ttcn3-bsc-test¶
Most of the testcases are passing, except the following ones:
- pass->FAIL BSC_Tests.TC_paging_500req
- pass->FAIL BSC_Tests.TC_paging_450req_no_paging_load_ind
- pass->FAIL BSC_Tests.TC_ho_out_fail_no_msc_response
- pass->FAIL BSC_Tests.TC_ho_out_fail_no_result_after_ho_cmd
- pass->FAIL BSC_Tests.TC_ho_in_fail_no_detect
- pass->FAIL BSC_Tests.TC_ho_neighbor_config_5
- pass->FAIL BSC_Tests.TC_ho_neighbor_config_6
- pass->FAIL BSC_Tests.TC_ratectr_all_available_allocated_dyn
- pass->FAIL BSC_Tests.TC_lost_sdcch_during_assignment
It took ~1.5h to execute the testsuite, so the time overhead is not significant.
ttcn3-bts-test¶
Most of the testcases are passing, except the following ones:
- pass->FAIL BTS_Tests.TC_chan_act_stress
- pass->FAIL BTS_Tests.TC_rsl_ms_pwr_ctrl
- pass->FAIL BTS_Tests.TC_paging_tmsi_200percent
- pass->FAIL BTS_Tests.TC_pcu_time_ind
- pass->FAIL BTS_Tests.TC_pcu_rts_req
- pass->FAIL BTS_Tests.TC_data_rtp_tchh48
- pass->FAIL BTS_Tests.TC_data_rtp_tchh24
- pass->FAIL BTS_Tests.TC_acch_overpower_always_on_sacch
- pass->FAIL BTS_Tests_SMSCB.TC_cbc_sdcch4_load_overload
- pass->FAIL BTS_Tests_SMSCB.TC_etws_p1ro
- pass->FAIL BTS_Tests_LAPDm.TC_iframe_timer_recovery
- pass->FAIL BTS_Tests_LAPDm.TC_rec_invalid_frame
- pass->FAIL BTS_Tests_LAPDm.TC_segm_concat_sacch
- pass->FAIL BTS_Tests_LAPDm.TC_t200_n200
- pass->FAIL BTS_Tests_LAPDm.TC_normal_reestablishment
The generic profile took ~1h to execute, so the time overhead is not significant.
