Osmocom Libraries » libosmo-abis

can we please get the back-ports to 1.4.x and 1.5.x and related news items out of the door soon-ish?

Please rephrase below:

On Sat, Mar 16, 2024 at 06:06:02AM +0000, fixeria wrote:

This issue primarily impacted production users operating IPA connections (A-bis/OML, A-bis/RSL, GSUP) over links with real-life delays, while it is unlikely to have affected the lab users.

over "links with delays with RTT approaching one second, such as overloaded/flakey wifi links or VSAT
back-haul. It is unlikely to have affected users with non-overloaded terrestrial/wired back-haul".

Apart from that: all good!

osmith wrote in #note-7:

I'm having a hard time getting e.g. osmo-bts into a state where it loses the abis link.

Likewise, I did some experiments and yet was unable to make TTCN-3 test cases fail by introducing artificial delays.

What I have tried:

• use netem to just slow down a specific test, which should in theory result in losing the TCP connection if the delay is >= 1s and the fix is not applied to libosmo-abis
• picked TC_est_dchan from the BTS testsuite, which has some abis traffic
• however this resulted in hitting unrelated timeouts in the testsuite, so I have tried another approach: writing a separate ttcn3 test that adds a delay in the TCP connection to get into the error state. As I looked closer into this, I realized that it is probably quite some effort to do this at the TCP level with our TTCN-3 testsuite. So I went back to the netem approach.
• I have adjusted a timeout in the testsuite that caused the unrelated failure, and then the test took significantly longer and was able to pass with an artificial delay of 900 ms, 1000ms, 1500ms.
• Unfortunately the test also passes with the delay if libosmo-abis is not patched...

So looking further into reproducing it. My plan is to make the patch releases after verifying that the default timeout now works as intended.

EDIT: tc command I'm using: "tc qdisc add dev eth0 root netem delay 1500ms"

Below is what I did:

• modify the the ttcn3-bts-test image to include the iproute2 package, which provides the tc-netem
• build the osmo-bts-master image with an old/unpatched version of libosmo-abis.git (v1.5.1)
• enable the TCP keepalive, which is disabled by default in osmo-bts.cfg

diff --git a/ttcn3-bts-test/generic/osmo-bts.confmerge b/ttcn3-bts-test/generic/osmo-bts.confmerge
index 9015904..17bd5ab 100644
--- a/ttcn3-bts-test/generic/osmo-bts.confmerge
+++ b/ttcn3-bts-test/generic/osmo-bts.confmerge
@@ -1,5 +1,7 @@
 line vty
  bind 0.0.0.0
+e1_input
+ e1_line 0 keepalive
 phy 0
  osmotrx ip local 172.18.9.20
  osmotrx ip remote 172.18.9.21

• patch ttcn3-bts-test/jenkins.sh
  • to use custom images (note the :netem tag)
  • to execute ttcn3-bts-test with --cap-add=NET_ADMIN (needed for tc-netem)
  • to configure an artificial delay before starting ttcn3-bts-test

diff --git a/ttcn3-bts-test/jenkins.sh b/ttcn3-bts-test/jenkins.sh
index f932b4a..e374044 100755
--- a/ttcn3-bts-test/jenkins.sh
+++ b/ttcn3-bts-test/jenkins.sh
@@ -50,7 +50,7 @@ start_bts() {
                        -e "SLEEP_BEFORE_RESPAWN=$sleep_time_respawn" \
                        --name ${BUILD_TAG}-bts -d \
                        $DOCKER_ARGS \
-                       $REPO_USER/osmo-bts-$IMAGE_SUFFIX \
+                       $REPO_USER/osmo-bts-$IMAGE_SUFFIX:netem \
                        /bin/sh -c "/usr/local/bin/respawn.sh osmo-bts-$variant -c /data/osmo-bts.gen.cfg >>/data/osmo-bts.log 2>&1" 
 }

@@ -117,8 +117,10 @@ start_testsuite() {
                        -v $VOL_BASE_DIR/bts-tester-${variant}:/data \
                        -v $VOL_BASE_DIR/unix:/data/unix \
                        --name ${BUILD_TAG}-ttcn3-bts-test \
+                       --cap-add=NET_ADMIN \
                        $DOCKER_ARGS \
-                       $REPO_USER/ttcn3-bts-test
+                       $REPO_USER/ttcn3-bts-test:netem \
+                       /bin/sh -c "tc qdisc add dev eth0 root netem delay 1000ms;  ttcn3-docker-run bts BTS_Tests;" 
 }

 set_pcuif_version() {

With that, I am able to execute BTS_Tests.TC_est_dchan and it's passing reliably (without any additional changes) with 500ms, 1000ms, 1500ms.
I confirmed the delay is applied correctly by running ping. The testcase execution time also increases significantly.

As a side effect, for some unknown reason, the testsuite fails to start tcpdump with those modifications:

fixeria@fixeria-dev:~/docker-playground/ttcn3-bts-test$ ls -la /tmp/tmp.IjrR8l78tv/bts-tester-generic/
total 7732
drwxr-xr-x  3 fixeria fixeria    4096 Mar 17 08:04 .
drwxr-xr-x 11 fixeria fixeria    4096 Mar 17 08:04 ..
-rw-r--r--  1 fixeria fixeria     751 Mar 17 08:04 BTS_Tests.cfg
-rw-r--r--  1 root    root    7882935 Mar 17 08:04 BTS_Tests.TC_est_dchan.merged
-rw-r--r--  1 root    root          0 Mar 17 08:02 BTS_Tests.TC_est_dchan.netcat.stderr
-rw-r--r--  1 root    root        151 Mar 17 08:02 BTS_Tests.TC_est_dchan.pcap.stderr
-rw-r--r--  1 root    root        151 Mar 17 08:02 BTS_Tests.TC_est_dchan.pcap.stdout
-rw-r--r--  1 root    root          0 Mar 17 08:04 BTS_Tests.TC_est_dchan.talloc
-rw-r--r--  1 root    root        204 Mar 17 08:04 junit-xml-22.log
drwxr-xr-x  2 root    root       4096 Mar 17 08:02 unix
fixeria@fixeria-dev:~/docker-playground/ttcn3-bts-test$ cat /tmp/tmp.IjrR8l78tv/bts-tester-generic/BTS_Tests.TC_est_dchan.pcap.stderr
Capturing on 'any'
dumpcap: The file to which the capture would be saved ("/data/BTS_Tests.TC_est_dchan.pcap") could not be opened: Permission denied.
fixeria@fixeria-dev:~/docker-playground/ttcn3-bts-test$ cat /tmp/tmp.IjrR8l78tv/bts-tester-generic/BTS_Tests.TC_est_dchan.pcap.stdout 
Capturing on 'any'
dumpcap: The file to which the capture would be saved ("/data/BTS_Tests.TC_est_dchan.pcap") could not be opened: Permission denied.

fixeria wrote in #note-14:

With that, I am able to execute BTS_Tests.TC_est_dchan and it's passing reliably (without any additional changes) with 500ms, 1000ms, 1500ms.

I did some experiments, running osmo-bsc and osmo-bts with TCP keepalive locally (not in Docker) and I was able to reproduce the problem: osmo-bts does terminate the A-bis link.

Yesterday I gave ttcn3-bts-test a spin with simulated delay of 600ms on the A-bis/RSL interface.

fixeria@fixeria-dev:/tmp/tmp.VKcOSQ8db6/bts-tester-generic$ ls -lah | grep TC_chan_act_stress
-rw-r--r--  1 root    root     870K Mar 18 21:20 BTS_Tests.TC_chan_act_stress.merged
-rw-r--r--  1 root    root        0 Mar 18 19:27 BTS_Tests.TC_chan_act_stress.netcat.stderr

Judging by this output, executing generic profile of the testsuite took ~2h. Executing the hopping profile would be +2h, so ~4h in total.
According to Jenkins, right now it takes ~2h to execute both profiles, so adding a delay of 600ms would roughly double the testsuite execution time.

As was agreed during the Jitsi call today, I will experiment further with lower delay values (<100ms) and other testsuites.

Update for OE: https://gerrit.osmocom.org/c/meta-telephony/+/36517