Cellular Network Infrastructure » OsmoGSMTester

It was spotted several times that all osmo-trx-lms tests in osmo-gsm-tester fail with message:

socket.c:367 unable to bind socket:10.42.42.117:4237: Address already in use

Close lookup shows osmo-trx-lms stil running but idle (not consuming CPU):

# ps -ef | grep osmo-trx-lms
root     14643 14604  0 11:47 pts/1    00:00:00 grep osmo-trx-lms
jenkins  55210     1  0 Aug06 ?        00:00:53 /osmo-gsm-tester-trx/last_run/osmo-trx/bin/osmo-trx-lms -C /osmo-gsm-tester-trx/last_run/osmo-trx.cfg

In order to get process creation time (too see which test caused the issue and if logs around it provide more information):

# ls -ld --time-style=full-iso  /proc/$(pidof osmo-trx-lms)
dr-xr-xr-x 9 jenkins jenkins 0 2019-08-06 10:37:14.274166715 +0200 /proc/55210

At that time, following run was in place:
https://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_run-prod/1926/

And the test: trial-1926 gprs:trx-lms+mod-bts0-numtrx2+mod-bts0-chanallocdescend cs_paging_gprs_active.py

The test runs and at some fails (expected since multi-trx is not yet supported in osmo-trx-lms) and then osmo-gsm-tester goes over regular procedure to kill all processes (in the case of osmo-trx-lms, it kills the ssh client, which should end up killing its child through the script handler):

10:38:00.558825 ---      ParallelTerminationStrategy: DBG: Scheduled to terminate 22 processes.  [process.py:108]
10:38:00.560001 ---      ParallelTerminationStrategy: DBG: Starting to kill with SIGTERM  [process.py:116]
...
10:38:00.669914 run           osmo-trx-lms(pid=1883): Terminating (SIGTERM)  [trial-1926↪gprs:trx-lms+mod-bts0-numtrx2+mod-bts0-chanallocdescend↪osmo-bts-trx↪osmo-trx-lms↪osmo-trx-lms(pid=1883)]  [process.py:236]
...
10:38:00.773158 ---      ParallelTerminationStrategy: PID 1883 died...  [process.py:75]
10:38:00.773706 run           osmo-trx-lms(pid=1883): DBG: Cleanup  [trial-1926↪gprs:trx-lms+mod-bts0-numtrx2+mod-bts0-chanallocdescend↪osmo-bts-trx↪osmo-trx-lms↪osmo-trx-lms(pid=1883)]  [process.py:265]
10:38:00.776101 run           osmo-trx-lms(pid=1883): Terminated {rc=36608}  [trial-1926↪gprs:trx-lms+mod-bts0-numtrx2+mod-bts0-chanallocdescend↪osmo-bts-trx↪osmo-trx-lms↪osmo-trx-lms(pid=1883)]  [process.py:270]

So my guess is not that ssh killing its child process is not working, but rather than when running with multi-trx we may end up in some race condition which somehow blocks osmo-trx-lms and prevents it from exiting.