Cellular Network Infrastructure » cni-legacy » OpenBSC » OsmoNITB

In osmo-gsm-tester-run jenkins job there was a FAILED test whith the following issue in test smpp/esme_ms_sms_storeforward.py:

11:11:55.154065 tst                       esme-22982: DBG: message received: {seq=665772384}  [trial-1824↪smpp↪esme-22982]  [esme.py:117]
11:11:55.190338 tst                       esme-22982: DBG: message received: {references_pending_receipt=[2, 3], user_message_reference=None}  [trial-1824↪smpp↪esme-22982]  [esme.py:121]
11:11:55.194636 tst   esme_ms_sms_storeforward.py:49: ERR: ValueError: list.remove(x): x not in list  [trial-1824↪smpp↪esme_ms_sms_storeforward.py:49]  [esme.py:122: self.references_pending_receipt.remove(pdu.user_message_reference)]
11:11:55.200672 tst   esme_ms_sms_storeforward.py:49: TRACEBACK: Traceback (most recent call last):
  File "/home/jenkins/workspace/osmo-gsm-tester_run/osmo-gsm-tester/src/osmo_gsm_tester/suite.py", line 105, in run
    self.path)
  File "/home/jenkins/workspace/osmo-gsm-tester_run/osmo-gsm-tester/src/osmo_gsm_tester/util.py", line 282, in run_python_file
    SourceFileLoader(module_name, path).load_module()
  File "<frozen importlib._bootstrap>", line 539, in _check_name_wrapper
  File "<frozen importlib._bootstrap>", line 1614, in load_module
  File "<frozen importlib._bootstrap>", line 596, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 1220, in load
  File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1129, in _exec
  File "<frozen importlib._bootstrap>", line 1471, in exec_module
  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
  File "/home/jenkins/workspace/osmo-gsm-tester_run/osmo-gsm-tester/suites/smpp/esme_ms_sms_storeforward.py", line 49, in <module>
    wait(ms.sms_was_received, msg)
  File "/home/jenkins/workspace/osmo-gsm-tester_run/osmo-gsm-tester/src/osmo_gsm_tester/test.py", line 47, in <lambda>
    wait = lambda *args, **kwargs: event_module.wait(suite_run, *args, **kwargs)
  File "/home/jenkins/workspace/osmo-gsm-tester_run/osmo-gsm-tester/src/osmo_gsm_tester/event_loop.py", line 59, in wait
    if not wait_no_raise(log_obj, condition, condition_args, condition_kwargs, timeout, timestep):
  File "/home/jenkins/workspace/osmo-gsm-tester_run/osmo-gsm-tester/src/osmo_gsm_tester/event_loop.py", line 50, in wait_no_raise
    poll()
  File "/home/jenkins/workspace/osmo-gsm-tester_run/osmo-gsm-tester/src/osmo_gsm_tester/event_loop.py", line 39, in poll
    func()
  File "/home/jenkins/workspace/osmo-gsm-tester_run/osmo-gsm-tester/src/osmo_gsm_tester/esme.py", line 78, in poll
    self.client.poll()
  File "/usr/local/lib/python3.4/dist-packages/smpplib/client.py", line 321, in poll
    self.read_once(ignore_error_codes)
  File "/usr/local/lib/python3.4/dist-packages/smpplib/client.py", line 297, in read_once
    self._message_received(p)
  File "/usr/local/lib/python3.4/dist-packages/smpplib/client.py", line 237, in _message_received
    self.message_received_handler(pdu=p)
  File "/home/jenkins/workspace/osmo-gsm-tester_run/osmo-gsm-tester/src/osmo_gsm_tester/esme.py", line 122, in _message_received_handler
    self.references_pending_receipt.remove(pdu.user_message_reference)
ValueError: list.remove(x): x not in list
  [trial-1824↪smpp↪esme_ms_sms_storeforward.py:49]  [suite.py:148]
11:11:55.208186 tst   esme_ms_sms_storeforward.py:49: Test FAILED (58.8 sec)  [trial-1824↪smpp↪esme_ms_sms_storeforward.py:49]  [suite.py:149]

The issue comes from the following fact --> user_message_reference=None
Which potentially means a deliverSM pdu was received which didn't contain the field user_message_reference.

The issue doesn't seem to happen usually, seems like sporadic (only saw it once for now). I attach the run trial which should provide some information. Due to issue #2413, the pcap file in the archive doesn't contain the SMPP traces, it may be a good idea to fix that issue first.

Try to reproduce the issue or have a look at related python-smpplib and openbsc SMSC code to see possible issues which can explain this.

Check the SMPP reference to see what it has to say about that PDU field too.