Project

General

Profile

Bug #5130

osmocom-repo-install-* fails for osmo-sgsn

Added by laforge 18 days ago. Updated 15 days ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
-
Target version:
-
Start date:
04/25/2021
Due date:
% Done:

100%

Spec Reference:

Description

The Osmocom-repo-install-* jenkins jobs are failing since April 23rd. It looks like the systemd service somehow fails to start with the default config file.

Associated revisions

Revision a45608fe (diff)
Added by osmith 17 days ago

examples/osmo-gbproxy.cfg: fix osmo-sgsn conflict

Adjust the config, so osmo-gbproxy and osmo-sgsn can be started with
their default configs on the same machine.

Fixes: OS#5130
Change-Id: I7959ea8ebd1f5c60177f37dd6eb01724308e0fc0

History

#1 Updated by laforge 18 days ago

+ osmo-sgsn --version
OsmoSGSN version 1.7.0.15-8880
...
+ systemctl --no-pager -l -n 200 status osmo-sgsn
● osmo-sgsn.service - Osmocom SGSN (Serving GPRS Support Node)
   Loaded: loaded (/lib/systemd/system/osmo-sgsn.service; disabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Sun 2021-04-25 10:45:04 UTC; 48ms ago
  Process: 16077 ExecStart=/usr/bin/osmo-sgsn -c /etc/osmocom/osmo-sgsn.cfg (code=exited, status=2)
 Main PID: 16077 (code=exited, status=2)

Apr 25 10:45:04 886f3aff64b7 systemd[1]: osmo-sgsn.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Apr 25 10:45:04 886f3aff64b7 systemd[1]: osmo-sgsn.service: Unit entered failed state.
Apr 25 10:45:04 886f3aff64b7 systemd[1]: osmo-sgsn.service: Failed with result 'exit-code'.
+ failed= osmo-sgsn
...
ERROR: services failed to start:  osmo-sgsn

#2 Updated by laforge 18 days ago

  • Status changed from New to In Progress

jenkins says it fails on debian9, debian10 and centos.

interestingly not reproducible locally in a debian9 lxc container:

root@debian9:~# osmo-sgsn --version
OsmoSGSN version 1.7.0.15-8880
root@debian9:~# systemctl status osmo-sgsn
● osmo-sgsn.service - Osmocom SGSN (Serving GPRS Support Node)
   Loaded: loaded (/lib/systemd/system/osmo-sgsn.service; disabled; vendor preset: enabled)
   Active: active (running) since Sun 2021-04-25 20:04:57 UTC; 3s ago
 Main PID: 8523 (osmo-sgsn)
    Tasks: 1 (limit: 629145)
   Memory: 2.9M
      CPU: 16ms
   CGroup: /system.slice/osmo-sgsn.service
           └─8523 /usr/bin/osmo-sgsn -c /etc/osmocom/osmo-sgsn.cfg

Apr 25 20:04:57 debian9 osmo-sgsn[8523]: <0020> sccp_user.c:548 OsmoSGSN: Using SS7 instance 0, pc:0.23.4
Apr 25 20:04:57 debian9 osmo-sgsn[8523]: <0020> sccp_user.c:555 OsmoSGSN: Creating AS instance
Apr 25 20:04:57 debian9 osmo-sgsn[8523]: <0020> sccp_user.c:566 OsmoSGSN: Using AS instance as-clnt-OsmoS
Apr 25 20:04:57 debian9 osmo-sgsn[8523]: <0020> sccp_user.c:571 OsmoSGSN: Creating default route
Apr 25 20:04:57 debian9 osmo-sgsn[8523]: <0020> sccp_user.c:611 OsmoSGSN: No unassociated ASP for m3ua, c
Apr 25 20:04:57 debian9 osmo-sgsn[8523]: <0020> sccp_user.c:638 OsmoSGSN: Using ASP instance asp-clnt-Osm
Apr 25 20:04:57 debian9 osmo-sgsn[8523]: <001f> osmo_ss7.c:447 0: Creating SCCP instance
Apr 25 20:04:58 debian9 osmo-sgsn[8523]: <001d> gsup_client.c:75 GSUP connecting to 127.0.0.1:4222
Apr 25 20:04:59 debian9 osmo-sgsn[8523]: <001d> gsup_client.c:75 GSUP connecting to 127.0.0.1:4222
Apr 25 20:05:00 debian9 osmo-sgsn[8523]: <001d> gsup_client.c:75 GSUP connecting to 127.0.0.1:4222

#3 Updated by laforge 18 days ago

could be some kind of resource conflict when starting all of the services at the same time. e.g. gbproxy vs sgsn vs pcu on Gb or sgsn vs ggsn?

#4 Updated by laforge 18 days ago

  • % Done changed from 0 to 10

even installing and starting all programs works in my debian9 lxc. after copy+pasting the 'systemctl start' command, I get all osmo services running:


root@debian9:~# systemctl | grep osmo
  osmo-bsc.service                                 loaded active     running      Osmocom Base Station Controller (BSC)                   
  osmo-bts-trx.service                             loaded activating auto-restart Osmocom osmo-bts for osmo-trx                           
  osmo-bts-virtual.service                         loaded active     running      Osmocom GSM BTS for virtual Um layer based on GSMTAP/UDP
  osmo-gbproxy.service                             loaded activating auto-restart Osmocom Gb proxy                                        
  osmo-ggsn.service                                loaded activating auto-restart OsmoGGSN                                                
  osmo-gtphub.service                              loaded active     running      Osmocom GTP Hub                                         
  osmo-hlr.service                                 loaded active     running      Osmocom Home Location Register (OsmoHLR)                
  osmo-hnbgw.service                               loaded active     running      Osmocom Home Nodeb Gateway (OsmoHNBGW)                  
  osmo-mgw.service                                 loaded active     running      Osmocom Media Gateway (MGW)                             
  osmo-msc.service                                 loaded active     running      Osmocom Mobile Switching Center (MSC)                   
  osmo-pcap-client.service                         loaded active     running      PCAP Client for the PCAP aggregation                    
  osmo-pcap-server.service                         loaded active     running      PCAP Server for the PCAP aggregation                    
  osmo-pcu.service                                 loaded active     running      Osmocom osmo-pcu                                        
  osmo-remsim-bankd.service                        loaded activating auto-restart Osmocom Remote SIM BankD                                
  osmo-remsim-server.service                       loaded active     running      Osmocom Remote SIM Server                               
  osmo-sgsn.service                                loaded active     running      Osmocom SGSN (Serving GPRS Support Node)                
  osmo-sip-connector.service                       loaded active     running      Osmo SIP Connector                                      
  osmo-stp.service                                 loaded active     running      Osmocom STP (Signal Transfer Point)

process wise it also looks like osmo-sgsn has no problems:

root@debian9:~# ps axuw | grep osmo-
root        8523  0.0  0.0 227304  9116 ?        Ss   20:04   0:00 /usr/bin/osmo-sgsn -c /etc/osmocom/osmo-sgsn.cfg
root       10259  0.0  0.0 122984  6256 ?        Ssl  20:13   0:00 /usr/bin/osmo-remsim-server
root       10357  0.0  0.0  77352  8756 ?        Ss   20:13   0:00 /usr/bin/osmo-sip-connector -c /etc/osmocom/osmo-sip-connector.cfg
root       10404  0.0  0.0  69976  5836 ?        Ss   20:13   0:00 /usr/bin/osmo-gtphub -c /etc/osmocom/osmo-gtphub.cfg
root       11355  0.0  0.0  77956  8512 ?        Ss   20:14   0:00 /usr/bin/osmo-pcu -c /etc/osmocom/osmo-pcu.cfg
root       11406  0.0  0.0  77756  9812 ?        Ss   20:14   0:00 /usr/bin/osmo-bsc -c /etc/osmocom/osmo-bsc.cfg -s
root       11407  0.0  0.0  75652  6620 ?        Ss   20:14   0:00 /usr/bin/osmo-mgw -s -c /etc/osmocom/osmo-mgw.cfg
root       11408  0.0  0.0  77016  8388 ?        Ss   20:14   0:00 /usr/bin/osmo-hlr -c /etc/osmocom/osmo-hlr.cfg -l /var/lib/osmocom/hlr.db
root       11409  0.0  0.0  97000 10948 ?        Ss   20:14   0:00 /usr/bin/osmo-msc -c /etc/osmocom/osmo-msc.cfg
root       11410  0.0  0.0  72084  8296 ?        Ss   20:14   0:00 /usr/bin/osmo-pcap-client -c /etc/osmocom/osmo-pcap-client.cfg
root       11411  0.0  0.0  66048  6316 ?        Ss   20:14   0:00 /usr/bin/osmo-stp -c /etc/osmocom/osmo-stp.cfg
root       11412  0.0  0.0  75064  6444 ?        Ss   20:14   0:01 /usr/bin/osmo-pcap-server -c /etc/osmocom/osmo-pcap-server.cfg
root       11413  0.0  0.0  79408  7912 ?        Ss   20:14   0:00 /usr/bin/osmo-hnbgw -c /etc/osmocom/osmo-hnbgw.cfg
root       11428  0.0  0.0  75180  7768 ?        Ss   20:14   0:03 /usr/bin/osmo-bts-virtual -s -c /etc/osmocom/osmo-bts-virtual.cfg

so it looks like something docker specific?

#5 Updated by laforge 18 days ago

  • Status changed from In Progress to New
  • Assignee changed from laforge to osmith

giving up at this point, re-assigning to osmith who created the docker job.

#6 Updated by osmith 17 days ago

  • Status changed from New to In Progress
  • % Done changed from 10 to 90

It's a conflict with osmo-gbproxy, where the default config was recently adjusted.

$ journalctl -u osmo-sgsn
...
Apr 26 07:58:56 052fe164ef7a osmo-sgsn[323]: <0013> socket.c:551 unable to bind socket: 127.0.0.1:23000: Address already in use                        
Apr 26 07:58:56 052fe164ef7a osmo-sgsn[323]: Error occurred during reading the below line:                                                             
Apr 26 07:58:56 052fe164ef7a osmo-sgsn[323]:   listen 127.0.0.1 23000                                                                                  
Apr 26 07:58:56 052fe164ef7a osmo-sgsn[323]: Failed to create the bind (rc -1)!                                                                        
Apr 26 07:58:56 052fe164ef7a osmo-sgsn[323]: Failed to parse the config file: '/etc/osmocom/osmo-sgsn.cfg'                                             
Apr 26 07:58:56 052fe164ef7a osmo-sgsn[323]: <0004> sgsn_main.c:449 Error in config file   

Fix:
https://gerrit.osmocom.org/c/osmo-gbproxy/+/23901

Additional patch to run journalctl for failed services (so we can see this in the jenkins log):
https://gerrit.osmocom.org/c/osmo-ci/+/23902

#7 Updated by osmith 15 days ago

  • Status changed from In Progress to Resolved
  • % Done changed from 90 to 100

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)