Actions
Bug #5130
closedosmocom-repo-install-* fails for osmo-sgsn
Start date:
04/25/2021
Due date:
% Done:
100%
Spec Reference:
Description
The Osmocom-repo-install-* jenkins jobs are failing since April 23rd. It looks like the systemd service somehow fails to start with the default config file.
Updated by laforge almost 3 years ago
+ osmo-sgsn --version OsmoSGSN version 1.7.0.15-8880 ... + systemctl --no-pager -l -n 200 status osmo-sgsn ● osmo-sgsn.service - Osmocom SGSN (Serving GPRS Support Node) Loaded: loaded (/lib/systemd/system/osmo-sgsn.service; disabled; vendor preset: enabled) Active: activating (auto-restart) (Result: exit-code) since Sun 2021-04-25 10:45:04 UTC; 48ms ago Process: 16077 ExecStart=/usr/bin/osmo-sgsn -c /etc/osmocom/osmo-sgsn.cfg (code=exited, status=2) Main PID: 16077 (code=exited, status=2) Apr 25 10:45:04 886f3aff64b7 systemd[1]: osmo-sgsn.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Apr 25 10:45:04 886f3aff64b7 systemd[1]: osmo-sgsn.service: Unit entered failed state. Apr 25 10:45:04 886f3aff64b7 systemd[1]: osmo-sgsn.service: Failed with result 'exit-code'. + failed= osmo-sgsn ... ERROR: services failed to start: osmo-sgsn
Updated by laforge almost 3 years ago
- Status changed from New to In Progress
jenkins says it fails on debian9, debian10 and centos.
interestingly not reproducible locally in a debian9 lxc container:
root@debian9:~# osmo-sgsn --version
OsmoSGSN version 1.7.0.15-8880
root@debian9:~# systemctl status osmo-sgsn
● osmo-sgsn.service - Osmocom SGSN (Serving GPRS Support Node)
Loaded: loaded (/lib/systemd/system/osmo-sgsn.service; disabled; vendor preset: enabled)
Active: active (running) since Sun 2021-04-25 20:04:57 UTC; 3s ago
Main PID: 8523 (osmo-sgsn)
Tasks: 1 (limit: 629145)
Memory: 2.9M
CPU: 16ms
CGroup: /system.slice/osmo-sgsn.service
└─8523 /usr/bin/osmo-sgsn -c /etc/osmocom/osmo-sgsn.cfg
Apr 25 20:04:57 debian9 osmo-sgsn[8523]: <0020> sccp_user.c:548 OsmoSGSN: Using SS7 instance 0, pc:0.23.4
Apr 25 20:04:57 debian9 osmo-sgsn[8523]: <0020> sccp_user.c:555 OsmoSGSN: Creating AS instance
Apr 25 20:04:57 debian9 osmo-sgsn[8523]: <0020> sccp_user.c:566 OsmoSGSN: Using AS instance as-clnt-OsmoS
Apr 25 20:04:57 debian9 osmo-sgsn[8523]: <0020> sccp_user.c:571 OsmoSGSN: Creating default route
Apr 25 20:04:57 debian9 osmo-sgsn[8523]: <0020> sccp_user.c:611 OsmoSGSN: No unassociated ASP for m3ua, c
Apr 25 20:04:57 debian9 osmo-sgsn[8523]: <0020> sccp_user.c:638 OsmoSGSN: Using ASP instance asp-clnt-Osm
Apr 25 20:04:57 debian9 osmo-sgsn[8523]: <001f> osmo_ss7.c:447 0: Creating SCCP instance
Apr 25 20:04:58 debian9 osmo-sgsn[8523]: <001d> gsup_client.c:75 GSUP connecting to 127.0.0.1:4222
Apr 25 20:04:59 debian9 osmo-sgsn[8523]: <001d> gsup_client.c:75 GSUP connecting to 127.0.0.1:4222
Apr 25 20:05:00 debian9 osmo-sgsn[8523]: <001d> gsup_client.c:75 GSUP connecting to 127.0.0.1:4222
Updated by laforge almost 3 years ago
could be some kind of resource conflict when starting all of the services at the same time. e.g. gbproxy vs sgsn vs pcu on Gb or sgsn vs ggsn?
Updated by laforge almost 3 years ago
- % Done changed from 0 to 10
even installing and starting all programs works in my debian9 lxc. after copy+pasting the 'systemctl start' command, I get all osmo services running:
root@debian9:~# systemctl | grep osmo osmo-bsc.service loaded active running Osmocom Base Station Controller (BSC) osmo-bts-trx.service loaded activating auto-restart Osmocom osmo-bts for osmo-trx osmo-bts-virtual.service loaded active running Osmocom GSM BTS for virtual Um layer based on GSMTAP/UDP osmo-gbproxy.service loaded activating auto-restart Osmocom Gb proxy osmo-ggsn.service loaded activating auto-restart OsmoGGSN osmo-gtphub.service loaded active running Osmocom GTP Hub osmo-hlr.service loaded active running Osmocom Home Location Register (OsmoHLR) osmo-hnbgw.service loaded active running Osmocom Home Nodeb Gateway (OsmoHNBGW) osmo-mgw.service loaded active running Osmocom Media Gateway (MGW) osmo-msc.service loaded active running Osmocom Mobile Switching Center (MSC) osmo-pcap-client.service loaded active running PCAP Client for the PCAP aggregation osmo-pcap-server.service loaded active running PCAP Server for the PCAP aggregation osmo-pcu.service loaded active running Osmocom osmo-pcu osmo-remsim-bankd.service loaded activating auto-restart Osmocom Remote SIM BankD osmo-remsim-server.service loaded active running Osmocom Remote SIM Server osmo-sgsn.service loaded active running Osmocom SGSN (Serving GPRS Support Node) osmo-sip-connector.service loaded active running Osmo SIP Connector osmo-stp.service loaded active running Osmocom STP (Signal Transfer Point)
process wise it also looks like osmo-sgsn has no problems:
root@debian9:~# ps axuw | grep osmo- root 8523 0.0 0.0 227304 9116 ? Ss 20:04 0:00 /usr/bin/osmo-sgsn -c /etc/osmocom/osmo-sgsn.cfg root 10259 0.0 0.0 122984 6256 ? Ssl 20:13 0:00 /usr/bin/osmo-remsim-server root 10357 0.0 0.0 77352 8756 ? Ss 20:13 0:00 /usr/bin/osmo-sip-connector -c /etc/osmocom/osmo-sip-connector.cfg root 10404 0.0 0.0 69976 5836 ? Ss 20:13 0:00 /usr/bin/osmo-gtphub -c /etc/osmocom/osmo-gtphub.cfg root 11355 0.0 0.0 77956 8512 ? Ss 20:14 0:00 /usr/bin/osmo-pcu -c /etc/osmocom/osmo-pcu.cfg root 11406 0.0 0.0 77756 9812 ? Ss 20:14 0:00 /usr/bin/osmo-bsc -c /etc/osmocom/osmo-bsc.cfg -s root 11407 0.0 0.0 75652 6620 ? Ss 20:14 0:00 /usr/bin/osmo-mgw -s -c /etc/osmocom/osmo-mgw.cfg root 11408 0.0 0.0 77016 8388 ? Ss 20:14 0:00 /usr/bin/osmo-hlr -c /etc/osmocom/osmo-hlr.cfg -l /var/lib/osmocom/hlr.db root 11409 0.0 0.0 97000 10948 ? Ss 20:14 0:00 /usr/bin/osmo-msc -c /etc/osmocom/osmo-msc.cfg root 11410 0.0 0.0 72084 8296 ? Ss 20:14 0:00 /usr/bin/osmo-pcap-client -c /etc/osmocom/osmo-pcap-client.cfg root 11411 0.0 0.0 66048 6316 ? Ss 20:14 0:00 /usr/bin/osmo-stp -c /etc/osmocom/osmo-stp.cfg root 11412 0.0 0.0 75064 6444 ? Ss 20:14 0:01 /usr/bin/osmo-pcap-server -c /etc/osmocom/osmo-pcap-server.cfg root 11413 0.0 0.0 79408 7912 ? Ss 20:14 0:00 /usr/bin/osmo-hnbgw -c /etc/osmocom/osmo-hnbgw.cfg root 11428 0.0 0.0 75180 7768 ? Ss 20:14 0:03 /usr/bin/osmo-bts-virtual -s -c /etc/osmocom/osmo-bts-virtual.cfg
so it looks like something docker specific?
Updated by laforge almost 3 years ago
- Status changed from In Progress to New
- Assignee changed from laforge to osmith
giving up at this point, re-assigning to osmith who created the docker job.
Updated by osmith almost 3 years ago
- Status changed from New to In Progress
- % Done changed from 10 to 90
It's a conflict with osmo-gbproxy, where the default config was recently adjusted.
$ journalctl -u osmo-sgsn ... Apr 26 07:58:56 052fe164ef7a osmo-sgsn[323]: <0013> socket.c:551 unable to bind socket: 127.0.0.1:23000: Address already in use Apr 26 07:58:56 052fe164ef7a osmo-sgsn[323]: Error occurred during reading the below line: Apr 26 07:58:56 052fe164ef7a osmo-sgsn[323]: listen 127.0.0.1 23000 Apr 26 07:58:56 052fe164ef7a osmo-sgsn[323]: Failed to create the bind (rc -1)! Apr 26 07:58:56 052fe164ef7a osmo-sgsn[323]: Failed to parse the config file: '/etc/osmocom/osmo-sgsn.cfg' Apr 26 07:58:56 052fe164ef7a osmo-sgsn[323]: <0004> sgsn_main.c:449 Error in config file
Fix:
https://gerrit.osmocom.org/c/osmo-gbproxy/+/23901
Additional patch to run journalctl for failed services (so we can see this in the jenkins log):
https://gerrit.osmocom.org/c/osmo-ci/+/23902
Updated by osmith almost 3 years ago
- Status changed from In Progress to Resolved
- % Done changed from 90 to 100
Actions
