Project

General

Profile

Actions

Feature #4107

open

Start systemd services as non-root user

Added by osmith over 4 years ago. Updated 4 months ago.

Status:
Stalled
Priority:
High
Assignee:
Target version:
-
Start date:
07/15/2019
Due date:
% Done:

40%

Spec Reference:

Description

laforge wrote in OS#3369:

Ideally, as far as possible, we should start them as non-root user
(which may require changes to our systemd service files, etc. in the
individual git repos - but that is fine!). Starting them as non-root
will also means that any writes to unintended directories like '/'will
be discovered as they then would make the program start fail.


Subtasks 1 (0 open1 closed)

Feature #5684: Create osmocom user via OE recipeRejectedosmith09/19/2022

Actions

Related issues

Related to Cellular Network Infrastructure - Bug #3369: no automatic testing of Debian/Ubuntu packagesResolvedosmith06/29/2018

Actions
Related to Cellular Network Infrastructure - Bug #4821: Update working dir in systemd unit files Resolvedmsuraev10/20/2020

Actions
Related to OsmoGGSN (former OpenGGSN) - Bug #2250: OpenGGSN requires to run as root for no apparent reasonClosedlaforge05/10/2017

Actions
Related to Cellular Network Infrastructure - Bug #5669: Test .deb packages built by our OBSNew08/30/2022

Actions
Related to Cellular Network Infrastructure - Bug #5685: Dropping debian 10 (buster)Feedbackosmith09/19/2022

Actions
Related to Cellular Network Infrastructure - Bug #5687: Document and implement realtime scheduling hierarchyClosedmsuraev09/20/2022

Actions
Related to Cellular Network Infrastructure - Feature #5722: Migrate jenkins build slaves from docker to podmanNew10/21/2022

Actions
Actions #1

Updated by osmith over 4 years ago

  • Related to Bug #3369: no automatic testing of Debian/Ubuntu packages added
Actions #2

Updated by laforge over 4 years ago

  • Priority changed from Normal to Low
Actions #3

Updated by laforge over 3 years ago

Programs like osmo-msc, osmo-sgsn, osmo-cbc, osmo-smlc, osmo-hlr have no real time requirements or special needs in terms of raw networks sockets or tun devices. All of those should be executed as normal, non-privileged user from the start. This could be done via the systemd unit files. This could be done via the systemd unit files, or explicitly inside the osmocom programs via a privilege dropping approach.

the only processes that need special privileges are (AFAICT):
  • osmo-gbproxy requires CAP_NET_RAW if IPPROTO_GTP sockets are required for FR/GRE/IP
  • osmo-trx, osmo-bts, osmo-pcu requires CAP_SYS_NICE if SCHED_RR is to be used per command line argument (and is not done by e.g. systemd before starting it)
  • osmo-ggsn requires CAP_NET_ADMIN for setting up the gtp0/tun0 devices (unless this is done externally before starting it)
  • any program requires CAP_SYS_NICE if it uses the relatively new libosmocore/src/vty/cpu_sched_vty.c code to have user-configured scheduling
For those above, we basically have three possible strategies:
  • at least drop all privileges except those we really ever need in the specific proram (CAP_NET_RAW / CAP_NET_ADMIN / CAP_SYS_NICE). We can first constrain the permitted capabilities using cap_set_flag, then use prctl(PR_SET_KEEPCAPS, 1L) to keep capabilities while changing from root to non-root, and then change the user ID / group ID. https://stackoverflow.com/a/13186076 has a nice example
  • if it is sufficient to perform those privileged operations once on start-up, we could even drop those capabilities after perfoming the operations like creating netdev, binding socket, changing scheduler policy. This would mean that no subsequent changes can be made later on.
Actions #4

Updated by laforge over 3 years ago

  • Assignee deleted (osmith)
  • Priority changed from Low to High
Actions #5

Updated by keith over 3 years ago

  • Related to Bug #4821: Update working dir in systemd unit files added
Actions #6

Updated by laforge about 3 years ago

  • Assignee set to osmith
IMHO, we should start by
  • create an osmocom user during package installation (if it doesn't exist yet)
    • alternatively call it osmo-cni if osmocom is deemed too generic
  • modify the systemd.service files to run the processes as that user
  • modify /etc/osmocom and its contents to be owned by that user
  • modify /var/lib/osmocom (HLR + SMS databases) to be owned by that user

For some programs, this is a no-brainer (e.g. BSC, MSC, SGSN)

For some others (TRX, BTS but possibly also MGW: SCHED_RR; GGSN: tun devices) we should work with capabilities, as described above.

Actions #7

Updated by laforge about 3 years ago

  • Related to Bug #2250: OpenGGSN requires to run as root for no apparent reason added
Actions #8

Updated by osmith about 3 years ago

laforge wrote:

IMHO, we should start by
  • modify /etc/osmocom and its contents to be owned by that user

That's untypical - do we want the programs to be able to change their own configs?

Actions #9

Updated by pespin about 3 years ago

osmith wrote:

laforge wrote:

IMHO, we should start by
  • modify /etc/osmocom and its contents to be owned by that user

That's untypical - do we want the programs to be able to change their own configs?

We should, otherwise the user cannot store back the running-config to the .cfg file through VTY command.

Actions #10

Updated by laforge over 1 year ago

  • Assignee changed from osmith to msuraev

re-assinging to msuraev as this has been without progress for too long.

Actions #11

Updated by msuraev over 1 year ago

osmith wrote in #note-8:

That's untypical - do we want the programs to be able to change their own configs?

Some configs in /etc has non-root group. Assuming we also create osmocom group, we can have /etc/osmocom owned by root:osmocom while /etc/osmocom/osmo.bsc.cfg owned by osmocom:osmocom - that's similar to how transmission-daemon handle its config files.

Actions #12

Updated by msuraev over 1 year ago

  • Related to Bug #5669: Test .deb packages built by our OBS added
Actions #13

Updated by msuraev over 1 year ago

laforge wrote in #note-6:

  • modify /var/lib/osmocom (HLR + SMS databases) to be owned by that user

Once #4821 is resolved, this point will be done automatically: systemd autoadjust the state dir permissions to match unit's User=/Group= settings.

Actions #14

Updated by msuraev over 1 year ago

To make sure no project is left behind let's summarize the current state

Repo Service Status Comment Implementation
osmo-hlr osmo-hlr review https://gerrit.osmocom.org/c/osmo-hlr/+/29311
osmo-ggsn osmo-ggsn WIP kernel GTP untested https://gerrit.osmocom.org/c/osmo-ggsn/+/29412
osmo-msc osmo-msc review https://gerrit.osmocom.org/c/osmo-msc/+/29709
osmo-gbproxy osmo-gbproxy WIP FR HDLC untested https://gerrit.osmocom.org/c/osmo-gbproxy/+/29834
osmo-bsc osmo-bsc review https://gerrit.osmocom.org/c/osmo-bsc/+/29710
osmo-mgw osmo-mgw WIP SCHED_RR untested with TTCN3 https://gerrit.osmocom.org/c/osmo-mgw/+/30094
osmo-sgsn osmo-sgsn review https://gerrit.osmocom.org/c/osmo-sgsn/+/29711
osmo-sgsn osmo-gtphub review https://gerrit.osmocom.org/c/osmo-sgsn/+/29711
osmo-hnbgw osmo-hnbgw review https://gerrit.osmocom.org/c/osmo-hnbgw/+/29712
osmo-hnodeb osmo-hnodeb review https://gerrit.osmocom.org/c/osmo-hnodeb/+/29713
osmo-upf osmo-upf review https://gerrit.osmocom.org/c/osmo-upf/+/29716
osmo-cbc osmo-cbc review https://gerrit.osmocom.org/c/osmo-cbc/+/29717
libosmo-sccp osmo-stp review https://gerrit.osmocom.org/c/libosmo-sccp/+/29722
osmo-bts *-mgr
osmo-bts osmo-bts-* WIP only trx and virtual variants are updated https://gerrit.osmocom.org/c/osmo-bts/+/30133
osmo-pcu osmo-pcu WIP SCHED_RR untested with TTCN3 https://gerrit.osmocom.org/c/osmo-pcu/+/30132
osmo-trx osmo-trx-* WIP SCHED_RR untested with TTCN3 https://gerrit.osmocom.org/c/osmo-trx/+/30131
osmo-pcap osmo-pcap-* review https://gerrit.osmocom.org/c/osmo-pcap/+/29724
osmo-smlc osmo-smlc review https://gerrit.osmocom.org/c/osmo-smlc/+/29720
osmo-sip-connector osmo-sip-connector review https://gerrit.osmocom.org/c/osmo-sip-connector/+/29721
osmo-python-tests osmo-ctrl2cgi WONTFIX See comment in #4821
osmo-python-tests osmo-trap2cgi WONTFIX See comment in #4821

Have I missed anything?

Actions #15

Updated by fixeria over 1 year ago

We may also want to run osmo-pcu with SCHED_RR.

Actions #17

Updated by msuraev over 1 year ago

The example code adding user/group is available in https://gerrit.osmocom.org/c/osmo-hlr/+/29311

The following tests were made:
  • clean install
  • upgrade from previous ("root") version
  • upgrade from previous ("user") version
  • writing config file via telnet
  • package uninstall
  • piuparts:

    sudo piuparts osmo-hlr_1.5.0_amd64.deb libosmo-gsup-client0_1.5.0_amd64.deb libosmo-mslookup0_1.5.0_amd64.deb libosmocore19_1.7.0_amd64.deb libosmogsm18_1.7.0_amd64.deb
    ...
    PASS: Installation, upgrade and purging tests.

In general, possible source of problem is mix-n-match between "root" and "user" packages where "root" package is installed after the "user", overriding permissions and disabling read/write access to config files. I'm not sure if it's worth investing time into dealing with that - seems like coordinating release so root->user transition happens simultaneously is easier.

Actions #18

Updated by msuraev over 1 year ago

laforge wrote in #note-3:

  • osmo-ggsn requires CAP_NET_ADMIN for setting up the gtp0/tun0 devices (unless this is done externally before starting it)

At least for tun0 device we can install corresponding .network file in addition to .service with proper User/Group settings.

Actions #19

Updated by msuraev over 1 year ago

  • % Done changed from 0 to 10
Actions #20

Updated by msuraev over 1 year ago

How should we deal with .spec files? Shall I update those as well?

Creating user during package install is a distro-specific thing. Are there some other distros we care about?

What about OE?

Actions #21

Updated by osmith over 1 year ago

msuraev wrote in #note-20:

How should we deal with .spec files? Shall I update those as well?

Creating user during package install is a distro-specific thing. Are there some other distros we care about?

What about OE?

As I understand, the systemd files get adjusted to expect the user to exist, and these systemd files are used in the rpms and on OE too. So we would need to make sure that the user exists there as well or else the systemd services wouldn't work there anymore.

Actions #22

Updated by msuraev over 1 year ago

  • Related to Bug #5685: Dropping debian 10 (buster) added
Actions #23

Updated by msuraev over 1 year ago

Do we have some kind of hierarchy with regards to realtime scheduling? Like "osmo-pcu should have higher priority than osmo-trx" and such?

Actions #24

Updated by pespin over 1 year ago

msuraev I personally use:
osmo-trx-uhd.cfg: "policy rr 18"
osmo-bts-trx.cfg: "policy rr 1"
osmo-pcu.cfg: "policy rr 1"

Actions #25

Updated by laforge over 1 year ago

On Tue, Sep 20, 2022 at 09:11:35AM +0000, msuraev wrote:

Do we have some kind of hierarchy with regards to realtime scheduling? Like "osmo-pcu should have higher priority than osmo-trx" and such?

no, but I think for CNI it's relatively "obvious" to me:
  • osmo-trx should be higher than anything else
  • osmo-bts-* below osmo-trx
  • osmo-mgw below osmo-bts-*
  • osmo-pcu below osmo-bts-*
  • everything else isn't really timing critical.
Actions #26

Updated by msuraev over 1 year ago

  • Related to Bug #5687: Document and implement realtime scheduling hierarchy added
Actions #27

Updated by pespin over 1 year ago

laforge osmo-pcu now depends on getting FNs on time to calculate when to send stuff regarding scheduling, that's why I use same prio for osmo-bts and osmo-pcu.

Actions #28

Updated by msuraev over 1 year ago

Seems like it's not that obvious so the topic deserve ticket of its own - see #5687.

Actions #29

Updated by msuraev over 1 year ago

Tested osmo-hlr.rpm built via https://obs.osmocom.org/project/show/home:msuraev:rpmtest on OpenSUSE Tumbleweed. User:Group are created as expected, the permissions are properly set during install time. The .rpm support matches that of .deb

Tested with:

zypper ar https://people.osmocom.org/packages/home:/msuraev:/rpmtest/openSUSE_Tumbleweed/ osmo
zypper in osmo-hlr
getent passwd osmocom
getent group osmocom
ls -alh /etc/osmocom
ls -alh /etc/ | grep osmo

Actions #30

Updated by msuraev over 1 year ago

laforge wrote in #note-3:

  • osmo-gbproxy requires CAP_NET_RAW if IPPROTO_GTP sockets are required for FR/GRE/IP

Looking through the code I couldn't find where this is used. It's also unclear why gbproxy would require it but OsmoSGSN and OsmoGGSN wouldn't. Could you please clarify?

Actions #31

Updated by laforge over 1 year ago

msuraev wrote in #note-30:

laforge wrote in #note-3:

  • osmo-gbproxy requires CAP_NET_RAW if IPPROTO_GTP sockets are required for FR/GRE/IP

Looking through the code I couldn't find where this is used. It's also unclear why gbproxy would require it but OsmoSGSN and OsmoGGSN wouldn't. Could you please clarify?

osmo-gbproxy is the only network element that "officially" supports Gb over frame relay over E1. We use it to convert from legacy RAN/BSS Gb/FR/E1 to Gb/UDP/IP, so that the SGSN can use normal Gb/UDP/IP.

Actions #32

Updated by msuraev over 1 year ago

laforge wrote in #note-31:

osmo-gbproxy is the only network element that "officially" supports Gb over frame relay over E1. We use it to convert from legacy RAN/BSS Gb/FR/E1 to Gb/UDP/IP, so that the SGSN can use normal Gb/UDP/IP.

How does that look like to Linux? Some specific network interface?

And how do we test it?

Would be nice to try and ensure it works instead of simply slapping CAN_NET_RAW and hoping it's enough.

Actions #33

Updated by laforge over 1 year ago

hdlcX net device. There's a wiki page documenting this, including how to set up a virtual loop back device like we use in Jenkins testing. libosmogb simply uses AF_PACKET sockets, so CAP_NET_RAW should do the trick.

Actions #34

Updated by msuraev over 1 year ago

  • Related to Feature #5722: Migrate jenkins build slaves from docker to podman added
Actions #35

Updated by msuraev over 1 year ago

  • % Done changed from 10 to 20

Testing OsmoGGSN in user mode (non GTP mode) https://gerrit.osmocom.org/c/osmo-ggsn/+/29412/ revealed following:

pass GGSN_Tests.TC_pdp4_act_deact
pass GGSN_Tests.TC_pdp4_act_deact_ipcp
pass GGSN_Tests.TC_pdp4_act_deact_ipcp_pap_broken
pass GGSN_Tests.TC_pdp4_act_deact_pcodns
pass GGSN_Tests.TC_pdp4_act_deact_gtpu_access
pass->FAIL GGSN_Tests.TC_pdp4_clients_interact_with_txseq
pass->FAIL GGSN_Tests.TC_pdp4_clients_interact_without_txseq
pass GGSN_Tests.TC_pdp4_act_deact_with_single_dns
pass GGSN_Tests.TC_pdp4_act_deact_with_separate_dns
pass GGSN_Tests.TC_pdp6_act_deact
pass GGSN_Tests.TC_pdp6_act_deact_pcodns
pass GGSN_Tests.TC_pdp6_act_deact_icmp6
pass->FAIL GGSN_Tests.TC_pdp6_act_deact_gtpu_access
pass GGSN_Tests.TC_pdp6_clients_interact
pass GGSN_Tests.TC_pdp46_act_deact
pass GGSN_Tests.TC_pdp46_act_deact_ipcp
pass GGSN_Tests.TC_pdp46_act_deact_icmp6
pass GGSN_Tests.TC_pdp46_act_deact_pcodns4
pass GGSN_Tests.TC_pdp46_act_deact_pcodns6
pass GGSN_Tests.TC_pdp46_act_deact_gtpu_access
pass GGSN_Tests.TC_pdp46_clients_interact
pass GGSN_Tests.TC_pdp46_act_deact_apn4
pass GGSN_Tests.TC_echo_req_resp
pass GGSN_Tests.TC_pdp_act2_recovery
pass GGSN_Tests.TC_act_deact_retrans_duplicate
pass GGSN_Tests.TC_pdp_act_restart_ctr_echo
NEW: PASS GGSN_Tests.TC_pdp4_act_deact_gtpu_access_wrong_saddr
NEW: PASS GGSN_Tests.TC_pdp4_act_deact_gtpu_access_ipv6_apn4
NEW: PASS GGSN_Tests.TC_pdp4_act_update_teic
NEW: PASS GGSN_Tests.TC_pdp4_act_update_teid
NEW: PASS GGSN_Tests.TC_pdp6_act_deact_gtpu_access_wrong_ll_saddr
NEW: PASS GGSN_Tests.TC_pdp6_act_deact_gtpu_access_wrong_global_saddr
NEW: PASS GGSN_Tests.TC_pdp6_act_deact_gtpu_access_ipv4_apn6
NEW: PASS GGSN_Tests.TC_pdp46_act_deact_gtpu_access_wrong_saddr_ipv4
NEW: PASS GGSN_Tests.TC_pdp46_act_deact_gtpu_access_wrong_ll_saddr_ipv6
NEW: PASS GGSN_Tests.TC_pdp46_act_deact_gtpu_access_wrong_global_saddr_ipv6
NEW: PASS GGSN_Tests.TC_echo_req_resp_gtpu
NEW: FAIL GGSN_Tests.TC_lots_of_concurrent_pdp_ctx
NEW: FAIL GGSN_Tests.TC_addr_pool_exhaustion

Summary:
pass->FAIL: 3
NEW: FAIL: 2
pass: 23
NEW: PASS: 11

Actions #36

Updated by msuraev over 1 year ago

  • Status changed from New to In Progress
Actions #37

Updated by msuraev over 1 year ago

  • % Done changed from 20 to 40

Note: realtime scheduling can be verified for a service with tuna --show_threads | grep RR

Actions #38

Updated by msuraev over 1 year ago

  • Status changed from In Progress to Stalled

Blocked by subtask.

Actions #39

Updated by laforge 5 months ago

  • Assignee changed from msuraev to osmith
  • Priority changed from High to Normal
Actions #40

Updated by laforge 4 months ago

  • Priority changed from Normal to High
Actions

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)