Cellular Network Infrastructure

The way we run the ttcn3 testsuites with docker-playground.git currently has several downsides. I'm listing them first and then show a proof of concept of how we could do it instead further below.

Downsides of current architecture¶

Overview¶

• the testsuite and SUT get built as part of the Dockerfiles:
  • we can't do incremental builds with previous build artifacts (bad for developers)
  • we can't use ccache (bad for developers and jenkins)
  • we have lots of Dockerfiles (bad for maintenance, and for adding new testsuites)

• configs are duplicated, e.g.:
  • osmo-ttcn3-hacks: mgw/osmo-mgw.cfg
  • docker-playground: ttcn3-mgw-test/osmo-mgw.cfg, osmo-mgw-{master,latest}/osmo-mgw.cfg

• master containers (e.g. osmo-mgw-master) mix master and nightly:
  • build programs from current master (or commit passed as argument), but install Osmocom libraries from osmocom:nightly binary package repositories (example)
  • this is a compromise for allowing to use a different source for the program and rebuild it, but it falls short:
    • you cannot change the source for the libraries (bad for developers)
    • you must spend extra time to build these programs instead of using binary packages (bad for jenkins)
    • this is not intuitive, can lead to unexpected errors when e.g. merging a libosmocore patch, then merging a program patch that depends on it and then building the container where program from git master gets built against libosmocore from nightly where the patch is missing and the build fails

For jenkins¶

There is lots of overhead with building so many containers and building from source inside of them. We need to rebuild them very often, whenever something in the git repositories or binary package repositories changes. It would be much faster if there was only one container which we start and then install packages from osmocom:{nightly,nightly:asan,osmocom:latest} right before they get used. And then build only the testsuite from source and use ccache while doing that.

For developers¶

• when running everything without docker:
  • fast but cannot use the same configurations as jenkins
  • configs get out of sync (leading to tests passing locally or in jenkins but not the other)
  • you need to make sure to start all test components properly yourself (which is some effort for e.g. osmo-bts)

• when running everything with docker:
  • best compatibility with jenkins
  • building all the containers takes a long time and extra disk space
  • without further hacks, cannot build libraries from source too
  • without further hacks, cannot use previous build artifacts or ccache
  • without further hacks, you need to push your changes to a remote branch first, then let the Dockerfile build that

New architecture¶

I've written a proof of concept script that could be used by both developers and jenkins. It is currently in osmo-ttcn3-hacks.git, osmith/wip-testenv branch.

The configs from osmo-ttcn3-hacks.git are used (osmo-mgw.cfg, MGCP_Test.cfg, etc.).

A developer would run it like this to let it run the components of the testsuite, and the testsuite itself, without containers. Before the testsuite starts, all components get built from source by using osmo-dev. This means, if there were no changes to osmo-mgw or libosmocore, it skips building anything, and if there were changes, it uses previous build artifacts.

$ ./testenv.py run mgw

Jenkins would run it like this:

$ ./testenv.py run mgw --podman -b osmocom:nightly

Instead of building from source, the osmocom:nightly repository gets configured and osmo-mgw gets installed from there.

It uses podman instead of docker, so we can run it rootless and avoid permission errors (no leftover files belonging to root that need to be cleaned up). One container is built if it does not exist. This container is not specific to the testsuite or SUT, or repository.

This uses the same configs from osmo-ttcn3-hacks.git.

Developers could also run commands with --podman -b, for example to quickly verify how a osmo-ttcn3-hacks change behaves with binaries from osmocom:latest. An apt cache gets mounted inside the container, so if running twice, previously downloaded deb files can be used again without fetching them from the server.

Instead of writing several Dockerfiles, each testsuite only needs a testenv.cfg:

[mgw]
program=osmo-mgw
packages=osmo-mgw  # debian packages to install
config=osmo-mgw.cfg

[testsuite]
program=MGCP_Test
config=MGCP_Test.cfg

osmo-bts has multiple test configurations (generic, oml, hopping). There we have e.g. testenv_oml.cfg:

[bts]
prepare=oml/prepare.sh  # existing script that runs osmo-config-merge
program=osmo-bts-trx
# libosmocore-utils: osmo-config-merge used in prepare.sh
packages=libosmocore-utils osmo-bts-trx
config=oml/osmo-bts.cfg

[fake-trx]
program=run_fake_trx.sh  # existing script that runs fake-trx

[trxcon]
program=trxcon

[testsuite]
program=BTS_Tests
config=oml/BTS_Tests.cfg

The testenv script runs all components from this file.

Extra features¶

While writing this proof of concept, I realized that without much effort we can have some more nice things:

• Added a parameter to run a specific test (testenv.py run mgw -t TC_crcx)
• Added a parameter to run a specific test config for osmo-bts (generic|oml|hopping)
• Display SUT output while the test is running
• Check if each component started up properly before starting the testsuite to catch config errors early (start it, sleep 200ms, does it still run?)
• Let ^C reliably kill all started processes

Status¶

• Proof of concept works (run without podman and build from source, run with podman and use binary repos)
• Wrote testenv.cfg files and tested with osmo-mgw, osmo-bts (generic, oml) (tests seem to run, but didn't run all tests yet)
• Tested that asan works.
• Currently everything runs in 1 container. For some tests we need to run components in multiple containers IIRC, this needs to be added to testenv.cfg. (I would still only build one container, but run it multiple times and install different programs into the instances.)
• Logic for extra "latest" configs need to be implemented
• Giving extra permissions (CAP_NET_ADMIN etc.) also needs to be added to testenv.cfg
• testenv.cfg files for other testsuites need to be written and tested