Titan TTCN3 Testsuites » History » Revision 11
Revision 10 (osmith, 09/05/2018 01:06 PM) → Revision 11/52 (osmith, 09/05/2018 01:06 PM)
{{>toc}}
h1. Osmocom Titan TTCN3 Testsuites
In 2017, Osmocom started to create extensive tests suites for the Cellular Network Infrastructure elements. Those suites are written in the [[TTCN-3]] programming language and use the compiler + executor of the [[Eclipse_TITAN]] project.
h2. Source Code / Patches
The source code of our test suites lives in the @osmo-ttcn3-hacks@ repository
* cgit web interface: http://git.osmocom.org/osmo-ttcn3-hacks/
* clone: @git clone git://git.osmocom.org/osmo-ttcn3-hacks@
* patch review: https://gerrit.osmocom.org/#/q/project:osmo-ttcn3-hacks
We're following the [[Gerrit]] process for patch review.
h2. Test Suites
|_.Implementation under Test|_.Testsuite|_.Jenkins|
|bsc-nat|http://git.osmocom.org/osmo-ttcn3-hacks/tree/bsc-nat|-|
|[[OsmoBSC:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/bsc|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bsc-test/|
|[[OsmoBTS:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/bts|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bts-test/|
|[[OpenGGSN:OsmoGGSN]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/ggsn_tests|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-ggsn-test/|
|[[OsmoHLR:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/hlr|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-hlr-test/|
|[[OsmoMGW:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/mgw|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-mgw-test/|
|[[OsmoMSC:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/msc|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-msc-test/|
|[[OsmoPCU:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/pcu|-|
|[[OsmoSGSN:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/sgsn|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-sgsn-test/|
|[[osmo-sip-connector:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/sip|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-sip-test/|
h2. Running a testsuite
You have multiple options on how to execute a test suite.
h3. Running test suite in the dockerized environment
This is the much easier way to run the test suite. No manual configuration of Osmocom programs or the test suite are required. Both the Osmocom program (Implementation Under Test) as well as the test suite are packaged as docker containers, which are then executed next to each other using a docker network with the "right" addresses between IUT and testsuite.
Also, using this setup you will be guaranteed to run in the exact same environment as the automatically-executed tests on jenkins.osmocom.org, i.e. your results should be exactly identical, without any differences introduced by your runtime environment, whether specific library versions or intentional or inadvertent configuration differences.
h4. Installing docker
Make sure you have "*docker-ce*":https://docs.docker.com/install/#supported-platforms installed (not to be confused with the possibly outdated version that may be shipped in your distribution's repositories). Check that the version string contains <code>-ce</code> as follows:
<pre>
$ docker -v
Docker version 18.06.1-ce, build e68fc7a
</pre>
Next, add yourself to the <code>docker</code> group, log off and on again, and verify that you can use docker with from your user:
<pre>
$ docker info
</pre>
h4. Cloning docker-playground
First install docker on your system and add yourself to the docker group. Then clone and build the containers from the "@docker-playground@ git repository":https://git.osmocom.org/docker-playground/:
<pre>
$ git clone git://git.osmocom.org/docker-playground
$ cd docker-playground
</pre>
In theory you could run @make@ now to build all containers. However, not all of them are maintained, so this will most likely fail. Instead take a look at the Jenkins script of a specific project to figure out its dependencies and then build these containers.
h4. Running a testsuite
Let's assume you'd want to run the testsuite for [[OsmoBSC:]]. You would need to
# look at its "jenkins.sh":http://git.osmocom.org/docker-playground/tree/ttcn3-bsc-test/jenkins.sh to understand which containers are required.
# build those respective containers - at time of writing, this would be:
<pre>
$ make osmo-stp-master osmo-bsc-master osmo-bts-master ttcn3-bsc-test
</pre>
# run the @jenkins.sh@ script of the @ttcn3-bsc-test@ suite:
<pre>
$ cd ttcn3-bsc-test && ./jenkins.sh
</pre>
The @jenkins.sh@ script will make sure to
* create the needed docker network[s]
* run all the required containers (IUT, helpers, testsuite)
* shut them down after test suite execution
* collect the log files after execution, @/tmp/logs@ will contain them
h4. Tagging images
If the build fails with a message like the following:
<pre>
repository docker.io/laforge/debian-jessie-build not found: does not exist or no pull access
</pre>
Then apply "this patch":https://gerrit.osmocom.org/#/c/docker-playground/+/10773/ and try again (sent to review, will likely be merged soon).
h4. Running only a sub-set of the test cases
FIXME
h3. Running it native on your machine
This is the most complicated bit to set-up, as you will have to run the respective Osmocom Program (Implementation Under Test) in the right configuration with all the IP addresses, port numbers, config file, etc. exactly like it's expected by the test suite.
You will need a deeper understanding about how the test suite works, and what its requirement are.
h4. Running only a sub-set of the test cases
FIXME
h2. Further information
* April 2018 talk by Harald on "State of he Osmocom TTCN-3 Test Suites":https://media.ccc.de/v/UGHHW3
