Project

General

Profile

Titan TTCN3 Testsuites » History » Version 39

osmith, 06/05/2019 08:54 AM

1 1 laforge
h1. Osmocom Titan TTCN3 Testsuites
2 36 fixeria
3
{{>toc}}
4 1 laforge
5 14 osmith
In 2017, Osmocom started to create extensive tests suites for the Cellular Network Infrastructure elements.  Those suites are written in the [[TTCN-3]] programming language and use the compiler and executor of the [[Eclipse_TITAN]] project. There are special cases, but in general, the idea is to test one component (e.g. @osmo-mgw@) independently.
6 2 laforge
7
h2. Source Code / Patches
8
9
The source code of our test suites lives in the @osmo-ttcn3-hacks@ repository
10
* cgit web interface: http://git.osmocom.org/osmo-ttcn3-hacks/
11
* clone: @git clone git://git.osmocom.org/osmo-ttcn3-hacks@
12
* patch review: https://gerrit.osmocom.org/#/q/project:osmo-ttcn3-hacks
13
14
We're following the [[Gerrit]] process for patch review.
15
16 32 laforge
h2. Proprietary APER<->BER transcoding library for Iu tests
17
18
As TITAN can only generate and parse ASN.1 BER encoding, but the Iu-CS, Iu-PS and Iuh interfaces are using APER encoding, sysmocom is providing a proprietary transcoding library called libfftranscode.  Debian9 amd64 packages are available from https://ftp.osmocom.org/binaries/libfftranscode/
19
20
We don't like to depend on proprietary software, but given that there is no FOSS ASN.1 compiler that can parse all ASN.1 constructs of the RANAP/RUA/HNBAP specs *and* parse + generate both BER and APER, this is the lesser evil.  Note that the dependency to this proprietary library is only required for the TTCN3 test suite and not a runtime dependency of any of the Osmocom programs itself.
21
22 2 laforge
h2. Test Suites
23
24
|_.Implementation under Test|_.Testsuite|_.Jenkins|
25
|bsc-nat|http://git.osmocom.org/osmo-ttcn3-hacks/tree/bsc-nat|-|
26
|[[OsmoBSC:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/bsc|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bsc-test/|
27
|[[OsmoBTS:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/bts|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bts-test/|
28
|[[OpenGGSN:OsmoGGSN]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/ggsn_tests|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-ggsn-test/|
29
|[[OsmoHLR:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/hlr|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-hlr-test/|
30
|[[OsmoMGW:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/mgw|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-mgw-test/|
31
|[[OsmoMSC:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/msc|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-msc-test/|
32
|[[OsmoPCU:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/pcu|-|
33
|[[OsmoSGSN:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/sgsn|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-sgsn-test/|
34
|[[osmo-sip-connector:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/sip|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-sip-test/|
35
36
h2. Running a testsuite
37
38
You have multiple options on how to execute a test suite.
39
40
h3. Running test suite in the dockerized environment
41 1 laforge
42 4 osmith
This is the much easier way to run the test suite.  No manual configuration of Osmocom programs or the test suite are required.  Both the Osmocom program (Implementation Under Test) as well as the test suite are packaged as docker containers, which are then executed next to each other using a docker network with the "right" addresses between IUT and testsuite.
43 2 laforge
44
Also, using this setup you will be guaranteed to run in the exact same environment as the automatically-executed tests on jenkins.osmocom.org, i.e. your results should be exactly identical, without any differences introduced by your runtime environment, whether specific library versions or intentional or inadvertent configuration differences.
45
46 10 osmith
h4. Installing docker
47 1 laforge
48 10 osmith
Make sure you have "*docker-ce*":https://docs.docker.com/install/#supported-platforms installed (not to be confused with the possibly outdated version that may be shipped in your distribution's repositories). Check that the version string contains <code>-ce</code> as follows:
49
50
<pre>
51
$ docker -v
52
Docker version 18.06.1-ce, build e68fc7a
53
</pre>
54
55
56 11 osmith
Next, add yourself to the <code>docker</code> group, log off and on again, and verify that you can use docker with your user:
57 10 osmith
58
<pre>
59
$ docker info
60
</pre>
61
62
h4. Cloning docker-playground
63
64 12 osmith
Osmocom related containers are stored in the "@docker-playground@":https://git.osmocom.org/docker-playground/ git repository.
65 5 osmith
66
<pre>
67
$ git clone git://git.osmocom.org/docker-playground
68
$ cd docker-playground
69 1 laforge
</pre>
70 6 osmith
71 1 laforge
h4. Running a testsuite
72
73 38 osmith
All testsuite directories start with @ttcn3@ or @nplab@. Run the following to build/update all required containers and start a specific testsuite:
74 24 osmith
75 1 laforge
<pre>
76 24 osmith
$ cd ttcn3-mgw-test
77
$ ./jenkins.sh
78 1 laforge
</pre>
79
80 24 osmith
Environment variables:
81
* @IMAGE_SUFFIX@: the version of the Osmocom stack to run the testsuite against. Default is `master`, set this to `latest` to test the last stable releases.
82 29 osmith
* @OSMO_TTCN3_BRANCH@: "osmo-ttcn3-hacks.git":https://git.osmocom.org/osmo-ttcn3-hacks/ branch, which will be used when building a `ttcn3-*` docker image. Defaults to `master`.
83 33 osmith
* @OSMO_BSC_BRANCH@, @OSMO_MSC_BRANCH@, ...: branch of the appropriate Osmocom project. Defaults to `master`.
84 24 osmith
* @NO_DOCKER_IMAGE_BUILD@: when set to @1@, it won't try to update the containers (see "README.md":https://git.osmocom.org/docker-playground/tree/README.md)
85 33 osmith
86 24 osmith
87 6 osmith
The @jenkins.sh@ script will make sure to
88 24 osmith
* create the needed docker network(s)
89 6 osmith
* run all the required containers (IUT, helpers, testsuite)
90
* shut them down after test suite execution
91
* collect the log files after execution, @/tmp/logs@ will contain them
92
93 2 laforge
h4. Running only a sub-set of the test cases
94
95 20 osmith
See running the testsuite outside of Docker for reference (below).
96
97 28 osmith
h3. Running it natively on your machine
98 2 laforge
99
This is the most complicated bit to set-up, as you will have to run the respective Osmocom Program (Implementation Under Test) in the right configuration with all the IP addresses, port numbers, config file, etc. exactly like it's expected by the test suite.
100 1 laforge
101 14 osmith
You will need a deeper understanding about how the test suite works, and what its requirement are. Oftentimes looking at how it's done in the Docker scripts is helpful.
102
103
h4. Preparation
104
105 34 Hoernchen
As stated [[Titan_TTCN3_Testsuites##Proprietary-APERBER-transcoding-library-for-Iu-tests|above]] certain parts of the testsuite require libfftranscode! If the build fails this might be the issue.
106 35 laforge
107
While you can try to use any version of Eclipse TITAN, for the best possible experience it's a good idea to use the same version as Osmocom uses in the dockerized environment.  As can be seen in the "Dockerfile":http://git.osmocom.org/docker-playground/tree/debian-stretch-titan/Dockerfile, this (currently) points to the eclipse-titan package from the [[Latest_Builds]] on Debian 9.0":.
108
109 14 osmith
<pre>
110
$ apt install eclipse-titan
111
$ git clone git://osmocom.org/osmo-ttcn3-hacks
112
$ cd osmo-ttc3-hacks/deps
113
$ make
114
</pre>
115
116
h4. Compile a testsuite
117
118
Let's compile the @mgw@ testsuite for example:
119
120
<pre>
121
$ cd mgw
122
$ ./gen_links.sh
123 26 msuraev
$ ./regen_makefile.sh
124 14 osmith
</pre>
125 1 laforge
126 23 osmith
The next command will transform the TTCN3 test data into C++ code. Do not use @-j@ here, that won't work (and it is pretty fast anyway).
127 14 osmith
<pre>
128
$ make compile
129
</pre>
130
131 27 msuraev
Note: if you observe errors during @make compile@, try running it from top-level as @make mgw@ which should trigger dependency update.
132
133 14 osmith
Finally compile the testsuite:
134
<pre>
135
$ make -j5
136
</pre>
137 27 msuraev
138 14 osmith
h4. Run a testsuite
139
140 37 osmith
Start the component that is about to get tested with the config that the testsuite expects. Again, with @mgw@ as example. A suitable config file should be in the same directory of osmo-ttcn3-hacks.git (otherwise one can use "the one from Jenkins":https://git.osmocom.org/docker-playground/tree/ttcn3-mgw-test/osmo-mgw.cfg and replace all IPs with @127.0.0.1@).
141 14 osmith
<pre>
142 21 osmith
$ osmo-mgw -c osmo-mgw.cfg
143 14 osmith
</pre>
144
145
Then run the testsuite:
146
<pre>
147
$ cd osmo-ttcn3-hacks/mgw
148
$ ../start-testsuite.sh ./MGCP_test MGCP_Test.cfg
149
</pre>
150
151
Afterwards you can merge and format the logs as follows:
152
<pre>
153
$ ttcn3_logmerge MGCP_test*.log > ./merged.log
154 1 laforge
$ ttcn3_logformat ./merged.log > ./result.log
155 30 msuraev
</pre>
156
or
157
<pre>
158 31 msuraev
$ ../log_merge.sh MGCP_test
159 1 laforge
</pre>
160 37 osmith
161
Depending on the component that you want to test, you may need to run multiple Osmocom programs. For OsmoMSC, you would run osmo-msc and osmo-stp for example. See the containers that "jenkins.sh":https://git.osmocom.org/docker-playground/tree/ttcn3-msc-test/jenkins.sh is running for reference.
162 2 laforge
163
h4. Running only a sub-set of the test cases
164
165 38 osmith
Running whole testsuites may take a long time. If you only want to run one specific test, or a subset of tests, open up the @_*Test.cfg@ file in the directory of the testsuite and edit the @[EXECUTE]@ section. For example, in "@MGCP_Test.cfg@":https://git.osmocom.org/osmo-ttcn3-hacks/tree/mgw/MGCP_Test.cfg from the @mgw@ directory it looks like the following: 
166 16 osmith
167
<pre>
168 1 laforge
...
169 17 osmith
[EXECUTE]
170
MGCP_Test.control
171
#MGCP_Test.TC_selftest
172
#MGCP_Test.TC_crcx
173
#MGCP_Test.TC_crcx_noprefix
174
...
175
</pre>
176 1 laforge
177 17 osmith
The only line that is *not* commented out is @MGCP_Test.control@. This @*.control@ line will cause all tests to be executed. So comment this line out, and then activate one of the lines of which you would like to execute the tests (see the example below). Then execute the testsuite as usually, and it will skip all commented out tests.
178
179
<pre>
180
...
181
[EXECUTE]
182
#MGCP_Test.control
183
#MGCP_Test.TC_selftest
184
MGCP_Test.TC_crcx
185
MGCP_Test.TC_crcx_noprefix
186
...
187 16 osmith
</pre>
188 2 laforge
189 15 osmith
h2. Reading the logs
190
191
* @xfail@ means, that a test is known to fail, @FAIL@ are unexpected test failures
192
* search for @setverdict(@ to find the results of single tests. @-> fail@ should be the location where a test is failing.
193
194 2 laforge
h2. Further information
195
196 22 osmith
* April 2018 talk by Harald on "State of the Osmocom TTCN-3 Test Suites":https://media.ccc.de/v/UGHHW3
197 39 osmith
* "Patch for colormake to colorize the ttcn3 compiler output":https://github.com/osmith42/Colormake/tree/osmith/ttcn3
Add picture from clipboard (Maximum size: 48.8 MB)