Project

General

Profile

Titan TTCN3 Testsuites » History » Version 41

daniel, 10/17/2019 02:23 PM
Add jenkins ttcn3 test links for osmo-bscnat, osmo-pcu

1 1 laforge
h1. Osmocom Titan TTCN3 Testsuites
2 36 fixeria
3
{{>toc}}
4 1 laforge
5 14 osmith
In 2017, Osmocom started to create extensive tests suites for the Cellular Network Infrastructure elements.  Those suites are written in the [[TTCN-3]] programming language and use the compiler and executor of the [[Eclipse_TITAN]] project. There are special cases, but in general, the idea is to test one component (e.g. @osmo-mgw@) independently.
6 2 laforge
7 40 Hoernchen
h2. Development
8
9
See [[Titan_TTCN3_Notes]]
10
11 2 laforge
h2. Source Code / Patches
12
13
The source code of our test suites lives in the @osmo-ttcn3-hacks@ repository
14
* cgit web interface: http://git.osmocom.org/osmo-ttcn3-hacks/
15
* clone: @git clone git://git.osmocom.org/osmo-ttcn3-hacks@
16
* patch review: https://gerrit.osmocom.org/#/q/project:osmo-ttcn3-hacks
17
18
We're following the [[Gerrit]] process for patch review.
19
20 32 laforge
h2. Proprietary APER<->BER transcoding library for Iu tests
21
22
As TITAN can only generate and parse ASN.1 BER encoding, but the Iu-CS, Iu-PS and Iuh interfaces are using APER encoding, sysmocom is providing a proprietary transcoding library called libfftranscode.  Debian9 amd64 packages are available from https://ftp.osmocom.org/binaries/libfftranscode/
23
24
We don't like to depend on proprietary software, but given that there is no FOSS ASN.1 compiler that can parse all ASN.1 constructs of the RANAP/RUA/HNBAP specs *and* parse + generate both BER and APER, this is the lesser evil.  Note that the dependency to this proprietary library is only required for the TTCN3 test suite and not a runtime dependency of any of the Osmocom programs itself.
25
26 2 laforge
h2. Test Suites
27
28
|_.Implementation under Test|_.Testsuite|_.Jenkins|
29 41 daniel
|[[OsmoBSCNAT:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/bsc-nat|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bscnat-test/|
30 2 laforge
|[[OsmoBSC:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/bsc|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bsc-test/|
31
|[[OsmoBTS:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/bts|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bts-test/|
32
|[[OpenGGSN:OsmoGGSN]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/ggsn_tests|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-ggsn-test/|
33
|[[OsmoHLR:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/hlr|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-hlr-test/|
34
|[[OsmoMGW:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/mgw|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-mgw-test/|
35
|[[OsmoMSC:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/msc|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-msc-test/|
36 41 daniel
|[[OsmoPCU:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/pcu|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-pcu-test/, https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-pcu-test-sns/|
37 2 laforge
|[[OsmoSGSN:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/sgsn|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-sgsn-test/|
38
|[[osmo-sip-connector:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/sip|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-sip-test/|
39
40
h2. Running a testsuite
41
42
You have multiple options on how to execute a test suite.
43
44
h3. Running test suite in the dockerized environment
45 1 laforge
46 4 osmith
This is the much easier way to run the test suite.  No manual configuration of Osmocom programs or the test suite are required.  Both the Osmocom program (Implementation Under Test) as well as the test suite are packaged as docker containers, which are then executed next to each other using a docker network with the "right" addresses between IUT and testsuite.
47 2 laforge
48
Also, using this setup you will be guaranteed to run in the exact same environment as the automatically-executed tests on jenkins.osmocom.org, i.e. your results should be exactly identical, without any differences introduced by your runtime environment, whether specific library versions or intentional or inadvertent configuration differences.
49
50 10 osmith
h4. Installing docker
51 1 laforge
52 10 osmith
Make sure you have "*docker-ce*":https://docs.docker.com/install/#supported-platforms installed (not to be confused with the possibly outdated version that may be shipped in your distribution's repositories). Check that the version string contains <code>-ce</code> as follows:
53
54
<pre>
55
$ docker -v
56
Docker version 18.06.1-ce, build e68fc7a
57
</pre>
58
59
60 11 osmith
Next, add yourself to the <code>docker</code> group, log off and on again, and verify that you can use docker with your user:
61 10 osmith
62
<pre>
63
$ docker info
64
</pre>
65
66
h4. Cloning docker-playground
67
68 12 osmith
Osmocom related containers are stored in the "@docker-playground@":https://git.osmocom.org/docker-playground/ git repository.
69 5 osmith
70
<pre>
71
$ git clone git://git.osmocom.org/docker-playground
72
$ cd docker-playground
73 1 laforge
</pre>
74 6 osmith
75 1 laforge
h4. Running a testsuite
76
77 38 osmith
All testsuite directories start with @ttcn3@ or @nplab@. Run the following to build/update all required containers and start a specific testsuite:
78 24 osmith
79 1 laforge
<pre>
80 24 osmith
$ cd ttcn3-mgw-test
81
$ ./jenkins.sh
82 1 laforge
</pre>
83
84 24 osmith
Environment variables:
85
* @IMAGE_SUFFIX@: the version of the Osmocom stack to run the testsuite against. Default is `master`, set this to `latest` to test the last stable releases.
86 29 osmith
* @OSMO_TTCN3_BRANCH@: "osmo-ttcn3-hacks.git":https://git.osmocom.org/osmo-ttcn3-hacks/ branch, which will be used when building a `ttcn3-*` docker image. Defaults to `master`.
87 33 osmith
* @OSMO_BSC_BRANCH@, @OSMO_MSC_BRANCH@, ...: branch of the appropriate Osmocom project. Defaults to `master`.
88 24 osmith
* @NO_DOCKER_IMAGE_BUILD@: when set to @1@, it won't try to update the containers (see "README.md":https://git.osmocom.org/docker-playground/tree/README.md)
89 33 osmith
90 24 osmith
91 6 osmith
The @jenkins.sh@ script will make sure to
92 24 osmith
* create the needed docker network(s)
93 6 osmith
* run all the required containers (IUT, helpers, testsuite)
94
* shut them down after test suite execution
95
* collect the log files after execution, @/tmp/logs@ will contain them
96
97 2 laforge
h4. Running only a sub-set of the test cases
98
99 20 osmith
See running the testsuite outside of Docker for reference (below).
100
101 28 osmith
h3. Running it natively on your machine
102 2 laforge
103
This is the most complicated bit to set-up, as you will have to run the respective Osmocom Program (Implementation Under Test) in the right configuration with all the IP addresses, port numbers, config file, etc. exactly like it's expected by the test suite.
104 1 laforge
105 14 osmith
You will need a deeper understanding about how the test suite works, and what its requirement are. Oftentimes looking at how it's done in the Docker scripts is helpful.
106
107
h4. Preparation
108
109 34 Hoernchen
As stated [[Titan_TTCN3_Testsuites##Proprietary-APERBER-transcoding-library-for-Iu-tests|above]] certain parts of the testsuite require libfftranscode! If the build fails this might be the issue.
110 35 laforge
111
While you can try to use any version of Eclipse TITAN, for the best possible experience it's a good idea to use the same version as Osmocom uses in the dockerized environment.  As can be seen in the "Dockerfile":http://git.osmocom.org/docker-playground/tree/debian-stretch-titan/Dockerfile, this (currently) points to the eclipse-titan package from the [[Latest_Builds]] on Debian 9.0":.
112
113 14 osmith
<pre>
114
$ apt install eclipse-titan
115
$ git clone git://osmocom.org/osmo-ttcn3-hacks
116
$ cd osmo-ttc3-hacks/deps
117
$ make
118
</pre>
119
120
h4. Compile a testsuite
121
122
Let's compile the @mgw@ testsuite for example:
123
124
<pre>
125
$ cd mgw
126
$ ./gen_links.sh
127 26 msuraev
$ ./regen_makefile.sh
128 14 osmith
</pre>
129 1 laforge
130 23 osmith
The next command will transform the TTCN3 test data into C++ code. Do not use @-j@ here, that won't work (and it is pretty fast anyway).
131 14 osmith
<pre>
132
$ make compile
133
</pre>
134
135 27 msuraev
Note: if you observe errors during @make compile@, try running it from top-level as @make mgw@ which should trigger dependency update.
136
137 14 osmith
Finally compile the testsuite:
138
<pre>
139
$ make -j5
140
</pre>
141 27 msuraev
142 14 osmith
h4. Run a testsuite
143
144 37 osmith
Start the component that is about to get tested with the config that the testsuite expects. Again, with @mgw@ as example. A suitable config file should be in the same directory of osmo-ttcn3-hacks.git (otherwise one can use "the one from Jenkins":https://git.osmocom.org/docker-playground/tree/ttcn3-mgw-test/osmo-mgw.cfg and replace all IPs with @127.0.0.1@).
145 14 osmith
<pre>
146 21 osmith
$ osmo-mgw -c osmo-mgw.cfg
147 14 osmith
</pre>
148
149
Then run the testsuite:
150
<pre>
151
$ cd osmo-ttcn3-hacks/mgw
152
$ ../start-testsuite.sh ./MGCP_test MGCP_Test.cfg
153
</pre>
154
155
Afterwards you can merge and format the logs as follows:
156
<pre>
157
$ ttcn3_logmerge MGCP_test*.log > ./merged.log
158 1 laforge
$ ttcn3_logformat ./merged.log > ./result.log
159 30 msuraev
</pre>
160
or
161
<pre>
162 31 msuraev
$ ../log_merge.sh MGCP_test
163 1 laforge
</pre>
164 37 osmith
165
Depending on the component that you want to test, you may need to run multiple Osmocom programs. For OsmoMSC, you would run osmo-msc and osmo-stp for example. See the containers that "jenkins.sh":https://git.osmocom.org/docker-playground/tree/ttcn3-msc-test/jenkins.sh is running for reference.
166 2 laforge
167
h4. Running only a sub-set of the test cases
168
169 38 osmith
Running whole testsuites may take a long time. If you only want to run one specific test, or a subset of tests, open up the @_*Test.cfg@ file in the directory of the testsuite and edit the @[EXECUTE]@ section. For example, in "@MGCP_Test.cfg@":https://git.osmocom.org/osmo-ttcn3-hacks/tree/mgw/MGCP_Test.cfg from the @mgw@ directory it looks like the following: 
170 16 osmith
171
<pre>
172 1 laforge
...
173 17 osmith
[EXECUTE]
174
MGCP_Test.control
175
#MGCP_Test.TC_selftest
176
#MGCP_Test.TC_crcx
177
#MGCP_Test.TC_crcx_noprefix
178
...
179
</pre>
180 1 laforge
181 17 osmith
The only line that is *not* commented out is @MGCP_Test.control@. This @*.control@ line will cause all tests to be executed. So comment this line out, and then activate one of the lines of which you would like to execute the tests (see the example below). Then execute the testsuite as usually, and it will skip all commented out tests.
182
183
<pre>
184
...
185
[EXECUTE]
186
#MGCP_Test.control
187
#MGCP_Test.TC_selftest
188
MGCP_Test.TC_crcx
189
MGCP_Test.TC_crcx_noprefix
190
...
191 16 osmith
</pre>
192 2 laforge
193 15 osmith
h2. Reading the logs
194
195
* @xfail@ means, that a test is known to fail, @FAIL@ are unexpected test failures
196
* search for @setverdict(@ to find the results of single tests. @-> fail@ should be the location where a test is failing.
197
198 2 laforge
h2. Further information
199
200 22 osmith
* April 2018 talk by Harald on "State of the Osmocom TTCN-3 Test Suites":https://media.ccc.de/v/UGHHW3
201 39 osmith
* "Patch for colormake to colorize the ttcn3 compiler output":https://github.com/osmith42/Colormake/tree/osmith/ttcn3
Add picture from clipboard (Maximum size: 48.8 MB)