Project

General

Profile

Titan TTCN3 Testsuites » History » Version 23

osmith, 10/01/2018 10:45 AM

1 2 laforge
{{>toc}}
2
3 1 laforge
h1. Osmocom Titan TTCN3 Testsuites
4
5 14 osmith
In 2017, Osmocom started to create extensive tests suites for the Cellular Network Infrastructure elements.  Those suites are written in the [[TTCN-3]] programming language and use the compiler and executor of the [[Eclipse_TITAN]] project. There are special cases, but in general, the idea is to test one component (e.g. @osmo-mgw@) independently.
6 2 laforge
7
h2. Source Code / Patches
8
9
The source code of our test suites lives in the @osmo-ttcn3-hacks@ repository
10
* cgit web interface: http://git.osmocom.org/osmo-ttcn3-hacks/
11
* clone: @git clone git://git.osmocom.org/osmo-ttcn3-hacks@
12
* patch review: https://gerrit.osmocom.org/#/q/project:osmo-ttcn3-hacks
13
14
We're following the [[Gerrit]] process for patch review.
15
16
h2. Test Suites
17
18
|_.Implementation under Test|_.Testsuite|_.Jenkins|
19
|bsc-nat|http://git.osmocom.org/osmo-ttcn3-hacks/tree/bsc-nat|-|
20
|[[OsmoBSC:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/bsc|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bsc-test/|
21
|[[OsmoBTS:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/bts|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bts-test/|
22
|[[OpenGGSN:OsmoGGSN]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/ggsn_tests|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-ggsn-test/|
23
|[[OsmoHLR:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/hlr|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-hlr-test/|
24
|[[OsmoMGW:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/mgw|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-mgw-test/|
25
|[[OsmoMSC:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/msc|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-msc-test/|
26
|[[OsmoPCU:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/pcu|-|
27
|[[OsmoSGSN:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/sgsn|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-sgsn-test/|
28
|[[osmo-sip-connector:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/sip|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-sip-test/|
29
30
h2. Running a testsuite
31
32
You have multiple options on how to execute a test suite.
33
34
h3. Running test suite in the dockerized environment
35 1 laforge
36 4 osmith
This is the much easier way to run the test suite.  No manual configuration of Osmocom programs or the test suite are required.  Both the Osmocom program (Implementation Under Test) as well as the test suite are packaged as docker containers, which are then executed next to each other using a docker network with the "right" addresses between IUT and testsuite.
37 2 laforge
38
Also, using this setup you will be guaranteed to run in the exact same environment as the automatically-executed tests on jenkins.osmocom.org, i.e. your results should be exactly identical, without any differences introduced by your runtime environment, whether specific library versions or intentional or inadvertent configuration differences.
39
40 10 osmith
h4. Installing docker
41 1 laforge
42 10 osmith
Make sure you have "*docker-ce*":https://docs.docker.com/install/#supported-platforms installed (not to be confused with the possibly outdated version that may be shipped in your distribution's repositories). Check that the version string contains <code>-ce</code> as follows:
43
44
<pre>
45
$ docker -v
46
Docker version 18.06.1-ce, build e68fc7a
47
</pre>
48
49
50 11 osmith
Next, add yourself to the <code>docker</code> group, log off and on again, and verify that you can use docker with your user:
51 10 osmith
52
<pre>
53
$ docker info
54
</pre>
55
56
h4. Cloning docker-playground
57
58 12 osmith
Osmocom related containers are stored in the "@docker-playground@":https://git.osmocom.org/docker-playground/ git repository.
59 5 osmith
60
<pre>
61
$ git clone git://git.osmocom.org/docker-playground
62
$ cd docker-playground
63 1 laforge
</pre>
64 6 osmith
65 1 laforge
In theory you could run @make@ now to build all containers. However, not all of them are maintained, so this will most likely fail. Instead take a look at the Jenkins script of a specific project to figure out its dependencies and then build these containers.
66 5 osmith
67 10 osmith
h4. Running a testsuite
68 5 osmith
69
Let's assume you'd want to run the testsuite for [[OsmoBSC:]].  You would need to 
70 6 osmith
# look at its "jenkins.sh":http://git.osmocom.org/docker-playground/tree/ttcn3-bsc-test/jenkins.sh to understand which containers are required.
71 7 osmith
# build those respective containers - at time of writing, this would be:
72 1 laforge
<pre>
73 9 osmith
$ make osmo-stp-master osmo-bsc-master osmo-bts-master ttcn3-bsc-test
74 1 laforge
</pre>
75
# run the @jenkins.sh@ script of the @ttcn3-bsc-test@ suite:
76 2 laforge
<pre>
77
$ cd ttcn3-bsc-test && ./jenkins.sh
78
</pre>
79
80 6 osmith
The @jenkins.sh@ script will make sure to
81
* create the needed docker network[s]
82
* run all the required containers (IUT, helpers, testsuite)
83
* shut them down after test suite execution
84
* collect the log files after execution, @/tmp/logs@ will contain them
85
86 2 laforge
h4. Running only a sub-set of the test cases
87
88 20 osmith
See running the testsuite outside of Docker for reference (below).
89
90 2 laforge
h3. Running it native on your machine
91
92
This is the most complicated bit to set-up, as you will have to run the respective Osmocom Program (Implementation Under Test) in the right configuration with all the IP addresses, port numbers, config file, etc. exactly like it's expected by the test suite.
93 1 laforge
94 14 osmith
You will need a deeper understanding about how the test suite works, and what its requirement are. Oftentimes looking at how it's done in the Docker scripts is helpful.
95
96
h4. Preparation
97
98
<pre>
99
$ apt install eclipse-titan
100
$ git clone git://osmocom.org/osmo-ttcn3-hacks
101
$ cd osmo-ttc3-hacks/deps
102
$ make
103
</pre>
104
105
h4. Compile a testsuite
106
107
Let's compile the @mgw@ testsuite for example:
108
109
<pre>
110
$ cd mgw
111
$ ./gen_links.sh
112
$ ./regen_makefile
113
</pre>
114 1 laforge
115 23 osmith
The next command will transform the TTCN3 test data into C++ code. Do not use @-j@ here, that won't work (and it is pretty fast anyway).
116 14 osmith
<pre>
117
$ make compile
118
</pre>
119
120
Finally compile the testsuite:
121
<pre>
122
$ make -j5
123
</pre>
124
125
h4. Run a testsuite
126
127 18 osmith
Start the component that is about to get tested with the config that the testsuite expects. Again, with @mgw@ as example. If you don't have a suitable config file, use "the one from Jenkins":https://git.osmocom.org/docker-playground/tree/ttcn3-mgw-test/osmo-mgw.cfg and replace all IPs with @127.0.0.1@.
128 14 osmith
<pre>
129 21 osmith
$ osmo-mgw -c osmo-mgw.cfg
130 14 osmith
</pre>
131
132
Then run the testsuite:
133
<pre>
134
$ cd osmo-ttcn3-hacks/mgw
135
$ ../start-testsuite.sh ./MGCP_test MGCP_Test.cfg
136
</pre>
137
138
Afterwards you can merge and format the logs as follows:
139
<pre>
140
$ ttcn3_logmerge MGCP_test*.log > ./merged.log
141
$ ttcn3_logformat ./merged.log > ./result.log
142
</pre>
143 2 laforge
144
h4. Running only a sub-set of the test cases
145
146 17 osmith
Running whole testsuites may take a long time. If you only want to run one specific test, or a subset of tests, open up the @_*Test.cfg@ file in the folder of the testsuite and edit the @[EXECUTE]@ section. For example, in "@MGCP_Test.cfg@":https://git.osmocom.org/osmo-ttcn3-hacks/tree/mgw/MGCP_Test.cfg from the @mgw@ folder it looks like the following: 
147 16 osmith
148
<pre>
149 1 laforge
...
150 17 osmith
[EXECUTE]
151
MGCP_Test.control
152
#MGCP_Test.TC_selftest
153
#MGCP_Test.TC_crcx
154
#MGCP_Test.TC_crcx_noprefix
155
...
156
</pre>
157 1 laforge
158 17 osmith
The only line that is *not* commented out is @MGCP_Test.control@. This @*.control@ line will cause all tests to be executed. So comment this line out, and then activate one of the lines of which you would like to execute the tests (see the example below). Then execute the testsuite as usually, and it will skip all commented out tests.
159
160
<pre>
161
...
162
[EXECUTE]
163
#MGCP_Test.control
164
#MGCP_Test.TC_selftest
165
MGCP_Test.TC_crcx
166
MGCP_Test.TC_crcx_noprefix
167
...
168 16 osmith
</pre>
169 2 laforge
170 15 osmith
h2. Reading the logs
171
172
* @xfail@ means, that a test is known to fail, @FAIL@ are unexpected test failures
173
* search for @setverdict(@ to find the results of single tests. @-> fail@ should be the location where a test is failing.
174
175 2 laforge
h2. Further information
176
177 22 osmith
* April 2018 talk by Harald on "State of the Osmocom TTCN-3 Test Suites":https://media.ccc.de/v/UGHHW3
Add picture from clipboard (Maximum size: 48.8 MB)