Project

General

Profile

Titan TTCN3 Testsuites » History » Revision 27

Revision 26 (msuraev, 10/28/2018 04:30 PM) → Revision 27/52 (msuraev, 12/11/2018 11:02 AM)

{{>toc}} 

 h1. Osmocom Titan TTCN3 Testsuites 

 In 2017, Osmocom started to create extensive tests suites for the Cellular Network Infrastructure elements.    Those suites are written in the [[TTCN-3]] programming language and use the compiler and executor of the [[Eclipse_TITAN]] project. There are special cases, but in general, the idea is to test one component (e.g. @osmo-mgw@) independently. 

 h2. Source Code / Patches 

 The source code of our test suites lives in the @osmo-ttcn3-hacks@ repository 
 * cgit web interface: http://git.osmocom.org/osmo-ttcn3-hacks/ 
 * clone: @git clone git://git.osmocom.org/osmo-ttcn3-hacks@ 
 * patch review: https://gerrit.osmocom.org/#/q/project:osmo-ttcn3-hacks 

 We're following the [[Gerrit]] process for patch review. 

 h2. Test Suites 

 |_.Implementation under Test|_.Testsuite|_.Jenkins| 
 |bsc-nat|http://git.osmocom.org/osmo-ttcn3-hacks/tree/bsc-nat|-| 
 |[[OsmoBSC:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/bsc|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bsc-test/| 
 |[[OsmoBTS:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/bts|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bts-test/| 
 |[[OpenGGSN:OsmoGGSN]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/ggsn_tests|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-ggsn-test/| 
 |[[OsmoHLR:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/hlr|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-hlr-test/| 
 |[[OsmoMGW:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/mgw|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-mgw-test/| 
 |[[OsmoMSC:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/msc|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-msc-test/| 
 |[[OsmoPCU:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/pcu|-| 
 |[[OsmoSGSN:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/sgsn|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-sgsn-test/| 
 |[[osmo-sip-connector:]]|http://git.osmocom.org/osmo-ttcn3-hacks/tree/sip|https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-sip-test/| 

 h2. Running a testsuite 

 You have multiple options on how to execute a test suite. 

 h3. Running test suite in the dockerized environment 

 This is the much easier way to run the test suite.    No manual configuration of Osmocom programs or the test suite are required.    Both the Osmocom program (Implementation Under Test) as well as the test suite are packaged as docker containers, which are then executed next to each other using a docker network with the "right" addresses between IUT and testsuite. 

 Also, using this setup you will be guaranteed to run in the exact same environment as the automatically-executed tests on jenkins.osmocom.org, i.e. your results should be exactly identical, without any differences introduced by your runtime environment, whether specific library versions or intentional or inadvertent configuration differences. 

 h4. Installing docker 

 Make sure you have "*docker-ce*":https://docs.docker.com/install/#supported-platforms installed (not to be confused with the possibly outdated version that may be shipped in your distribution's repositories). Check that the version string contains <code>-ce</code> as follows: 

 <pre> 
 $ docker -v 
 Docker version 18.06.1-ce, build e68fc7a 
 </pre> 


 Next, add yourself to the <code>docker</code> group, log off and on again, and verify that you can use docker with your user: 

 <pre> 
 $ docker info 
 </pre> 

 h4. Cloning docker-playground 

 Osmocom related containers are stored in the "@docker-playground@":https://git.osmocom.org/docker-playground/ git repository. 

 <pre> 
 $ git clone git://git.osmocom.org/docker-playground 
 $ cd docker-playground 
 </pre> 

 h4. Running a testsuite 

 All testsuite folders start with @ttcn3@ or @nplab@. Run the following to build/update all required containers and start a specific testsuite: 

 <pre> 
 $ cd ttcn3-mgw-test 
 $ ./jenkins.sh 
 </pre> 

 Environment variables: 
 * @IMAGE_SUFFIX@: the version of the Osmocom stack to run the testsuite against. Default is `master`, set this to `latest` to test the last stable releases. 
 * @NO_DOCKER_IMAGE_BUILD@: when set to @1@, it won't try to update the containers (see "README.md":https://git.osmocom.org/docker-playground/tree/README.md) 

 The @jenkins.sh@ script will make sure to 
 * create the needed docker network(s) 
 * run all the required containers (IUT, helpers, testsuite) 
 * shut them down after test suite execution 
 * collect the log files after execution, @/tmp/logs@ will contain them 

 h4. Running only a sub-set of the test cases 

 See running the testsuite outside of Docker for reference (below). 

 h3. Running it native on your machine 

 This is the most complicated bit to set-up, as you will have to run the respective Osmocom Program (Implementation Under Test) in the right configuration with all the IP addresses, port numbers, config file, etc. exactly like it's expected by the test suite. 

 You will need a deeper understanding about how the test suite works, and what its requirement are. Oftentimes looking at how it's done in the Docker scripts is helpful. 

 h4. Preparation 

 <pre> 
 $ apt install eclipse-titan 
 $ git clone git://osmocom.org/osmo-ttcn3-hacks 
 $ cd osmo-ttc3-hacks/deps 
 $ make 
 </pre> 

 h4. Compile a testsuite 

 Let's compile the @mgw@ testsuite for example: 

 <pre> 
 $ cd mgw 
 $ ./gen_links.sh 
 $ ./regen_makefile.sh 
 </pre> 

 The next command will transform the TTCN3 test data into C++ code. Do not use @-j@ here, that won't work (and it is pretty fast anyway). 
 <pre> 
 $ make compile 
 </pre> 

 Note: if you observe errors during @make compile@, try running it from top-level as @make mgw@ which should trigger dependency update. 

 Finally compile the testsuite: 
 <pre> 
 $ make -j5 
 </pre> 


 

 h4. Run a testsuite 

 Start the component that is about to get tested with the config that the testsuite expects. Again, with @mgw@ as example. If you don't have a suitable config file, use "the one from Jenkins":https://git.osmocom.org/docker-playground/tree/ttcn3-mgw-test/osmo-mgw.cfg and replace all IPs with @127.0.0.1@. 
 <pre> 
 $ osmo-mgw -c osmo-mgw.cfg 
 </pre> 

 Then run the testsuite: 
 <pre> 
 $ cd osmo-ttcn3-hacks/mgw 
 $ ../start-testsuite.sh ./MGCP_test MGCP_Test.cfg 
 </pre> 

 Afterwards you can merge and format the logs as follows: 
 <pre> 
 $ ttcn3_logmerge MGCP_test*.log > ./merged.log 
 $ ttcn3_logformat ./merged.log > ./result.log 
 </pre> 

 h4. Running only a sub-set of the test cases 

 Running whole testsuites may take a long time. If you only want to run one specific test, or a subset of tests, open up the @_*Test.cfg@ file in the folder of the testsuite and edit the @[EXECUTE]@ section. For example, in "@MGCP_Test.cfg@":https://git.osmocom.org/osmo-ttcn3-hacks/tree/mgw/MGCP_Test.cfg from the @mgw@ folder it looks like the following:  

 <pre> 
 ... 
 [EXECUTE] 
 MGCP_Test.control 
 #MGCP_Test.TC_selftest 
 #MGCP_Test.TC_crcx 
 #MGCP_Test.TC_crcx_noprefix 
 ... 
 </pre> 

 The only line that is *not* commented out is @MGCP_Test.control@. This @*.control@ line will cause all tests to be executed. So comment this line out, and then activate one of the lines of which you would like to execute the tests (see the example below). Then execute the testsuite as usually, and it will skip all commented out tests. 

 <pre> 
 ... 
 [EXECUTE] 
 #MGCP_Test.control 
 #MGCP_Test.TC_selftest 
 MGCP_Test.TC_crcx 
 MGCP_Test.TC_crcx_noprefix 
 ... 
 </pre> 

 h2. Reading the logs 

 * @xfail@ means, that a test is known to fail, @FAIL@ are unexpected test failures 
 * search for @setverdict(@ to find the results of single tests. @-> fail@ should be the location where a test is failing. 

 h2. Further information 

 * April 2018 talk by Harald on "State of the Osmocom TTCN-3 Test Suites":https://media.ccc.de/v/UGHHW3
Add picture from clipboard (Maximum size: 48.8 MB)