Cellular Network Infrastructure » OsmoGSMTester

After having support for running multiple scenarios in parallel we ran into a new issue.

When the test session is started and hence the N scenarios are started, as soon as the first scenario has finished a test result is generated in a test-report.txt file.

As a result of that, the shell script that is used by jenkins will think that the complete session has ended, it will rsync the test session directory and delete it after that from the test system.

One of the symptoms of that are exceptions like these, where a ongoing scenario is doing something with the test session directory but fails because it does not longer exist.

Jul 27 11:32:22 apu-roh logger: 20160727113222    INFO: "TestSession<tmp.ETuKTKWLIa-build-233>:StandardTestScenario<0x7f5ca8a68b50>" Creating report for test object
Jul 27 11:32:22 apu-roh logger: 20160727113222    INFO: "TestSession<tmp.ETuKTKWLIa-build-233>" Stopping test session process "tcpdump" 
Jul 27 11:32:22 apu-roh logger: 20160727113222    INFO: "TestSession<tmp.ETuKTKWLIa-build-233>:ProcessNurse<tcpdump>" Trying to stop process "tcpdump" 
Jul 27 11:32:22 apu-roh logger: 20160727113222    INFO: "TestSession<tmp.ETuKTKWLIa-build-233>:ProcessNurse<tcpdump>" Sending SIGTERM to 1195 ("tcpdump")
Jul 27 11:32:22 apu-roh logger: 20160727113222    INFO: "TestSession<tmp.ETuKTKWLIa-build-233>:ProcessNurse<tcpdump>" Wainting for process "tcpdump" 
Jul 27 11:32:22 apu-roh logger: 20160727113222    INFO: "TestSession<tmp.ETuKTKWLIa-build-233>:ProcessNurse<tcpdump>" Process "tcpdump" has terminated
Jul 27 11:32:22 apu-roh logger: 20160727113222    INFO: --------------------------------------------------------------------------------
Jul 27 11:32:22 apu-roh logger: 20160727113222    INFO: SendMOSmsTestCase ok (10 seconds)
Jul 27 11:32:22 apu-roh logger: 20160727113222    INFO: SendMOSmsTestCase ok (10 seconds)
Jul 27 11:32:22 apu-roh logger: 20160727113222    INFO: SendMOSmsTestCase ok (11 seconds)
Jul 27 11:32:22 apu-roh logger: 20160727113222    INFO: StandardTestScenario ok (245 seconds)
Jul 27 11:32:22 apu-roh logger: 20160727113222    INFO: StandardTestScenario error!
Jul 27 11:32:22 apu-roh logger: 20160727113222    INFO: 
Jul 27 11:32:22 apu-roh logger: 20160727113222    INFO: --------------------------------------------------------------------------------
Jul 27 11:32:24 apu-roh logger: 20160727113224   ERROR: /var/tmp/osmo-gsm-tester/tmp.ETuKTKWLIa-build-233 has no manifest file!
Jul 27 11:33:05 apu-roh logger: 20160727113305    INFO: "TestSession<tmp.ETuKTKWLIa-build-233>:StandardTestScenario<0x7f5ca8a68b50>:SendMOSmsTestCase<0x7f5ca8a68bd0>" success! took 61 seconds to deliver SMS to 7808
Jul 27 11:33:06 apu-roh logger: 20160727113306    INFO: "TestRunner" Releasing test case resources
Jul 27 11:33:06 apu-roh logger: 20160727113306    INFO: releasing Modem resource /wavecom_6
Jul 27 11:33:06 apu-roh logger: 20160727113306    INFO: "TestSession<tmp.ETuKTKWLIa-build-233>:StandardTestScenario<0x7f5ca8a68b50>:SendMOSmsTestCase<0x7f5ca8a68bd0>" released Modem /wavecom_6
Jul 27 11:33:06 apu-roh logger: 20160727113306    INFO: releasing Modem resource /wavecom_7
Jul 27 11:33:06 apu-roh logger: 20160727113306    INFO: "TestSession<tmp.ETuKTKWLIa-build-233>:StandardTestScenario<0x7f5ca8a68b50>:SendMOSmsTestCase<0x7f5ca8a68bd0>" released Modem /wavecom_7
...
zip
...
Jul 27 11:33:10 apu-roh logger: 20160727113310    INFO: "TestSession<tmp.ETuKTKWLIa-build-233>:StandardTestScenario<0x7f5ca8a68b50>" Tearing down test scenario dependencies
Jul 27 11:33:10 apu-roh logger: 20160727113310    INFO: "TestSession<tmp.ETuKTKWLIa-build-233>:StandardTestScenario<0x7f5ca8a68b50>" Tearing down NITB
Jul 27 11:33:10 apu-roh logger: 20160727113310    INFO: "TestSession<tmp.ETuKTKWLIa-build-233>:StandardTestScenario<0x7f5ca8a68b50>:NITB<0x7f5ca8a78d90>" Tearing down NITB
Jul 27 11:33:10 apu-roh logger: 20160727113310    INFO: "TestSession<tmp.ETuKTKWLIa-build-233>:StandardTestScenario<0x7f5ca8a68b50>" Tearing down BTS
Jul 27 11:33:10 apu-roh logger: 20160727113310    INFO: "TestSession<tmp.ETuKTKWLIa-build-233>:StandardTestScenario<0x7f5ca8a68b50>:NanoBTS<0x7f5ca8a78e10>" Tearing down BTS
Jul 27 11:33:10 apu-roh logger: 20160727113310 WARNING: log directory not existing yet! creating it /var/tmp/osmo-gsm-tester/tmp.ETuKTKWLIa-build-233/20160727112805-StandardTestScenario-nanoBTS1900
Jul 27 11:33:10 apu-roh logger: Traceback (most recent call last):
Jul 27 11:33:10 apu-roh logger:   File "/usr/local/src/osmo-gsm-tester/manager.py", line 204, in tick
Jul 27 11:33:10 apu-roh logger:     self.tr.tick()
Jul 27 11:33:10 apu-roh logger:   File "/usr/local/src/osmo-gsm-tester/runner.py", line 277, in tick
Jul 27 11:33:10 apu-roh logger:     self.clean_testsession(ts)
Jul 27 11:33:10 apu-roh logger:   File "/usr/local/src/osmo-gsm-tester/runner.py", line 94, in clean_testsession
Jul 27 11:33:10 apu-roh logger:     ts.teardown()
Jul 27 11:33:10 apu-roh logger:   File "/usr/local/src/osmo-gsm-tester/session.py", line 178, in teardown
Jul 27 11:33:10 apu-roh logger:     sc.teardown()
Jul 27 11:33:10 apu-roh logger:   File "/usr/local/src/osmo-gsm-tester/scenario.py", line 292, in teardown
Jul 27 11:33:10 apu-roh logger:     self._teardown_deps(ts)
Jul 27 11:33:10 apu-roh logger:   File "/usr/local/src/osmo-gsm-tester/scenario.py", line 286, in _teardown_deps
Jul 27 11:33:10 apu-roh logger:     resobj.teardown(ts)
Jul 27 11:33:10 apu-roh logger:   File "/usr/local/src/osmo-gsm-tester/bts.py", line 371, in teardown
Jul 27 11:33:10 apu-roh logger:     self.set_oml_ip(ts, nitb_addr)
Jul 27 11:33:10 apu-roh logger:   File "/usr/local/src/osmo-gsm-tester/bts.py", line 352, in set_oml_ip
Jul 27 11:33:10 apu-roh logger:     self.ipaccess_config(ts, ipacfg_set_oml_ip_args)
Jul 27 11:33:10 apu-roh logger:   File "/usr/local/src/osmo-gsm-tester/bts.py", line 335, in ipaccess_config
Jul 27 11:33:10 apu-roh logger:     p = subprocess.Popen(cmd, stdout=fd['stdout'], stderr=fd['stderr'], env=env)
Jul 27 11:33:10 apu-roh logger:   File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
Jul 27 11:33:10 apu-roh logger:     errread, errwrite)
Jul 27 11:33:10 apu-roh logger:   File "/usr/lib/python2.7/subprocess.py", line 1259, in _execute_child
Jul 27 11:33:10 apu-roh logger:     raise child_exception
Jul 27 11:33:10 apu-roh logger: OSError: [Errno 2] No such file or directory

In order to fix that issue the following things should be done:

• make sure the test-result.txt (and .xml) is only generated after all scenarios of a test session have finished
• make sure the contents of the test-result.txt (and .xml) is unique in such a way that you can tell the results various test scenarios appart (which is currently not the case)