Bug #1783
closedjenkins deletes session before session complete
0%
Description
After having support for running multiple scenarios in parallel we ran into a new issue.
When the test session is started and hence the N scenarios are started, as soon as the first scenario has finished a test result is generated in a test-report.txt file.
As a result of that, the shell script that is used by jenkins will think that the complete session has ended, it will rsync the test session directory and delete it after that from the test system.
One of the symptoms of that are exceptions like these, where a ongoing scenario is doing something with the test session directory but fails because it does not longer exist.
Jul 27 11:32:22 apu-roh logger: 20160727113222 INFO: "TestSession<tmp.ETuKTKWLIa-build-233>:StandardTestScenario<0x7f5ca8a68b50>" Creating report for test object Jul 27 11:32:22 apu-roh logger: 20160727113222 INFO: "TestSession<tmp.ETuKTKWLIa-build-233>" Stopping test session process "tcpdump" Jul 27 11:32:22 apu-roh logger: 20160727113222 INFO: "TestSession<tmp.ETuKTKWLIa-build-233>:ProcessNurse<tcpdump>" Trying to stop process "tcpdump" Jul 27 11:32:22 apu-roh logger: 20160727113222 INFO: "TestSession<tmp.ETuKTKWLIa-build-233>:ProcessNurse<tcpdump>" Sending SIGTERM to 1195 ("tcpdump") Jul 27 11:32:22 apu-roh logger: 20160727113222 INFO: "TestSession<tmp.ETuKTKWLIa-build-233>:ProcessNurse<tcpdump>" Wainting for process "tcpdump" Jul 27 11:32:22 apu-roh logger: 20160727113222 INFO: "TestSession<tmp.ETuKTKWLIa-build-233>:ProcessNurse<tcpdump>" Process "tcpdump" has terminated Jul 27 11:32:22 apu-roh logger: 20160727113222 INFO: -------------------------------------------------------------------------------- Jul 27 11:32:22 apu-roh logger: 20160727113222 INFO: SendMOSmsTestCase ok (10 seconds) Jul 27 11:32:22 apu-roh logger: 20160727113222 INFO: SendMOSmsTestCase ok (10 seconds) Jul 27 11:32:22 apu-roh logger: 20160727113222 INFO: SendMOSmsTestCase ok (11 seconds) Jul 27 11:32:22 apu-roh logger: 20160727113222 INFO: StandardTestScenario ok (245 seconds) Jul 27 11:32:22 apu-roh logger: 20160727113222 INFO: StandardTestScenario error! Jul 27 11:32:22 apu-roh logger: 20160727113222 INFO: Jul 27 11:32:22 apu-roh logger: 20160727113222 INFO: -------------------------------------------------------------------------------- Jul 27 11:32:24 apu-roh logger: 20160727113224 ERROR: /var/tmp/osmo-gsm-tester/tmp.ETuKTKWLIa-build-233 has no manifest file! Jul 27 11:33:05 apu-roh logger: 20160727113305 INFO: "TestSession<tmp.ETuKTKWLIa-build-233>:StandardTestScenario<0x7f5ca8a68b50>:SendMOSmsTestCase<0x7f5ca8a68bd0>" success! took 61 seconds to deliver SMS to 7808 Jul 27 11:33:06 apu-roh logger: 20160727113306 INFO: "TestRunner" Releasing test case resources Jul 27 11:33:06 apu-roh logger: 20160727113306 INFO: releasing Modem resource /wavecom_6 Jul 27 11:33:06 apu-roh logger: 20160727113306 INFO: "TestSession<tmp.ETuKTKWLIa-build-233>:StandardTestScenario<0x7f5ca8a68b50>:SendMOSmsTestCase<0x7f5ca8a68bd0>" released Modem /wavecom_6 Jul 27 11:33:06 apu-roh logger: 20160727113306 INFO: releasing Modem resource /wavecom_7 Jul 27 11:33:06 apu-roh logger: 20160727113306 INFO: "TestSession<tmp.ETuKTKWLIa-build-233>:StandardTestScenario<0x7f5ca8a68b50>:SendMOSmsTestCase<0x7f5ca8a68bd0>" released Modem /wavecom_7 ... zip ... Jul 27 11:33:10 apu-roh logger: 20160727113310 INFO: "TestSession<tmp.ETuKTKWLIa-build-233>:StandardTestScenario<0x7f5ca8a68b50>" Tearing down test scenario dependencies Jul 27 11:33:10 apu-roh logger: 20160727113310 INFO: "TestSession<tmp.ETuKTKWLIa-build-233>:StandardTestScenario<0x7f5ca8a68b50>" Tearing down NITB Jul 27 11:33:10 apu-roh logger: 20160727113310 INFO: "TestSession<tmp.ETuKTKWLIa-build-233>:StandardTestScenario<0x7f5ca8a68b50>:NITB<0x7f5ca8a78d90>" Tearing down NITB Jul 27 11:33:10 apu-roh logger: 20160727113310 INFO: "TestSession<tmp.ETuKTKWLIa-build-233>:StandardTestScenario<0x7f5ca8a68b50>" Tearing down BTS Jul 27 11:33:10 apu-roh logger: 20160727113310 INFO: "TestSession<tmp.ETuKTKWLIa-build-233>:StandardTestScenario<0x7f5ca8a68b50>:NanoBTS<0x7f5ca8a78e10>" Tearing down BTS Jul 27 11:33:10 apu-roh logger: 20160727113310 WARNING: log directory not existing yet! creating it /var/tmp/osmo-gsm-tester/tmp.ETuKTKWLIa-build-233/20160727112805-StandardTestScenario-nanoBTS1900 Jul 27 11:33:10 apu-roh logger: Traceback (most recent call last): Jul 27 11:33:10 apu-roh logger: File "/usr/local/src/osmo-gsm-tester/manager.py", line 204, in tick Jul 27 11:33:10 apu-roh logger: self.tr.tick() Jul 27 11:33:10 apu-roh logger: File "/usr/local/src/osmo-gsm-tester/runner.py", line 277, in tick Jul 27 11:33:10 apu-roh logger: self.clean_testsession(ts) Jul 27 11:33:10 apu-roh logger: File "/usr/local/src/osmo-gsm-tester/runner.py", line 94, in clean_testsession Jul 27 11:33:10 apu-roh logger: ts.teardown() Jul 27 11:33:10 apu-roh logger: File "/usr/local/src/osmo-gsm-tester/session.py", line 178, in teardown Jul 27 11:33:10 apu-roh logger: sc.teardown() Jul 27 11:33:10 apu-roh logger: File "/usr/local/src/osmo-gsm-tester/scenario.py", line 292, in teardown Jul 27 11:33:10 apu-roh logger: self._teardown_deps(ts) Jul 27 11:33:10 apu-roh logger: File "/usr/local/src/osmo-gsm-tester/scenario.py", line 286, in _teardown_deps Jul 27 11:33:10 apu-roh logger: resobj.teardown(ts) Jul 27 11:33:10 apu-roh logger: File "/usr/local/src/osmo-gsm-tester/bts.py", line 371, in teardown Jul 27 11:33:10 apu-roh logger: self.set_oml_ip(ts, nitb_addr) Jul 27 11:33:10 apu-roh logger: File "/usr/local/src/osmo-gsm-tester/bts.py", line 352, in set_oml_ip Jul 27 11:33:10 apu-roh logger: self.ipaccess_config(ts, ipacfg_set_oml_ip_args) Jul 27 11:33:10 apu-roh logger: File "/usr/local/src/osmo-gsm-tester/bts.py", line 335, in ipaccess_config Jul 27 11:33:10 apu-roh logger: p = subprocess.Popen(cmd, stdout=fd['stdout'], stderr=fd['stderr'], env=env) Jul 27 11:33:10 apu-roh logger: File "/usr/lib/python2.7/subprocess.py", line 679, in __init__ Jul 27 11:33:10 apu-roh logger: errread, errwrite) Jul 27 11:33:10 apu-roh logger: File "/usr/lib/python2.7/subprocess.py", line 1259, in _execute_child Jul 27 11:33:10 apu-roh logger: raise child_exception Jul 27 11:33:10 apu-roh logger: OSError: [Errno 2] No such file or directory
In order to fix that issue the following things should be done:
- make sure the test-result.txt (and .xml) is only generated after all scenarios of a test session have finished
- make sure the contents of the test-result.txt (and .xml) is unique in such a way that you can tell the results various test scenarios appart (which is currently not the case)
Related issues
Updated by lazlo about 8 years ago
- Related to Feature #1773: parallel scenario execution added
Updated by lazlo about 8 years ago
- Status changed from New to Resolved
Fixed in b5444b2. Required some fixups so should be safe to say that from 99cb3b5 on it is working.