Project

General

Profile

Actions

Bug #4383

open

Jenkins build verification is non-deterministic

Added by fixeria about 4 years ago. Updated almost 3 years ago.

Status:
Stalled
Priority:
High
Assignee:
Category:
-
Target version:
-
Start date:
01/29/2020
Due date:
% Done:

40%

Spec Reference:

Description

Recently I submitted a patch that is not supposed to change the unit test output:

https://gerrit.osmocom.org/c/pysim/+/16982/

and it actually does not, but Jenkins verification fails:

https://jenkins.osmocom.org/jenkins/job/gerrit-pysim/261/

Verifying card ...
Card contents do not match the test data:
Expected: sysmoUSIM-SJS1.ok
------------8<------------
Using PC/SC reader (dev=1) interface
Reading ...
ICCID: 1122334455667788990
IMSI: 001010000000102
SMSP: ffffffffffffffffffffffffffffffffffffffffffffffffe1ffffffffffffffffffffffff0581005155f5ffffffffffff000000
SPN: 
Display HPLMN: False
Display OPLMN: False
PLMNsel: fff11fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
PLMNwAcT:
    fff11fffff # MCC: 1651 MNC: 151 AcT: UTRAN, E-UTRAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused

OPLMNwAcT:
    fff11fffff # MCC: 1651 MNC: 151 AcT: UTRAN, E-UTRAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused

HPLMNAcT:
    ffffffffff # unused
    ffffffffff # unused
    ffffffffff # unused
    ffffffffff # unused
    ffffffffff # unused
    ffffffffff # unused
    ffffffffff # unused
    ffffffffff # unused
    ffffffffff # unused
    ffffffffff # unused
    ffffffffff # unused
    ffffffffff # unused

ACC: 0008
MSISDN: Not available
AD: 00000002
Done !

------------8<------------
Got:
------------8<------------
Using PC/SC reader (dev=2) interface
Reading ...
ICCID: 1122334455667788990
IMSI: 001010000000102
SMSP: ffffffffffffffffffffffffffffffffffffffffffffffffe1ffffffffffffffffffffffff0581005155f5ffffffffffff000000
SPN: Magic  <-- The contents of this file do not match the expectations
Display HPLMN: True
Display OPLMN: True
PLMNsel: fff11fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
PLMNwAcT:
    fff11fffff # MCC: 1651 MNC: 151 AcT: UTRAN, E-UTRAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused

OPLMNwAcT:
    fff11fffff # MCC: 1651 MNC: 151 AcT: UTRAN, E-UTRAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused

HPLMNAcT:  <-- The contents of this file do not match the expectations
    fff11fffff # MCC: 1651 MNC: 151 AcT: UTRAN, E-UTRAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused
    ffffff0000 # unused

ACC: 0008
MSISDN: Not available
AD: 00000002
Done !

------------8<------------
Build step 'Execute shell' marked build as failure
[WARNINGS]Skipping publisher since build result is FAILURE
Finished: FAILURE

As I figured out, this is a side effect of the other changes previously submitted to Gerrit:

https://gerrit.osmocom.org/c/pysim/+/16941
https://gerrit.osmocom.org/c/pysim/+/16972

The problem is that a patch submitted to Gerrit may change the contents of SIM card's file system, so after that Jenkins fails to verify any other changes because the unit test output has changed. In this case both SPN and HPLMNAcT have been changed. Perhaps we need a 'smarter' way (than just comparing stdout and a file) to do build verification, e.g. we could match only those files which we're programming in the unit test. Python has a nice framework for unit tests - https://docs.python.org/3/library/unittest.html, so we could have one unit test per file.


Related issues

Related to pySim - Feature #4384: Jenkins program-read verification on daily (or weekly) basisStalleddexter01/29/2020

Actions
Related to pySim - Feature #5127: tool for "re-formatting" [sysmocom] cards in testerNewlaforge04/23/2021

Actions
Actions

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)