Project

General

Profile

Feature #4384

Jenkins program-read verification on daily (or weekly) basis

Added by fixeria about 1 year ago. Updated about 1 month ago.

Status:
Feedback
Priority:
Normal
Assignee:
Target version:
-
Start date:
01/29/2020
Due date:
% Done:

0%

Spec Reference:

Description

We already have build verification for some projects running once per day:

https://jenkins.osmocom.org/jenkins/view/master/

It would be nice to have daily or at least weekly (given the low amount of contributions) master build verification for pySim too.


Related issues

Related to pySim - Bug #4383: Jenkins build verification is non-deterministicStalled01/29/2020

History

#1 Updated by fixeria about 1 year ago

  • Tracker changed from Bug to Feature

#2 Updated by laforge 10 months ago

  • Status changed from New to Feedback
  • Assignee changed from dexter to fixeria

fixeria what would be "build verifcation" of a python program? It's not like we're compiling anything...

#3 Updated by fixeria 10 months ago

  • Related to Bug #4383: Jenkins build verification is non-deterministic added

#4 Updated by fixeria 10 months ago

  • Subject changed from Jenkins build verification on daily (or weekly) basis to Jenkins program-read verification on daily (or weekly) basis

The idea to have a regular testing was proposed in the IRC while discussing #4383. If I remember correctly, back then Jenkins verification was broken (some EF was changed) by a change submitted to Gerrit, and this remained undiscovered for some time. So if we had automatic daily or weekly verification, the problem could be spotted and fixed earlier. Right now one would need to walk over the list of all changes trying to find the culprit.

#5 Updated by laforge about 1 month ago

fixeria wrote:

The idea to have a regular testing was proposed in the IRC while discussing #4383. If I remember correctly, back then Jenkins verification was broken (some EF was changed) by a change submitted to Gerrit, and this remained undiscovered for some time. So if we had automatic daily or weekly verification, the problem could be spotted and fixed earlier. Right now one would need to walk over the list of all changes trying to find the culprit.

I'm not really following you here. What happened:
  • a patch was submitted to jenkins, changing some unexpected data on the card
  • all follow-up verifications failed due to the wrong contents on the card
This kind of problem can only be avoided if we either
  1. completely re-format the cards between each two test runs. (re-install the OS, card profile, dynamic data)
    • is only possible for sysmocom cards, as we don't have the kind of data/tools/docs for others
    • would introduce a lot of unusual flash wear, so I'd assume the cards would not survive all too long. Once again not a problem for sysmocom cards which we have thousands, but others we only have few
  2. keep a backup of all "well-known" (3GPP standard files) of each card, and verify that at start-up. If any files differ, re-write selectively only those files from the backup
    • would work with probably almost any card (maybe except the super cheap magicSIM that may not permit regular write operations to all files?
    • would not help in case of unexpected modifications were made to non-standard/proprietary files that we cannot backup/restore in a generic way
  3. keep a completely separate physical test setip with different card readers + cards for testing master and jenkins. Then any breakage introduced by random jenkins patches would not render us unable to see if master still works
  • '1' is out of the question
  • '2' is doable, I think I could hack up a backup-and-restore tool relatively quickl (for SIM/USIM/ISIM standard files)
  • '3' sounds like it's not a lot of effort, physically. We could even use one of our sysmoOCTSIM now. However, I don'y think we can use the exact same test expectations in both setups, as the card ICCID will be different, and possibly also ADM pin, etc.

dexter, any input to this?

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)