Project

General

Profile

Bug #3767

Most ttcn3-*-test-latest jenkins jobs are failing

Added by osmith over 2 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
High
Assignee:
Target version:
-
Start date:
01/24/2019
Due date:
% Done:

100%

Spec Reference:

Description

New Osmocom version were released yesterday, and we expected that the -latest jobs would finally reach the same state as the master ones (the ones without a suffix).

However, most tests are still failing for various reasons:
See https://jenkins.osmocom.org/jenkins/view/TTCN3/

This was partially caused by updating eclipse-titan from 6.3.1 to 6.5.0 in the binary package repository, which then meant that our tests were not building anymore (they couldn't find TTCN3.hh anymore). So the testing docker containers were not building anymore, and a following bug in jenkins_common.sh caused the jobs not to abort there, but to continue with the last image that it was able to build (patch to fix that submitted here). The eclipse-titan update was reverted here and now the docker container builds again.

But the problem is not resolved yet, ttcn3-bts-test-latest for example is still failing.

I think daniel mentioned that he saw a "command not found" in the log? I can't find it though.

I will look more into this tomorrow.


Checklist

  • fix ttcn3-hlr-test-latest in my branch
  • fix ttcn3-msc-test-latest in my branch
  • fix ttcn3-sgsn-test-latest in my branch
  • fix ttcn3-sip-test-latest in my branch
  • get changes merged into master
  • fix ttcn3-bts-test-latest in my branch

Related issues

Related to Cellular Network Infrastructure - Bug #3268: execute TTCN3 test suites against "latest" feedsResolved05/15/2018

Related to Cellular Network Infrastructure - Bug #3769: docker-playground: RUN_ARGS in Makefile leads to unexpected behaviorRejected01/28/2019

Related to Cellular Network Infrastructure - Bug #3771: osmo-msc Debian package does not depend on libdbd-sqlite3Resolved01/29/2019

Associated revisions

Revision c9fcc6e8 (diff)
Added by osmith over 2 years ago

jenkins-common.sh: exit if image fails to build

docker_images_require() must fail when a new image can not be built,
even if an old image already exists.

We have this situation in a lot of the -latest tests right now, which
makes it harder to find the actual error.

Note that we can't simply use #!/bin/sh -e in all the jenkins.sh
scripts that source jenkins_common.sh, because they must be able to
clean up the docker containers on failure. So they can't abort
somewhere in the middle of the scripts. We could rewrite them to do the
clean up in a trap, but this should be done in a separate commit.

Related: OS#3767
Change-Id: I7039089457b62b8798a79c5efd62bd91622986d3

Revision 761b7acd (diff)
Added by osmith over 2 years ago

ttcn3-bts-test-latest: use osmocom-bb-host-master

Do not try to run a osmocom-bb-host-latest docker image, which does not
exist as we are not tagging release versions for osmocom-bb (yet).

This was done right in the docker_images_require call at the top
already (which is why it did not try to build such a "-latest" image),
but then the script tried to use the "-latest" images anyway (instead
of hardcoding "-master").

Related: OS#3767
Change-Id: Ib743f09c71d93bf4c4ad947ef62b37de6b2d7ecc

Revision 6687896c (diff)
Added by osmith over 2 years ago

ttcn3-*-test: don't hardcode /usr/local/bin/

Remove hardcoded /usr/local/bin/ paths from jenkins.sh. This caused the
*-latest tests to fail, as they have the Osmocom binaries in /usr/bin
instead of /usr/local/bin.

Related: OS#3767
Change-Id: I9d1427e66028ffd15fed211b2dab5adc37718666

Revision 2279152b (diff)
Added by osmith over 2 years ago

osmo-msc-latest: install libdbd-sqlite3

Fix osmo-msc crash at start up:
<0009> db.c:621 Failed to create database connection to sqlite3 db 'sms.db'; Is the sqlite3 database driver for libdbi installed on this system?
DB: Failed to init database: sms.db

It would be better if the Debian package pulled in this dependency
automatically, see OS#3771.

Related: OS#3767
Change-Id: Ia816aed8dd3b86f44f4454af89cec6f0b9d55dd1

Revision 624dd4fb (diff)
Added by osmith over 2 years ago

osmo-msc.cfg: remove auth policy, LU reject cause

Delete legacy config lines. They have been deleted in osmo-msc.git's
example config files in [1] and [2] almost a year ago.

When running locally, osmo-msc refuses to start with these lines. The
ttcn3-msc-test dir has its own osmo-msc.cfg file, so it works there
without this patch.

$ osmo-msc
There is no such command.
Error occurred during reading the below line:
auth policy closed

<0006> msc_main.c:573 Failed to parse the config file: 'osmo-msc.cfg'

[1]: change-id I825e659da529257e5edec94d9d59f0e10c1b4c63
[2]: change-id Ie1f39e706477aaf42051877b52d4b3ae1c5f138e

Related: OS#3767
Change-Id: I101b69c5c25252cac346259d8d88f3e02bc943d6

History

#1 Updated by osmith over 2 years ago

  • Related to Bug #3268: execute TTCN3 test suites against "latest" feeds added

#2 Updated by osmith over 2 years ago

  • Priority changed from Normal to High

#3 Updated by osmith over 2 years ago

Next bug figured out, more to come: ttcn3-bts-test-latest tried to use a non-existing "osmocom-bb-host-master" image.

Patches related to this issue are here:
https://gerrit.osmocom.org/#/q/topic:ttcn3-latest-fixes+(status:open+OR+status:merged)

#4 Updated by osmith over 2 years ago

  • Checklist item ttcn3-hlr-test-latest added
  • Checklist item ttcn3-msc-test-latest added
  • Checklist item ttcn3-sgsn-test-latest added
  • Checklist item ttcn3-sip-test-latest added

#5 Updated by osmith over 2 years ago

I think daniel mentioned that he saw a "command not found" in the log? I can't find it though.

Could be this message from https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-hlr-test-latest/96/console:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"/usr/local/bin/osmo-hlr\": stat /usr/local/bin/osmo-hlr: no such file or directory": unknown.

#6 Updated by osmith over 2 years ago

  • Checklist item changed from ttcn3-hlr-test-latest to fix ttcn3-hlr-test-latest in my branch
  • Checklist item changed from ttcn3-msc-test-latest to fix ttcn3-msc-test-latest in my branch
  • Checklist item changed from ttcn3-sgsn-test-latest to fix ttcn3-sgsn-test-latest in my branch
  • Checklist item changed from ttcn3-sip-test-latest to fix ttcn3-sip-test-latest in my branch
  • Checklist item get changes merged into master added
  • Checklist item fix ttcn3-hlr-test-latest in my branch set to Done

#7 Updated by osmith over 2 years ago

  • Checklist item fix ttcn3-bts-test-latest in my branch added

#8 Updated by osmith over 2 years ago

  • % Done changed from 0 to 30

It seems like the remaining errors are related to docker containers that are still running on the Jenkins slave, and therefore block spawning of new containers.
I've asked somebody with root access to kill those.

#9 Updated by osmith over 2 years ago

  • Checklist item fix ttcn3-sgsn-test-latest in my branch set to Done
  • Checklist item fix ttcn3-sip-test-latest in my branch set to Done

#10 Updated by osmith over 2 years ago

  • Related to Bug #3769: docker-playground: RUN_ARGS in Makefile leads to unexpected behavior added

#11 Updated by osmith over 2 years ago

  • Related to Bug #3771: osmo-msc Debian package does not depend on libdbd-sqlite3 added

#12 Updated by osmith over 2 years ago

  • Checklist item fix ttcn3-msc-test-latest in my branch set to Done
  • % Done changed from 30 to 90

All -latest tests are running on my branch now!

Now we only need to get the patches merged (review welcome):
https://gerrit.osmocom.org/#/q/topic:ttcn3-latest-fixes+status:open

#13 Updated by osmith over 2 years ago

  • Checklist item get changes merged into master set to Done
  • Status changed from In Progress to Resolved
  • % Done changed from 90 to 100

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)