Project

General

Profile

Bug #4379

ttcn3-ggsn-test(-latest) sporadic all-test failures

Added by pespin 22 days ago. Updated 22 days ago.

Status:
Feedback
Priority:
Normal
Assignee:
Target version:
-
Start date:
01/28/2020
Due date:
% Done:

50%


Description

See for instance ttcn3-ggsn-test-latest run 467 or ttcn3-ggsn-test 928.

All tests fail with:

Verdict: fail reason: VTY Timeout for prompt

That's because osmo-ggsn is not running. It happens because osmo-ggsn is not started (there's no ggsn.log created) by the osmo-ggsn-{master,latest} docker container. It doesn't start osmo-ggsn because the docker container fail during setup:

+ docker run --cap-add=NET_ADMIN --device /dev/net/tun:/dev/net/tun --sysctl net.ipv6.conf.all.disable_ipv6=0 --rm --network ttcn3-ggsn-test --ip 172.18.3.201 -v /home/osmocom-build/jenkins/workspace/ttcn3-ggsn-test-latest/logs/ggsn:/data --name jenkins-ttcn3-ggsn-test-latest-467-ggsn -d osmocom-build/osmo-ggsn-latest /bin/sh -c osmo-ggsn -c /data/osmo-ggsn.cfg >/data/osmo-ggsn.log 2>&1
4c6702e4fed587d1c44687f7cb247d872ed4e0f5e0b37cf151b011d74367f482
docker: Error response from daemon: linux runtime spec devices: error gathering device information while adding custom device "/dev/net/tun": no such file or directory.

The error line doesn't appear when tests run fine. So docker container is really failing to start.

I could find this same error here: https://drone.spritsail.io/spritsail/iodine/4/1/5

The error line seems to suggest that /dev/net/tun doesn't exist in the host running docker, which is a bit strange because some times it does have it (tests pass) and sometimes doesn't.

The issue seems to be that when tests succeed, they are run on:

Building remotely on build2-deb9build-ansible (ttcn3 obs osmo-gsm-tester-build osmocom-gerrit-debian9 osmocom-master-debian9 coverity) in workspace /home/osmocom-build/jenkins/workspace/ttcn3-ggsn-test-latest

But when they fail, they are run on:

Building remotely on admin2-deb9build (ttcn3 obs osmo-gsm-tester-build osmocom-gerrit-debian9 osmocom-master-debian9 coverity) in workspace /home/osmocom-build/jenkins/workspace/ttcn3-ggsn-test-latest

So "build2-deb9build-ansible" vs "admin2-deb9build" . I'd assume admin2-deb9build is missing the tun kernel module (installation or modprobe).

History

#1 Updated by pespin 22 days ago

The host does have the run module loaded, but no /dev/net exists...

root@admin2-deb9build:~# lsmod | grep tun
tun                    28672  2
root@admin2-deb9build:~# ls /dev/net
ls: cannot access '/dev/net': No such file or directory

#2 Updated by pespin 22 days ago

  • Assignee changed from pespin to roh

I tried removing the module and loading it again, but it looks like a kernel upgrade took place or new kernel modules are not available?

# modprobe -r tun
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.9.0-9-amd64/modules.dep.bin'
root@admin2-deb9build:/dev# uname -a
Linux admin2-deb9build 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u2 (2019-05-13) x86_64 GNU/Linux
root@admin2-deb9build:/dev# ls /lib/modules/4.9.0-
4.9.0-5-amd64/ 4.9.0-8-amd64/

Assigning the ticket to roh since he usually takes care of upgrading the systems and similar stuff.

#3 Updated by roh 22 days ago

this is a lxc containers, so module loading inside the vm doesn't happen.

the module is loaded for the host, and should be available to the vm as well.

https://discuss.linuxcontainers.org/t/openvpn-error-cannot-open-tun-tap-dev-dev-net-tun-no-such-file-or-directory-errno-2-solved/1614/4

suggests we can add some config to make it available.

#4 Updated by roh 22 days ago

  • Status changed from New to Feedback
  • Assignee changed from roh to pespin
  • % Done changed from 0 to 50

added to the lxc vm config:

lxc.mount.entry = /dev/net/tun /var/lib/lxc/osmo-build-debian9/rootfs/dev/net/tun none bind,create=file

and the device node seems there now.

please test if it also is working/accessible or if we need to do something about permissions as well. if its working this can be closed

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)