Project

General

Profile

Actions

Bug #3208

closed

automatic test setup for OsmoGGSN with kernel-gtp-u

Added by laforge over 3 years ago. Updated 4 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
-
Start date:
04/24/2018
Due date:
% Done:

100%

Spec Reference:
Tags:
GTP

Description

We do have some tests in GGSN_Tests.ttcn which are executed at https://jenkins.osmocom.org/jenkins/job/ttcn3-ggsn-test/

However, this runs osmo-ggsn only in userspace mode, i.e. we're not testing the interaction with kernel-gtp-u.

Let's create a setup in which we can test with [various version of the] kernel gtp-u module. I guess this would then mean we're running some qemu-kvm VM with a given kernel + distribution, in which we're running a specified version of osmo-ggsn.

Turning the VM into a jenkins build slave is not a good idea, as that would mean it's not possible to build + restart a different kernel as needed. So we should rather have something like a build slave which then (inside a jenkins job) starts the VM with OsmoGGSN, possibly after rebuilding kernel and/or osmo-ggsn.


Files

.config .config 61.2 KB proposed .config file laforge, 01/19/2021 01:50 PM

Related issues

Related to OsmoGGSN (former OpenGGSN) - Bug #3215: GGSN_Tests.ttcn GTP-U sequence number handling incompatible with kernel GTP-UResolvedlaforge04/25/2018

Actions
Related to Linux Kernel GTP-U - Feature #1943: test setup for Linux kernel GTP-U tunneling moduleNewosmith02/06/2017

Actions
Actions #1

Updated by laforge over 3 years ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 10
I'm not working with an automatic setup yet, but what I have managed so far is:
  • script to create a qumu-kvm VM with Debian 9 and all required packages, plus the libmnl, libgtpnl and osmo-ggsn source code
  • another script to
    • build a given [newer] Linux kernel version on the host, followed by
    • scp the resulting kernel+modules into the running VM and
    • using update-grub to activate the new kernel [only works if new version is higher]

This setup can then be used to run GGSN_Tests.ttcn against it.

Actions #2

Updated by laforge over 3 years ago

  • Related to Bug #3215: GGSN_Tests.ttcn GTP-U sequence number handling incompatible with kernel GTP-U added
Actions #3

Updated by laforge over 3 years ago

  • Tags set to GTP
Actions #4

Updated by laforge over 3 years ago

  • Status changed from In Progress to Stalled
Actions #5

Updated by laforge about 3 years ago

  • Priority changed from Urgent to Normal
Actions #6

Updated by laforge 10 months ago

  • Assignee changed from laforge to osmith

laforge wrote:

I'm not working with an automatic setup yet, but what I have managed so far is:
[...]

Not sure if I will find those scripts again, I will try.

Meanwhile, I'm convinced that a different approach might be more applicable:
  • build a very minimal kernel from a user-sepcified git tree (git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git is a good deafult)
  • build a initrd containing the kernel modules, osmo-ggsn and the dependencies (I guess network:osmocom:nightly is fine)
  • build a libvirt XML description for the qemu-kvm VM that can be used with virsh
  • attach that VM to the same network as the ttcn3-ggsn-test conatiner
  • execute the TTCN3 test suite like normal, but against the GGSN in the VM, not in the osmo-ggsn-master docker image

In this approach there would be no need for a block storage device at all. The initrd would contain everything needed.

Some useful links I'm currently trying to come up with a relatively minimal kernel .config that includes only what we need for running OsmoGGSN and the GTP kernel module. Will update this ticket accordingly.
Actions #7

Updated by laforge 10 months ago

I would expect the attached .config file to build a kernel with everything we need. It was created by starting with 'allnoconfig' and then manually enabling various features. For sure one could go even smaller than this, but I enabled features which might come in handy at some later point.

Actions #8

Updated by osmith 10 months ago

  • Status changed from Stalled to In Progress
Actions #9

Updated by osmith 10 months ago

  • % Done changed from 10 to 20

After looking at the links above and @pespin's testing scripts, this is my plan:

  • put kernel testing scripts into scripts/ subdir of docker-playground.git (based on @pespin's scripts, they do pretty much all we need already)
  • ttcn3-ggsn-test/jenkins.sh:
    • accept new arguments for kernel git repo and branch, and if set:
    • build the minimal kernel (have some shared code in jenkins-common.sh and/or in scripts/ subdir of docker-playground)
    • run osmo-ggsn-master or osmo-ggsn-latest docker images, depending on $IMAGE_SUFFIX as usually
    • add parameters to "docker run":
      • mount the kernel we just built into the image as file
      • mount the testing scripts subdir
      • let it run the test script as entry point, instead of osmo-ggsn directly
    • the test scripts shall do the following:
      • start qemu with the kernel and, instead of providing a separate initramfs, just use the rootfs from the docker image
        • this is how the existing scripts work
      • run osmo-ggsn with the kernel module inside qemu
      • set up networking, so the testsuite can talk to osmo-ggsn through qemu through docker, as if it was the other container where it runs without qemu
  • osmo-ci.git jobs/ttcn3-testsuites.yml:
    • add jobs to test osmo-ggsn latest and master with the kernel module
Actions #10

Updated by pespin 10 months ago

osmith wrote:

  • build the minimal kernel (have some shared code in jenkins-common.sh and/or in scripts/ subdir of docker-playground)

I would make this optional: Add the possibility to run directly from stock kernel zImage provided in the docker image.

So, IIUC, the summary is you extend docker image osmo-ggsn-master to run qemu (which in turn runs osmo-ggsn inside) instead of running osmo-ggsn directly in docker.
Sounds good to me. Not sure if running qemu under docker can have any issue, I'd google that first.

Actions #11

Updated by osmith 10 months ago

pespin wrote:

I would make this optional: Add the possibility to run directly from stock kernel zImage provided in the docker image.

Good idea!

So, IIUC, the summary is you extend docker image osmo-ggsn-master to run qemu (which in turn runs osmo-ggsn inside) instead of running osmo-ggsn directly in docker.

Yes.

Sounds good to me. Not sure if running qemu under docker can have any issue, I'd google that first.

From a quick search, it seems to be fine.

Actions #12

Updated by osmith 10 months ago

  • % Done changed from 20 to 40

Proof of concept successful \o/

osmo-ggsn runs with the kernel module inside qemu inside docker.

Needs lots of polishing, and not all tests are passing yet. But ttcn3 is talking to osmo-ggsn in this setup, as I can see in the osmo-ggsn log and because 10 tests are passing.

+ cat /tmp/tmp.uhC1XcOUK7/ggsn-tester/junit-xml-20.log
<?xml version="1.0"?>
<testsuite name='GGSN_Tests' tests='26' failures='4' errors='12' skipped='0' inconc='0' time='95.00'>
  <testcase classname='GGSN_Tests' name='TC_pdp4_act_deact' time='2.397944'>
    <failure type='fail-verdict'>"VTY Timeout for prompt: enable" 
      GGSN_Tests.ttcn:1476 GGSN_Tests control part
      GGSN_Tests.ttcn:962 TC_pdp4_act_deact testcase
    </failure>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp4_act_deact_ipcp' time='0.070644'/>
  <testcase classname='GGSN_Tests' name='TC_pdp4_act_deact_ipcp_pap_broken' time='0.028127'/>
  <testcase classname='GGSN_Tests' name='TC_pdp4_act_deact_pcodns' time='0.027913'/>
  <testcase classname='GGSN_Tests' name='TC_pdp4_act_deact_gtpu_access' time='9.056108'>
    <failure type='fail-verdict'>
      GGSN_Tests.ttcn:1480 GGSN_Tests control part
      GGSN_Tests.ttcn:1073 TC_pdp4_act_deact_gtpu_access testcase
    </failure>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp4_clients_interact_with_txseq' time='3.049691'>
    <failure type='fail-verdict'>
      GGSN_Tests.ttcn:1481 GGSN_Tests control part
      GGSN_Tests.ttcn:1095 TC_pdp4_clients_interact_with_txseq testcase
    </failure>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp4_clients_interact_without_txseq' time='3.042250'>
    <failure type='fail-verdict'>
      GGSN_Tests.ttcn:1482 GGSN_Tests control part
      GGSN_Tests.ttcn:1101 TC_pdp4_clients_interact_without_txseq testcase
    </failure>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp4_act_deact_with_single_dns' time='0.043029'/>
  <testcase classname='GGSN_Tests' name='TC_pdp4_act_deact_with_separate_dns' time='0.035822'/>
  <testcase classname='GGSN_Tests' name='TC_pdp6_act_deact' time='0.027302'>
    <error type='DTE'>Dynamic test case error: Assignment of an unbound octetstring value to a template.</error>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp6_act_deact_pcodns' time='0.021316'>
    <error type='DTE'>Dynamic test case error: Using the value of an optional field containing omit.</error>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp6_act_deact_icmp6' time='0.022701'>
    <error type='DTE'>Dynamic test case error: Using the value of an optional field containing omit.</error>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp6_act_deact_gtpu_access' time='0.025706'>
    <error type='DTE'>Dynamic test case error: Using the value of an optional field containing omit.</error>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp6_clients_interact' time='0.030426'>
    <error type='DTE'>Dynamic test case error: Using the value of an optional field containing omit.</error>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp46_act_deact' time='0.030937'>
    <error type='DTE'>Dynamic test case error: Assignment of an unbound octetstring value to a template.</error>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp46_act_deact_ipcp' time='0.025506'>
    <error type='DTE'>Dynamic test case error: Using the value of an optional field containing omit.</error>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp46_act_deact_icmp6' time='0.029307'>
    <error type='DTE'>Dynamic test case error: Using the value of an optional field containing omit.</error>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp46_act_deact_pcodns4' time='0.027207'>
    <error type='DTE'>Dynamic test case error: Using the value of an optional field containing omit.</error>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp46_act_deact_pcodns6' time='0.025411'>
    <error type='DTE'>Dynamic test case error: Using the value of an optional field containing omit.</error>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp46_act_deact_gtpu_access' time='0.012181'>
    <error type='DTE'>Dynamic test case error: Using the value of an optional field containing omit.</error>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp46_clients_interact' time='0.015149'>
    <error type='DTE'>Dynamic test case error: Using the value of an optional field containing omit.</error>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp46_act_deact_apn4' time='0.045058'/>
  <testcase classname='GGSN_Tests' name='TC_echo_req_resp' time='0.021418'/>
  <testcase classname='GGSN_Tests' name='TC_pdp_act2_recovery' time='0.038146'/>
  <testcase classname='GGSN_Tests' name='TC_act_deact_retrans_duplicate' time='1.044541'/>
  <testcase classname='GGSN_Tests' name='TC_pdp_act_restart_ctr_echo' time='10.040006'/>
</testsuite>
Some notes:
  • Using the pre-built kernel from debian does not work, as it doesn't have CONFIG_NET_9P_VIRTIO etc., on which this setup relies to pass the docker rootfs to qemu
    • I've already added logic to cache the cloned linux repository. Not sure if that makes sense to keep it on jenkins, but while developing it's certainly not useful to download several GB for each run.
    • EDIT: it has CONFIG_NET_9P_VIRTIO=m, not =y, but we need =y since this is used to make the rootfs available where the kernel modules are.
  • Kernel config: I've tried @laforge's config first, but it did not work with @pespin's scripts, even after I enabled some virtio options. Before wasting too much time on this, I just went with @pespin's minimal config (as shipped with the scripts) and enabled GTP there.
    • laforge: is there anything else you'd like to have in the kernel config?

WIP branch: osmith/wip-ggsn-kmod

Actions #13

Updated by laforge 10 months ago

Hi oliver,

On Fri, Feb 05, 2021 at 12:42:52PM +0000, wrote:

Proof of concept successful \o/

great.

  • Using the pre-built kernel from debian does not work, as it doesn't have CONFIG_NET_9P_VIRTIO etc., on which this setup relies to pass the docker rootfs to qemu

sad. If there's another way to pass the rootfs without spending days of extra effort,
it would be worth investigating. Most users will run distribution kernels, after all.

  • I've already added logic to cache the cloned linux repository. Not sure if that makes sense to keep it on jenkins, but while developing it's certainly not useful to download several GB for each run.

Does it cache the specific repo and invalidate it when the URL changes? I would expect
the job is typically run with different repos / branches at each run, testing code that
different people have submitted.

Ideally the git clone would --reference the cached previoous cloned repo, basically using
all the local objects whenever possible and only download the smaller incremental parts
that have modified from the remote repo.

  • Kernel config: I've tried @laforge's config first, but it did not work with @pespin's scripts, even after I enabled some virtio options. Before wasting too much time on this, I just went with @pespin's minimal config (as shipped with the scripts) and enabled GTP there.

Once we have a working setup, I'm happy to try to strip down pespin's cofig further (if possible/needed),
the main point of this primarily is to reduce build time and not build tons of things not needed.

Regards,
Harald

Actions #14

Updated by osmith 10 months ago

laforge wrote:

  • Using the pre-built kernel from debian does not work, as it doesn't have CONFIG_NET_9P_VIRTIO etc., on which this setup relies to pass the docker rootfs to qemu

sad. If there's another way to pass the rootfs without spending days of extra effort,
it would be worth investigating. Most users will run distribution kernels, after all.

Ah, I had assumed you would want to always build a kernel for this test. It shouldn't be too much effort to make it possible to use a pre-built kernel as well.

Ideally the git clone would --reference the cached previoous cloned repo, basically using
all the local objects whenever possible and only download the smaller incremental parts
that have modified from the remote repo.

That's exactly how it works. It creates an empty git repository, adds the given git url as remote if it does not exist, fetches it, checks out the desired branch. As we can add multiple remotes to the same git repo, it doesn't download all shared commits/objects over and over. (jenkins-common.sh:kernel_build() in the wip branch)

Once we have a working setup, I'm happy to try to strip down pespin's cofig further (if possible/needed),
the main point of this primarily is to reduce build time and not build tons of things not needed.

Sure.

Actions #15

Updated by laforge 10 months ago

  • Related to Feature #1943: test setup for Linux kernel GTP-U tunneling module added
Actions #16

Updated by osmith 10 months ago

laforge: is it important to you, that we use the kernel from debian's packages? Or is having a pre-built version of what we ship with the minimal configs fine as well?

In case of the latter, we could just upload the kernel as built from the minimal config to somewhere like https://ftp.osmocom.org/binaries/ and have the scripts download it from there, unless building was explicitly specified. It saves time when building the docker container (no need to install debian's kernel and let it build debian's initramfs twice (!), done automatically by apt), we don't need to maintain the extra code path to generate an initramfs for the test with at least the virtio kernel modules, and we don't need to generate that initramfs which also costs time. So all in all, it would be much faster, but at the "cost" of not having exactly the same kernel as used in debian.

Actions #17

Updated by laforge 10 months ago

On Mon, Feb 08, 2021 at 12:50:04PM +0000, wrote:

is it important to you, that we use the kernel from debian's packages? Or is having a pre-built version of what we ship with the minimal configs fine as well?

As I mentioned, I think 99.9% of our users will use the distribution packaged kernel.

There are two use cases for running those tests:

a) ensure distribution kernels work as expected
b) ensure various git trees (like net-next.git) with patches from ongoing kernel development
still work as expected

Those could very well be two different jobs or dockerfiles, if that solves the speed problem?

Actions #18

Updated by osmith 9 months ago

  • % Done changed from 40 to 80
Actions #20

Updated by osmith 9 months ago

  • Status changed from In Progress to Resolved
  • % Done changed from 90 to 100
Actions #21

Updated by laforge 9 months ago

  • Status changed from Resolved to In Progress
  • % Done changed from 100 to 90

Thanks. Can you please also let me know how I can specify manually a specific git tree to test? Let's say somebody submits some patches for review. I'd want to use that person's git tree as kernel source to manually run our tests against. Or if the person has no git tree, I would apply the patches, push the resulting kernel tree to some repo somewhere and then trigger that jenkins job, specifying the git tree.

Also, please document the entire setup somewhere at an appropriate wiki page. Sorry if I may have missed that.

Actions #22

Updated by osmith 9 months ago

laforge wrote:

Thanks. Can you please also let me know how I can specify manually a specific git tree to test? Let's say somebody submits some patches for review. I'd want to use that person's git tree as kernel source to manually run our tests against. Or if the person has no git tree, I would apply the patches, push the resulting kernel tree to some repo somewhere and then trigger that jenkins job, specifying the git tree.

It's possible to set a custom git repository with KERNEL_REMOTE_NAME, KERNEL_URL, KERNEL_BRANCH environment variables. So far it was not possible to set these in the jenkins job, but I've just renamed ttcn3-ggsn-test-kernel-net-next to ttcn3-ggsn-test-kernel-git and added the parameters:

New safety check to make sure KERNEL_REMOTE_NAME is not used with two different KERNEL_URL:

Also, please document the entire setup somewhere at an appropriate wiki page. Sorry if I may have missed that.

The setup and all related environment variables are described in the "Kernel test" section of the README.md: I've updated the Titan TTCN3 Testsuites wiki page to reference README.md for env vars and the kernel module test, so we don't need to keep the information up to date in two locations:

laforge: anything missing in the documentation or otherwise?

EDIT: related issue #3209 is still open, I'll do that later today after fixing more urgent issues.

Actions #23

Updated by osmith 9 months ago

As discussed, this patch moves the -kernel-git job into its own file, so the parameters don't show up in the other jobs:

https://gerrit.osmocom.org/c/osmo-ci/+/23194

Actions #24

Updated by laforge 9 months ago

osmith wrote:

laforge: anything missing in the documentation or otherwise?

I was mostly thinking of a description on the (several) GGSN jenkins jobs, what they do exactly (which kernel, whcih ggsn) and how to manually specify (in the Jenkins UI) how to trigger testing a certain branch/repo.

The background for this is that I would likely give some other Linux developers access to our jenkins so they can trigger builds/tests themselves. Assume they know nothing about Osmocom or how our test suites work. They just know about the kernel GTP and want to have a certain repo/branch tested and check the results.

In terms of the jobs: I think we should add one more job that periodically (daily?) tests net-next. So in total,

  • automatic debian kernel + ggsn-latest (exists)
  • automatic debian kernel + ggsn-nightly (exists)
  • manual user-specified kernel + ggsn-? (exists)
  • automatic net-next kernel + ggsn-latest (missing)
  • automatic torvalds/linux.git kernel master + ggsn-latest (missing)
Actions #25

Updated by osmith 9 months ago

Requested changes made: https://gerrit.osmocom.org/q/topic:ttcn3-ggsn-kmod+status:open

  • manual user-specified kernel + ggsn-? (exists)

This was using ggsn master, but since the job is manually triggered now, I've added a parameter to switch between master and latest.

Actions #26

Updated by laforge 9 months ago

wiki page with documentation is now at Automatic_Testing

Actions #27

Updated by osmith 8 months ago

Thanks for writing the wiki page and ML announcement!

So is there anything left to do for this issue, or can we mark it as resolved?

Actions #28

Updated by osmith 4 months ago

  • Status changed from In Progress to Resolved
  • % Done changed from 90 to 100
Actions

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)