Cellular Network Infrastructure » Core Network (CN) » OsmoGGSN (former OpenGGSN)

• script to create a qumu-kvm VM with Debian 9 and all required packages, plus the libmnl, libgtpnl and osmo-ggsn source code
• another script to
  • build a given [newer] Linux kernel version on the host, followed by
  • scp the resulting kernel+modules into the running VM and
  • using update-grub to activate the new kernel [only works if new version is higher]

This setup can then be used to run GGSN_Tests.ttcn against it.

laforge wrote:

I'm not working with an automatic setup yet, but what I have managed so far is:
[...]

Not sure if I will find those scripts again, I will try.

• build a very minimal kernel from a user-sepcified git tree (git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git is a good deafult)
• build a initrd containing the kernel modules, osmo-ggsn and the dependencies (I guess network:osmocom:nightly is fine)
• build a libvirt XML description for the qemu-kvm VM that can be used with virsh
• attach that VM to the same network as the ttcn3-ggsn-test conatiner
• execute the TTCN3 test suite like normal, but against the GGSN in the VM, not in the osmo-ggsn-master docker image

In this approach there would be no need for a block storage device at all. The initrd would contain everything needed.

• README file on how to build minimal kernels for execution in VM: https://gist.github.com/chrisdone/02e165a0004be33734ac2334f215380e
• ktest (https://github.com/YADRO-KNS/ktest) really sounds like it does most of what we want, is also aimed for integration with jenkins
  • seems to focus on building a single out-of-tree module rather than a full kernel
  • seems to be targeted to opensuse (not sure how much it matters)
  • has zero documentation besides https://ostconf.com/system/attachments/files/000/001/543/original/ktest.pdf?1541150201
• rapido (https://github.com/rapido-linux/rapido) - bunch of shell scripts doing most of the required steps, but meant for human user, not automatization

• video of rapido talk https://media.ccc.de/v/1416-rapido-quick-kernel-testing-from-source/related

I would expect the attached .config file to build a kernel with everything we need. It was created by starting with 'allnoconfig' and then manually enabling various features. For sure one could go even smaller than this, but I enabled features which might come in handy at some later point.

After looking at the links above and @pespin's testing scripts, this is my plan:

• put kernel testing scripts into scripts/ subdir of docker-playground.git (based on @pespin's scripts, they do pretty much all we need already)
• ttcn3-ggsn-test/jenkins.sh:
  • accept new arguments for kernel git repo and branch, and if set:
  • build the minimal kernel (have some shared code in jenkins-common.sh and/or in scripts/ subdir of docker-playground)
  • run osmo-ggsn-master or osmo-ggsn-latest docker images, depending on $IMAGE_SUFFIX as usually
  • add parameters to "docker run":
    • mount the kernel we just built into the image as file
    • mount the testing scripts subdir
    • let it run the test script as entry point, instead of osmo-ggsn directly
  • the test scripts shall do the following:
    • start qemu with the kernel and, instead of providing a separate initramfs, just use the rootfs from the docker image
      • this is how the existing scripts work
    • run osmo-ggsn with the kernel module inside qemu
    • set up networking, so the testsuite can talk to osmo-ggsn through qemu through docker, as if it was the other container where it runs without qemu
• osmo-ci.git jobs/ttcn3-testsuites.yml:
  • add jobs to test osmo-ggsn latest and master with the kernel module

osmith wrote:

• build the minimal kernel (have some shared code in jenkins-common.sh and/or in scripts/ subdir of docker-playground)

I would make this optional: Add the possibility to run directly from stock kernel zImage provided in the docker image.

So, IIUC, the summary is you extend docker image osmo-ggsn-master to run qemu (which in turn runs osmo-ggsn inside) instead of running osmo-ggsn directly in docker.
Sounds good to me. Not sure if running qemu under docker can have any issue, I'd google that first.

pespin wrote:

I would make this optional: Add the possibility to run directly from stock kernel zImage provided in the docker image.

Good idea!

So, IIUC, the summary is you extend docker image osmo-ggsn-master to run qemu (which in turn runs osmo-ggsn inside) instead of running osmo-ggsn directly in docker.

Yes.

Sounds good to me. Not sure if running qemu under docker can have any issue, I'd google that first.

From a quick search, it seems to be fine.

Proof of concept successful \o/

osmo-ggsn runs with the kernel module inside qemu inside docker.

Needs lots of polishing, and not all tests are passing yet. But ttcn3 is talking to osmo-ggsn in this setup, as I can see in the osmo-ggsn log and because 10 tests are passing.

+ cat /tmp/tmp.uhC1XcOUK7/ggsn-tester/junit-xml-20.log
<?xml version="1.0"?>
<testsuite name='GGSN_Tests' tests='26' failures='4' errors='12' skipped='0' inconc='0' time='95.00'>
  <testcase classname='GGSN_Tests' name='TC_pdp4_act_deact' time='2.397944'>
    <failure type='fail-verdict'>"VTY Timeout for prompt: enable" 
      GGSN_Tests.ttcn:1476 GGSN_Tests control part
      GGSN_Tests.ttcn:962 TC_pdp4_act_deact testcase
    </failure>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp4_act_deact_ipcp' time='0.070644'/>
  <testcase classname='GGSN_Tests' name='TC_pdp4_act_deact_ipcp_pap_broken' time='0.028127'/>
  <testcase classname='GGSN_Tests' name='TC_pdp4_act_deact_pcodns' time='0.027913'/>
  <testcase classname='GGSN_Tests' name='TC_pdp4_act_deact_gtpu_access' time='9.056108'>
    <failure type='fail-verdict'>
      GGSN_Tests.ttcn:1480 GGSN_Tests control part
      GGSN_Tests.ttcn:1073 TC_pdp4_act_deact_gtpu_access testcase
    </failure>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp4_clients_interact_with_txseq' time='3.049691'>
    <failure type='fail-verdict'>
      GGSN_Tests.ttcn:1481 GGSN_Tests control part
      GGSN_Tests.ttcn:1095 TC_pdp4_clients_interact_with_txseq testcase
    </failure>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp4_clients_interact_without_txseq' time='3.042250'>
    <failure type='fail-verdict'>
      GGSN_Tests.ttcn:1482 GGSN_Tests control part
      GGSN_Tests.ttcn:1101 TC_pdp4_clients_interact_without_txseq testcase
    </failure>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp4_act_deact_with_single_dns' time='0.043029'/>
  <testcase classname='GGSN_Tests' name='TC_pdp4_act_deact_with_separate_dns' time='0.035822'/>
  <testcase classname='GGSN_Tests' name='TC_pdp6_act_deact' time='0.027302'>
    <error type='DTE'>Dynamic test case error: Assignment of an unbound octetstring value to a template.</error>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp6_act_deact_pcodns' time='0.021316'>
    <error type='DTE'>Dynamic test case error: Using the value of an optional field containing omit.</error>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp6_act_deact_icmp6' time='0.022701'>
    <error type='DTE'>Dynamic test case error: Using the value of an optional field containing omit.</error>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp6_act_deact_gtpu_access' time='0.025706'>
    <error type='DTE'>Dynamic test case error: Using the value of an optional field containing omit.</error>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp6_clients_interact' time='0.030426'>
    <error type='DTE'>Dynamic test case error: Using the value of an optional field containing omit.</error>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp46_act_deact' time='0.030937'>
    <error type='DTE'>Dynamic test case error: Assignment of an unbound octetstring value to a template.</error>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp46_act_deact_ipcp' time='0.025506'>
    <error type='DTE'>Dynamic test case error: Using the value of an optional field containing omit.</error>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp46_act_deact_icmp6' time='0.029307'>
    <error type='DTE'>Dynamic test case error: Using the value of an optional field containing omit.</error>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp46_act_deact_pcodns4' time='0.027207'>
    <error type='DTE'>Dynamic test case error: Using the value of an optional field containing omit.</error>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp46_act_deact_pcodns6' time='0.025411'>
    <error type='DTE'>Dynamic test case error: Using the value of an optional field containing omit.</error>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp46_act_deact_gtpu_access' time='0.012181'>
    <error type='DTE'>Dynamic test case error: Using the value of an optional field containing omit.</error>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp46_clients_interact' time='0.015149'>
    <error type='DTE'>Dynamic test case error: Using the value of an optional field containing omit.</error>
  </testcase>
  <testcase classname='GGSN_Tests' name='TC_pdp46_act_deact_apn4' time='0.045058'/>
  <testcase classname='GGSN_Tests' name='TC_echo_req_resp' time='0.021418'/>
  <testcase classname='GGSN_Tests' name='TC_pdp_act2_recovery' time='0.038146'/>
  <testcase classname='GGSN_Tests' name='TC_act_deact_retrans_duplicate' time='1.044541'/>
  <testcase classname='GGSN_Tests' name='TC_pdp_act_restart_ctr_echo' time='10.040006'/>
</testsuite>

• Using the pre-built kernel from debian does not work, as it doesn't have CONFIG_NET_9P_VIRTIO etc., on which this setup relies to pass the docker rootfs to qemu
  • I've already added logic to cache the cloned linux repository. Not sure if that makes sense to keep it on jenkins, but while developing it's certainly not useful to download several GB for each run.
  • EDIT: it has CONFIG_NET_9P_VIRTIO=m, not =y, but we need =y since this is used to make the rootfs available where the kernel modules are.
• Kernel config: I've tried @laforge's config first, but it did not work with @pespin's scripts, even after I enabled some virtio options. Before wasting too much time on this, I just went with @pespin's minimal config (as shipped with the scripts) and enabled GTP there.
  • laforge: is there anything else you'd like to have in the kernel config?

WIP branch: osmith/wip-ggsn-kmod

laforge wrote:

• Using the pre-built kernel from debian does not work, as it doesn't have CONFIG_NET_9P_VIRTIO etc., on which this setup relies to pass the docker rootfs to qemu

sad. If there's another way to pass the rootfs without spending days of extra effort,
it would be worth investigating. Most users will run distribution kernels, after all.

Ah, I had assumed you would want to always build a kernel for this test. It shouldn't be too much effort to make it possible to use a pre-built kernel as well.

Ideally the git clone would --reference the cached previoous cloned repo, basically using
all the local objects whenever possible and only download the smaller incremental parts
that have modified from the remote repo.

That's exactly how it works. It creates an empty git repository, adds the given git url as remote if it does not exist, fetches it, checks out the desired branch. As we can add multiple remotes to the same git repo, it doesn't download all shared commits/objects over and over. (jenkins-common.sh:kernel_build() in the wip branch)

Once we have a working setup, I'm happy to try to strip down pespin's cofig further (if possible/needed),
the main point of this primarily is to reduce build time and not build tons of things not needed.

Sure.

laforge: is it important to you, that we use the kernel from debian's packages? Or is having a pre-built version of what we ship with the minimal configs fine as well?

In case of the latter, we could just upload the kernel as built from the minimal config to somewhere like https://ftp.osmocom.org/binaries/ and have the scripts download it from there, unless building was explicitly specified. It saves time when building the docker container (no need to install debian's kernel and let it build debian's initramfs twice (!), done automatically by apt), we don't need to maintain the extra code path to generate an initramfs for the test with at least the virtio kernel modules, and we don't need to generate that initramfs which also costs time. So all in all, it would be much faster, but at the "cost" of not having exactly the same kernel as used in debian.

https://gerrit.osmocom.org/q/topic:ttcn3-ggsn-kmod

• https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-ggsn-test-kernel/
• https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-ggsn-test-kernel-latest/
• https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-ggsn-test-kernel-net-next/

Thanks. Can you please also let me know how I can specify manually a specific git tree to test? Let's say somebody submits some patches for review. I'd want to use that person's git tree as kernel source to manually run our tests against. Or if the person has no git tree, I would apply the patches, push the resulting kernel tree to some repo somewhere and then trigger that jenkins job, specifying the git tree.

Also, please document the entire setup somewhere at an appropriate wiki page. Sorry if I may have missed that.

laforge wrote:

Thanks. Can you please also let me know how I can specify manually a specific git tree to test? Let's say somebody submits some patches for review. I'd want to use that person's git tree as kernel source to manually run our tests against. Or if the person has no git tree, I would apply the patches, push the resulting kernel tree to some repo somewhere and then trigger that jenkins job, specifying the git tree.

It's possible to set a custom git repository with KERNEL_REMOTE_NAME, KERNEL_URL, KERNEL_BRANCH environment variables. So far it was not possible to set these in the jenkins job, but I've just renamed ttcn3-ggsn-test-kernel-net-next to ttcn3-ggsn-test-kernel-git and added the parameters:

• https://jenkins.osmocom.org/jenkins/job/ttcn3-ggsn-test-kernel-git/ (see "Build with parameters")
• https://gerrit.osmocom.org/c/osmo-ci/+/23149
• https://gerrit.osmocom.org/c/osmo-ci/+/23150

• https://gerrit.osmocom.org/c/docker-playground/+/23151

Also, please document the entire setup somewhere at an appropriate wiki page. Sorry if I may have missed that.

• https://git.osmocom.org/docker-playground/tree/README.md

• https://osmocom.org/projects/cellular-infrastructure/wiki/Titan_TTCN3_Testsuites#Running-jenkinssh
• https://gerrit.osmocom.org/c/docker-playground/+/23152

laforge: anything missing in the documentation or otherwise?

EDIT: related issue #3209 is still open, I'll do that later today after fixing more urgent issues.

As discussed, this patch moves the -kernel-git job into its own file, so the parameters don't show up in the other jobs:

https://gerrit.osmocom.org/c/osmo-ci/+/23194

osmith wrote:

laforge: anything missing in the documentation or otherwise?

I was mostly thinking of a description on the (several) GGSN jenkins jobs, what they do exactly (which kernel, whcih ggsn) and how to manually specify (in the Jenkins UI) how to trigger testing a certain branch/repo.

The background for this is that I would likely give some other Linux developers access to our jenkins so they can trigger builds/tests themselves. Assume they know nothing about Osmocom or how our test suites work. They just know about the kernel GTP and want to have a certain repo/branch tested and check the results.

In terms of the jobs: I think we should add one more job that periodically (daily?) tests net-next. So in total,

• automatic debian kernel + ggsn-latest (exists)
• automatic debian kernel + ggsn-nightly (exists)
• manual user-specified kernel + ggsn-? (exists)
• automatic net-next kernel + ggsn-latest (missing)
• automatic torvalds/linux.git kernel master + ggsn-latest (missing)

Requested changes made: https://gerrit.osmocom.org/q/topic:ttcn3-ggsn-kmod+status:open

• manual user-specified kernel + ggsn-? (exists)

This was using ggsn master, but since the job is manually triggered now, I've added a parameter to switch between master and latest.

wiki page with documentation is now at Automatic_Testing

Thanks for writing the wiki page and ML announcement!

So is there anything left to do for this issue, or can we mark it as resolved?