Project

General

Profile

Actions

Support #4021

closed

long-term end-to-end osmo-remsim test setup

Added by laforge almost 5 years ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Immediate
Assignee:
Category:
tests
Target version:
-
Start date:
05/23/2019
Due date:
% Done:

100%

Spec Reference:

Description

let's build a setup consisting of:
  • 5-slot CCID card reader
  • sysmoQMOD with four Gobi2000 modems
    • four random SIM cards
  • APU2 with Debian9
  • one instance of remsim-server from osmocom:nightly feed running on the host
  • Debian 9 VM for remsim-bankd
    • USB port towards card reader routed to this VM
    • one instance of osmo-bankd from osmocom:nightly feed running
  • Debian 9 VM for remsim-client
    • USB port towards QMOD routed to this VM
    • four instances of osmo-remsmim from osmocom:nightly feed running

The host machine should run tc netem to simulate a non-zero latency, such as 200ms between client/server/bankd

All software should automatically start up after boot.


Checklist

  • CP2102 USB UARTs for QMOD
Actions #1

Updated by roh almost 5 years ago

  • assembled hardware
  • configured host system for libvirt, remsim-server, sshd
  • moved to osmocom-network segment (vlan25)
  • added cardreader
Actions #2

Updated by roh almost 5 years ago

done:
  • 2x kvm debian9 inside the debian9 on apu2 for remsimbank and remsimclient(s)
  • added nightly feeds to all debian
  • usb from acs reader routed into remsimbank vm
  • autostart vms
  • installed osmo-* daemons in respective vm
  • network segment for vms: 10.9.99.0/24 - .1 server .10 client .30 simbank
todo:
  • tc netem
  • systemd servicefiles for all daemons
  • qmod/modems
  • simcards
Actions #3

Updated by roh almost 5 years ago

done:
  • systemd servicefiles for -bankd and -server
  • qmod/modems connected - may need to move to a different usb controller (issues pushing all parts of the device to vm)
Actions #4

Updated by roh almost 5 years ago

done:
  • tested usb forwarding via complete seperate usb port - no go on apu due to hw limits.
  • swapped the vm for -client and host for -server due to usb/pci forwarding limitations (qmod needs to be on host)
  • systemd servicefile template (copy 4 times and configure for specific modems)
Actions #5

Updated by roh almost 5 years ago

done:
  • systemd servicefile for osmo-remsim-client using proper templating (no copying servicefile around anymore), just add more configs.
todo:
  • netem
  • manual testing/verification
Actions #6

Updated by roh over 4 years ago

done:
  • updated all install/vm
  • the clients now have proper ids (-n 0-3)
  • basic netem setup (inmarsat-style 300ms / 150ms jitter / 50% corrlation / 512kBit)
    on network output of remsimserver-vm - do we need a more complicated setup here? it feels proper laggy to me.
todo:
  • real tests/load/traffic on all components (do we have a list of things to probe already?)
  • which testrunner to use?
Actions #7

Updated by roh over 4 years ago

Actions #8

Updated by laforge over 4 years ago

  • Priority changed from Normal to High
Actions #9

Updated by roh over 4 years ago

i think we need a bigger ssd for this machine. it ran out of disk twice now.

it has 2 images for the vm (4g each) and 4g swap, which results in about 500mb free when i delete all old logs.
i tried adding it to the monitoring but we have another blocking issue there. (#4664)

kevin got access a few days ago, so he can help get some traffic on those simcards.

how will this work exactly? jenkins will log in via ssh and key and trigger some script?

Actions #10

Updated by roh over 4 years ago

done:
  • swapped the simcards to sysmoUSIM.
Actions #11

Updated by roh over 4 years ago

done: * upgraded ssd to 240G, copied system, fixed uuid, resized / to 100Gb, grown swap to 16Gb, added remaining diskspace to /space

Actions #12

Updated by laforge over 4 years ago

I cannot find any mention of the IP adresss of thee system. roh please add.

Actions #13

Updated by roh over 4 years ago

its on 10.9.25.90/24

i noted it in https://osmocom.org/projects/osmo-remsim/wiki/Osmo-remsim#Test-Suite (only on the graphic)

Actions #14

Updated by laforge over 4 years ago

  • Status changed from New to In Progress
  • Assignee changed from roh to laforge
  • % Done changed from 0 to 50

I'm currently upgrading the system and the VMs to buster. We shouldn't start to deploy new systems based on oldstable, it just means we need to migrate at some point sooner than neccessary.

Actions #15

Updated by roh over 4 years ago

sorry, yes.
i think this was due to buster being released in july and the install happening beforehand.

Actions #16

Updated by laforge over 4 years ago

  • Checklist item CP2102 USB UARTs for QMOD added
  • Assignee changed from laforge to roh
  • % Done changed from 50 to 0
we should extend this testbed as follows:
  • CP2102 USB UART for ST12 of QMOD
  • CP2102 USB UART for ST34 of QMOD
Actions #17

Updated by roh about 4 years ago

  • added a usb-hub (apu only has 2 ports and both were used)
  • added 2x cp2102
  • soldered pinheader
  • added 2.5mm to pinheader adapters.

it took me quite a while to get the osmo-remsim-client working again (the usb path changed obviously) and to figure out that the cp2102 i used were broken before or have problems with the high baudrate (looks like binary)

is there any (standard distro shipped) tool to display usb-paths besides lsusb.py?

packaging is now a cardboard box to contain the cp2102 and usbhub.

for some reason i cannot edit the checklist item (grayed out).

also somebody disabled the start of osmo-remsim-client@[0-3] for reasons i do not know.

Actions #18

Updated by laforge about 4 years ago

On Wed, Jan 15, 2020 at 07:54:18PM +0000, roh [REDMINE] wrote:

is there any (standard distro shipped) tool to display usb-paths besides lsusb.py?

I don't know, but in this specific use case simtrace2-list (part of simtrace2 host tools)
should work just fine.

also somebody disabled the start of osmo-remsim-client@[0-3] for reasons i do not know.

That was me while doing some manual testing

Actions #19

Updated by laforge about 4 years ago

  • Status changed from In Progress to Stalled
  • Assignee changed from roh to laforge
  • % Done changed from 0 to 50
Actions #20

Updated by laforge about 4 years ago

  • Assignee changed from laforge to roh
  • Priority changed from High to Immediate

roh, please upgrade the system to latest dfu bootloader + cardem, and install latest nightly packages of libosmo-simtrace2 + osmo-remsim-*. Might be a good idea to try the script tsaitgaist produced (where is it? in some ticket here?) for full upgrade with multiple steps, including DFU loader upgrade via DFU. mschramm might know more?

Note: The client side package name has changed from osmo-remsim-client to osmo-remsim-client-st2

Actions #21

Updated by laforge about 4 years ago

minimum osmo-remsim software version:0.2.2.80.b797
minimum cardem/dfu firmware: 0.7.0.54-823f

Actions #22

Updated by roh about 4 years ago

done

the packages on the 3 debian machines were updated to -nightly

hardware was manually updated according to https://projects.sysmocom.de/projects/quad-modem-board/wiki/Board_Update

pushed https://gerrit.osmocom.org/c/osmo-remsim/+/17389 to fix the systemd startscripts - paramter for ip was -s and is -i now, and proper client ids

Actions #23

Updated by tsaitgaist about 4 years ago

laforge wrote:

roh, please upgrade the system to latest dfu bootloader + cardem, and install latest nightly packages of libosmo-simtrace2 + osmo-remsim-*. Might be a good idea to try the script tsaitgaist produced (where is it? in some ticket here?) for full upgrade with multiple steps, including DFU loader upgrade via DFU. mschramm might know more?

Note: The client side package name has changed from osmo-remsim-client to osmo-remsim-client-st2

the script taking care/simplifying of updating can be found here: https://git.osmocom.org/simtrace2/tree/contrib/flash.py

Actions #24

Updated by roh over 3 years ago

i think this can be closed or have i missed a feature?

Actions #25

Updated by laforge over 3 years ago

On Thu, Jun 25, 2020 at 06:07:38PM +0000, roh [REDMINE] wrote:

i think this can be closed or have i missed a feature?

it can be closed.

Actions #26

Updated by roh over 3 years ago

  • Status changed from Stalled to Resolved
  • % Done changed from 50 to 100
Actions #27

Updated by mschramm about 1 year ago

Actions

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)