Project

General

Profile

Support #4021

long-term end-to-end osmo-remsim test setup

Added by laforge over 1 year ago. Updated 3 months ago.

Status:
Resolved
Priority:
Immediate
Assignee:
Category:
tests
Target version:
-
Start date:
05/23/2019
Due date:
% Done:

100%

Spec Reference:

Description

let's build a setup consisting of:
  • 5-slot CCID card reader
  • sysmoQMOD with four Gobi2000 modems
    • four random SIM cards
  • APU2 with Debian9
  • one instance of remsim-server from osmocom:nightly feed running on the host
  • Debian 9 VM for remsim-bankd
    • USB port towards card reader routed to this VM
    • one instance of osmo-bankd from osmocom:nightly feed running
  • Debian 9 VM for remsim-client
    • USB port towards QMOD routed to this VM
    • four instances of osmo-remsmim from osmocom:nightly feed running

The host machine should run tc netem to simulate a non-zero latency, such as 200ms between client/server/bankd

All software should automatically start up after boot.


Checklist

  • CP2102 USB UARTs for QMOD

History

#1 Updated by roh over 1 year ago

  • assembled hardware
  • configured host system for libvirt, remsim-server, sshd
  • moved to osmocom-network segment (vlan25)
  • added cardreader

#2 Updated by roh over 1 year ago

done:
  • 2x kvm debian9 inside the debian9 on apu2 for remsimbank and remsimclient(s)
  • added nightly feeds to all debian
  • usb from acs reader routed into remsimbank vm
  • autostart vms
  • installed osmo-* daemons in respective vm
  • network segment for vms: 10.9.99.0/24 - .1 server .10 client .30 simbank
todo:
  • tc netem
  • systemd servicefiles for all daemons
  • qmod/modems
  • simcards

#3 Updated by roh over 1 year ago

done:
  • systemd servicefiles for -bankd and -server
  • qmod/modems connected - may need to move to a different usb controller (issues pushing all parts of the device to vm)

#4 Updated by roh over 1 year ago

done:
  • tested usb forwarding via complete seperate usb port - no go on apu due to hw limits.
  • swapped the vm for -client and host for -server due to usb/pci forwarding limitations (qmod needs to be on host)
  • systemd servicefile template (copy 4 times and configure for specific modems)

#5 Updated by roh over 1 year ago

done:
  • systemd servicefile for osmo-remsim-client using proper templating (no copying servicefile around anymore), just add more configs.
todo:
  • netem
  • manual testing/verification

#6 Updated by roh about 1 year ago

done:
  • updated all install/vm
  • the clients now have proper ids (-n 0-3)
  • basic netem setup (inmarsat-style 300ms / 150ms jitter / 50% corrlation / 512kBit)
    on network output of remsimserver-vm - do we need a more complicated setup here? it feels proper laggy to me.
todo:
  • real tests/load/traffic on all components (do we have a list of things to probe already?)
  • which testrunner to use?

#7 Updated by roh about 1 year ago

#8 Updated by laforge about 1 year ago

  • Priority changed from Normal to High

#9 Updated by roh about 1 year ago

i think we need a bigger ssd for this machine. it ran out of disk twice now.

it has 2 images for the vm (4g each) and 4g swap, which results in about 500mb free when i delete all old logs.
i tried adding it to the monitoring but we have another blocking issue there. (#4664)

kevin got access a few days ago, so he can help get some traffic on those simcards.

how will this work exactly? jenkins will log in via ssh and key and trigger some script?

#10 Updated by roh about 1 year ago

done:
  • swapped the simcards to sysmoUSIM.

#11 Updated by roh 11 months ago

done: * upgraded ssd to 240G, copied system, fixed uuid, resized / to 100Gb, grown swap to 16Gb, added remaining diskspace to /space

#12 Updated by laforge 10 months ago

I cannot find any mention of the IP adresss of thee system. roh please add.

#13 Updated by roh 10 months ago

its on 10.9.25.90/24

i noted it in https://osmocom.org/projects/osmo-remsim/wiki/Osmo-remsim#Test-Suite (only on the graphic)

#14 Updated by laforge 10 months ago

  • Status changed from New to In Progress
  • Assignee changed from roh to laforge
  • % Done changed from 0 to 50

I'm currently upgrading the system and the VMs to buster. We shouldn't start to deploy new systems based on oldstable, it just means we need to migrate at some point sooner than neccessary.

#15 Updated by roh 10 months ago

sorry, yes.
i think this was due to buster being released in july and the install happening beforehand.

#16 Updated by laforge 10 months ago

  • Checklist item CP2102 USB UARTs for QMOD added
  • Assignee changed from laforge to roh
  • % Done changed from 50 to 0
we should extend this testbed as follows:
  • CP2102 USB UART for ST12 of QMOD
  • CP2102 USB UART for ST34 of QMOD

#17 Updated by roh 8 months ago

  • added a usb-hub (apu only has 2 ports and both were used)
  • added 2x cp2102
  • soldered pinheader
  • added 2.5mm to pinheader adapters.

it took me quite a while to get the osmo-remsim-client working again (the usb path changed obviously) and to figure out that the cp2102 i used were broken before or have problems with the high baudrate (looks like binary)

is there any (standard distro shipped) tool to display usb-paths besides lsusb.py?

packaging is now a cardboard box to contain the cp2102 and usbhub.

for some reason i cannot edit the checklist item (grayed out).

also somebody disabled the start of osmo-remsim-client@[0-3] for reasons i do not know.

#18 Updated by laforge 8 months ago

On Wed, Jan 15, 2020 at 07:54:18PM +0000, roh [REDMINE] wrote:

is there any (standard distro shipped) tool to display usb-paths besides lsusb.py?

I don't know, but in this specific use case simtrace2-list (part of simtrace2 host tools)
should work just fine.

also somebody disabled the start of osmo-remsim-client@[0-3] for reasons i do not know.

That was me while doing some manual testing

#19 Updated by laforge 7 months ago

  • Status changed from In Progress to Stalled
  • Assignee changed from roh to laforge
  • % Done changed from 0 to 50

#20 Updated by laforge 7 months ago

  • Assignee changed from laforge to roh
  • Priority changed from High to Immediate

roh, please upgrade the system to latest dfu bootloader + cardem, and install latest nightly packages of libosmo-simtrace2 + osmo-remsim-*. Might be a good idea to try the script tsaitgaist produced (where is it? in some ticket here?) for full upgrade with multiple steps, including DFU loader upgrade via DFU. mschramm might know more?

Note: The client side package name has changed from osmo-remsim-client to osmo-remsim-client-st2

#21 Updated by laforge 7 months ago

minimum osmo-remsim software version:0.2.2.80.b797
minimum cardem/dfu firmware: 0.7.0.54-823f

#22 Updated by roh 7 months ago

done

the packages on the 3 debian machines were updated to -nightly

hardware was manually updated according to https://projects.sysmocom.de/projects/quad-modem-board/wiki/Board_Update

pushed https://gerrit.osmocom.org/c/osmo-remsim/+/17389 to fix the systemd startscripts - paramter for ip was -s and is -i now, and proper client ids

#23 Updated by tsaitgaist 7 months ago

laforge wrote:

roh, please upgrade the system to latest dfu bootloader + cardem, and install latest nightly packages of libosmo-simtrace2 + osmo-remsim-*. Might be a good idea to try the script tsaitgaist produced (where is it? in some ticket here?) for full upgrade with multiple steps, including DFU loader upgrade via DFU. mschramm might know more?

Note: The client side package name has changed from osmo-remsim-client to osmo-remsim-client-st2

the script taking care/simplifying of updating can be found here: https://git.osmocom.org/simtrace2/tree/contrib/flash.py

#24 Updated by roh 3 months ago

i think this can be closed or have i missed a feature?

#25 Updated by laforge 3 months ago

On Thu, Jun 25, 2020 at 06:07:38PM +0000, roh [REDMINE] wrote:

i think this can be closed or have i missed a feature?

it can be closed.

#26 Updated by roh 3 months ago

  • Status changed from Stalled to Resolved
  • % Done changed from 50 to 100

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)