Project

General

Profile

Support #4365

Create testsuite to test osmo-bts-trx+osmo-trx under high channel load

Added by pespin about 1 month ago. Updated 12 days ago.

Status:
In Progress
Priority:
Normal
Assignee:
Category:
LimeSDR
Target version:
-
Start date:
01/15/2020
Due date:
% Done:

50%

Spec Reference:

Description

The final aim of this task is to check if we run into CPU limitations of the Raspi CM3 of the LimeNet-micro when maximizing the channel load even of a single TRX.

Say, for example, 14 concurrent TCH/H with AMR inside on the 7 TS. It would be very interesting to see if that works, and if there is any margin on the CPU left, etc.

Quick way to test manually: Use osmo-bsc connected to osmo-bts-trx and use VTY command "bts <0-255> trx <0-255> timeslot <0-7> sub-slot <0-7> (activate|deactivate) (hr|fr|efr|amr) [<0-7>]"

"Automatic" testing: Use TTCN3, add test to BTS_Tests.ttcn:
As we even have that RTP source and sink you could even send some (random payload) RTP messages if you'd want, they just need to match in size and in terms of the RTP payload type. We don't care about the content of the RTP at all here.
Make sure the config/vty is configured for 7 timeslots at TCH/H and unlimited radio link timeout and then send the 14x CHAN ACT with the right channel mode (and if you want to add RTP, the IPA CRCX/MDCX).


Related issues

Related to OsmoTRX - Bug #4366: Create testsuite to test osmo-bts-trx+osmo-trx under high channel load (w/ RACH correlation)New01/15/2020

History

#1 Updated by pespin about 1 month ago

  • Related to Bug #4366: Create testsuite to test osmo-bts-trx+osmo-trx under high channel load (w/ RACH correlation) added

#2 Updated by pespin about 1 month ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 50

I started some work in osmo-ttcn3-hacks.git and docker-playground.git branch "pespin/bts-perf".

https://git.osmocom.org/osmo-ttcn3-hacks/commit/?h=pespin/bts-perf
https://git.osmocom.org/docker-playground/commit/?h=pespin/bts-perf

In there I have a passing test which uses an osmo-bsc.cfg with all channels 1..7 set to TCH/H and the TC_pespin test activates speech AMR on all of them in the BTS, waits 10 seconds, then deactivates them.

Next step is to launch the test against osmo-bts-trx+osmo-trx-lms running on my own laptop to see if there's any issue. Later, run it against same setup but running in LimeNet-micro.

#3 Updated by ipse about 1 month ago

Note, that in normal operation, osmo-bts-trx also consumes a considerable amount of CPU. Not as much as osmo-trx but still quite noticeable. But this only happens when osmo-trx produces bursts for osmo-bts-trx to process. If you just enable channels, osmo-trx will produce idle bursts (or no bursts, depending on the version and configuration), which means osmo-bts-trx won't have anything to process.

Also note that osmo-bts-trx CPU usage might depend on the uplink signal quality because it involves Viterbi decoding which iterates over a signal. I personally haven't tested this but this is the theory.

#4 Updated by pespin 12 days ago

I have a ttcn3 test opening 14 TCH/H channels from BSC side towards a bts-trx->LimeNet-micro all orchestrated with osmo-gsm-tester, and it seems to be working fine
In LimeNet-micro, top shows a load average of: 0.77, 0.78, 0.51
That's with osmo-bts-trx running outside of LimeNet-micro and without RTP processing though, but looks like there's enough load to keep it going anyway.

In osmo-gsm-tester prod main unit: jenkins ~/test/:

./run_locally.sh -s ttcn3_bts_tests:trx-lms-limenet-micro -l dbg -T -t highchanload_tchh

It worked fine for a while (20 mins) with no overruns or similar, while I could monitor it though VTY, top, etc.
I then tried to run stress-ng in the RPI CM3 while osmo-trx was running, and osmo-bts-trx quickly dropped the connection probably due to CLOCK IND arriving to late, despite osmo-trx-lms running with rt-prio 18. That's expected I'd say though because I did really put the whole system into a crazy stress which shouldn't happen in usual ways. I'll try do run with more realistic stress next times. that's what I used so far:

stress-ng --vm 4 --hdd 4 --cpu 4 --open 4 --sem 4 --sock 4 --io 2 --timer 2

https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/17098 bts: Introduce new module for performance tests

Next steps:
  • Support running osmo-bts-trx remotely by osmo-gsm-tester into RPI CM3 module (same as we do for osmo-trx-lms, run the packaged binary directly).
  • Add an RTP source/sink to the TTCN3 code to make sure some data is being sent so that osmo-bts-trx does some work (first in docker setup branch pespin/bts-perf, then in real HW with osmo-gsm-tester).

The problem is that we don't have an easy way to feed the uplink in this case afaiu, only the downlink.

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)