Project

General

Profile

Feature #1849

osmo-bts-trx integration to osmo-gsm-tester

Added by laforge 5 months ago. Updated 5 months ago.

Status:
Stalled
Priority:
Normal
Assignee:
Category:
osmo-bts-trx
Target version:
Start date:
11/18/2016
Due date:
% Done:

90%

Spec Reference:

Description

we should make sure we test osmo-bts-trx just as much as we test osmo-bts-sysmo in the osmo-gsm-tester setup. The osmo-gsm-tester will have to be extended with osmo-bts-trx specifics.

Testing should probably also include testing with different hardware, such as
  • USRP B2x0
  • BladeRF

It may also make sense to test against different osmo-trx versions, but that would then rather be a feature for osmo-trx, not so much osmo-bts-trx.


Related issues

Related to Cellular Infrastructure - Feature #1850: migrate osmo-gsm-tester from sysmocom internal jenkins to public jenkins New 11/18/2016
Related to OsmoGSMTester - Feature #1980: Review osmo-gsm-tester code Resolved 03/13/2017
Blocked by OsmoTRX - Bug #1869: osmo-trx binary cannot be moved to similar CPU In Progress 12/05/2016

History

#1 Updated by laforge 5 months ago

  • Related to Feature #1850: migrate osmo-gsm-tester from sysmocom internal jenkins to public jenkins added

#2 Updated by neels 5 months ago

  • Status changed from New to In Progress
  • Assignee set to neels
  • % Done changed from 0 to 30

The preparatory builds to create the binaries to test are complete.
However, to be able to run osmo-bts-trx on the gsm-tester, our APU
probably needs to be upgraded to Debian 8, mostly for UHD.

Waiting a day for a reply to my mail (intern@sysmocom); at some point
I will simply kick off the upgrade and hope for the best.

Note that currently the jenkins builds for the gsm-tester as well as all the
hardware are internal to sysmocom, so this issue might be in the wrong scope.

#3 Updated by neels 5 months ago

neels wrote:

Note that currently the jenkins builds for the gsm-tester as well as all the
hardware are internal to sysmocom, so this issue might be in the wrong scope.

Well, in the light of #1850, the scope is spot on

#4 Updated by neels 5 months ago

  • % Done changed from 30 to 80

The gsm-tester APU has been upgraded to Debian 8 in order to accomodate osmo-bts-trx.
This caused some fallout concerning the general setup and other BTS models:

In line with the APU upgrade, the jenkins jobs to build the binaries are now also built on
the Debian8 build slave (on sysmocom's internal jenkins). To facilitate that, various
packages and the sysmo SDK had to be installed on this build slave.

The Debian8 gsm-tester APU now uses systemd scripts (found in the osmo-gsm-tester.git) to
run ofono and the gsm-tester daemon.

The jenkins-bts-sysmo.sh script building the sysmoBTS binaries for the gsm-tester was
reviewed -- it now uses the firmware headers provided by the SDK instead of manually
provided firmware headers, among other things.

All bts-model specific builds for the gsm-tester now succeed.

The openbsc build for the gsm-tester still fails, but this seems to be due to a
broken openbsc build in general. Now investigating.

#5 Updated by neels 5 months ago

(The openbsc build breakage was not specific to osmo-gsm-tester, fixed by a revert in libosmocore.
https://lists.osmocom.org/pipermail/openbsc/2016-December/009906.html )

The gsm-tester build job successfully gathers binaries and launches a gsm-tester run on the APU again.
ofono communication is also working, though in need of a tweaked systemd service file that announces
the dbus org.ofono service. This file apparently came from 'make install' in the ofono source tree??

# cat /lib/systemd/system/ofono.service 
[Unit]
Description=Telephony service
After=syslog.target

[Service]
Type=dbus
BusName=org.ofono
ExecStart=/usr/local/sbin/ofonod -n
StandardError=null

[Install]
WantedBy=multi-user.target

Completely deleted the debian7 build slave; the build job to transfer the most
recent gsm-tester python code to the APU was still run on it. Moved to Debian8.

It seems the remote gsm-tester daemon restart isn't working yet.

Also, the octphy bts fails to start, cause not yet investigated.

#6 Updated by neels 5 months ago

octphy was missing libortp, but now is failing with:

l1_oml.c:1227 Rx APP-INFO.resp (name='octsdr_gsm', desc='Software Define Radio', ver='02.05.00-B818', ver_hdr='02.07.00-B1039')
l1_oml.c:1232 Invalid header-file / dsp-firmware combination, exiting...

#7 Updated by neels 5 months ago

Upgraded the octbts DSP firmware to 02.07.00, and the octbts runs successfully now.
This is actually the first time I see an osmo-gsm-tester result light up blue,
meaning full-on success in jenkins!

So, in summary:

  • The APU has been upgraded to Debian8, to prepare for osmo-bts-trx.
  • All build jobs are now built by the Debian8 build slave.
  • All "previous" BTS models are back to successful tests on the upgraded APU:
    • nanoBTS
    • sysmoBTS
    • octBTS
  • The osmo-bts-trx binaries are built by new jenkins build jobs and already transferred to the APU.
  • STILL TODO: configure the gsm-tester python to actually run the osmo-bts-trx in a test scenario.

#8 Updated by neels 5 months ago

  • % Done changed from 80 to 90

osmo-bts-trx is added to the gsm-tester py, but still on a branch.
The point being that the APU reports that no UHD device is connected,
though last time I checked I believe to have seen a B210 (?) plugged into it.

# uhd_find_devices 
linux; GNU C++ version 4.9.2; Boost_105500; UHD_003.009.005-0-unknown

No UHD Devices Found

libuhd-dev and uhd-host as well as the firmwares are installed,
as described at https://osmocom.org/projects/osmotrx/wiki/OsmoTRX#Dependencies

FTR, osmo-trx says:

ALERT 140551759832896 19:36:06.9 UHDDevice.cpp:776:open: No UHD devices found with address ''
ALERT 140551759832896 19:36:06.9 osmo-trx.cpp:530:main: Failed to create radio device

#9 Updated by neels 5 months ago

In fact no BS210 was connected to the gsmtester prototype.

I connected one temporarily to test.

Startup worked:

  • osmo-trx starts without problems
  • osmo-bts-trx starts and connects with the osmo-nitb

Modem registering with the osmo-bts-trx times out, does not work.

A core file for osmo-trx is found, osmo-bts-trx ends with error.

20161205102906    INFO         procnurse.py:137: "NITB[0x7f077b434c10]:ProcessNurse[osmo-nitb]" Starting process "osmo-nitb" 
20161205102906    INFO         procnurse.py:148: "NITB[0x7f077b434c10]:ProcessNurse[osmo-nitb]" Started process "osmo-nitb" with PID 26020
20161205102910    INFO         procnurse.py:137: "OsmoTrxBTS[0x7f077b434bd0]:ProcessNurse[osmo-trx]" Starting process "osmo-trx" 
20161205102910    INFO         procnurse.py:148: "OsmoTrxBTS[0x7f077b434bd0]:ProcessNurse[osmo-trx]" Started process "osmo-trx" with PID 26021
20161205102914    INFO         procnurse.py:137: "OsmoTrxBTS[0x7f077b434bd0]:ProcessNurse[osmo-bts-trx]" Starting process "osmo-bts-trx" 
20161205102914    INFO         procnurse.py:148: "OsmoTrxBTS[0x7f077b434bd0]:ProcessNurse[osmo-bts-trx]" Started process "osmo-bts-trx" with PID 26032 
20161205102921    INFO            runner.py:133: "OsmoTrxBTS[0x7f077b434bd0]" Performing test scenario post-start setup
20161205102921    INFO          scenario.py:246: "OsmoTrxBTS[0x7f077b434bd0]" Checking number of BTSes connected to NITB
20161205102921    INFO          scenario.py:256: "OsmoTrxBTS[0x7f077b434bd0]" Found 1 BTS connected to NITB
20161205102921    INFO  resource_manager.py:201: "OsmoTrxBTS[0x7f077b434bd0]:SendMOSmsTestCase[0x7f077c0bdd10]" allocated Modem None
20161205102921    INFO  resource_manager.py:201: "OsmoTrxBTS[0x7f077b434bd0]:SendMOSmsTestCase[0x7f077c0bdd10]" allocated Modem None
20161205102921    INFO          scenario.py:239: "OsmoTrxBTS[0x7f077b434bd0]:SendMOSmsTestCase[0x7f077c0bdd10]" prepare HLR for NITB with subscriber data
20161205102921    INFO          scenario.py:226: "OsmoTrxBTS[0x7f077b434bd0]" Add subscriber IMSI 901700000007801 (modem /wavecom_0, ext 7801, algo comp128v1, ki D620F48487B1B782DA55DF6717F08FF9)
20161205102922    INFO          scenario.py:226: "OsmoTrxBTS[0x7f077b434bd0]" Add subscriber IMSI 901700000007802 (modem /wavecom_1, ext 7802, algo comp128v1, ki 47FDB2D55CE6A10A85ABDAD034A5B7B3)
20161205102922    INFO            runner.py:159: "OsmoTrxBTS[0x7f077b434bd0]:SendMOSmsTestCase[0x7f077c0bdd10]" Setting up test case
20161205102945    INFO              case.py:102: "OsmoTrxBTS[0x7f077b434bd0]:SendMOSmsTestCase[0x7f077c0bdd10]" Waiting for modem to register with network (with 120 seconds timeout)
20161205103148   ERROR              case.py:108: "OsmoTrxBTS[0x7f077b434bd0]:SendMOSmsTestCase[0x7f077c0bdd10]" Timeout when waiting for modems to register with network!
20161205103148   ERROR            runner.py:164: "OsmoTrxBTS[0x7f077b434bd0]:SendMOSmsTestCase[0x7f077c0bdd10]" Setting up test case failed!
20161205103148    INFO  resource_manager.py:201: "OsmoTrxBTS[0x7f077b434bd0]:SendMOSmsTestCase[0x7f077c0bdd50]" allocated Modem None
20161205103148    INFO  resource_manager.py:201: "OsmoTrxBTS[0x7f077b434bd0]:SendMOSmsTestCase[0x7f077c0bdd50]" allocated Modem None
20161205103148    INFO          scenario.py:239: "OsmoTrxBTS[0x7f077b434bd0]:SendMOSmsTestCase[0x7f077c0bdd50]" prepare HLR for NITB with subscriber data
20161205103148    INFO          scenario.py:226: "OsmoTrxBTS[0x7f077b434bd0]" Add subscriber IMSI 901700000007803 (modem /wavecom_2, ext 7803, algo comp128v1, ki ABBED4C91417DF710F60675B6EE2C8D2)
20161205103148    INFO          scenario.py:226: "OsmoTrxBTS[0x7f077b434bd0]" Add subscriber IMSI 901700000007804 (modem /wavecom_3, ext 7804, algo comp128v1, ki 8BA541179156F2BF0918CA3CFF9351B0)
20161205103148    INFO            runner.py:159: "OsmoTrxBTS[0x7f077b434bd0]:SendMOSmsTestCase[0x7f077c0bdd50]" Setting up test case
20161205103209    INFO              case.py:102: "OsmoTrxBTS[0x7f077b434bd0]:SendMOSmsTestCase[0x7f077c0bdd50]" Waiting for modem to register with network (with 120 seconds timeout)
20161205103410   ERROR              case.py:108: "OsmoTrxBTS[0x7f077b434bd0]:SendMOSmsTestCase[0x7f077c0bdd50]" Timeout when waiting for modems to register with network!
20161205103410   ERROR            runner.py:164: "OsmoTrxBTS[0x7f077b434bd0]:SendMOSmsTestCase[0x7f077c0bdd50]" Setting up test case failed!
20161205103410    INFO         procnurse.py:112: "OsmoTrxBTS[0x7f077b434bd0]:ProcessNurse[osmo-trx]" Found core file /var/tmp/osmo-gsm-tester/tmp.DE0ZLpRaLs/20161205102906-StandardTestScenario-osmoTrxBTS/tmp.DE0ZLpRaLs-osmo-bts-trx/core
20161205103410    INFO         procnurse.py:125: "OsmoTrxBTS[0x7f077b434bd0]:ProcessNurse[osmo-trx]" Renamed core file to /var/tmp/osmo-gsm-tester/tmp.DE0ZLpRaLs/20161205102906-StandardTestScenario-osmoTrxBTS/tmp.DE0ZLpRaLs-osmo-bts-trx/osmo-trx-core-26021
20161205103410   ERROR            netres.py: 99: Process "osmo-trx" exited unexpected!
20161205103410   ERROR          scenario.py:270: "OsmoTrxBTS[0x7f077b434bd0]" test scenario resource "BTS" exited unexpected with "-4"!
20161205103410   ERROR            runner.py:184: "OsmoTrxBTS[0x7f077b434bd0]" Calling tick on test scenario failed!
20161205103410    INFO            runner.py:273: "OsmoTrxBTS[0x7f077b434bd0]" Preparing to remove test scenario because state is "failure" 
20161205103410    INFO            runner.py:112: "OsmoTrxBTS[0x7f077b434bd0]" Stopping test scenario
20161205103410    INFO            netres.py: 88: "NITB[0x7f077b434c10]" Stopping process "osmo-nitb" 
20161205103410    INFO         procnurse.py:164: "NITB[0x7f077b434c10]:ProcessNurse[osmo-nitb]" Trying to stop process "osmo-nitb" 
20161205103410    INFO         procnurse.py:171: "NITB[0x7f077b434c10]:ProcessNurse[osmo-nitb]" Sending SIGTERM to 26020 ("osmo-nitb") 
20161205103410    INFO         procnurse.py:174: "NITB[0x7f077b434c10]:ProcessNurse[osmo-nitb]" Wainting for process "osmo-nitb" 
20161205103410    INFO         procnurse.py:177: "NITB[0x7f077b434c10]:ProcessNurse[osmo-nitb]" Process "osmo-nitb" has terminated
20161205103410    INFO            netres.py: 88: "OsmoTrxBTS[0x7f077b434bd0]" Stopping process "osmo-trx" 
20161205103410    INFO         procnurse.py:164: "OsmoTrxBTS[0x7f077b434bd0]:ProcessNurse[osmo-trx]" Trying to stop process "osmo-trx" 
20161205103410   ERROR         procnurse.py:180: "OsmoTrxBTS[0x7f077b434bd0]:ProcessNurse[osmo-trx]" Process 26021 ("osmo-trx") already returned with rc -4
20161205103410    INFO            netres.py: 88: "OsmoTrxBTS[0x7f077b434bd0]" Stopping process "osmo-bts-trx" 
20161205103410    INFO         procnurse.py:164: "OsmoTrxBTS[0x7f077b434bd0]:ProcessNurse[osmo-bts-trx]" Trying to stop process "osmo-bts-trx" 
20161205103410    INFO         procnurse.py:171: "OsmoTrxBTS[0x7f077b434bd0]:ProcessNurse[osmo-bts-trx]" Sending SIGTERM to 26032 ("osmo-bts-trx")
20161205103410    INFO         procnurse.py:174: "OsmoTrxBTS[0x7f077b434bd0]:ProcessNurse[osmo-bts-trx]" Wainting for process "osmo-bts-trx" 
20161205103410    INFO         procnurse.py:177: "OsmoTrxBTS[0x7f077b434bd0]:ProcessNurse[osmo-bts-trx]" Process "osmo-bts-trx" has terminated
20161205103410    INFO            runner.py:115: "OsmoTrxBTS[0x7f077b434bd0]" Removed test scenario from test runner 
20161205103410    INFO            runner.py:278: "tmp.DE0ZLpRaLs" Tearing down test session
20161205103410    INFO            runner.py: 92: "tmp.DE0ZLpRaLs" Cleaning up test session
20161205103410    INFO           session.py:188: "tmp.DE0ZLpRaLs" Tearing down test session
20161205103410    INFO           session.py:190: "OsmoTrxBTS[0x7f077b434bd0]" Tearing down test scenario
20161205103410    INFO          scenario.py:278: "OsmoTrxBTS[0x7f077b434bd0]" Tearing down test scenario dependencies
20161205103410    INFO          scenario.py:288: "OsmoTrxBTS[0x7f077b434bd0]" Tearing down NITB
20161205103410    INFO              nitb.py:303: "NITB[0x7f077b434c10]" Tearing down NITB

The core file is renamed from "core", which leaves some doubt whether it
really comes from osmo-trx. But osmo-trx also ends with rc=-4, so it's
pretty clearly osmo-bts-trx:

root@apu-roh:/var/tmp/osmo-gsm-tester/tmp.DE0ZLpRaLs/20161205102906-StandardTestScenario-osmoTrxBTS/tmp.DE0ZLpRaLs-osmo-bts-trx# LD_LIBRARY_PATH="$PWD/lib" gdb bin/osmo-trx osmo-trx-core-26021
[...]
Core was generated by `/var/tmp/osmo-gsm-tester/tmp.DE0ZLpRaLs/20161205102906-StandardTestScenario-osm'.
Program terminated with signal SIGILL, Illegal instruction.
#0  _mm_shuffle_ps (__mask=34, __B=..., __A=...) at /usr/lib/gcc/x86_64-linux-gnu/4.9/include/xmmintrin.h:743
743      return (__m128) __builtin_ia32_shufps ((__v4sf)__A, (__v4sf)__B, __mask);
(gdb) bt
#0  _mm_shuffle_ps (__mask=34, __B=..., __A=...) at /usr/lib/gcc/x86_64-linux-gnu/4.9/include/xmmintrin.h:743
#1  sse_conv_real4 (x=0xdb4e58, h=0xd5bfd0, y=0xd95020, len=41) at convolve.c:60
#2  0x0000000000446434 in convolve_real (x=0xdb4e70, x_len=41, h=0xd5bfd0, h_len=4, y=0xd95020, y_len=41, start=0, len=41, step=1, offset=0) at convolve.c:564
#3  0x000000000043dc10 in convolve (x=x@entry=0xd83500, h=h@entry=0xd8e9c0, y=0xd5b170, y@entry=0x0, spanType=spanType@entry=START_ONLY, start=start@entry=0, len=41, len@entry=0, step=1, offset=0)
    at sigProcLib.cpp:476
#4  0x000000000043edf9 in modulateBurstBasic (sps=1, guard_len=<optimized out>, bits=...) at sigProcLib.cpp:1180
#5  modulateBurst (wBurst=..., guardPeriodLength=<optimized out>, sps=1, emptyPulse=<optimized out>) at sigProcLib.cpp:1196
#6  0x000000000044241b in generateRACHSequence (sps=<optimized out>) at sigProcLib.cpp:1629
#7  sigProcLibSetup () at sigProcLib.cpp:2128
#8  0x000000000041ca04 in Transceiver::init (this=this@entry=0xd95b60, filler=1, rtsc=0, rach_delay=0, edge=<optimized out>) at Transceiver.cpp:181
#9  0x000000000040f4db in makeTransceiver (config=config@entry=0x7ffdf146aa10, radio=radio@entry=0xd82970) at osmo-trx.cpp:287
#10 0x000000000040b991 in main (argc=<optimized out>, argv=<optimized out>) at osmo-trx.cpp:540
(gdb) 

Next up: verify that libraries and UHD versions match...

#10 Updated by neels 5 months ago

At least libc and libsqlite3 had mismatching versions between build slave and APU.
Upgrading however did not have a visible effect, still getting a core dump from osmo-trx.

#11 Updated by neels 5 months ago

root@apu-roh:~# osmo-trx
linux; GNU C++ version 4.9.2; Boost_105500; UHD_003.009.005-0-unknown

opening configuration table from path :memory:
Config Settings
   Log Level............... NOTICE
   Device args............. 
   TRX Base Port........... 5700
   TRX Address............. 127.0.0.1
   Channels................ 1
   Tx Samples-per-Symbol... 4
   Rx Samples-per-Symbol... 1
   EDGE support............ Disabled
   Reference............... Internal
   C0 Filler Table......... Disabled
   Multi-Carrier........... Disabled
   Diversity............... Disabled
   Tuning offset........... 0
   RSSI to dBm offset...... 0
   Swap channels........... 0

-- Detected Device: B210
-- Operating over USB 2.
-- Initialize CODEC control...
-- Initialize Radio control...
-- Performing register loopback test... pass
-- Performing register loopback test... pass
-- Performing CODEC loopback test... pass
-- Performing CODEC loopback test... pass
-- Asking for clock rate 16.000000 MHz... 
-- Actually got clock rate 16.000000 MHz.
-- Performing timer loopback test... pass
-- Performing timer loopback test... pass
-- Setting master clock rate selection to 'automatic'.
-- Asking for clock rate 26.000000 MHz... 
-- Actually got clock rate 26.000000 MHz.
-- Performing timer loopback test... pass
-- Performing timer loopback test... pass
-- Setting B210 4/1 Tx/Rx SPS
Illegal instruction

hmm.

#12 Updated by neels 5 months ago

  • Blocked by Bug #1869: osmo-trx binary cannot be moved to similar CPU added

#13 Updated by neels 5 months ago

  • Status changed from In Progress to Stalled

#1869 blocks this because jenkins wants to build a new osmo-trx for every gsm-tester run.

We could technically instruct jenkins to build directly on the APU, hoping that the
CPU feature in question is not used, but that would be extra effort, complicating
the build and completely obsolete work when #1869 is fixed.

#14 Updated by neels about 1 month ago

Also available in: Atom PDF