https://osmocom.org/https://osmocom.org/favicon.ico?16647414092018-06-18T13:00:23ZOpen Source Mobile CommunicationsOsmoGSMTester - Bug #3273: osmo-bts-octphy failing to start in prod setuphttps://osmocom.org/issues/3273?journal_id=99502018-06-18T13:00:23Zdexter
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Feedback</i></li><li><strong>Assignee</strong> changed from <i>dexter</i> to <i>pespin</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>50</i></li></ul><p>I have now checked back on this problem. The DSP (both!) were crashed, so that they became unresponsive. A powercycle helped to get it working again.</p>
<p>In order to test I temporarily connected the LTHW Octbts to my test laptop. I got the exact same results as shown in the log above. So it is definitely nothing wrong with the tester.</p>
<p>I fear that this problem will come back from time to time. Both DSPs have their reset lines connected to a GPIO of the Phytec board. Those reset lines can be controlled via dsp_rst.sh, which comes pre-installed on the phytec board. The reset procedure would look like:</p>
<pre>
dsp_rst.sh on p
dsp_rst.sh off p
dsp_rst.sh on s
dsp_rst.sh off s
</pre>
<p>Where p = Primary DSP, s = Secondary DSP, on = Reset is asserted, off = Reset is de-asserted</p>
<p>My suggestion is to integrate the reset procedure in the testing process. Maybe we only should reset only when necessary. We have discussed earlier if it makes sense to reset the DSPs before each test, but we came to the conclusion that this is not wise as this is not a normal operating condition. The dsp firmware should survive multiple restarts of openbts.</p> OsmoGSMTester - Bug #3273: osmo-bts-octphy failing to start in prod setuphttps://osmocom.org/issues/3273?journal_id=99512018-06-18T13:15:32Zpespin
<ul></ul><p>Ok Thanks, let's not add the reset by default before each tests unless we see it's really necessary. Good to have it documented here, so we can come back to it if it happens again. I'll close the ticket once we have next test run passing with octphy.</p> OsmoGSMTester - Bug #3273: osmo-bts-octphy failing to start in prod setuphttps://osmocom.org/issues/3273?journal_id=100682018-06-25T19:31:04Zdexter
<ul></ul><p>I have now inspected the log files in the jenkins build artefacts (run.2018-06-25_10-03-06). The current problem seems not to be octphy related. osmo-bts-octphy fails to start because it has problems to bind the telnet/ctrl interface. I have looked into all octphy related tests. The error is the same everywhere. I do not know the environment well enough, but maybe there were some changes or there is still another instance running when osmo-bsc-octphy starts.</p>
<pre>
(launched: 2018-06-25_11:24:21.828663)
<0006> l1_if.c:770 model_init()
[0;m20180625112421929 [1;31mDLGLOBAL[0;m <0010> socket.c:360 unable to bind socket:10.42.42.52:4238: Cannot assign requested address
[0;m20180625112421930 [1;31mDLGLOBAL[0;m <0010> socket.c:371 no suitable addr found for: 10.42.42.52:4238
[0;m20180625112421930 [1;31mDLGLOBAL[0;m <0010> socket.c:360 unable to bind socket:10.42.42.52:4241: Cannot assign requested address
[0;m20180625112421930 [1;31mDLGLOBAL[0;m <0010> socket.c:371 no suitable addr found for: 10.42.42.52:4241
[0;m20180625112421930 [1;31mDLGLOBAL[0;m <0010> telnet_interface.c:100 Cannot bind telnet at 10.42.42.52 4241
[0;mError initializing telnet
</pre> OsmoGSMTester - Bug #3273: osmo-bts-octphy failing to start in prod setuphttps://osmocom.org/issues/3273?journal_id=101052018-06-27T09:30:45Zpespin
<ul><li><strong>Assignee</strong> changed from <i>pespin</i> to <i>dexter</i></li></ul><p>Previous issues fixed.</p>
<p>Current issue:</p>
<pre>
[0;m[1;36m20180627051913862 [1;34mDOML[0;m[1;36m <0001> oml.c:1001 OC=RADIO-CARRIER(02) INST=(00,00,ff) [0;m[1;36mRx OPSTART
[0;m20180627051913862 [1;32mDL1C[0;m <0006> l1_oml.c:1228 Tx APP-INFO.req
[0;m20180627051913862 [1;32mDL1C[0;m <0006> l1_oml.c:1159 Tx APP-INFO-SYSTEM.req
[0;m20180627051913862 [1;32mDL1C[0;m <0006> l1_oml.c:1364 Tx TRX-OPEN.req(trx=0, rf_port=0, arfcn=868, center=868, tsc=7, rx_gain=70, tx_atten=0)
[0;m20180627051913863 [1;32mDL1C[0;m <0006> l1_oml.c:1180 Rx APP-INFO.resp (name='octsdr_gsm', desc='Software Define Radio', ver='02.11.00-B1927', ver_hdr='02.11.00-B1927')
[0;m20180627051913864 [1;32mDL1C[0;m <0006> l1_oml.c:1128 Rx APP-INFO-SYSTEM.resp (platform='Opus2', version='OCTSYS_VERSION=01.02.26.B1;OCTODK_VERSION=01.17.05-B7;OCTADF_VERSION=04.10.01-B3387;')
[0;m20180627051913864 [1;32mDL1C[0;m <0006> l1_oml.c:1131 Note: compiled with multi-trx support.
[0;m20180627051913864 [1;32mDL1C[0;m <0006> l1_oml.c:1291 TRX-OPEN.resp(trx=0) = cOCTVC1_GSM_RC_TRX_RADIO_CONFIG
[0;m20180627051913865 [1;31mDL1C[0;m <0006> l1_oml.c:1296 TRX-OPEN failed: cOCTVC1_GSM_RC_TRX_RADIO_CONFIG
</pre> OsmoGSMTester - Bug #3273: osmo-bts-octphy failing to start in prod setuphttps://osmocom.org/issues/3273?journal_id=101082018-06-27T15:45:31Zdexter
<ul><li><strong>Assignee</strong> changed from <i>dexter</i> to <i>pespin</i></li></ul><p>I have tested the octbts locally again. I got the same error. A powercycle helped to resolve the problem. The octbts is now up and running again.</p>
<p>(I just wonder if there is a license option that limits the runtime...)</p> OsmoGSMTester - Bug #3273: osmo-bts-octphy failing to start in prod setuphttps://osmocom.org/issues/3273?journal_id=101652018-07-02T10:09:59Zpespin
<ul><li><strong>Assignee</strong> changed from <i>pespin</i> to <i>dexter</i></li></ul><p>It seems the Firmware crashed again. It basically crashes every time we run a round of tests in one jenkins job. We should investigate how/when the firmware crashes (if it's our fault) and contact OCTOBTS support to get it fixed.</p> OsmoGSMTester - Bug #3273: osmo-bts-octphy failing to start in prod setuphttps://osmocom.org/issues/3273?journal_id=102632018-07-09T08:01:28Zdexter
<ul></ul><p>Unfortunately I do not have a response from Octasic yet.</p>
<p>I think I will leave test board I have on my desktop on over night to see if the problem is even connected to any outer factors.</p>
<p>Is there a way to tell which test is causing the fault. Does it hang always at the same test or is it always different?</p> OsmoGSMTester - Bug #3273: osmo-bts-octphy failing to start in prod setuphttps://osmocom.org/issues/3273?journal_id=102662018-07-09T08:27:37Zpespin
<ul></ul><p>The way to check if a specific test is triggering the issue is to restart the firmware (having it in a working state), then running the hourly set of tests in osmo-gsm-tester, then check the logs to see the first test failing, and repeating the entire procedure some times. If it always fails in the same test, it means probably either that one or the previous one is crashing the firmware. Then manually reproducing same test can help discover more information. However, it could be that we are unable to manually trigger the issue due to timing constrains (ie. osmo-gsm-tester triggering events quickly automatically as a response to other events).</p> OsmoGSMTester - Bug #3273: osmo-bts-octphy failing to start in prod setuphttps://osmocom.org/issues/3273?journal_id=108792018-08-20T16:43:38Zdexter
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Stalled</i></li></ul> OsmoGSMTester - Bug #3273: osmo-bts-octphy failing to start in prod setuphttps://osmocom.org/issues/3273?journal_id=118072018-10-01T07:20:33Zdexter
<ul><li><strong>Status</strong> changed from <i>Stalled</i> to <i>Resolved</i></li></ul> OsmoGSMTester - Bug #3273: osmo-bts-octphy failing to start in prod setuphttps://osmocom.org/issues/3273?journal_id=118082018-10-01T07:20:44Zdexter
<ul><li><strong>Status</strong> changed from <i>Resolved</i> to <i>Rejected</i></li></ul>