Cellular Network Infrastructure » Radio Access Network » OsmoBSC

bsc-e1.pcap starts shortly after bring up, to all chans being in BORKEN state

Nack filter: gsm_abis_rsl.msg_type == 0x23

osmo-bsc Config.

For what it may be worth, simply transitioning to WAIT_RF_RELEASE_ACK instead of BORKEN does seem to eventually return the state to UNUSED and allows further use of the channels.

--- a/src/osmo-bsc/lchan_fsm.c
+++ b/src/osmo-bsc/lchan_fsm.c
@@ -730,7 +730,7 @@ static void lchan_fsm_wait_activ_ack(struct osmo_fsm_inst *fi, uint32_t event, v
                        lchan->release.rr_cause = bsc_gsm48_rr_cause_from_rsl_cause(lchan->release.rsl_error_cause);
                        lchan->release.in_error = true;
                        if (lchan->release.rsl_error_cause != RSL_ERR_RCH_ALR_ACTV_ALLOC)
-                               next_state = LCHAN_ST_BORKEN;
+                               next_state = LCHAN_ST_WAIT_RF_RELEASE_ACK;
                        else
                                /* Taking this over from legacy code: send an RF Chan Release even though
                                 * the Activ was NACKed. Is this really correct? */

I had a look at the capture file, and I think I know what happens:

• from time to time the BSC gets CHANnel ReQuireD with Access Delay 65 (!)
  • the usual range is 0 .. 63, and 65 * 550 m == 35.8 km is quite far...
  • AFAIR, values exceeding this range are allowed for GSM-450 only
• osmo-bsc tries to allocate a channel by sending RSL CHANnel ACTIVation message
  • this message contains the Timing Advance IE with the expected delay
  • apparently, your BTS is not happy about such a huge delay
• the BTS rejects RSL CHANnel ACTIVation by sending NACK
• osmo-bsc gets confused and treats the timeslot as 'BROKEN'

In osmo-bts we have configurable filtering for the delay / quality of the Access and Normal bursts, so 'outsiders' never reach the BSC. And I guess you need to configure / implement something similar for your Ericsson BTS. I have no idea if this BTS itself can do filtering, but in the worst case we can implement it in osmo-bsc.

Feel free to cherry-pick this patch and give it a try:

https://git.osmocom.org/osmo-bsc/commit/?h=fixeria/rach&id=f2d3cddef982c0c8592b9b4e5ef4089a1910ea3c

Nice, I tested your patch adding a log line and certainly can see it happening from time to time, however, changing the MNC and bringing up the network again with 1,000s of phones then attempting to LUR, I still end up with most if not all BORKEN SDCCH after a few minutes.

What other information would help?

fixeria wrote:

I had a look at the capture file, and I think I know what happens:

thanks for the debugging.

• from time to time the BSC gets CHANnel ReQuireD with Access Delay 65 (!)
  • the usual range is 0 .. 63, and 65 * 550 m == 35.8 km is quite far...
  • AFAIR, values exceeding this range are allowed for GSM-450 only
• osmo-bsc tries to allocate a channel by sending RSL CHANnel ACTIVation message
  • this message contains the Timing Advance IE with the expected delay
  • apparently, your BTS is not happy about such a huge delay

I think the assignment of "extended range" timing advance is something that likely requires additional [proprietary] siganling. We don't need to bother as we don't want to implement it.

• the BTS rejects RSL CHANnel ACTIVation by sending NACK
• osmo-bsc gets confused and treats the timeslot as 'BROKEN'

osmo-bsc should for sure not get confused by a CHAN ACT NACK, is there a separate bug here?

In osmo-bts we have configurable filtering for the delay / quality of the Access and Normal bursts, so 'outsiders' never reach the BSC. And I guess you need to configure / implement something similar for your Ericsson BTS. I have no idea if this BTS itself can do filtering, but in the worst case we can implement it in osmo-bsc.

I think osmo-bsc should do a proper check, let's call it "input validation". It should be done generically so it will work for any type of BTS.

keith wrote:

What other information would help?

if this fix worked as expected, the log+pcap should now look quite different. please update, thanks!

So looking at the original pcap again, I see that, as you rightly spotted, all those Nacks are indeed the results of TA > 63 - so looks like that patch is a runner. Should I prepare this, with associated vty config? rach-max-ta or some such?

Adding a pcap of bringing the network up with a new MNC..

I left it run a little longer just now, and interestingly it's only on TRX 0 that everything gets b0rked:

OsmoBSC# show lchan summary
BTS 0, TRX 0, Timeslot 1 SDCCH8, Lchan 0, Type NONE, State BORKEN - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 0, Timeslot 1 SDCCH8, Lchan 1, Type NONE, State BORKEN - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 0, Timeslot 1 SDCCH8, Lchan 2, Type NONE, State BORKEN - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 0, Timeslot 1 SDCCH8, Lchan 3, Type NONE, State BORKEN - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 0, Timeslot 1 SDCCH8, Lchan 4, Type NONE, State BORKEN - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 0, Timeslot 1 SDCCH8, Lchan 5, Type SDCCH, State WAIT_ACTIV_ACK - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 0, Timeslot 1 SDCCH8, Lchan 6, Type NONE, State BORKEN - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 0, Timeslot 1 SDCCH8, Lchan 7, Type NONE, State BORKEN - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 0, Timeslot 2 SDCCH8, Lchan 0, Type NONE, State BORKEN - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 0, Timeslot 2 SDCCH8, Lchan 1, Type NONE, State BORKEN - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 0, Timeslot 2 SDCCH8, Lchan 2, Type NONE, State BORKEN - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 0, Timeslot 2 SDCCH8, Lchan 3, Type NONE, State BORKEN - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 0, Timeslot 2 SDCCH8, Lchan 4, Type NONE, State BORKEN - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 0, Timeslot 2 SDCCH8, Lchan 5, Type NONE, State BORKEN - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 0, Timeslot 2 SDCCH8, Lchan 6, Type NONE, State BORKEN - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 0, Timeslot 2 SDCCH8, Lchan 7, Type NONE, State BORKEN - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 0, Timeslot 3 TCH/F, Lchan 0, Type NONE, State BORKEN - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 0, Timeslot 4 TCH/F, Lchan 0, Type NONE, State BORKEN - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 0, Timeslot 5 TCH/F, Lchan 0, Type NONE, State BORKEN - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 0, Timeslot 6 TCH/F, Lchan 0, Type NONE, State BORKEN - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 0, Timeslot 7 TCH/F, Lchan 0, Type NONE, State BORKEN - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 1, Timeslot 0 SDCCH8, Lchan 0, Type SDCCH, State WAIT_RLL_RTP_ESTABLISH - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 1, Timeslot 0 SDCCH8, Lchan 1, Type SDCCH, State WAIT_AFTER_ERROR - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 1, Timeslot 0 SDCCH8, Lchan 2, Type SDCCH, State WAIT_RLL_RTP_ESTABLISH - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 1, Timeslot 0 SDCCH8, Lchan 3, Type SDCCH, State WAIT_RLL_RTP_ESTABLISH - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 1, Timeslot 0 SDCCH8, Lchan 4, Type SDCCH, State WAIT_AFTER_ERROR - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 1, Timeslot 0 SDCCH8, Lchan 5, Type SDCCH, State WAIT_RLL_RTP_ESTABLISH - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 1, Timeslot 0 SDCCH8, Lchan 6, Type SDCCH, State WAIT_AFTER_ERROR - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 1, Timeslot 0 SDCCH8, Lchan 7, Type SDCCH, State WAIT_RLL_RTP_ESTABLISH - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 1, Timeslot 1 SDCCH8, Lchan 0, Type SDCCH, State WAIT_RLL_RTP_ESTABLISH - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 1, Timeslot 1 SDCCH8, Lchan 1, Type SDCCH, State WAIT_RLL_RTP_ESTABLISH - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 1, Timeslot 1 SDCCH8, Lchan 2, Type SDCCH, State WAIT_RLL_RTP_ESTABLISH - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 1, Timeslot 1 SDCCH8, Lchan 3, Type SDCCH, State WAIT_AFTER_ERROR - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 1, Timeslot 1 SDCCH8, Lchan 4, Type SDCCH, State WAIT_RLL_RTP_ESTABLISH - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 1, Timeslot 1 SDCCH8, Lchan 5, Type SDCCH, State WAIT_RLL_RTP_ESTABLISH - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 1, Timeslot 1 SDCCH8, Lchan 6, Type SDCCH, State WAIT_RLL_RTP_ESTABLISH - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 1, Timeslot 1 SDCCH8, Lchan 7, Type SDCCH, State WAIT_AFTER_ERROR - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 1, Timeslot 2 TCH/F, Lchan 0, Type TCH_F, State WAIT_RLL_RTP_ESTABLISH - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 1, Timeslot 3 TCH/F, Lchan 0, Type TCH_F, State WAIT_RLL_RTP_ESTABLISH - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 1, Timeslot 4 TCH/F, Lchan 0, Type TCH_F, State WAIT_RLL_RTP_ESTABLISH - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 1, Timeslot 5 TCH/F, Lchan 0, Type TCH_F, State WAIT_RLL_RTP_ESTABLISH - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 1, Timeslot 6 TCH/F, Lchan 0, Type TCH_F, State WAIT_RLL_RTP_ESTABLISH - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm
BTS 0, TRX 1, Timeslot 7 TCH/F, Lchan 0, Type TCH_F, State WAIT_RLL_RTP_ESTABLISH - L1 MS Power: 0 dBm RXL-FULL-dl: -110 dBm RXL-FULL-ul: -110 dBm

I also see chans actually leaving the and re-entering the borken state, which I thought was not possible. I guess full osmo-bsc log would be a useful thing here. I'm not sure how to deal with the lack of gsmtap logs embedded in the pcap.

Adding a DEBUG level osmo-bsc.log, Maybe grepping on it might help. I just let it run until it was about 100M

[Note this log does not correspond directly to lur-bomb.pcap]

Hi keith,

keith wrote:

So looking at the original pcap again, I see that, as you rightly spotted, all those Nacks are indeed the results of TA > 63 - so looks like that patch is a runner. Should I prepare this, with associated vty config?

yes, would definitely be nice to get this patch finished and merged.

rach-max-ta or some such?

I think "rach max-delay" would be better. In case of the Access Burst we're dealing with delay (in symbol periods), not an advance yet. Regarding the reasonable default value, I think 63 would be safe to assume.

Adding a DEBUG level osmo-bsc.log, Maybe grepping on it might help. I just let it run until it was about 100M

This time it happens because of a different problem:

DCHAN DEBUG <000f> fsm.c:322 lchan(0-0-2-SDCCH8-4)[0x55e1591e21b0]{WAIT_RF_RELEASE_ACK}: Timeout of T3111
DCHAN DEBUG <000f> lchan_fsm.c:1574 lchan(0-0-2-SDCCH8-4)[0x55e1591e21b0]{WAIT_RF_RELEASE_ACK}: (type=SDCCH) Handling failure, will then transition to state BORKEN
DCHAN ERROR <000f> lchan_fsm.c:81 lchan(0-0-2-SDCCH8-4)[0x55e1591e21b0]{WAIT_RF_RELEASE_ACK}: (type=SDCCH) lchan failure in state WAIT_RF_RELEASE_ACK: Timeout
DCHAN DEBUG <000f> lchan_fsm.c:1574 lchan(0-0-2-SDCCH8-4)[0x55e1591e21b0]{WAIT_RF_RELEASE_ACK}: state_chg to BORKEN
DCHAN DEBUG <000f> lchan_fsm.c:382 lchan(0-0-2-SDCCH8-4)[0x55e1591e21b0]{BORKEN}: (type=SDCCH) Clearing lchan state
...
DCHAN DEBUG <000f> abis_rsl.c:1232 lchan(0-0-2-SDCCH8-4)[0x55e1591e21b0]{BORKEN}: (type=NONE) Rx MEAS_RES
DCHAN DEBUG <000f> abis_rsl.c:1096 lchan(0-0-2-SDCCH8-4)[0x55e1591e21b0]{BORKEN}: (type=NONE) MEAS RES for inactive channel
...
DCHAN DEBUG <000f> abis_rsl.c:1232 lchan(0-0-2-SDCCH8-4)[0x55e1591e21b0]{BORKEN}: (type=NONE) Rx RF_CHAN_REL_ACK
DCHAN DEBUG <000f> abis_rsl.c:1264 lchan(0-0-2-SDCCH8-4)[0x55e1591e21b0]{BORKEN}: Received Event LCHAN_EV_RSL_RF_CHAN_REL_ACK
DCHAN DEBUG <000f> lchan_fsm.c:1300 lchan(0-0-2-SDCCH8-4)[0x55e1591e21b0]{BORKEN}: state_chg to WAIT_AFTER_ERROR

The value of T3111 appears to be too tight for your setup? Assuming that you're using the default:

OsmoBSC# show timer net T3111
net: T3111 = 2 s        Wait time before RSL RF Channel Release (default: 2 s)

Please try again with a higher value, like 4 or even 8.

laforge wrote:

• the BTS rejects RSL CHANnel ACTIVation by sending NACK
• osmo-bsc gets confused and treats the timeslot as 'BROKEN'

osmo-bsc should for sure not get confused by a CHAN ACT NACK, is there a separate bug here?

TBH, I don't know. The current approach seems to be logical and safe from the FSM point of view. What would probably make sense is some kind of watchdog that would try to un-BORKE BORKen timeslots after a certain time. This can be done by activating a broken sub-slot and releasing it immediately. If the BTS still refuses to activate it, then it's completely BORKen and the second attempt to un-BORKE can be postponed further. What do you think?

Submitted https://gerrit.osmocom.org/c/osmo-bsc/+/23574

If that flies, re this ticket, I would still like to see the watchdog idea implemented.

Patch merged, resolved.