Cellular Network Infrastructure » Radio Access Network » OsmoPCU

Can you share the MAC block provoking osmo-pcu crash here as hex? I would be interested to have a look.

As far as I can see, in gprs_rlcmac_pdch::packet_paging_request() we first encode the paging message using the low-level bitvec API, and then decode the encoding result in the same function, I guess to make sure that the message is correct. Using the bitvec API for encoding kind of makes sense, because we cannot know in advance how many unique identities would fit.

I was quite busy at the time also fixing several other stuff.
I think it should be easy to reproduce by running the entire PCU_Tests.ttcn.control with libosmocore+osmo-pcu with "--enable-sanitize". I can do it whenever I have more time and check which test exactly throws that issue, and provide pcap file.

But I saw this similar ASan report around 2-3 times in osmo-pcu.log for one PCU_Tests.control run, that's why I'm saying it should be easy to reproduce.

Hi Pau,

I think it should be easy to reproduce by running the entire PCU_Tests.ttcn.control with libosmocore+osmo-pcu with "--enable-sanitize".

I could not reproduce this segfault. I even hacked a new TTCN-3 test case verifying handling of multiple MIs:

https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/20997 library: fix Repeated Page Info IE in PacketPagingReq
https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/20998 PCU_Tests: WIP: add f_TC_paging_cs_multi_tuwat

and everything works like a charm:

04:09 < fixeria_> pespin: re OS#4838: I came up with a TTCN-3 test case that sends several PS PAGING requests
04:09 < fixeria_> pespin: I am running sanitized osmo-pcu, and so far sending 4 or even 6 requests works fine
04:16 < fixeria_> pespin_: 56 (7 unique TFIs * 8 TRXes) concurrent TBFs, and 56 CS PAGING requests also work fine ;)

I can reproduce this by running TC_paging_cs_from_bts or TC_paging_cs_from_sgsn_sign (but I think other tests running TC_paging_* also trigger it).

So for TC_paging_cs_from_sgsn_sign I attach pcap file (but ofc the rlcmac resulting message cannot be seen because it crashes while generating it. Still the incoming BSSGP one can be seen).

Interestingly, the TTCN3 test also seems to be crashing somehow, probably when receiving the immediateAssignment and trying to decode it in RLCMAC (form backtrace it seems it crashes in the Unix Domain socket handler?):

MTC@f50471d3be1b: BSSGP successfully initialized
MTC@f50471d3be1b: Sending RACH.ind on fn=1337 with RA=120, TA=0
MTC@f50471d3be1b: Start timer T: 2 s
MTC@f50471d3be1b: Rx Immediate Assignment: { header := { l2_plen := { l2_plen := 11, zero_one := '01'B }, skip_indicator := 0, rr_protocol_discriminator := 6, message_type := IMMEDIATE_ASSIGNMENT (63) }, payload := { imm_ass := { ded_or_tbf := { spare := '0'B, tma := false, downlink := false, tbf := true }, page_mode := PAGE_MODE_NORMAL (0), chan_desc := omit, pkt_chan_desc := { channel_Type_spare := 1, tn := 7, tsc := 7, presence := '0'B, zero := { hopping := '0'B, spare := '0'B, arfcn := 871, indirect := omit }, one := omit }, req_ref := { ra := '01111000'B, t1p := 1, t3 := 11, t2 := 11 }, timing_advance := 0, mobile_allocation := { len := 0, ma := ''B }, rest_octets := { presence := '11'B, ll := omit, lh := omit, hl := omit, hh := { pa_disc := '0'B, pa := { uldl := { ass_disc := '0'B, ass := { ul := { presence := '1'B, dynamic := { tfi_assignment := 0, polling := '0'B, spare := '0'B, usf := 0, usf_granularity := '0'B, p0_present := '0'B, p0 := omit, pr_mode := omit, ch_coding_cmd := CH_CODING_CS2 (1), tlli_block_chan_coding := '1'B, alpha_present := '0'B, alpha := omit, gamma := 0, ta_index_present := '0'B, ta_index := omit, tbf_starting_time_present := '0'B, tbf_starting_time := omit }, single := omit } } } } } } } } }
MTC@f50471d3be1b: Rx Uplink TBF GPRS assignment: { presence := '1'B, dynamic := { tfi_assignment := 0, polling := '0'B, spare := '0'B, usf := 0, usf_granularity := '0'B, p0_present := '0'B, p0 := omit, pr_mode := omit, ch_coding_cmd := CH_CODING_CS2 (1), tlli_block_chan_coding := '1'B, alpha_present := '0'B, alpha := omit, gamma := 0, ta_index_present := '0'B, ta_index := omit, tbf_starting_time_present := '0'B, tbf_starting_time := omit }, single := omit }
close_fd10
 /osmo-ttcn3-hacks/pcu/PCU_Tests: Segmentation fault occurred
/usr/lib/titan/libttcn3-parallel-dynamic.so(_Z14signal_handleri+0xa3)[0x7f9582b39c03]
/lib/x86_64-linux-gnu/libc.so.6(+0x33060)[0x7f9580f8f060]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x14)[0x7f9580fd7524]
/usr/lib/titan/libttcn3-parallel-dynamic.so(Free+0xe)[0x7f9582b297fe]
/osmo-ttcn3-hacks/pcu/UD_PT.so(_ZN12UD__PortType15UD__PT_PROVIDER24Handle_Fd_Event_ReadableEi+0x1f8)[0x7f95839df7d8]
/usr/lib/titan/libttcn3-parallel-dynamic.so(_ZN19Fd_And_Timeout_User13call_handlersEi+0x56d)[0x7f9582afdebd]
/usr/lib/titan/libttcn3-parallel-dynamic.so(_ZN13TTCN_Snapshot8take_newEb+0x387)[0x7f9582afe5f7]
/osmo-ttcn3-hacks/pcu/PCUIF_Components.so(_ZN17PCUIF__Components21f__PCUIF__CT__handlerERK10CHARSTRINGRK7BOOLEAN+0xd89)[0x7f95a6bdb26b]
/osmo-ttcn3-hacks/pcu/PCUIF_Components.so(_ZN17PCUIF__Components18start_ptc_functionEPKcR8Text_Buf+0x254)[0x7f95a6bdbf32]
/usr/lib/titan/libttcn3-parallel-dynamic.so(_ZN11Module_List14start_functionEPKcS1_R8Text_Buf+0x2b)[0x7f9582ae477b]
/usr/lib/titan/libttcn3-parallel-dynamic.so(_ZN12TTCN_Runtime14start_functionEPKcS1_R8Text_Buf+0x25)[0x7f9582af68e5]
/usr/lib/titan/libttcn3-parallel-dynamic.so(_ZN18TTCN_Communication13process_startEv+0x42)[0x7f9582abac32]
/usr/lib/titan/libttcn3-parallel-dynamic.so(_ZN18TTCN_Communication23process_all_messages_tcEv+0x2f5)[0x7f9582abb485]
/usr/lib/titan/libttcn3-parallel-dynamic.so(_ZN12TTCN_Runtime8ptc_mainEv+0xdf)[0x7f9582afa4ff]
/usr/lib/titan/libttcn3-parallel-dynamic.so(main+0x365)[0x7f95827f3665]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f9580f7c2e1]
/osmo-ttcn3-hacks/pcu/PCU_Tests(+0x229a)[0x559c36bba29a]
MC@f50471d3be1b: Unexpected end of PTC connection (8) from f50471d3be1b [172.18.13.10].
BTS(9)@f50471d3be1b: Warning: The last outgoing messages on port PCUIF may be lost.
BTS(9)@f50471d3be1b: Dynamic test case error: Port PCUIF has neither connections nor mappings. Message cannot be sent on it.
ClckGen-0(11)@f50471d3be1b: Dynamic test case error: Port CLCK has neither connections nor mappings. Message cannot be sent on it.

Offending message crashing decoder:

40 88 3c 14 93 12 00 00 00 00 12 00 00 00 00 00 00 00 00 00 00 00 00

I can reproduce the ASan issue in a unit test with this patch:

diff --git a/tests/rlcmac/RLCMACTest.cpp b/tests/rlcmac/RLCMACTest.cpp
index f1b6508..2c7d1da 100644
--- a/tests/rlcmac/RLCMACTest.cpp
+++ b/tests/rlcmac/RLCMACTest.cpp
@@ -110,6 +110,7 @@ void testRlcMacDownlink(void *test_ctx)
        "4913e00850884013a8048b2b2b2b2b2b2b2b2b2b2b2b2b", // Polling Request (malformed)
        "412430007fffffffffffffffefd19c7ba12b2b2b2b2b2b", // Packet Uplink Ack Nack?
        "41942b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b", // System Info 13?
+       "40883c1493120000000012000000000000000000000000", // Paging request (OS#4838)
        };

        int testDataSize = sizeof(testData)/sizeof(testData[0]);

This change seems to fix the bug, but I still need to see if it's the proper fix:

diff --git a/src/csn1.c b/src/csn1.c
index fa29e27..881a32a 100644
--- a/src/csn1.c
+++ b/src/csn1.c
@@ -1216,7 +1216,7 @@ csnStreamDecoder(csnStream_t* ar, const CSN_DESCR* pDescr, struct bitvec *vector

           while (count > 0)
           {
-            readIndex -= 8;
+            *readIndex -= 8;
            *pui8 = bitvec_read_field(vector, readIndex, 8);
             LOGPC(DCSN1, LOGL_DEBUG, "%s = %u | ", pDescr->sz , (unsigned)*pui8);
             pui8++;

The problem was readIndex pointer being changed instead of its content, so when bitvec_read_field tried reading from it, it was reading from somewherse else, because address changed from 0x7ffffffdfb80 to 0x7ffffffdfb60

Proper fix should be here, I still need to adjust the unit test expectancies:
https://gerrit.osmocom.org/c/osmo-pcu/+/21305 csn1: Fix readIndex pointer change in CSN_VARIABLE_ARRAY

Merged, closing.