Actions
Bug #6262
closeduse-after-free when io_uring backend is used
Start date:
11/20/2023
Due date:
% Done:
0%
Spec Reference:
Description
when running the TTCN3 test suite against osmo-stp with the osmo_io_sctp branches of libosmocore/netif/sccp and the io_uring backend enabled, I'm running into the following use-after-free:
DLSS7 <000c> osmo_ss7_asp.c:904 XUA_ASP(asp-sender){ASP_DOWN}: Received Event SCTP-COMM_DOWN.ind
DLSS7 <000c> xua_asp_fsm.c:679 XUA_ASP(asp-sender){ASP_DOWN}: state_chg to ASP_DOWN
DLSS7 <000c> xua_asp_fsm.c:360 XUA_AS(as-sender){AS_DOWN}: Received Event ASPAS-ASP_DOWN.ind
DLSS7 <000c> xua_asp_fsm.c:113 0: asp-asp-sender: No Layer Manager, dropping M-ASP_DOWN.indication
DLSS7 <000c> xua_asp_fsm.c:113 0: asp-asp-sender: No Layer Manager, dropping M-SCTP_RELEASE.indication
DLSS7 <000c> osmo_ss7_asp.c:754 0: asp-asp-receiver0: ss7_asp_xua_srv_conn_cb(): sctp_recvmsg() returned 0 (flags=0x80)
DLSS7 <000c> osmo_ss7_asp.c:703 0: asp-asp-receiver0: xUA CLNT SCTP NOTIFICATION 32769 flags=0x0
DLSS7 <000c> osmo_ss7_asp.c:711 0: asp-asp-receiver0: xUA CLNT SCTP_ASSOC_CHANGE: SHUTDOWN_COMP
DLSS7 <000c> osmo_ss7_asp.c:754 0: asp-asp-receiver1: ss7_asp_xua_srv_conn_cb(): sctp_recvmsg() returned 0 (flags=0x80)
DLSS7 <000c> osmo_ss7_asp.c:703 0: asp-asp-receiver1: xUA CLNT SCTP NOTIFICATION 32773 flags=0x0
DLSS7 <000c> osmo_ss7_asp.c:726 0: asp-asp-receiver1: xUA CLNT SHUTDOWN_EVENT
DLSS7 <000c> osmo_ss7_asp.c:898 asp-receiver0: connection closed
DLSS7 <000c> osmo_ss7_asp.c:904 XUA_ASP(asp-receiver0){ASP_DOWN}: Received Event SCTP-COMM_DOWN.ind
DLSS7 <000c> xua_asp_fsm.c:679 XUA_ASP(asp-receiver0){ASP_DOWN}: state_chg to ASP_DOWN
DLSS7 <000c> xua_asp_fsm.c:360 XUA_AS(as-receiver){AS_DOWN}: Received Event ASPAS-ASP_DOWN.ind
DLSS7 <000c> xua_asp_fsm.c:113 0: asp-asp-receiver0: No Layer Manager, dropping M-ASP_DOWN.indication
DLSS7 <000c> xua_asp_fsm.c:113 0: asp-asp-receiver0: No Layer Manager, dropping M-SCTP_RELEASE.indication
DLSS7 <000c> osmo_ss7_asp.c:754 0: asp-asp-receiver1: ss7_asp_xua_srv_conn_cb(): sctp_recvmsg() returned 0 (flags=0x80)
DLSS7 <000c> osmo_ss7_asp.c:703 0: asp-asp-receiver1: xUA CLNT SCTP NOTIFICATION 32769 flags=0x0
DLSS7 <000c> osmo_ss7_asp.c:711 0: asp-asp-receiver1: xUA CLNT SCTP_ASSOC_CHANGE: SHUTDOWN_COMP
DLSS7 <000c> osmo_ss7_asp.c:898 asp-receiver1: connection closed
DLSS7 <000c> osmo_ss7_asp.c:904 XUA_ASP(asp-receiver1){ASP_DOWN}: Received Event SCTP-COMM_DOWN.ind
DLSS7 <000c> xua_asp_fsm.c:679 XUA_ASP(asp-receiver1){ASP_DOWN}: state_chg to ASP_DOWN
DLSS7 <000c> xua_asp_fsm.c:360 XUA_AS(as-receiver){AS_DOWN}: Received Event ASPAS-ASP_DOWN.ind
DLSS7 <000c> xua_asp_fsm.c:113 0: asp-asp-receiver1: No Layer Manager, dropping M-ASP_DOWN.indication
DLSS7 <000c> xua_asp_fsm.c:113 0: asp-asp-receiver1: No Layer Manager, dropping M-SCTP_RELEASE.indication
DLSS7 <000c> osmo_ss7_xua_srv.c:237 (Re)binding ipa Server to :::5000
DLSS7 <000c> osmo_ss7_xua_srv.c:72 (r=::ffff:127.0.0.1:20100<->l=::ffff:127.0.0.1:5000): New ipa connection accepted
DLSS7 <000c> osmo_ss7_xua_srv.c:108 (r=::ffff:127.0.0.1:20100<->l=::ffff:127.0.0.1:5000): ipa connection without matching ASP definition and no dynamic registration enabled, terminating
DLSS7 <000c> osmo_ss7_asp.c:898 ?: connection closed
=================================================================
==3194240==ERROR: AddressSanitizer: heap-use-after-free on address 0x6140000791c0 at pc 0x7fc18926b413 bp 0x7fff489d84f0 sp 0x7fff489d84e8
READ of size 8 at 0x6140000791c0 thread T0
#0 0x7fc18926b412 in iofd_uring_handle_completion (/usr/local/lib/libosmocore.so.21+0x26b412) (BuildId: f5d73d624783a4048f5066f9c5922b925a6aa0bb)
#1 0x7fc18926bc34 in iofd_uring_cqe (/usr/local/lib/libosmocore.so.21+0x26bc34) (BuildId: f5d73d624783a4048f5066f9c5922b925a6aa0bb)
#2 0x7fc189269311 in iofd_uring_poll_cb (/usr/local/lib/libosmocore.so.21+0x269311) (BuildId: f5d73d624783a4048f5066f9c5922b925a6aa0bb)
#3 0x7fc1891fad0d in poll_disp_fds (/usr/local/lib/libosmocore.so.21+0x1fad0d) (BuildId: f5d73d624783a4048f5066f9c5922b925a6aa0bb)
#4 0x7fc1891fae9a in _osmo_select_main (/usr/local/lib/libosmocore.so.21+0x1fae9a) (BuildId: f5d73d624783a4048f5066f9c5922b925a6aa0bb)
#5 0x7fc1891faeb6 in osmo_select_main (/usr/local/lib/libosmocore.so.21+0x1faeb6) (BuildId: f5d73d624783a4048f5066f9c5922b925a6aa0bb)
#6 0x55d1534d0116 in main /space/home/laforge/projects/git/libosmo-sccp/stp/stp_main.c:269
#7 0x7fc188e456c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
#8 0x7fc188e45784 in __libc_start_main_impl ../csu/libc-start.c:360
#9 0x55d1534cf380 in _start (/space/home/laforge/projects/git/libosmo-sccp/stp/.libs/osmo-stp+0x3380) (BuildId: 9b28e0c01de34a9326cb0f0c3f3146a2b134dd3c)
0x6140000791c0 is located 384 bytes inside of 392-byte region [0x614000079040,0x6140000791c8)
freed by thread T0 here:
#0 0x7fc189ed7288 in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:52
#1 0x7fc18a5135b1 in _tc_free_internal ../../talloc.c:1222
#2 0x7fc1895631fe in osmo_stream_srv_destroy (/usr/local/lib/libosmonetif.so.11+0xe31fe) (BuildId: a355aed21f2c416d5f7d2a90ba6d9bc0d1cbd346)
#3 0x7fc1899b8823 in xua_accept_cb (/space/home/laforge/projects/git/libosmo-sccp/src/.libs/libosmo-sigtran.so.9+0x1b8823) (BuildId: c81249073e8805fb9691fe25445dd993fecbeeb9)
#4 0x7fc18955b32e in osmo_stream_srv_link_ofd_cb (/usr/local/lib/libosmonetif.so.11+0xdb32e) (BuildId: a355aed21f2c416d5f7d2a90ba6d9bc0d1cbd346)
#5 0x7fc1891fad0d in poll_disp_fds (/usr/local/lib/libosmocore.so.21+0x1fad0d) (BuildId: f5d73d624783a4048f5066f9c5922b925a6aa0bb)
#6 0x7fc1891fae9a in _osmo_select_main (/usr/local/lib/libosmocore.so.21+0x1fae9a) (BuildId: f5d73d624783a4048f5066f9c5922b925a6aa0bb)
#7 0x7fc1891faeb6 in osmo_select_main (/usr/local/lib/libosmocore.so.21+0x1faeb6) (BuildId: f5d73d624783a4048f5066f9c5922b925a6aa0bb)
#8 0x55d1534d0116 in main /space/home/laforge/projects/git/libosmo-sccp/stp/stp_main.c:269
#9 0x7fc188e456c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
previously allocated by thread T0 here:
#0 0x7fc189ed85bf in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
#1 0x7fc18a514df3 in __talloc_with_prefix ../../talloc.c:783
SUMMARY: AddressSanitizer: heap-use-after-free (/usr/local/lib/libosmocore.so.21+0x26b412) (BuildId: f5d73d624783a4048f5066f9c5922b925a6aa0bb) in iofd_uring_handle_completion
Shadow bytes around the buggy address:
0x614000078f00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x614000078f80: fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa
0x614000079000: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
0x614000079080: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x614000079100: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x614000079180: fd fd fd fd fd fd fd fd[fd]fa fa fa fa fa fa fa
0x614000079200: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x614000079280: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x614000079300: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x614000079380: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x614000079400: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==3194240==ABORTING
So it looks like we're in a completion handler for something that we already destroyed via osmo_stream_srv_destroy in a xua_accept_cb earlier.
It's to be determined whose fault this is exactly.
Related issues
Updated by daniel 5 months ago
Ah, the joy of async handling. I think this might be osmo_stream_srv_destroy() also freeing the iofd and related data, and the uring entry (which completes later) then causing a read of a structure there.
Then again, we're protecting against that in osmo_io, so maybe it's also something similar in netif _stream. If osmo_io if the iofd is marked closed then completions shouldn't call the callback functions so them dereferencing user data shouldn't cause this.
Actions
