Project

General

Profile

Actions

Bug #6262

open

use-after-free when io_uring backend is used

Added by laforge 3 months ago. Updated 17 days ago.

Status:
New
Priority:
Normal
Assignee:
Target version:
-
Start date:
11/20/2023
Due date:
% Done:

0%

Spec Reference:
Tags:

Description

when running the TTCN3 test suite against osmo-stp with the osmo_io_sctp branches of libosmocore/netif/sccp and the io_uring backend enabled, I'm running into the following use-after-free:

DLSS7 <000c> osmo_ss7_asp.c:904 XUA_ASP(asp-sender){ASP_DOWN}: Received Event SCTP-COMM_DOWN.ind
DLSS7 <000c> xua_asp_fsm.c:679 XUA_ASP(asp-sender){ASP_DOWN}: state_chg to ASP_DOWN
DLSS7 <000c> xua_asp_fsm.c:360 XUA_AS(as-sender){AS_DOWN}: Received Event ASPAS-ASP_DOWN.ind
DLSS7 <000c> xua_asp_fsm.c:113 0: asp-asp-sender: No Layer Manager, dropping M-ASP_DOWN.indication
DLSS7 <000c> xua_asp_fsm.c:113 0: asp-asp-sender: No Layer Manager, dropping M-SCTP_RELEASE.indication
DLSS7 <000c> osmo_ss7_asp.c:754 0: asp-asp-receiver0: ss7_asp_xua_srv_conn_cb(): sctp_recvmsg() returned 0 (flags=0x80)
DLSS7 <000c> osmo_ss7_asp.c:703 0: asp-asp-receiver0: xUA CLNT SCTP NOTIFICATION 32769 flags=0x0
DLSS7 <000c> osmo_ss7_asp.c:711 0: asp-asp-receiver0: xUA CLNT SCTP_ASSOC_CHANGE: SHUTDOWN_COMP
DLSS7 <000c> osmo_ss7_asp.c:754 0: asp-asp-receiver1: ss7_asp_xua_srv_conn_cb(): sctp_recvmsg() returned 0 (flags=0x80)
DLSS7 <000c> osmo_ss7_asp.c:703 0: asp-asp-receiver1: xUA CLNT SCTP NOTIFICATION 32773 flags=0x0
DLSS7 <000c> osmo_ss7_asp.c:726 0: asp-asp-receiver1: xUA CLNT SHUTDOWN_EVENT
DLSS7 <000c> osmo_ss7_asp.c:898 asp-receiver0: connection closed
DLSS7 <000c> osmo_ss7_asp.c:904 XUA_ASP(asp-receiver0){ASP_DOWN}: Received Event SCTP-COMM_DOWN.ind
DLSS7 <000c> xua_asp_fsm.c:679 XUA_ASP(asp-receiver0){ASP_DOWN}: state_chg to ASP_DOWN
DLSS7 <000c> xua_asp_fsm.c:360 XUA_AS(as-receiver){AS_DOWN}: Received Event ASPAS-ASP_DOWN.ind
DLSS7 <000c> xua_asp_fsm.c:113 0: asp-asp-receiver0: No Layer Manager, dropping M-ASP_DOWN.indication
DLSS7 <000c> xua_asp_fsm.c:113 0: asp-asp-receiver0: No Layer Manager, dropping M-SCTP_RELEASE.indication
DLSS7 <000c> osmo_ss7_asp.c:754 0: asp-asp-receiver1: ss7_asp_xua_srv_conn_cb(): sctp_recvmsg() returned 0 (flags=0x80)
DLSS7 <000c> osmo_ss7_asp.c:703 0: asp-asp-receiver1: xUA CLNT SCTP NOTIFICATION 32769 flags=0x0
DLSS7 <000c> osmo_ss7_asp.c:711 0: asp-asp-receiver1: xUA CLNT SCTP_ASSOC_CHANGE: SHUTDOWN_COMP
DLSS7 <000c> osmo_ss7_asp.c:898 asp-receiver1: connection closed
DLSS7 <000c> osmo_ss7_asp.c:904 XUA_ASP(asp-receiver1){ASP_DOWN}: Received Event SCTP-COMM_DOWN.ind
DLSS7 <000c> xua_asp_fsm.c:679 XUA_ASP(asp-receiver1){ASP_DOWN}: state_chg to ASP_DOWN
DLSS7 <000c> xua_asp_fsm.c:360 XUA_AS(as-receiver){AS_DOWN}: Received Event ASPAS-ASP_DOWN.ind
DLSS7 <000c> xua_asp_fsm.c:113 0: asp-asp-receiver1: No Layer Manager, dropping M-ASP_DOWN.indication
DLSS7 <000c> xua_asp_fsm.c:113 0: asp-asp-receiver1: No Layer Manager, dropping M-SCTP_RELEASE.indication
DLSS7 <000c> osmo_ss7_xua_srv.c:237 (Re)binding ipa Server to :::5000
DLSS7 <000c> osmo_ss7_xua_srv.c:72 (r=::ffff:127.0.0.1:20100<->l=::ffff:127.0.0.1:5000): New ipa connection accepted
DLSS7 <000c> osmo_ss7_xua_srv.c:108 (r=::ffff:127.0.0.1:20100<->l=::ffff:127.0.0.1:5000): ipa connection without matching ASP definition and no dynamic registration enabled, terminating
DLSS7 <000c> osmo_ss7_asp.c:898 ?: connection closed
=================================================================
==3194240==ERROR: AddressSanitizer: heap-use-after-free on address 0x6140000791c0 at pc 0x7fc18926b413 bp 0x7fff489d84f0 sp 0x7fff489d84e8
READ of size 8 at 0x6140000791c0 thread T0
    #0 0x7fc18926b412 in iofd_uring_handle_completion (/usr/local/lib/libosmocore.so.21+0x26b412) (BuildId: f5d73d624783a4048f5066f9c5922b925a6aa0bb)
    #1 0x7fc18926bc34 in iofd_uring_cqe (/usr/local/lib/libosmocore.so.21+0x26bc34) (BuildId: f5d73d624783a4048f5066f9c5922b925a6aa0bb)
    #2 0x7fc189269311 in iofd_uring_poll_cb (/usr/local/lib/libosmocore.so.21+0x269311) (BuildId: f5d73d624783a4048f5066f9c5922b925a6aa0bb)
    #3 0x7fc1891fad0d in poll_disp_fds (/usr/local/lib/libosmocore.so.21+0x1fad0d) (BuildId: f5d73d624783a4048f5066f9c5922b925a6aa0bb)
    #4 0x7fc1891fae9a in _osmo_select_main (/usr/local/lib/libosmocore.so.21+0x1fae9a) (BuildId: f5d73d624783a4048f5066f9c5922b925a6aa0bb)
    #5 0x7fc1891faeb6 in osmo_select_main (/usr/local/lib/libosmocore.so.21+0x1faeb6) (BuildId: f5d73d624783a4048f5066f9c5922b925a6aa0bb)
    #6 0x55d1534d0116 in main /space/home/laforge/projects/git/libosmo-sccp/stp/stp_main.c:269
    #7 0x7fc188e456c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #8 0x7fc188e45784 in __libc_start_main_impl ../csu/libc-start.c:360
    #9 0x55d1534cf380 in _start (/space/home/laforge/projects/git/libosmo-sccp/stp/.libs/osmo-stp+0x3380) (BuildId: 9b28e0c01de34a9326cb0f0c3f3146a2b134dd3c)

0x6140000791c0 is located 384 bytes inside of 392-byte region [0x614000079040,0x6140000791c8)
freed by thread T0 here:
    #0 0x7fc189ed7288 in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:52
    #1 0x7fc18a5135b1 in _tc_free_internal ../../talloc.c:1222
    #2 0x7fc1895631fe in osmo_stream_srv_destroy (/usr/local/lib/libosmonetif.so.11+0xe31fe) (BuildId: a355aed21f2c416d5f7d2a90ba6d9bc0d1cbd346)
    #3 0x7fc1899b8823 in xua_accept_cb (/space/home/laforge/projects/git/libosmo-sccp/src/.libs/libosmo-sigtran.so.9+0x1b8823) (BuildId: c81249073e8805fb9691fe25445dd993fecbeeb9)
    #4 0x7fc18955b32e in osmo_stream_srv_link_ofd_cb (/usr/local/lib/libosmonetif.so.11+0xdb32e) (BuildId: a355aed21f2c416d5f7d2a90ba6d9bc0d1cbd346)
    #5 0x7fc1891fad0d in poll_disp_fds (/usr/local/lib/libosmocore.so.21+0x1fad0d) (BuildId: f5d73d624783a4048f5066f9c5922b925a6aa0bb)
    #6 0x7fc1891fae9a in _osmo_select_main (/usr/local/lib/libosmocore.so.21+0x1fae9a) (BuildId: f5d73d624783a4048f5066f9c5922b925a6aa0bb)
    #7 0x7fc1891faeb6 in osmo_select_main (/usr/local/lib/libosmocore.so.21+0x1faeb6) (BuildId: f5d73d624783a4048f5066f9c5922b925a6aa0bb)
    #8 0x55d1534d0116 in main /space/home/laforge/projects/git/libosmo-sccp/stp/stp_main.c:269
    #9 0x7fc188e456c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58

previously allocated by thread T0 here:
    #0 0x7fc189ed85bf in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0x7fc18a514df3 in __talloc_with_prefix ../../talloc.c:783

SUMMARY: AddressSanitizer: heap-use-after-free (/usr/local/lib/libosmocore.so.21+0x26b412) (BuildId: f5d73d624783a4048f5066f9c5922b925a6aa0bb) in iofd_uring_handle_completion
Shadow bytes around the buggy address:
  0x614000078f00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x614000078f80: fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa
  0x614000079000: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x614000079080: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x614000079100: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x614000079180: fd fd fd fd fd fd fd fd[fd]fa fa fa fa fa fa fa
  0x614000079200: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x614000079280: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x614000079300: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x614000079380: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x614000079400: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==3194240==ABORTING

So it looks like we're in a completion handler for something that we already destroyed via osmo_stream_srv_destroy in a xua_accept_cb earlier.

It's to be determined whose fault this is exactly.


Related issues

Related to libosmocore - Bug #6264: osmo_iofd_write_msgb unconditionally dereferences io_ops.write_cbResolveddaniel11/20/2023

Actions
Actions #1

Updated by laforge 3 months ago

FYI, the bug is reproducible. Running the test suite will make it crahs at some point.

Actions #2

Updated by laforge 3 months ago

  • Related to Bug #6264: osmo_iofd_write_msgb unconditionally dereferences io_ops.write_cb added
Actions #3

Updated by laforge 3 months ago

  • Tags set to osmo_io
Actions #4

Updated by daniel 3 months ago

Ah, the joy of async handling. I think this might be osmo_stream_srv_destroy() also freeing the iofd and related data, and the uring entry (which completes later) then causing a read of a structure there.

Then again, we're protecting against that in osmo_io, so maybe it's also something similar in netif _stream. If osmo_io if the iofd is marked closed then completions shouldn't call the callback functions so them dereferencing user data shouldn't cause this.

Actions #5

Updated by laforge 17 days ago

  • Assignee changed from laforge to jolly

I think this might already have been resolved by jolly?

Actions

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)