https://osmocom.org/https://osmocom.org/favicon.ico?16647414092021-12-06T08:27:18ZOpen Source Mobile CommunicationsOsmoBSC - Bug #5337: ttcn3-bsc-test: leaked struct bsc_subscr in BSC_Tests.TC_no_mschttps://osmocom.org/issues/5337?journal_id=231532021-12-06T08:27:18Zlaforge
<ul><li><strong>Assignee</strong> set to <i>fixeria</i></li></ul> OsmoBSC - Bug #5337: ttcn3-bsc-test: leaked struct bsc_subscr in BSC_Tests.TC_no_mschttps://osmocom.org/issues/5337?journal_id=231642021-12-07T20:50:30Zfixeria
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li><li><strong>Priority</strong> changed from <i>Normal</i> to <i>Low</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>10</i></li></ul><p>The following leak:</p>
<pre>
IMSI TMSI Use
001010000100001 ffffffff 1 (conn)
</pre>
<p>can be reproduced by running the BSC_Tests.TC_no_msc.</p> OsmoBSC - Bug #5337: ttcn3-bsc-test: leaked struct bsc_subscr in BSC_Tests.TC_no_mschttps://osmocom.org/issues/5337?journal_id=231662021-12-07T20:50:48Zfixeria
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-7 priority-1 priority-lowest" href="/issues/4832">Bug #4832</a>: osmo-bsc hard-releases lchan if no MSC is found</i> added</li></ul> OsmoBSC - Bug #5337: ttcn3-bsc-test: leaked struct bsc_subscr in BSC_Tests.TC_no_mschttps://osmocom.org/issues/5337?journal_id=231672021-12-07T21:23:02Zfixeria
<ul></ul><p>fixeria wrote in <a href="#note-2">#note-2</a>:</p>
<blockquote>
<p>The following leak:</p>
<p>[...]</p>
<p>can be reproduced by running the BSC_Tests.TC_no_msc.</p>
</blockquote>
<p>So in gsm_08_08.c/bsc_compl_l3() we allocate:</p>
<ul>
<li>a 'struct bsc_subscr' with IMSI=001010000100001, and</li>
<li>a 'struct gsm_subscriber_connection' for the allocated subscriber.</li>
</ul>
<p>I was interested to see if the new connection can be listed using 'show conns' command, and boom!</p>
<pre>
bsc_vty.c:725:2: runtime error: member access within null pointer of type 'struct bsc_msc_data'
AddressSanitizer:DEADLYSIGNAL
=================================================================
==711471==ERROR: AddressSanitizer: SEGV on unknown address 0x00000000003c (pc 0x55c82025b5b6 bp 0x7ffc86eacc50 sp 0x7ffc86eacc20 T0)
==711471==The signal is caused by a READ memory access.
==711471==Hint: address points to the zero page.
#0 0x55c82025b5b6 in dump_one_subscr_conn /home/wmn/wmn/osmocom/osmo-bsc/src/osmo-bsc/bsc_vty.c:725
#1 0x55c82025bf31 in show_subscr_conn /home/wmn/wmn/osmocom/osmo-bsc/src/osmo-bsc/bsc_vty.c:757
#2 0x7fca696030d2 in cmd_execute_command_real ../../../../src/libosmocore/src/vty/command.c:2604
#3 0x7fca69606448 in vty_command ../../../../src/libosmocore/src/vty/vty.c:464
#4 0x7fca69606448 in vty_execute ../../../../src/libosmocore/src/vty/vty.c:729
#5 0x7fca69606448 in vty_read ../../../../src/libosmocore/src/vty/vty.c:1471
#6 0x7fca69608e6d in client_data ../../../../src/libosmocore/src/vty/telnet_interface.c:154
#7 0x7fca695c9907 in poll_disp_fds ../../../src/libosmocore/src/select.c:361
#8 0x7fca695c9907 in _osmo_select_main ../../../src/libosmocore/src/select.c:393
#9 0x7fca695c9a0e in osmo_select_main_ctx ../../../src/libosmocore/src/select.c:449
#10 0x55c8201250e3 in main /home/wmn/wmn/osmocom/osmo-bsc/src/osmo-bsc/osmo_bsc_main.c:1087
#11 0x7fca68998b24 in __libc_start_main (/usr/lib/libc.so.6+0x27b24)
#12 0x55c82011b12d in _start (/home/wmn/wmn/osmocom/osmo-bsc/src/osmo-bsc/osmo-bsc+0x74512d)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /home/wmn/wmn/osmocom/osmo-bsc/src/osmo-bsc/bsc_vty.c:725 in dump_one_subscr_conn
==711471==ABORTING
</pre> OsmoBSC - Bug #5337: ttcn3-bsc-test: leaked struct bsc_subscr in BSC_Tests.TC_no_mschttps://osmocom.org/issues/5337?journal_id=232122021-12-10T18:34:19Zfixeria
<ul><li><strong>% Done</strong> changed from <i>10</i> to <i>20</i></li></ul><p>The following leak:</p>
<blockquote>
<pre>
IMSI TMSI Use
001019876543210 ffffffff 3 (3*paging-start)
</pre>
</blockquote>
<p>can be reproduced by running:</p>
<ul>
<li>BSC_Tests.TC_lcs_loc_req_for_active_ms_ta_req,</li>
<li>BSC_Tests.TC_lcs_loc_req_for_active_ms_le_timeout2,</li>
<li>BSC_Tests.TC_cm_service_during_lcs_loc_req.</li>
</ul>
<p>In order to identify them, I hacked BSC_Tests.f_gen_test_hdlr_pars() to randomize IMSI for each test case, because all test cases the same use IMSI '001019876543210'H by default. It might be a good idea to get this patch merged:</p>
<p><a class="external" href="https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/26506">https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/26506</a> BSC_Tests: ramdomize IMSI in f_gen_test_hdlr_pars() [NEW]</p> OsmoBSC - Bug #5337: ttcn3-bsc-test: leaked struct bsc_subscr in BSC_Tests.TC_no_mschttps://osmocom.org/issues/5337?journal_id=232132021-12-10T18:51:09Zfixeria
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>New</i></li><li><strong>Assignee</strong> changed from <i>fixeria</i> to <i>neels</i></li></ul><p>I don't feel competent enough to fix this myself, so handing this ticket over to <a class="user active" href="https://osmocom.org/users/91">neels</a>.</p> OsmoBSC - Bug #5337: ttcn3-bsc-test: leaked struct bsc_subscr in BSC_Tests.TC_no_mschttps://osmocom.org/issues/5337?journal_id=232152021-12-10T20:31:10Zfixeria
<ul></ul><p>Looks like we have even more leaks.</p>
<blockquote>
<p><a class="external" href="https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/26506">https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/26506</a> BSC_Tests: ramdomize IMSI in f_gen_test_hdlr_pars() [NEW]</p>
</blockquote>
<p>With this patch applied I am getting the following output:</p>
<pre>
OsmoBSC# show subscriber all
IMSI TMSI Use
001018839845904 ffffffff 1 (paging-start) # TC_lcs_loc_req_for_active_ms_ta_req
001015247946574 ffffffff 1 (paging-start) # TC_lcs_loc_req_for_active_ms_le_timeout2
001019330051280 ffffffff 1 (paging-start) # TC_lcs_loc_req_for_active_ms_ta_req
001019060050196 ffffffff 1 (paging-start) # TC_lcs_loc_req_for_active_ms_le_timeout2
001017749471063 ffffffff 1 (paging-start) # TC_cm_service_during_lcs_loc_req
001010000100001 ffffffff 1 (conn) # TC_no_msc
</pre>
<p>So both TC_lcs_loc_req_for_active_ms_{ta_req,le_timeout2} trigger two subscriber leaks each.</p> OsmoBSC - Bug #5337: ttcn3-bsc-test: leaked struct bsc_subscr in BSC_Tests.TC_no_mschttps://osmocom.org/issues/5337?journal_id=232242021-12-13T12:08:32Zneelsnhofmeyr@sysmocom.de
<ul><li><strong>Subject</strong> changed from <i>ttcn3-bsc-test: leaked struct bsc_subscr</i> to <i>ttcn3-bsc-test: leaked struct bsc_subscr in BSC_Tests.TC_no_msc</i></li></ul><p>splitting up reported leaks into separate issues</p> OsmoBSC - Bug #5337: ttcn3-bsc-test: leaked struct bsc_subscr in BSC_Tests.TC_no_mschttps://osmocom.org/issues/5337?journal_id=235692022-02-05T16:08:48Zfixeria
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Feedback</i></li><li><strong>Priority</strong> changed from <i>Low</i> to <i>Normal</i></li></ul><p>Adding a quick status update here:</p>
<ul>
<li>We started check to the IUT's talloc report in ttcn3-bsc-test (see <code>f_verify_talloc_count()</code>)
<ul>
<li><a class="external" href="https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/26619">https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/26619</a> bsc: detect subscr and conn leaks during f_shutdown_helper()</li>
</ul>
</li>
<li>osmo-bsc master has been fixed by Neels, so everything is green now
<ul>
<li><a class="external" href="https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bsc-test/">https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bsc-test/</a></li>
</ul></li>
</ul>
<p>This is nice, but (as expected) we started to see regressions in several Jenkins jobs:</p>
<ul>
<li><a class="external" href="https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bsc-test-latest/">https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bsc-test-latest/</a> (+25 since build 1210)</li>
<li><a class="external" href="https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bsc-test-sccplite-latest/">https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bsc-test-sccplite-latest/</a> (+16 since build 1103)</li>
<li><a class="external" href="https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bsc-test-vamos/">https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bsc-test-vamos/</a> (+4 since build 245)</li>
<li>The respective CentOS jobs are affected too, as well as <code>2021q1</code> and <code>2021q4</code></li>
</ul>
<p>As a quick solution, we can disable mamleak checking for everything except '-master'. I prepared a change for osmo-ttcn3-hacks:</p>
<p><a class="external" href="https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/27073">https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/27073</a> BSC_Tests: add module parameter 'mp_verify_talloc_count' [NEW]</p>
<p>However I think a proper solution would be back-porting the patches from master. <a class="user active" href="https://osmocom.org/users/7">laforge</a>, <a class="user active" href="https://osmocom.org/users/91">neels</a> what do you think?</p>
<p>As this problem was brought up several times during the weekly review, I am setting the normal priority.</p> OsmoBSC - Bug #5337: ttcn3-bsc-test: leaked struct bsc_subscr in BSC_Tests.TC_no_mschttps://osmocom.org/issues/5337?journal_id=235702022-02-05T16:13:32Zfixeria
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-3 priority-2 priority-default closed" href="/issues/5355">Bug #5355</a>: ttcn3-bsc-test: leaked struct bsc_subscr in LCS tests</i> added</li></ul> OsmoBSC - Bug #5337: ttcn3-bsc-test: leaked struct bsc_subscr in BSC_Tests.TC_no_mschttps://osmocom.org/issues/5337?journal_id=235782022-02-05T18:10:27Zlaforge
<ul></ul><p>On Sat, Feb 05, 2022 at 04:08:50PM +0000, <a class="email" href="mailto:redmine@osmocom.org">redmine@osmocom.org</a> wrote:</p>
<blockquote>
<p>As a quick solution, we can disable mamleak checking for everything except '-master'. I prepared a change for osmo-ttcn3-hacks:</p>
<p><a class="external" href="https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/27073">https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/27073</a> BSC_Tests: add module parameter 'mp_verify_talloc_count' [NEW]</p>
<p>However I think a proper solution would be back-porting the patches from master. <a class="user active" href="https://osmocom.org/users/7">laforge</a>, <a class="user active" href="https://osmocom.org/users/91">neels</a> what do you think?</p>
</blockquote>
<p>The question is how many those are and how much risk we think those patches pose.</p> OsmoBSC - Bug #5337: ttcn3-bsc-test: leaked struct bsc_subscr in BSC_Tests.TC_no_mschttps://osmocom.org/issues/5337?journal_id=235872022-02-07T08:47:59Zfixeria
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-3 priority-2 priority-default closed" href="/issues/5444">Bug #5444</a>: ttcn3-bsc-test-vamos: leaked 'struct bsc_subscr'</i> added</li></ul> OsmoBSC - Bug #5337: ttcn3-bsc-test: leaked struct bsc_subscr in BSC_Tests.TC_no_mschttps://osmocom.org/issues/5337?journal_id=235922022-02-07T18:06:18Zneelsnhofmeyr@sysmocom.de
<ul><li><strong>Related to</strong> <i><a class="issue tracker-2 status-1 priority-2 priority-default" href="/issues/5446">Feature #5446</a>: correlate git version to ttcn3 tests</i> added</li></ul> OsmoBSC - Bug #5337: ttcn3-bsc-test: leaked struct bsc_subscr in BSC_Tests.TC_no_mschttps://osmocom.org/issues/5337?journal_id=235942022-02-07T18:08:59Zneelsnhofmeyr@sysmocom.de
<ul></ul><blockquote>
<p>However I think a proper solution would be back-porting the patches from master. <a class="user active" href="https://osmocom.org/users/7">laforge</a>, <a class="user active" href="https://osmocom.org/users/91">neels</a> what do you think?</p>
</blockquote>
<p>My opinion on this is found here: <a class="external" href="https://osmocom.org/issues/5446">https://osmocom.org/issues/5446</a></p>
<p>pasting:</p>
<p>This happens a lot: we improve ttcn3 testing and enhance osmo-foo master, and then<br />obviously the latest binaries cannot possibly pass the tests. We invent and managa<br />shims in jenkins.sh and ttcn3 config to not do something or other on latest.</p>
<p>A way to not have this burden would be that the ttcn3 test suite for a program<br />is correlated to the git version being tested, for example if the ttcn3 is kept<br />in the same git tree as the program. We should use the 'latest' version of<br />ttcn3 tests for a latest binary. With an implicit correlation, we can always<br />use exactly the tests that match the specific git revision that was built, no<br />matter if it was released or we're just rebasing a local branch.</p>
<p>I think in the long run it would save us a lot of grunt work, and it would<br />definitely save a lot of code cruft to make specific parts of the tests<br />optional.</p> OsmoBSC - Bug #5337: ttcn3-bsc-test: leaked struct bsc_subscr in BSC_Tests.TC_no_mschttps://osmocom.org/issues/5337?journal_id=238212022-03-24T19:09:52Zfixeria
<ul><li><strong>Related to</strong> <i><a class="issue tracker-2 status-4 priority-2 priority-default" href="/issues/2781">Feature #2781</a>: Extend OsmBSC TTCN-3 test coverage regarding resource leaks</i> added</li></ul> OsmoBSC - Bug #5337: ttcn3-bsc-test: leaked struct bsc_subscr in BSC_Tests.TC_no_mschttps://osmocom.org/issues/5337?journal_id=247162022-08-23T16:03:31Zneelsnhofmeyr@sysmocom.de
<ul><li><strong>Assignee</strong> changed from <i>neels</i> to <i>fixeria</i></li></ul><p>re-reading this issue that is assigned to me, i find that i'm not sure of the status.<br />Vadim, do you recall more about this issue?</p> OsmoBSC - Bug #5337: ttcn3-bsc-test: leaked struct bsc_subscr in BSC_Tests.TC_no_mschttps://osmocom.org/issues/5337?journal_id=247292022-08-23T21:01:26Zfixeria
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Resolved</i></li><li><strong>% Done</strong> changed from <i>20</i> to <i>100</i></li></ul><p>fixeria wrote in <a href="#note-9">#note-9</a>:</p>
<blockquote>
<p>This is nice, but (as expected) we started to see regressions in several Jenkins jobs:</p>
<ul>
<li><a class="external" href="https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bsc-test-latest/">https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bsc-test-latest/</a> (+25 since build 1210)</li>
<li><a class="external" href="https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bsc-test-sccplite-latest/">https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bsc-test-sccplite-latest/</a> (+16 since build 1103)</li>
<li><a class="external" href="https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bsc-test-vamos/">https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bsc-test-vamos/</a> (+4 since build 245)</li>
</ul>
</blockquote>
<p>osmo-bsc v1.9.0 was tagged 8 weeks ago, so the -latest already does contain the memleak fixes. No regressions seen anymore.</p>
<blockquote>
<ul>
<li>The respective CentOS jobs are affected too, as well as <code>2021q1</code> and <code>2021q4</code></li>
</ul>
</blockquote>
<p>We stopped running TTCN-3 tests for <code>2021q1</code> and <code>2021q4</code>; <code>2022q2</code> is not affected.</p>