https://osmocom.org/https://osmocom.org/favicon.ico?16647414092018-03-14T12:48:14ZOpen Source Mobile CommunicationsOsmoGSMTester - Bug #3064: ofono: crash after merging osmo-gsm-tester patch to read IMSIhttps://osmocom.org/issues/3064?journal_id=82212018-03-14T12:48:14Zpespin
<ul><li><strong>File</strong> <a href="/attachments/3007">trial-215-run.tgz</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/3007/trial-215-run.tgz">trial-215-run.tgz</a> added</li></ul> OsmoGSMTester - Bug #3064: ofono: crash after merging osmo-gsm-tester patch to read IMSIhttps://osmocom.org/issues/3064?journal_id=82242018-03-14T13:03:41Zpespin
<ul><li><strong>File</strong> <a href="/attachments/3010">ofono-gdb-session.txt</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/3010/ofono-gdb-session.txt">ofono-gdb-session.txt</a> added</li></ul><pre>
(gdb) bt
#0 0x00007ffff7b20517 in g_queue_is_empty () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#1 0x00005555556acea0 in sim_fs_op_free (pointer=0x555555974270) at src/simfs.c:101
#2 0x00007ffff7b205fc in g_queue_foreach () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3 0x00007ffff7b2065b in g_queue_free_full () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#4 0x00005555556acf44 in sim_fs_free (fs=0x5555559995b0) at src/simfs.c:125
#5 0x0000555555681ae1 in sim_remove (atom=0x555555971bc0) at src/sim.c:3167
#6 0x000055555564e3a8 in flush_atoms (modem=0x5555559ed690, new_state=MODEM_STATE_POWER_OFF) at src/modem.c:432
#7 0x000055555564e5f6 in modem_change_state (modem=0x5555559ed690, new_state=MODEM_STATE_POWER_OFF)
at src/modem.c:510
#8 0x000055555564f1d2 in set_powered (modem=0x5555559ed690, powered=0) at src/modem.c:896
#9 0x000055555564f985 in modem_set_property (conn=0x55555596b8d0, msg=0x5555559719e0, data=0x5555559ed690)
at src/modem.c:1120
#10 0x00005555556c6800 in process_message (connection=0x55555596b8d0, message=0x5555559719e0,
method=0x555555940128 <modem_methods+40>, iface_user_data=0x5555559ed690) at gdbus/object.c:259
#11 0x00005555556c8305 in generic_message (connection=0x55555596b8d0, message=0x5555559719e0,
user_data=0x5555559bf3b0) at gdbus/object.c:1070
#12 0x00007ffff7899733 in ?? () from /lib/x86_64-linux-gnu/libdbus-1.so.3
#13 0x00007ffff788ad84 in dbus_connection_dispatch () from /lib/x86_64-linux-gnu/libdbus-1.so.3
#14 0x00005555556c41b4 in message_dispatch (data=0x55555596b8d0) at gdbus/mainloop.c:72
#15 0x00007ffff7b0f6aa in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#16 0x00007ffff7b0fa60 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#17 0x00007ffff7b0fd82 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#18 0x000055555564cb3a in main (argc=1, argv=0x7fffffffec18) at src/main.c:306
(gdb) frame 1
#1 0x00005555556acea0 in sim_fs_op_free (pointer=0x555555974270) at src/simfs.c:101
101 if (fs->session && g_queue_is_empty(fs->op_q)) {
(gdb) print fs
$1 = (struct sim_fs *) 0x555555a7aad0
(gdb) print *fs
$2 = {op_q = 0x1b0, op_source = 32,
bitmap = "\000\000\000\000\200֖UUU", '\000' <repeats 11 times>, "O\261\367\377\177\000\000!\000\000", fd = 0,
sim = 0x7ffff72a0030, driver = 0x55555599c2b0, contexts = 0x20, session = 0x51, session_id = 1937339183,
watch_id = 1986356271}
</pre>
<p>I'd say the fs pointer contains garbage, as if it was freed already before that point.</p> OsmoGSMTester - Bug #3064: ofono: crash after merging osmo-gsm-tester patch to read IMSIhttps://osmocom.org/issues/3064?journal_id=82312018-03-14T15:41:17Zpespin
<ul></ul><p>I rebased our osmo-gsm-tester branch on top of ofono 1.23 upstream because I saw some sim related fixes, but the bug is still there:</p>
<pre>
#0 0x00007ffff7b20517 in g_queue_is_empty () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#1 0x00005555556adcdd in sim_fs_op_free (pointer=0x5555559cb990) at src/simfs.c:101
#2 0x00007ffff7b205fc in g_queue_foreach () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3 0x00007ffff7b2065b in g_queue_free_full () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#4 0x00005555556add81 in sim_fs_free (fs=0x5555559c0780) at src/simfs.c:125
#5 0x00005555556828f3 in sim_remove (atom=0x5555559cb000) at src/sim.c:3175
#6 0x000055555564f16f in flush_atoms (modem=0x555555a8fb00, new_state=MODEM_STATE_POWER_OFF) at src/modem.c:432
#7 0x000055555564f3bd in modem_change_state (modem=0x555555a8fb00, new_state=MODEM_STATE_POWER_OFF)
at src/modem.c:510
#8 0x000055555564ff99 in set_powered (modem=0x555555a8fb00, powered=0) at src/modem.c:896
#9 0x000055555565074c in modem_set_property (conn=0x55555596c8d0, msg=0x55555596e460, data=0x555555a8fb00)
at src/modem.c:1120
#10 0x00005555556c76b6 in process_message (connection=0x55555596c8d0, message=0x55555596e460,
method=0x555555941128 <modem_methods+40>, iface_user_data=0x555555a8fb00) at gdbus/object.c:259
#11 0x00005555556c91bb in generic_message (connection=0x55555596c8d0, message=0x55555596e460,
user_data=0x5555559e0430) at gdbus/object.c:1070
#12 0x00007ffff7899733 in ?? () from /lib/x86_64-linux-gnu/libdbus-1.so.3
#13 0x00007ffff788ad84 in dbus_connection_dispatch () from /lib/x86_64-linux-gnu/libdbus-1.so.3
#14 0x00005555556c506a in message_dispatch (data=0x55555596c8d0) at gdbus/mainloop.c:72
#15 0x00007ffff7b0f6aa in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#16 0x00007ffff7b0fa60 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#17 0x00007ffff7b0fd82 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#18 0x000055555564d901 in main (argc=1, argv=0x7fffffffebf8) at src/main.c:306
(gdb) frame 1
#1 0x00005555556adcdd in sim_fs_op_free (pointer=0x5555559cb990) at src/simfs.c:101
101 if (fs->session && g_queue_is_empty(fs->op_q)) {
(gdb) print fs
$1 = (struct sim_fs *) 0x5555559c8fd0
(gdb) print *fs
$2 = {op_q = 0x150, op_source = 32,
bitmap = "\000\000\000\000\220\t\227UUU", '\000' <repeats 11 times>, "O\261\367\377\177\000\000\001\002\000",
fd = 0, sim = 0x5555559c9160, driver = 0x0, contexts = 0x5555559c90a0, session = 0xffffff9c, session_id = 0,
watch_id = 0}
</pre> OsmoGSMTester - Bug #3064: ofono: crash after merging osmo-gsm-tester patch to read IMSIhttps://osmocom.org/issues/3064?journal_id=82352018-03-14T17:02:19Zpespin
<ul></ul><p>I can easily reproduce this running ussd:sysmo in RnD.<br />I guess we only see the issue with tests running with 1 MS, since when 2 MS are used there's more time in between we used the SIM service and we power off the same MS.</p> OsmoGSMTester - Bug #3064: ofono: crash after merging osmo-gsm-tester patch to read IMSIhttps://osmocom.org/issues/3064?journal_id=82382018-03-14T18:17:33Zpespin
<ul></ul><p>According to ofono maintainer, it may be a similar issue as discussed in <a class="external" href="https://www.mail-archive.com/ofono@ofono.org/msg17488.html">https://www.mail-archive.com/ofono@ofono.org/msg17488.html</a></p> OsmoGSMTester - Bug #3064: ofono: crash after merging osmo-gsm-tester patch to read IMSIhttps://osmocom.org/issues/3064?journal_id=82402018-03-14T18:45:40Zpespin
<ul></ul><p>I added a patch to hopefully workaround the issue until we and/or upstream find a proper fix: <a class="external" href="https://gerrit.osmocom.org/#/c/7291/">https://gerrit.osmocom.org/#/c/7291/</a><br />I'll leave it running tests in RnD for a while and see if I can still crash ofono with it. I'll try to change the prod jenkins job this night to use branch pespin/workaround-3064 which contains the workaround on top of current master, and see if tests pass fine during night.</p> OsmoGSMTester - Bug #3064: ofono: crash after merging osmo-gsm-tester patch to read IMSIhttps://osmocom.org/issues/3064?journal_id=82422018-03-15T10:22:06Zpespin
<ul></ul><p>I didn't see the crash since I added the 1sec delay between using SImManager and powering off. Merging it for now as a workaround, but the issue is still there and should be fixed.</p>
<p>Way to reproduce: Remove the sleep 1 before set_power(false) in modem.py power_cycle(), then run in a loop ussd suite and it will eventually crash after a few runs. A standalone python script to cause the crash can probably be generated too, by SetProperty Powered true, then call GetProperties() until the property containing the IMSI appears, then call setProperty Powered False.</p> OsmoGSMTester - Bug #3064: ofono: crash after merging osmo-gsm-tester patch to read IMSIhttps://osmocom.org/issues/3064?journal_id=83482018-03-19T16:43:21Zpespin
<ul></ul><p>Denis shared following patch which seems to be fixing the issue: <a class="external" href="https://pastebin.ca/4003426">https://pastebin.ca/4003426</a></p>
<p>I expect it to be merged soon.</p>
<pre>
diff --git a/src/simfs.c b/src/simfs.c
index efee13e5..c7ddbb38 100644
--- a/src/simfs.c
+++ b/src/simfs.c
@@ -98,10 +98,8 @@ static void sim_fs_op_free(gpointer pointer)
struct sim_fs *fs = node->context->fs;
/* only release the session if there are no pending reads */
- if (fs->session && g_queue_is_empty(fs->op_q)) {
+ if (fs->watch_id && g_queue_is_empty(fs->op_q))
__ofono_sim_remove_session_watch(fs->session, fs->watch_id);
- fs->watch_id = 0;
- }
g_free(node->buffer);
g_free(node);
@@ -900,6 +898,13 @@ static void session_read_info_cb(const struct ofono_error *error,
}
}
+static void session_destroy_cb(void *userdata)
+{
+ struct sim_fs *fs = userdata;
+
+ fs->watch_id = 0;
+}
+
static void get_session_cb(ofono_bool_t active, int session_id,
void *data)
{
@@ -955,7 +960,7 @@ static gboolean sim_fs_op_next(gpointer user_data)
else
fs->watch_id = __ofono_sim_add_session_watch(
fs->session, get_session_cb,
- fs, NULL);
+ fs, session_destroy_cb);
}
} else {
switch (op->structure) {
</pre> OsmoGSMTester - Bug #3064: ofono: crash after merging osmo-gsm-tester patch to read IMSIhttps://osmocom.org/issues/3064?journal_id=83502018-03-19T16:59:25Zpespin
<ul></ul><p>Merged in <a class="external" href="https://git.kernel.org/pub/scm/network/ofono/ofono.git/commit/?id=a99c0be535410a92773ffdfbebb766bec66b66fe">https://git.kernel.org/pub/scm/network/ofono/ofono.git/commit/?id=a99c0be535410a92773ffdfbebb766bec66b66fe</a>.</p>
<p>TODO: <br />- Rebase osmo-gsm-tester branch on top of this commit, and deploy it in RnD and prod. It was already tested in RnD that indeed it fixes the crash.<br />- Submit patch to gerrit reverting osmo-gsm-tester f8d12196f05b20aa28b3103db26ea37ba6849362</p> OsmoGSMTester - Bug #3064: ofono: crash after merging osmo-gsm-tester patch to read IMSIhttps://osmocom.org/issues/3064?journal_id=87812018-04-10T14:31:33Zlynxis
<ul><li><strong>Assignee</strong> changed from <i>lynxis</i> to <i>pespin</i></li></ul><p><a class="user active" href="https://osmocom.org/users/30187">pespin</a>: I'm unsure, what I should do here. It seems to fixed everything :).</p> OsmoGSMTester - Bug #3064: ofono: crash after merging osmo-gsm-tester patch to read IMSIhttps://osmocom.org/issues/3064?journal_id=87872018-04-10T15:47:24Zpespin
<ul><li><strong>Assignee</strong> changed from <i>pespin</i> to <i>lynxis</i></li></ul><p>The TODO list is still in TODO ;)</p>
<p>- Rebase osmo-gsm-tester branch on top of this commit <- this may have already been done. However, there were several QMI mem leaks fixed recently, I think it makes sense to rebase on top of master.<br />- Deploy it in RnD and prod. It was already tested in RnD that indeed it fixes the crash. <- Once rebased, this needs to be done.<br />- Submit patch to gerrit reverting osmo-gsm-tester f8d12196f05b20aa28b3103db26ea37ba6849362 <- Once new ofono is in place, this needs to be done.</p> OsmoGSMTester - Bug #3064: ofono: crash after merging osmo-gsm-tester patch to read IMSIhttps://osmocom.org/issues/3064?journal_id=138432019-04-08T19:12:26Zpespin
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Feedback</i></li><li><strong>Assignee</strong> changed from <i>lynxis</i> to <i>pespin</i></li></ul><p>Revert submitted here:</p>
<p><a class="external" href="https://gerrit.osmocom.org/#/c/osmo-gsm-tester/+/13550">https://gerrit.osmocom.org/#/c/osmo-gsm-tester/+/13550</a> Revert "modem: workaround ofono crash"</p> OsmoGSMTester - Bug #3064: ofono: crash after merging osmo-gsm-tester patch to read IMSIhttps://osmocom.org/issues/3064?journal_id=139822019-04-15T13:35:18Zpespin
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Resolved</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li></ul><p>No crash spotted during few days after reverting the patch, closing the ticket.</p>