https://osmocom.org/https://osmocom.org/favicon.ico?16647414092023-03-11T07:51:31ZOpen Source Mobile CommunicationsOsmoBTS - Bug #5944: DTXu + AMR on TCH/H appears to be not usable on osmo-bts-sysmohttps://osmocom.org/issues/5944?journal_id=263492023-03-11T07:51:31Zlaforge
<ul></ul>Some Random Ideas (not saying Keith should do all of that):
<ul>
<li>look at the dsp trace log (Play with the flags) to see if there's anything useful in it</li>
<li>capture what's actually happening on the air interface using osmocomBB fixeria/burst_ind for detailed analysis.</li>
</ul> OsmoBTS - Bug #5944: DTXu + AMR on TCH/H appears to be not usable on osmo-bts-sysmohttps://osmocom.org/issues/5944?journal_id=264292023-03-20T23:24:13Zfixeria
<ul><li><strong>Category</strong> set to <i>osmo-bts-sysmo</i></li></ul> OsmoBTS - Bug #5944: DTXu + AMR on TCH/H appears to be not usable on osmo-bts-sysmohttps://osmocom.org/issues/5944?journal_id=264732023-03-26T17:11:01Zfalconia
<ul></ul><p>Question: was this problem seen at CCC? I seem to recall hearing that the GSM network at CCC was run with sysmoBTS units, and I definitely remember reading that they used the AMR codec exclusively, not supporting FR1 or EFR. Did they perhaps disable DTXu, or did the combo of sysmoBTS+AMR+DTXu actually work for other people?</p> OsmoBTS - Bug #5944: DTXu + AMR on TCH/H appears to be not usable on osmo-bts-sysmohttps://osmocom.org/issues/5944?journal_id=264832023-03-26T21:33:10Zkeith
<ul></ul><p>I just checked the congress config.</p>
<p>DTXu, unsurprisingly, was not enabled.</p> OsmoBTS - Bug #5944: DTXu + AMR on TCH/H appears to be not usable on osmo-bts-sysmohttps://osmocom.org/issues/5944?journal_id=264852023-03-26T21:42:41Zkeith
<ul></ul><p>laforge wrote in <a href="#note-1">#note-1</a>:</p>
<blockquote>
Some Random Ideas (not saying Keith should do all of that):
<ul>
<li>look at the dsp trace log (Play with the flags) to see if there's anything useful in it</li>
<li>capture what's actually happening on the air interface using osmocomBB fixeria/burst_ind for detailed analysis.</li>
</ul>
</blockquote>
<p>I did play around with the dsp trace flags, including piping that output into pcaps to correlate with gsmtap/osmo-bts-sysmo logging. <br />So far that did not reveal anything extra about what was happening.</p>
<p>So capturing the uplink off the air would reveal more. I'm not sure I have the hardware to do that. I haven't looked at it in a long time.</p>
<p>Assuming, for the moment, (giving everything I tried with hacking on osmo-bts) the problem is in the L1 and therefore might be a cannot_fix issue, sniffing the UL might be a nice investigation project for somebody with a lot of free time.</p> OsmoBTS - Bug #5944: DTXu + AMR on TCH/H appears to be not usable on osmo-bts-sysmohttps://osmocom.org/issues/5944?journal_id=264962023-03-27T10:02:24Zlaforge
<ul></ul><p>AFAIR, there are several modes in which the L1 can be operated. We're using a mode where we get more or less the RTP payload, as that's rather convenient. So I wouldn't give up on not being able to fis this at all.</p>
<p>Getting air interface traces via osmocom-bb <code>tnt/burst_ind</code> is rather easy, I've just done it very recently for getting CSD call traces. Still not saying that you (Keith) should do that.</p> OsmoBTS - Bug #5944: DTXu + AMR on TCH/H appears to be not usable on osmo-bts-sysmohttps://osmocom.org/issues/5944?journal_id=264972023-03-27T11:26:42Zlaforge
<ul><li><strong>File</strong> <a href="/attachments/6688">amr-sid.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/6688/amr-sid.png">amr-sid.png</a> added</li><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul><a name="Really-a-generic-AMR-or-only-a-TCHH-only-problem"></a>
<h3 >Really a generic AMR or only a TCH/H only problem?<a href="#Really-a-generic-AMR-or-only-a-TCHH-only-problem" class="wiki-anchor">¶</a></h3>
<p>First of all, can you please clarify whether the problem only appers on TCH/H?</p>
<p>So far, this bug subject or title doesn't constrain it to any channel type, so it clams <em>all</em> AMR is affected when DTXu is in place.</p>
<p>However, the <code>SID_*_INH</code> types are specific to TCH/H. So if there's a problem related to those, it should not exist on TCH/F. Please clarify.</p>
<a name="Suspicious-treatment-of-AMR-seq-during-working-DTX-with-no-inhibit"></a>
<h3 >Suspicious treatment of AMR seq during working DTX with no inhibit<a href="#Suspicious-treatment-of-AMR-seq-during-working-DTX-with-no-inhibit" class="wiki-anchor">¶</a></h3>
<p>Looking at a snippet of your pcap file:<br /><img src="https://osmocom.org/attachments/download/6688/amr-sid.png" alt="" /></p>
<ul>
<li>we can see from packet 514 onwards, that every PH-DATA.ind generates one AMR/RTP frame.
<ul>
<li>FN is +4 / +5, as expected, equalling 18 .. 22.5ms interval</li>
<li>sequence number monotonically increasing</li>
<li>timestamps incrementing 160 ticks (= 20ms)</li>
</ul>
</li>
<li>packet 526..528 (FN=256407) show the SID_FIRST_P1 frame
<ul>
<li>AMR: seq + timestamp to previous RTP frame as expected (+1 / +160)</li>
</ul>
</li>
<li>packet 529..530 (FN=256412, +5) show the SID_FIRST_P2 frame
<ul>
<li>no AMR is generated</li>
</ul>
</li>
<li>packet 531 (FN=256416, +4)
<ul>
<li>empty payload reported despite excellent RSSI and quality</li>
<li>no AMR is generated</li>
</ul>
</li>
<li>packet 532..534 (FN=256420, +4) shows a SID frame
<ul>
<li>AMR seq +1 / timestamp + 480 (= 60ms). Matches the ~59 ms for FN difference of 13</li>
</ul>
</li>
<li>packet 535 (FN=256416, +4)
<ul>
<li>empty payload reported despite excellent RSSI and quality</li>
</ul>
</li>
<li>packet 536..540
<ul>
<li>empty payload reported, RSSI -110 dBm, so the MS really didn't transmit</li>
</ul>
</li>
<li>packet 541
<ul>
<li>empty payload reported despite excellent RSSI and quality</li>
</ul>
</li>
<li>packet 542..544 (FN=256455) shows a SID frame
<ul>
<li>AMR seq + 4, timestamp + 1280 (= 160ms). Matches the ~158ms for FN difference of 35</li>
<li><strong>why is AMR seq +4?</strong> There were 7 blocks in between (FN 256425, 256429, 256433, 256438, 256442, 256446, 256451) which we didn't generate any AMR for, i.e. which were dropped.</li>
</ul></li>
</ul> OsmoBTS - Bug #5944: DTXu + AMR on TCH/H appears to be not usable on osmo-bts-sysmohttps://osmocom.org/issues/5944?journal_id=265072023-03-27T18:15:58Zlaforge
<ul></ul><p>I really cannot explain some parts of the trace. I cannot reconcile how our code would behave that way. Do you have some local modifications to the AMR DTX logic in place?</p>
<p>Fore example, packet 828..830 seem to indicate that an ONSET received from the PHY would generate a RTP packet. Based on the osmo-bts code in master, I fail to see how that could ever happen. The ONSET would not set any rmsg, and hence add_l1sap_header() should never be called.</p> OsmoBTS - Bug #5944: DTXu + AMR on TCH/H appears to be not usable on osmo-bts-sysmohttps://osmocom.org/issues/5944?journal_id=265082023-03-27T18:20:25Zlaforge
<ul></ul><p>possibly related: <a class="external" href="https://gitlab.com/nrw_noa/osmo-bts/-/commit/0aa2a89af15fa532c585873aaa95da6353e670f8">https://gitlab.com/nrw_noa/osmo-bts/-/commit/0aa2a89af15fa532c585873aaa95da6353e670f8</a></p> OsmoBTS - Bug #5944: DTXu + AMR on TCH/H appears to be not usable on osmo-bts-sysmohttps://osmocom.org/issues/5944?journal_id=265092023-03-27T18:40:17Zlaforge
<ul></ul><p>Another oddity:</p>
<pre>
1481 0 22:52:02.595433279 GSMTAP 258366/194/04/00/10 Rx PH-DATA.ind TCH/H (hL2 000200bb): 09 04 00 50 44 00 00 00 00 0a , Meas: RSSI -43.97 dBm, Qual 20.41 dB, BER 0.00, Timing -3
1482 0 22:52:02.595491257 GSMTAP 258366/194/04/00/10 DTX: received SID_FIRST_P1 from L1 (9 bytes)
1483 0 22:52:02.595612741 AMR PT=AMR, SSRC=0x4178B092, Seq=32959, Time=2092022532
</pre>
So what we see here is that
<ul>
<li>SID_FIRST_P1 is signaled as Amr_SidFirstP1 (instead of expected Amr)</li>
<li>9 bytes of payload 09 04 00 50 44 00 00 00 00 0a
<ul>
<li><code>09</code>: Amr_SidFirstP1</li>
<li><code>04</code>: CMI</li>
<li><code>00</code>: CMR</li>
<li><code>50 44 00 00 00 00 0a</code> is actually the RTP payload the L1 sends us</li>
</ul>
</li>
<li>our code ignores this and instead calls <code>osmo_amr_rtp_enc(sid_first, 0, AMR_SID, AMR_GOOD) which appears to render @50 44 00 00 00</code> without the <code>00 0a</code> suffix. This seems an artefact of first merging <code>c3fb0dcc8cd01a84942d06267003478b972feadb</code> and then <code>c77b574d1fbe43ca19db0e5e041b3b5e2a71b856</code> by msuraev in 2016. While P2 indeed has no payload, P1 does!
<ul>
<li><strong>Why is the RTP payload from L1 different than our <code>osmo_amr_rtp_enc</code>?</strong></li>
</ul>
</li>
<li>our transmitted RTP payload is (after RTP header) is <code>50 44 00 00 00</code>
<ul>
<li><strong>this is wrong. There is no CMR and no TOC bytes preceding</strong>. The total packet size should be 7 bytes: CMR (1) + TOC (1) + 39bits-SID+padbits (5).</li>
</ul></li>
</ul> OsmoBTS - Bug #5944: DTXu + AMR on TCH/H appears to be not usable on osmo-bts-sysmohttps://osmocom.org/issues/5944?journal_id=265102023-03-27T18:59:30Zkeith
<ul></ul><p>laforge wrote in <a href="#note-8">#note-8</a>:</p>
<blockquote>
<p>I really cannot explain some parts of the trace. I cannot reconcile how our code would behave that way. Do you have some local modifications to the AMR DTX logic in place?</p>
</blockquote>
<p>Yes!, As I wrote in the description: "The pcap comes from an osmo-bts with some modifications to logging and RTP marking, (ignore the BAD AMR frames following ONSET"</p>
<p>I should have been more explicit, or include a link to the actual source used. I've since been hacking on that without making any commits so I don't actualy have whatever was used at the time, but as I said in the description also, I didn't think that this RTP packet you are seeing generated as a result of the ONSET is relevant to the issue.</p>
<blockquote>
<p>Fore example, packet 828..830 seem to indicate that an ONSET received from the PHY would generate a RTP packet. Based on the osmo-bts code in master, I fail to see how that could ever happen. The ONSET would not set any rmsg, and hence add_l1sap_header() should never be called.</p>
</blockquote>
<p>Correct, I was playing around with many things at the time, trying to understand and eliminate the cause of so much "RTP clock out of sync with lower layer"</p>
<p>What I was doing was counting from 0 to 10, along with the "LSB" of wall clock seconds. I try to maintain silence between the numbers and I'm listening to the B-leg and I note which numbers I did not hear. This helps to look at the trace and identify where I need to look. This one worked quite well, but I happened to be running code at the time that was sending this RTP packet you identify.</p>
<p>Now you might say, oh well this is invalid if you are sending RTP packets that osmo-bts master would never send. You'll have to believe me, it makes NO difference whatsoever.</p>
<p>At the time I made this pcap, I had already spent about 3 days!! (granted, of my rather slow process of analysis and understanding) looking at the osmo-bts code related to RX from L1 and generation of RTP. So at this point I am 99.9999% convinced that it has nothing to do with that, what I wanted to demonstrate here is that I am speaking, but the L1 is not sending any speech data. This is why I wrote "that's a hack but not relevant".</p>
<p>I'm sorry if this is confusing or has caused you to waste valuable debugging time, but I think I am quite explicit in the description that this is not osmo-bts master</p> OsmoBTS - Bug #5944: DTXu + AMR on TCH/H appears to be not usable on osmo-bts-sysmohttps://osmocom.org/issues/5944?journal_id=265112023-03-27T19:09:40Zlaforge
<ul></ul><p>laforge wrote in <a href="#note-10">#note-10</a>:</p>
<blockquote>
<ul>
<li><strong>this is wrong. There is no CMR and no TOC bytes preceding</strong>. The total packet size should be 7 bytes: CMR (1) + TOC (1) + 39bits-SID+padbits (5).</li>
</ul>
</blockquote>
<p>Untested fix in <a class="external" href="https://gerrit.osmocom.org/c/osmo-bts/+/32088">https://gerrit.osmocom.org/c/osmo-bts/+/32088</a></p> OsmoBTS - Bug #5944: DTXu + AMR on TCH/H appears to be not usable on osmo-bts-sysmohttps://osmocom.org/issues/5944?journal_id=265152023-03-27T20:49:14Zkeith
<ul></ul><p>laforge wrote in <a href="#note-7">#note-7</a>:</p>
<blockquote>
<a name="Really-a-generic-AMR-or-only-a-TCHH-only-problem"></a>
<h3 >Really a generic AMR or only a TCH/H only problem?<a href="#Really-a-generic-AMR-or-only-a-TCHH-only-problem" class="wiki-anchor">¶</a></h3>
<p>First of all, can you please clarify whether the problem only appers on TCH/H?</p>
</blockquote>
<p>I have clarified, that <strong>the problem only exists on TCH/H</strong>.</p>
<p>Using TCH/F, I did various (listening) tests of MS -> MS calls and also MS -> Sip Endpoint (with libopencore-amrnb)</p>
<p>Audio is clear and complete.</p> OsmoBTS - Bug #5944: DTXu + AMR on TCH/H appears to be not usable on osmo-bts-sysmohttps://osmocom.org/issues/5944?journal_id=265162023-03-27T20:51:24Zkeith
<ul><li><strong>Subject</strong> changed from <i>DTXu + AMR appears to be not usable on osmo-bts-sysmo</i> to <i>DTXu + AMR on TCH/H appears to be not usable on osmo-bts-sysmo</i></li></ul> OsmoBTS - Bug #5944: DTXu + AMR on TCH/H appears to be not usable on osmo-bts-sysmohttps://osmocom.org/issues/5944?journal_id=265222023-03-28T00:00:19Zlaforge
<ul><li><strong>File</strong> <a href="/attachments/6693">sysmobts-amr-hr-dtx-working.pcap</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/6693/sysmobts-amr-hr-dtx-working.pcap">sysmobts-amr-hr-dtx-working.pcap</a> added</li><li><strong>File</strong> <a href="/attachments/6694">sysmobts-amr-hr-dtx-some_dropouts_counting.pcap</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/6694/sysmobts-amr-hr-dtx-some_dropouts_counting.pcap">sysmobts-amr-hr-dtx-some_dropouts_counting.pcap</a> added</li></ul>I've also done some tests here in a local setup:
<ul>
<li>osmo-bts-sysmo + osmo-pcu from current 201705-nightly on sysmoBTS 1002 @ 192.168.100.233</li>
<li>osmo-bsc (current mater, built from source) + osmo-mgw on my laptop (192.168.100.149)</li>
<li>osmo-msc,hlr,stp,... on 192.168.11.4</li>
<li>2x MS (Motorola C1xx with Calypos chipset, stock firmware)</li>
</ul>
<p>I'm attaching two pcap files containing GSMTAP from BTS+BSC, Abis, OML, RSL. My results were not very conclusive. The first attempt (<a class="attachment" href="https://osmocom.org/attachments/6693">sysmobts-amr-hr-dtx-working.pcap</a>) sounded fine to me. The second attempt, I was counting up from 1 to 32 (or so), with long breaks between the words. About 5 of those words spoken on the caller side didn't make it to the called party. See <a class="attachment" href="https://osmocom.org/attachments/6694">sysmobts-amr-hr-dtx-some_dropouts_counting.pcap</a></p>
<p>One thing I noticed looking at the dsp trace:<br /><pre>
TchAhs_DecodeSpeech() - MS CMI (2) is not in ACS [3,0,0,0], using init values (InitCodecMode = 0, rxCodec = 3)
TchAhs_DecodeSpeech() - MS CMI (2) is not in ACS [3,0,0,0], using init values (InitCodecMode = 0, rxCodec = 3)
</pre></p>
<p>occurred virtually at every word after a period of silence. So the MS is sending CMI=2 (5.9kbps) but the active codec set doesn't contain that? This looks rather odd and deserves further analysis.</p> OsmoBTS - Bug #5944: DTXu + AMR on TCH/H appears to be not usable on osmo-bts-sysmohttps://osmocom.org/issues/5944?journal_id=265232023-03-28T00:11:10Zlaforge
<ul></ul><p>laforge wrote in <a href="#note-15">#note-15</a>:</p>
<blockquote>
<p>occurred virtually at every word after a period of silence. So the MS is sending CMI=2 (5.9kbps) but the active codec set doesn't contain that? This looks rather odd and deserves further analysis.</p>
</blockquote>
The MultiRateConfig IE in the RSL CHAN ACT looks fine, as per wireshark:
<ul>
<li>ICMI: implicit</li>
<li>Start Mode: 0</li>
<li>5.90k is the only mode part of the subset</li>
</ul>
<p>The same is also sent to the MS in the RR ASSIGNMENT COMMAND. So if there's any miscommunication/misinterpretation, it appears to be between osmo-bts and the DSP.</p> OsmoBTS - Bug #5944: DTXu + AMR on TCH/H appears to be not usable on osmo-bts-sysmohttps://osmocom.org/issues/5944?journal_id=265252023-03-28T01:59:32Zkeith
<ul><li><strong>File</strong> <a href="/attachments/6695">iphone5.pcapng</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/6695/iphone5.pcapng">iphone5.pcapng</a> added</li></ul><p>Here's a pcap of an iPhone 5c, in general, this phone is working so much better than anything else I have tried, i very rarely have any lost or clipped words.</p>
<p>(quick hint: "!gsmtap_log.string == "\n" && !gsmtap_log.string contains "[RACH]____[ACCESS TYPE" is a good base filter to remove that noise.)</p>
<p>This is call (the b-leg is a sip endpoint), answer, then mute the MS microphone and then unmute, hangup.</p>
<p>It is interesting to note (gsmtap_log.string contains "DTX:") that there is not one single ONSET. The iPhone seems to send pure SID_FIRST_INH all the time. (actually I later saw other behaviour, but only after the mic was muted for quite some time)</p>
<p>The CMI not in ActiveCodecSet issue (gsmtap_log.string contains "not in ACS") gets even more curious here, as I also see:</p>
<pre>
MS CMI (3) is not in ACS [3,6,0,0], using init values (InitCodecMode = 0, rxCodec = 3)
</pre>
<p>3 not in 3? wtf?</p>
<p>also, seems the L1 counts codecs from 1, we count from 0 (my CHAN ACT with Multirate config is in the pcap.</p>
<p>AND, to be CLEAR: This is what I was running: <a class="external" href="https://cgit.osmocom.org/osmo-bts/?h=keith%2Fdtx-hacking">https://cgit.osmocom.org/osmo-bts/?h=keith%2Fdtx-hacking</a></p>
<p>+ two lines of extra silly logging from libosmo-abis when I was trying to get a handle on the code flow<br />+ I'm using l1fwd-proxy and osmo-bts-sysmo-remote on x86</p> OsmoBTS - Bug #5944: DTXu + AMR on TCH/H appears to be not usable on osmo-bts-sysmohttps://osmocom.org/issues/5944?journal_id=265262023-03-28T02:01:00Zkeith
<ul></ul><p>laforge wrote in <a href="#note-15">#note-15</a>:</p>
<blockquote>
<p>I was counting up from 1 to 32 (or so), with long breaks between the words. About 5 of those words spoken on the caller side didn't make it to the called party.</p>
</blockquote>
<p>5 out of ~30, sounds similar to my experience.</p> OsmoBTS - Bug #5944: DTXu + AMR on TCH/H appears to be not usable on osmo-bts-sysmohttps://osmocom.org/issues/5944?journal_id=265402023-03-28T15:56:14Zlaforge
<ul></ul><p>regarding the <em>CMI not in ACS</em> messages, I'm now dumping how we configure the DSP/PHY during MPH-ACTIVATE.req:</p>
<ul>
<li>amrCmiPhase=1 (GsmL1_AmrCmiPhase_Odd)</li>
<li>amrInitCodec=0 (GsmL1_AmrCodecMode_1; first mode of the modeset)</li>
<li>amrActriveCodecSet={3,0,0,0} (GsmL1_AmrCodec_5_9, GsmL1_AmrCodec_Unset, GsmL1_AmrCodec_Unset, GsmL1_AmrCodec_Unset, GsmL1_AmrCodec_Unset)</li>
</ul>
<p>This looks all good, and as expected. Not sure why we get those strange log messages from the DSP.</p>