https://osmocom.org/https://osmocom.org/favicon.ico?16647414092020-06-02T22:32:31ZOpen Source Mobile CommunicationsCore testing infrastructure - Bug #4576: ttcn3-hlr-tests need to route multicast traffic between ttcn3-hlr-test and osmo-hlr-master containershttps://osmocom.org/issues/4576?journal_id=185182020-06-02T22:32:31Zneelsnhofmeyr@sysmocom.de
<ul></ul><p>Looking on the docker-hosting machine, I see some restrictive rules in "DOCKER-ISOLATION-STAGE-1",<br />maybe some custom iptables could make the multicast pass through?<br />OTOH, do we really need to resort to manual iptables rules?</p>
<pre>
sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy DROP)
target prot opt source destination
DOCKER-USER all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain DOCKER (1 references)
target prot opt source destination
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DROP all -- !172.18.10.0/24 anywhere
DROP all -- anywhere !172.18.10.0/24
DROP all -- !172.18.2.0/24 anywhere
DROP all -- anywhere !172.18.2.0/24
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
RETURN all -- anywhere anywhere
Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target prot opt source destination
DROP all -- anywhere anywhere
RETURN all -- anywhere anywhere
Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere
</pre>
<p>(BTW, during this iptables, ttcn3-bsc-tests was also active on 172.18.2.0/24. ttcn3-hlr-tests is using 172.18.10.0/24)</p> Core testing infrastructure - Bug #4576: ttcn3-hlr-tests need to route multicast traffic between ttcn3-hlr-test and osmo-hlr-master containershttps://osmocom.org/issues/4576?journal_id=185192020-06-02T22:59:54Zneelsnhofmeyr@sysmocom.de
<ul></ul><p>On the docker-hosting machine, when I run</p>
<pre><code>iptables -I DOCKER-ISOLATION-STAGE-1 -s 172.18.10.0/24 -d 239.192.23.42 -j ACCEPT</code></pre>
<p>the traffic reaches the osmo-hlr-master container and osmo-hlr responds as expected.<br />However, the response still doesn't trigger the ttcn3 tester.</p>
<pre><code>316 15.062310 Jun 3, 2020 00:50:04.994885000 CEST 172.18.10.103 4266 239.192.23.42 4266 Standard query 0x7463 ANY sip.voice.491615433824.msisdn.mdns.osmocom.org<br />327 15.063073 Jun 3, 2020 00:50:04.995648000 CEST 172.18.10.20 4266 239.192.23.42 4266 Standard query response 0x7463 TXT A 66.66.66.66 TXT</code></pre>
<p>The response also goes from 172.18.10.0/24 to 239.192.23.42, and I can also see it in a tcpdump running in the nonjenkins-ttcn3-hlr-test container.<br />So there is still some reason why the tester doesn't pick up the response, even though it is apparently reaching inside the container.</p>
<p>Would be nice to get a hint from <a class="user active" href="https://osmocom.org/users/301771">osmith</a> about how stuff used to work. I'll ping him tomorrow.</p> Core testing infrastructure - Bug #4576: ttcn3-hlr-tests need to route multicast traffic between ttcn3-hlr-test and osmo-hlr-master containershttps://osmocom.org/issues/4576?journal_id=185202020-06-02T23:00:24Zneelsnhofmeyr@sysmocom.de
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Feedback</i></li><li><strong>Assignee</strong> set to <i>osmith</i></li></ul> Core testing infrastructure - Bug #4576: ttcn3-hlr-tests need to route multicast traffic between ttcn3-hlr-test and osmo-hlr-master containershttps://osmocom.org/issues/4576?journal_id=185212020-06-03T08:10:18Zlaforge
<ul></ul><p>On Tue, Jun 02, 2020 at 10:59:56PM +0000, neels [REDMINE] wrote:</p>
<blockquote>
<p>The response also goes from 172.18.10.0/24 to 239.192.23.42, and I can also see it in a tcpdump running in the nonjenkins-ttcn3-hlr-test container.</p>
</blockquote>
<p>maybe the multicast TTL (set by the sender) is not high enough and it times out (reaches zero)<br />in the target (destination, receiver) container and hence is dropped before being delivered?</p>
<p>Unfortunately the pcap of the tcpdump was not attached here, otherwise I could have looked at<br />it quickly.</p>
<p>There is also the possibility of iptables/ntftables rules inside the destination container,<br />maybe also set up by docker? There is one separate instance of all the packet filter<br />tables/chains/hooks/rules in every network namespace, i.e. one on the host and one in each of the<br />containers.</p> Core testing infrastructure - Bug #4576: ttcn3-hlr-tests need to route multicast traffic between ttcn3-hlr-test and osmo-hlr-master containershttps://osmocom.org/issues/4576?journal_id=185222020-06-03T13:05:16Zneelsnhofmeyr@sysmocom.de
<ul><li><strong>File</strong> <a href="/attachments/4189">on_docker_host.pcapng</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/4189/on_docker_host.pcapng">on_docker_host.pcapng</a> added</li><li><strong>File</strong> <a href="/attachments/4190">nonjenkins-ttcn3-hlr-test.pcap</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/4190/nonjenkins-ttcn3-hlr-test.pcap">nonjenkins-ttcn3-hlr-test.pcap</a> added</li><li><strong>File</strong> <a href="/attachments/4191">nonjenkins-hlr.pcap</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/4191/nonjenkins-hlr.pcap">nonjenkins-hlr.pcap</a> added</li></ul><p>interesting to note, when I restarted the hlr test docker containers today, the iptables rule to accept the multicast address was still there, but docker inserted new DROP rules before my rule, rendering my rule ineffective.<br />So I need to re-add the multicast ACCEPT rule as first rule every time the test docker containers restart.</p>
<p>This time I did only -d, no -s</p>
<pre><code>iptables -I DOCKER-ISOLATION-STAGE-1 -d 239.192.23.42 -j ACCEPT</code></pre>
<p>Took three pcaps, on the docker host, inside the ttcn3 tester container, inside the osmo-hlr container (attached).</p>
<p>The UDP "Time to live" is 1. Seems too low considering a bridge between the containers?<br />But still, the MSLookup request is actually received by osmo-hlr, only the response (also UDP ttl == 1) is not received by the tester.</p> Core testing infrastructure - Bug #4576: ttcn3-hlr-tests need to route multicast traffic between ttcn3-hlr-test and osmo-hlr-master containershttps://osmocom.org/issues/4576?journal_id=185232020-06-03T13:34:54Zneelsnhofmeyr@sysmocom.de
<ul></ul><p>neels wrote:</p>
<blockquote>
<p>The UDP "Time to live" is 1.</p>
</blockquote>
<p>of course IP ttl</p> Core testing infrastructure - Bug #4576: ttcn3-hlr-tests need to route multicast traffic between ttcn3-hlr-test and osmo-hlr-master containershttps://osmocom.org/issues/4576?journal_id=185262020-06-03T14:39:08Zneelsnhofmeyr@sysmocom.de
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>In Progress</i></li><li><strong>Assignee</strong> changed from <i>osmith</i> to <i>neels</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>40</i></li></ul><p>about the question why it worked for osmith but not for jenkins nor me:<br />osmith tells me that he probably ran the tests without docker containers, with ttcn tester and osmo-hlr all running on the host machine directly.<br />So indeed we need to figure out an all-new way how docker allows the MC UDP traffic across containers.</p>
<p>I'm at the point where I am now seeing the mDNS response on the tcpdump run right next to the ttcn tester program,<br />but still the tester doesn't seem to pick them up.</p>
<p>In terms of the IP TTL, the tcpdump is on the same stage as the tester, right?<br />If the tcpdump in that container sees it, the tester should, too?</p>
<p>I still need to examine the iptables within the docker container, haven't done that yet.</p> Core testing infrastructure - Bug #4576: ttcn3-hlr-tests need to route multicast traffic between ttcn3-hlr-test and osmo-hlr-master containershttps://osmocom.org/issues/4576?journal_id=185312020-06-03T18:00:13Zlaforge
<ul></ul><p>On Wed, Jun 03, 2020 at 02:39:09PM +0000, neels [REDMINE] wrote:</p>
<blockquote>
<p>In terms of the IP TTL, the tcpdump is on the same stage as the tester, right?</p>
</blockquote>
<p>no. tcpdump works (despite its name) on the Ethernet layer, or more precisely at <br />the boundary between Layer2 and Layer3. This means all of the local IP stack, UDP and socket<br />code has not yet been traversed. So it may very well be that the IP stack is decrementing<br />the TTL to 0 and then dropping the packet. There might be some related counters visible in<br />'lnstat', but I think the easiest thing is to increase the TTL on the sender side (socket option)<br />and see if it arrives.</p>
<p>The above is also the reason why you see packets in tcpdump which are then subsequenty dropped<br />by packet filter rules.</p>
<blockquote>
<p>If the tcpdump in that container sees it, the tester should, too?</p>
</blockquote>
<p>No, see above.</p> Core testing infrastructure - Bug #4576: ttcn3-hlr-tests need to route multicast traffic between ttcn3-hlr-test and osmo-hlr-master containershttps://osmocom.org/issues/4576?journal_id=185332020-06-03T18:45:11Zlaforge
<ul></ul><p>On Wed, Jun 03, 2020 at 07:59:39PM +0200, Harald Welte wrote:</p>
<blockquote>
<p>On Wed, Jun 03, 2020 at 02:39:09PM +0000, neels [REDMINE] wrote:</p>
<blockquote>
<p>In terms of the IP TTL, the tcpdump is on the same stage as the tester, right?</p>
</blockquote>
<p>There might be some related counters visible in 'lnstat', [...]</p>
</blockquote>
<p>not in lnstat, but in /proc/net/snmp</p>
<pre>
cat /proc/net/snmp
</pre><br />inside the receiving container should give you the following two lines:
<pre>
Ip: Forwarding DefaultTTL InReceives InHdrErrors InAddrErrors ForwDatagrams InUnknownProtos InDiscards InDelivers OutRequests OutDiscards OutNoRoutes ReasmTimeout ReasmReqds ReasmOKs ReasmFails FragOKs FragFails FragCreates
Ip: 1 64 21422602 0 0 1931698 0 0 19101351 11638738 6387 2468 0 0 0 0 0 33 0
</pre>
<p>So the fourth numeric field in the second line is InHdrErrors, which is the value that's incremented every<br />time an IP packet TTL exceeds (or other IP header problems are encountered, but as you can see in the above<br />example, it's still 0 on my laptop with an uptime of 24 days, so you shouldn't see to many other errors<br />incrementing that value.</p> Core testing infrastructure - Bug #4576: ttcn3-hlr-tests need to route multicast traffic between ttcn3-hlr-test and osmo-hlr-master containershttps://osmocom.org/issues/4576?journal_id=186082020-06-08T14:49:02Zneelsnhofmeyr@sysmocom.de
<ul></ul><p>I hacked a TTL 2 into osmo-hlr's osmo_mdns_sock_init(), the answer still does not reach the tester.<br />Now the tester sends out an mDNS query with TTL 1 that reaches osmo-hlr fine,<br />then osmo-hlr sends back an mDNS response with TTL 2 that does not reach the tester.</p>
<p>(Just to make sure I also tried with TTL 3)</p>
<p>Seems like the TTL is not the problem.</p>
<p>(We should probably anyway consider adding the multicast TTL as config option to osmo-hlr and osmo-mslookup-client, independently from this issue)</p> Core testing infrastructure - Bug #4576: ttcn3-hlr-tests need to route multicast traffic between ttcn3-hlr-test and osmo-hlr-master containershttps://osmocom.org/issues/4576?journal_id=186092020-06-08T15:03:54Zneelsnhofmeyr@sysmocom.de
<ul></ul><p>running iptables inside the docker containers seems not to be an option at all.<br />I installed the 'iptables' package, then get:</p>
<pre>
root@4c22177bd320:/data# iptables -L
iptables v1.6.0: can't initialize iptables table `filter': Permission denied (you must be root)
</pre>
<p>(but obviously am root in the container)</p> Core testing infrastructure - Bug #4576: ttcn3-hlr-tests need to route multicast traffic between ttcn3-hlr-test and osmo-hlr-master containershttps://osmocom.org/issues/4576?journal_id=186102020-06-08T15:10:53Zneelsnhofmeyr@sysmocom.de
<ul></ul><p>considering: could it make sense to, just for those tests, configure osmo-hlr to listen and respond to mslookup mDNS in <strong>unicast</strong>, not multicast? Seems ugly, but if it helps with testing.....</p> Core testing infrastructure - Bug #4576: ttcn3-hlr-tests need to route multicast traffic between ttcn3-hlr-test and osmo-hlr-master containershttps://osmocom.org/issues/4576?journal_id=186132020-06-08T15:33:20Zneelsnhofmeyr@sysmocom.de
<ul></ul><p>Looking closely in the ttcn3 log output, I see the mDNS port being set up and messages going out on it, and also the same sent messages being received right back on mDNS (as expected in multicast),<br />but no answer coming from osmo-hlr is received.</p>
<p>...maybe change the HLR tests so that osmo-hlr and the ttcn tester run in the same docker image?</p> Core testing infrastructure - Bug #4576: ttcn3-hlr-tests need to route multicast traffic between ttcn3-hlr-test and osmo-hlr-master containershttps://osmocom.org/issues/4576?journal_id=201692020-10-28T20:13:35Zneelsnhofmeyr@sysmocom.de
<ul></ul><p>in some forum i got this hint:</p>
<pre>
Use the '--network host' option, and the container will use same network stack as the host operating system. This only works on Linux, though.
</pre> Core testing infrastructure - Bug #4576: ttcn3-hlr-tests need to route multicast traffic between ttcn3-hlr-test and osmo-hlr-master containershttps://osmocom.org/issues/4576?journal_id=201702020-10-28T20:14:46Zneelsnhofmeyr@sysmocom.de
<ul></ul><p>erm that '--network host' was actually an answer to the question of how to run a service in a docker container that should be fully visible on a public network interface, not necessarily about multicast between containers with separate networks</p> Core testing infrastructure - Bug #4576: ttcn3-hlr-tests need to route multicast traffic between ttcn3-hlr-test and osmo-hlr-master containershttps://osmocom.org/issues/4576?journal_id=201802020-10-28T20:40:08Zfixeria
<ul></ul><p>Unfortunately, yes: '--network host' is the only way to get multicast traffic working in Docker:</p>
<p><a class="external" href="https://stackoverflow.com/questions/51737969/how-to-support-multicast-network-in-docker">https://stackoverflow.com/questions/51737969/how-to-support-multicast-network-in-docker</a><br /><a class="external" href="https://github.com/moby/libnetwork/issues/552">https://github.com/moby/libnetwork/issues/552</a></p>
<p>This is another example why (at least now) Docker is not a good fit for building telecom infrastructure on top of it.</p> Core testing infrastructure - Bug #4576: ttcn3-hlr-tests need to route multicast traffic between ttcn3-hlr-test and osmo-hlr-master containershttps://osmocom.org/issues/4576?journal_id=201892020-10-28T21:48:27Zneelsnhofmeyr@sysmocom.de
<ul></ul><p>i'm intuitively thinking about a simplistic "multicast repeater" thing that could forward DGSM multicast requests between two separate networks via unicast UDP...<br />we'd set that up between two docker containers to get our DGSM tests working.<br />maybe such could even be useful in some or other real world deployments one day, say two villages separated by some NAT or something?<br />(another less simple idea is to add the unicast option directly to osmo-hlr and the osmo-mslookup-client used by hook scripts,<br />i.e. allow configuring a specific unicast target for DGSM instead of, or maybe in addition to, multicast?)<br /><a class="user active" href="https://osmocom.org/users/7">laforge</a> your views on this would be welcome, you invariably have a bigger picture on these things than i do.</p> Core testing infrastructure - Bug #4576: ttcn3-hlr-tests need to route multicast traffic between ttcn3-hlr-test and osmo-hlr-master containershttps://osmocom.org/issues/4576?journal_id=201902020-10-28T21:48:37Zneelsnhofmeyr@sysmocom.de
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Stalled</i></li></ul>