Project

General

Profile

Actions

Bug #2746

closed

prod env: osmo-ggsn unable to get IPv6 link-local address

Added by pespin over 6 years ago. Updated about 6 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
-
Start date:
12/13/2017
Due date:
% Done:

100%

Spec Reference:

Description

gprs tests are currently failing when trying to create an ipv6 pdp context.

The issue is actually in osmo-ggsn returning an error when sgsn sends a Pdp Activate with IPv4v6. When the modem asks for an ipv4, it still uses a IPv4v6 (see #2713) but when the error is returned it re-tries with a IPv4-only EUA and then it works. However for IPv6 it doesn't and it returns a failure through the dbus interface.

Now the interesting part for this ticket: How osmo-ggsn fails.

When receiving a request for an ipv4v6 ctx, osmo-ggsn tries to get one IP of each type from the pools in the APN. Before looking in the pool, it checks if the APN actually supports this kind of address / has this type of pool. When checking for the IPv6 pool, it fails printing the following + sending a packet back with an error:

20171213174828668 DGGSN <0002> ggsn.c:635 PDP(901700000015252:6): APN doesn't support requested EUA / AF type

That happens because the following path triggers:

            /* does this APN actually have an IPv6 pool? */
            if (!apn_supports_ipv6(apn))
                goto err_wrong_af;

which means the ipv6 pool was not configured successfully at startup. We can actually see it in log file:

20171213174741477 DGGSN <0002> ggsn.c:201 APN(internet): Starting
20171213174741477 DGGSN <0002> ggsn.c:204 APN(internet): Opening TUN device tun4
20171213174741478 DGGSN <0002> ggsn.c:209 APN(internet): Opened TUN device tun4
20171213174741478 DGGSN <0002> ggsn.c:220 APN(internet): Setting tun IP address 176.16.222.1/24
20171213174741480 DGGSN <0002> ggsn.c:283 APN(internet): Creating IPv4 pool 176.16.222.0/24
20171213174741481 DGGSN <0002> ggsn.c:185 APN(internet): Blacklist tun IP 176.16.222.1/24
20171213174741481 DIP <0000> ippool.c:295 addr blacklisted from pool: 176.16.222.1
20171213174741481 DGGSN <0002> ggsn.c:314 APN(internet): Successfully started
20171213174741482 DGGSN <0002> ggsn.c:201 APN(inet6): Starting
20171213174741482 DGGSN <0002> ggsn.c:204 APN(inet6): Opening TUN device tun6
20171213174741482 DGGSN <0002> ggsn.c:209 APN(inet6): Opened TUN device tun6
20171213174741482 DGGSN <0002> ggsn.c:232 APN(inet6): Setting tun IPv6 address fde4:8dba:82e1:2000::/56
20171213174741484 DGGSN <0002> ggsn.c:252 APN(inet6): Cannot obtain IPv6 link-local address of interface: Success
20171213174741484 DGGSN <0002> ggsn.c:115 APN(inet6): Stopping
20171213174741484 DGGSN <0002> ggsn.c:128 APN(inet6): Closing TUN device tun6
20171213174741503 DGGSN <0002> ggsn.c:201 APN(inet46): Starting
20171213174741503 DGGSN <0002> ggsn.c:204 APN(inet46): Opening TUN device tun46
20171213174741504 DGGSN <0002> ggsn.c:209 APN(inet46): Opened TUN device tun46
20171213174741504 DGGSN <0002> ggsn.c:220 APN(inet46): Setting tun IP address 176.16.46.0/24
20171213174741505 DGGSN <0002> ggsn.c:232 APN(inet46): Setting tun IPv6 address fde4:8dba:82e1:2000::/56
20171213174741506 DGGSN <0002> ggsn.c:252 APN(inet46): Cannot obtain IPv6 link-local address of interface: Success
20171213174741506 DGGSN <0002> ggsn.c:115 APN(inet46): Stopping
20171213174741506 DGGSN <0002> ggsn.c:128 APN(inet46): Closing TUN device tun46
20171213174741535 DGGSN <0002> ggsn.c:812 GGSN(ggsn0): Starting GGSN

As you can see, it fails to configure ipv6 pools because it fails to get the IPv6 link-local address from the tun iface:

20171213174741506 DGGSN <0002> ggsn.c:252 APN(inet46): Cannot obtain IPv6 link-local address of interface: Success

Later on it fails with different errno:

20171213174741549 DGGSN <0002> ggsn.c:252 APN(inet6): Cannot obtain IPv6 link-local address of interface: No such file or directory

The only path in tun/netdev_ip_local_get which can return an error and issues a syscall is:

    if (getifaddrs(&ifaddr) == -1) {
        return -1;
    }

This function only receives a pointer and has no context associated with our code. It actually works fine in R&D env (debian9 in R&D vs debian 8 in Prod), which means there's a bug in libc or kernel in debian8.

debian8 (Prod):
- Debian GLIBC 2.19-18+deb8u10
- #1 SMP Debian 3.16.43-2+deb8u2 (2017-06-26)

debian9 (R&D):
- Debian GLIBC 2.24-11+deb9u1
- kernel: 1 SMP Debian 4.9.30-2+deb9u3 (2017-08-06)

"man getifaddrs":

RETURN VALUE
       On success, getifaddrs() returns zero; on error, -1 is returned, and errno is set appropriately.

ERRORS
       getifaddrs() may fail and set errno for any of the errors specified for socket(2), bind(2), getsockname(2), recvmsg(2),  sendto(2),
       malloc(3), or realloc(3).
VERSIONS
       The  getifaddrs()  function  first appeared in glibc 2.3, but before glibc 2.3.3, the implementation supported only IPv4 addresses;
       IPv6 support was added in glibc 2.3.3.  Support of address families other than IPv4 is  available  only  on  kernels  that  support
       netlink.

So regarding glibc version it should support ipv6 addresses (it doesn't mean there aren't bugs in there though), and "kernel that support" netlink it's a big difficult to match to a specific version.

In any case, it seems like another good reason to upgrade production environment to debian 9.


Files

bla bla 52.2 KB pespin, 12/13/2017 07:44 PM
bla-rnd bla-rnd 134 KB pespin, 12/13/2017 07:56 PM
Actions #1

Updated by pespin over 6 years ago

Upgrading latest packages with apt-get (staying in debian8) + rebooting didn't help solving the issue.

Actions #2

Updated by pespin over 6 years ago

I did a quick test by running the following program while the test was running and it never failed:

#include <stdlib.h>
#include <stdio.h>
#include <sys/types.h>
#include <ifaddrs.h>

int main(int argc, char** argv) {
        struct ifaddrs *ifaddr;

        if (getifaddrs(&ifaddr) == -1) {
                fpritnf(stderr, "fail\n");
                return 1;
        } else fprintf(stderr, "success\n");

return 0;
}

Closer look showed me that actually the check is "tun_ip_local_get(apn->tun.tun, &ipv6_tun_linklocal_ip, 1, IP_TYPE_IPv6_LINK) < 1" which means it checks not only for error, but asserts that the IP was found. Which means the getifaddrs is working fine but the address is not found. I run "ifconfig" and "ip l" while the test was running, and I only saw tun4 appearing. However, according to osmo-ggsn logs we see no error appearing.

Actions #3

Updated by pespin over 6 years ago

After stracing osmo-ggsn, there seems to be something weird in the output of while setting ipv6 SIOCSIFADDR tun addr (tun_setaddr6):

write(2, "20171213202024691 DGGSN <0002> g"..., 6820171213202024691 DGGSN <0002> ggsn.c:201 APN(inet6): Starting
) = 68
write(2, "20171213202024693 DGGSN <0002> g"..., 8320171213202024693 DGGSN <0002> ggsn.c:204 APN(inet6): Opening TUN device tun6
) = 83
open("/dev/net/tun", O_RDWR)            = 6
ioctl(6, TUNSETIFF, 0x7ffe4d18a690)     = 0
ioctl(6, TUNSETNOCSUM, 0x1)             = 0
write(2, "20171213202024696 DGGSN <0002> g"..., 8220171213202024696 DGGSN <0002> ggsn.c:209 APN(inet6): Opened TUN device tun6
) = 82
fcntl(6, F_GETFL)                       = 0x8002 (flags O_RDWR|O_LARGEFILE)
fcntl(6, F_SETFL, O_RDWR|O_NONBLOCK|O_LARGEFILE) = 0
fcntl(6, F_GETFD)                       = 0
fcntl(6, F_SETFD, FD_CLOEXEC)           = 0
write(2, "20171213202024697 DGGSN <0002> g"..., 10920171213202024697 DGGSN <0002> ggsn.c:232 APN(inet6): Setting tun IPv6 address fde4:8dba:82e1:2000::/56
) = 109
access("/proc/net", R_OK)               = 0
access("/proc/net/unix", R_OK)          = 0
socket(PF_LOCAL, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 7
ioctl(7, SIOCGIFINDEX, {ifr_name="tun6", ifr_index=42}) = 0
close(7)                                = 0
socket(PF_INET6, SOCK_DGRAM, IPPROTO_IP) = 7
ioctl(7, SIOCSIFADDR, {ifr_name="<FD>䍺<82><E1> ", ifr_addr={0x38 /* AF_??? */, "\0\0*\0\0\0\1\0\0\0\0\0\0\0"}}) = 0
close(7)                                = 0
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 7
ioctl(7, SIOCSIFFLAGS, {ifr_name="tun6", ifr_flags=IFF_UP|IFF_RUNNING}) = 0
close(7)                                = 0
socket(PF_NETLINK, SOCK_RAW, NETLINK_ROUTE) = 7
bind(7, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
getsockname(7, {sa_family=AF_NETLINK, pid=4567, groups=00000000}, [12]) = 0
sendto(7, "\24\0\0\0\22\0\1\3x}1Z\0\0\0\0\0\0\0\0", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20
recvmsg(7, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"l\4\0\0\20\0\2\0x}1Z\327\21\0\0\0\0\4\3\1\0\0\0I\0\1\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 3412
recvmsg(7, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"t\4\0\0\20\0\2\0x}1Z\327\21\0\0\0\0\1\0\4\0\0\0\3\20\0\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 3412
recvmsg(7, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"t\4\0\0\20\0\2\0x}1Z\327\21\0\0\0\0\1\0\7\0\0\0\2\20\0\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 3412
recvmsg(7, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"p\4\0\0\20\0\2\0x}1Z\327\21\0\0\0\0\1\0\n\0\0\0\2\20\0\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 3392
recvmsg(7, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"\24\0\0\0\3\0\2\0x}1Z\327\21\0\0\0\0\0\0", 4096}], msg_controllen=0, msg_flags=0}, 0) = 20
sendto(7, "\24\0\0\0\26\0\1\3y}1Z\0\0\0\0\0\0\0\0", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20
recvmsg(7, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"L\0\0\0\24\0\2\0y}1Z\327\21\0\0\2\10\200\376\1\0\0\0\10\0\1\0\177\0\0\1"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 1468
recvmsg(7, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"H\0\0\0\24\0\2\0y}1Z\327\21\0\0\n\200\200\376\1\0\0\0\24\0\1\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 288
recvmsg(7, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"\24\0\0\0\3\0\2\0y}1Z\327\21\0\0\0\0\0\0", 4096}], msg_controllen=0, msg_flags=0}, 0) = 20
close(7)                                = 0
write(2, "20171213202024703 DGGSN <0002> g"..., 11920171213202024703 DGGSN <0002> ggsn.c:252 APN(inet6): Cannot obtain IPv6 link-local address of interface: Success

However all return codes look fine (matches what we see in the logs).

I attach full strace.

EDIT: I forgot to say, the strace is taken as root, and I verified that indeed only tun4 was created, so it's not a permissions issue.

Actions #4

Updated by pespin over 6 years ago

Here's the same output in R&D (which creates the tuns correctly):

write(2, "20171213204857407 DGGSN <0002> g"..., 6820171213204857407 DGGSN <0002> ggsn.c:201 APN(inet6): Starting
ESC[0;m) = 68
write(2, "20171213204857408 DGGSN <0002> g"..., 8320171213204857408 DGGSN <0002> ggsn.c:204 APN(inet6): Opening TUN device tun6
ESC[0;m) = 83
open("/dev/net/tun", O_RDWR)            = 6
ioctl(6, TUNSETIFF, 0x7fff7c1ba790)     = 0
ioctl(6, TUNSETNOCSUM, 0x1)             = 0
write(2, "20171213204857411 DGGSN <0002> g"..., 8220171213204857411 DGGSN <0002> ggsn.c:209 APN(inet6): Opened TUN device tun6
ESC[0;m) = 82
fcntl(6, F_GETFL)                       = 0x8002 (flags O_RDWR|O_LARGEFILE)
fcntl(6, F_SETFL, O_RDWR|O_NONBLOCK|O_LARGEFILE) = 0
fcntl(6, F_GETFD)                       = 0
fcntl(6, F_SETFD, FD_CLOEXEC)           = 0
write(2, "20171213204857412 DGGSN <0002> g"..., 10920171213204857412 DGGSN <0002> ggsn.c:232 APN(inet6): Setting tun IPv6 address fde4:8dba:82e1:2000::/56
ESC[0;m) = 109
access("/proc/net", R_OK)               = 0
access("/proc/net/unix", R_OK)          = 0
socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 7
ioctl(7, SIOCGIFINDEX, {ifr_name="tun6", }) = 0
close(7)                                = 0
socket(AF_INET6, SOCK_DGRAM, IPPROTO_IP) = 7
ioctl(7, SIOCSIFADDR, {ifr_name="\375\344\215\272\202\341 ", ifr_addr={sa_family=0x38 /* AF_??? */, sa_data="\0\0\272\1\0\0\1\0\0\0\0\0\0\0"}}) = 0
close(7)                                = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 7
ioctl(7, SIOCSIFFLAGS, {ifr_name="tun6", ifr_flags=IFF_UP|IFF_RUNNING}) = 0
close(7)                                = 0
socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE) = 7
bind(7, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 0
getsockname(7, {sa_family=AF_NETLINK, nl_pid=25384, nl_groups=00000000}, [12]) = 0
sendto(7, {{len=20, type=0x12 /* NLMSG_??? */, flags=NLM_F_REQUEST|0x300, seq=1513194537, pid=0}, "\0\0\0\0"}, 20, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 20
recvmsg(7, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=1232, type=0x10 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194537, pid=25384}, "\0\0\4\3\1\0\0\0I\0\1\0\0\0\0\0\7\0\3\0lo\0\0\10\0\r\0\1\0\0\0"...}, {{len=1240, type=0x10 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194537, pid=25384}, "\0\0\1\0\2\0\0\0C\20\1\0\0\0\0\0\t\0\3\0eth0\0\0\0\0\10\0\r\0"...}, {{len=1240, type=0x10 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194537, pid=25384}, "\0\0\1\0\3\0\0\0C\20\1\0\0\0\0\0\t\0\3\0eth1\0\0\0\0\10\0\r\0"...}, {{len=3352834104, type=0x7fdd /* NLMSG_??? */, flags=0, seq=3374275523, pid=32733}, "\334\5\0\0\0\0\0\00080\330\307\335\177\0\0\260z?\311\335\177\0\0\310\244\33|\377\177\0\0"...}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 3712
recvmsg(7, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=1240, type=0x10 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194537, pid=25384}, "\0\0\1\0\4\0\0\0\3\20\0\0\0\0\0\0\t\0\3\0eth2\0\0\0\0\10\0\r\0"...}, {{len=1236, type=0x10 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194537, pid=25384}, "\0\0\1\0\5\0\0\0\2\20\0\0\0\0\0\0\n\0\3\0wlan0\0\0\0\10\0\r\0"...}, {{len=1244, type=0x10 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194537, pid=25384}, "\0\0\1\0Q\0\0\0\2\20\0\0\0\0\0\0\22\0\3\0wwp0s18f2u1u"...}, {{len=3374275523, type=0x7fdd /* NLMSG_??? */, flags=0, seq=1500, pid=0}, "80\330\307\335\177\0\0\260z?\311\335\177\0\0\310\244\33|\377\177\0\0\304\244\33|\377\177\0\0"...}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 3720
recvmsg(7, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=1248, type=0x10 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194537, pid=25384}, "\0\0\1\0R\0\0\0\2\20\0\0\0\0\0\0\24\0\3\0wwp0s18f2u1u"...}, {{len=1220, type=0x10 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194537, pid=25384}, "\0\0\376\377S\0\0\0\220\20\0\0\0\0\0\0\24\0\3\0wwp0s18f2u1u"...}, {{len=1236, type=0x10 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194537, pid=25384}, "\0\0\1\0T\0\0\0\2\20\0\0\0\0\0\0\n\0\3\0wwan1\0\0\0\10\0\r\0"...}, {{len=0, type=0 /* NLMSG_??? */, flags=0, seq=524293, pid=0}}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 3704
recvmsg(7, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=1224, type=0x10 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194537, pid=25384}, "\0\0\376\377U\0\0\0\220\20\0\0\0\0\0\0\24\0\3\0wwp0s18f2u1u"...}, {{len=1236, type=0x10 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194537, pid=25384}, "\0\0\1\0V\0\0\0\2\20\0\0\0\0\0\0\n\0\3\0wwan2\0\0\0\10\0\r\0"...}, {{len=1228, type=0x10 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194537, pid=25384}, "\0\0\376\377\271\1\0\0Q\0\1\0\0\0\0\0\t\0\3\0tun4\0\0\0\0\10\0\r\0"...}, {{len=0, type=0 /* NLMSG_??? */, flags=0, seq=524293, pid=0}}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 3688
recvmsg(7, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=1228, type=0x10 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194537, pid=25384}, "\0\0\376\377\272\1\0\0Q\0\1\0\0\0\0\0\t\0\3\0tun6\0\0\0\0\10\0\r\0"...}, {{len=131088, type=0x8429 /* NLMSG_??? */, flags=NLM_F_REQUEST|NLM_F_DUMP_INTR|NLM_F_DUMP_FILTERED|0x5a00, seq=25384, pid=65536}, "V\0\0\0\2\20\0\0\0\0\0\0\n\0\3\0wwan2\0\0\0\10\0\r\0\350\3\0\0"...}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 1228
recvmsg(7, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=20, type=NLMSG_DONE, flags=NLM_F_MULTI, seq=1513194537, pid=25384}, "\0\0\0\0"}, {{len=442, type=0x51 /* NLMSG_??? */, flags=NLM_F_REQUEST, seq=0, pid=196617}, "tun6\0\0\0\0\10\0\r\0\364\1\0\0\5\0\20\0\0\0\0\0\5\0\21\0\0\0\0\0"...}, {{len=0, type=0 /* NLMSG_??? */, flags=0, seq=0, pid=0}}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 20
sendto(7, {{len=20, type=0x16 /* NLMSG_??? */, flags=NLM_F_REQUEST|0x300, seq=1513194538, pid=0}, "\0\0\0\0"}, 20, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 20
recvmsg(7, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=76, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\10\200\376\1\0\0\0\10\0\1\0\177\0\0\1\10\0\2\0\177\0\0\1\7\0\3\0lo\0\0"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\30\200\0\2\0\0\0\10\0\1\0\n\t\31e\10\0\2\0\n\t\31e\10\0\4\0\n\t\31\377"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\30\200\0\3\0\0\0\10\0\1\0\n**\1\10\0\2\0\n**\1\10\0\4\0\n**\377"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\30\201\0\3\0\0\0\10\0\1\0\n**\2\10\0\2\0\n**\2\10\0\4\0\n**\377"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\30\201\0\3\0\0\0\10\0\1\0\n**\3\10\0\2\0\n**\3\10\0\4\0\n**\377"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\30\201\0\3\0\0\0\10\0\1\0\n**\4\10\0\2\0\n**\4\10\0\4\0\n**\377"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\30\201\0\3\0\0\0\10\0\1\0\n**\5\10\0\2\0\n**\5\10\0\4\0\n**\377"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\30\201\0\3\0\0\0\10\0\1\0\n**\6\10\0\2\0\n**\6\10\0\4\0\n**\377"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\30\201\0\3\0\0\0\10\0\1\0\n**\7\10\0\2\0\n**\7\10\0\4\0\n**\377"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\30\201\0\3\0\0\0\10\0\1\0\n**\10\10\0\2\0\n**\10\10\0\4\0\n**\377"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\30\201\0\3\0\0\0\10\0\1\0\n**\t\10\0\2\0\n**\t\10\0\4\0\n**\377"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\30\201\0\3\0\0\0\10\0\1\0\n**2\10\0\2\0\n**2\10\0\4\0\n**\377"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\30\201\0\3\0\0\0\10\0\1\0\n**3\10\0\2\0\n**3\10\0\4\0\n**\377"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\30\200\0\4\0\0\0\10\0\1\0\n@\1\1\10\0\2\0\n@\1\1\10\0\4\0\n@\1\377"...}, {{len=80, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\30\200\0\4\0\0\0\10\0\1\0\300\250\1\310\10\0\2\0\300\250\1\310\t\0\3\0eth2"...}, {{len=80, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\30\200\0\271\1\0\0\10\0\1\0\260\20\336\1\10\0\2\0\260\20\336\1\t\0\3\0tun4"...}, {{len=0, type=0x24 /* NLMSG_??? */, flags=NLM_F_MULTI|NLM_F_ACK|NLM_F_ECHO, seq=0, pid=0}}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 1380
recvmsg(7, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\n\200\200\376\1\0\0\0\24\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\24\0\6\0"...}, {{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\n@\200\375\2\0\0\0\24\0\1\0\376\200\0\0\0\0\0\0\2\r\271\377\3765:@\24\0\6\0"...}, {{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\n@\200\375\3\0\0\0\24\0\1\0\376\200\0\0\0\0\0\0\2\r\271\377\3765:A\24\0\6\0"...}, {{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\n@\200\375\271\1\0\0\24\0\1\0\376\200\0\0\0\0\0\0\216l\342-s>\364\265\24\0\6\0"...}, {{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\n8\200\0\272\1\0\0\24\0\1\0\375\344\215\272\202\341 \0\0\0\0\0\0\0\0\0\24\0\6\0"...}, {{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\n@\200\375\272\1\0\0\24\0\1\0\376\200\0\0\0\0\0\0A@B1l\317\247\232\24\0\6\0"...}, {{len=131092, type=0x842a /* NLMSG_??? */, flags=NLM_F_REQUEST|NLM_F_DUMP_INTR|NLM_F_DUMP_FILTERED|0x5a00, seq=25384, pid=8460290}, "\3\0\0\0\10\0\1\0\n**\4\10\0\2\0\n**\4\10\0\4\0\n**\377\v\0\3\0"...}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 432
recvmsg(7, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=20, type=NLMSG_DONE, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\0\0\0\0"}, {{len=1, type=0x14 /* NLMSG_??? */, flags=NLM_F_REQUEST, seq=0, pid=0}}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 20
close(7)                                = 0

I see no big differences. I attach the full strace log.
Actions #5

Updated by pespin over 6 years ago

On the Prod machine, I checked how other tools set the IP (successfully).

I can create the tunnel with:

ip tuntap add dev tun6 mode tun user root group root

ip addr uses the entlink interface to set the ip.

ifconfig uses the same bits as us:

ioctl(5, SIOCGIFINDEX, {ifr_name="tun6", ifr_index=56}) = 0
ioctl(5, SIOCSIFADDR, {ifr_name="<FD>䍺<82><E1> ", ifr_addr={0x38 /* AF_??? */, "\0\0008\0\0\0\10\5\223\366\332\177\0\0"}}) = 0

I am probably not seeing the interface being created by osmo-ggsn because it is destroyed after trying to fetch the link-local ip, since apn_stop() is called which in turn calls tun_free().

Thinking more about it, that's what I think could be happening: IIRC, setting the IP is not immediate/synchronous, specially because some routing entries are added automatically. Which means we are probably setting up the tunnel correctly, but when we call getifaddrs() to get the list of IPs, the IP we just set is not yet there and we fail to get it. Then we destroy the iface.

We could try waiting a bit more or using the debugger to stop it for a while and see what happens (and if the iface is indeed correctly created and the IP is set).

Actions #6

Updated by pespin over 6 years ago

With gdb I can see that the interface is created and that the IP is assigned. However, no link-local IP is assigned to the interface.

Prod:

62: tun46: <POINTOPOINT,UP,LOWER_UP> mtu 1500 qlen 500
    inet6 fde4:8dba:82e1:2000::/56 scope global
       valid_lft forever preferred_lft foreve

R&D:

447: tun46: <POINTOPOINT,UP,LOWER_UP> mtu 1500 state UNKNOWN qlen 500
    inet6 fde4:8dba:82e1:2000::/56 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::1d0:8ef4:21ff:fcc4/64 scope link flags 800
       valid_lft forever preferred_lft forever

I'm not sure how is that link-local IP assigned though.

Actions #7

Updated by pespin over 6 years ago

  • Status changed from New to Feedback
  • % Done changed from 0 to 80

I added code to osmo-ggsn to manually set up a link-local ip.

It can be used in osmo-gsm-tester as soon as it is merged + the trial is built: https://gerrit.osmocom.org/#/c/5349/

Then, we need to add the following line to osmo-ggsn.cfg in osmo-gsm-tester, for inet6:

ipv6 link-local fe80::1111:1111:1111:1111/64

and to inet64:

ipv6 link-local fe80::1111:1111:1111:1112/64

After this options are used, osmo-ggsn tester should start and configure ipv6 pools correctly, and we can enable ipv6 ctx tests again.

Actions #8

Updated by pespin over 6 years ago

  • Status changed from Feedback to Resolved
  • % Done changed from 80 to 100

osmo-ggsn patches were merged. I pushed just now a commit for osmo-gsm-tester in gerrit to use the option. I already tried in a branch running in Prod setup that it works fine and that ipv6 context can be created successfully using ofono+osmo-ggsn.

Actions #9

Updated by laforge about 6 years ago

  • Status changed from Resolved to Closed
Actions

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)