Cellular Network Infrastructure » osmo-ePDG - VoWifi Evolved Packet Data Gateway

This POC looks related:
https://gitea.osmocom.org/osmith/ipsec-gtpu-poc/src/branch/master/ipsec-gtp

AFAIU we need, at osmo-epdg, to create/manage one GTP tun dev for each <remote_pgw_ip_addr, APN> key tuple.

Then, on each of those, we have a complete TEID namespace, and we need to configure 1 entry for each bearer created during CreateSessionResponse, containing:
- local_teid: Selected by osmo-epdg when sending CreateSessionRequest
- remote_teid: Found in UE IP in CreateSessionResponse
- local inner (ms) IP address: Found in UE IP in CreateSessionResponse
- local outter (gsn) IP address: Set up in config file
- remote outter (gsn) IP address: Set up in config file and perhaps updated in CreateSessionResponse

So upon receiving CreateSession Response, if we didn't yet create any tun dev for that <remote_pgw_ip_addr, APN>,we first create the tun dev. Then, we set up the tunnel config above.

On Wed, Oct 25, 2023 at 05:00:28PM +0000, pespin wrote:

We also need to think about the best approach to run the erlang code with NET_ADMIN capabilities which will probably be needed to create the tunnels and set up netfiler rules.

I would expect the ergw folks have some solution for that?

• Most of them pull github links with "git://" proto, which is not supported anymore. I submitted several PR upstream to fix some of them, but I had to end up using this to workaround it:
  "git config --global url."https://github.com/".insteadOf git://github.com/".
• Some are pulling an old version of lager which fails to build here with OTP-26. I submitted some patches upgrading lager to last upstream 3.9.2 release.
• gen_socket is linking against -lerl_interface, which doesn't exist apparently since OTP-23. I submitted a PR dropping that link argument, since that fixes it here locally.

So I now have my own fork of gen_socket, gen_netlink, gtplib and gtp_u_kmod repositories, which the patches submitted as PR plus updates to rebar.config pointing to my forked repo/branches.

With that, I can build everything inside osmo-epdg.git (branch "pespin/master"), but I'm having problems at runtime too:

18:29:57.856 [info] RegName: port_grx
18:29:57.873 [notice] VrfOpts: [{routes,[{{10,180,0,0},16}]}]
18:29:57.873 [notice] FDesc: {file_descriptor,raw_file_io_list,{file_descriptor,prim_file,#{handle => #Ref<0.114031393.2719088661.121660>,owner => <0.526.0>,r_buffer => #Ref<0.114031393.2719088642.121370>,r_ahead_size => 0}}}
terminating ss7_routes with reason shutdownterminating ss7_links with reason shutdown18:29:57.873 [error] CRASH REPORT Process <0.526.0> with 1 neighbours crashed with reason: no match of right hand value {file_descriptor,raw_file_io_list,{file_descriptor,prim_file,#{handle => #Ref<0.114031393.2719088661.121660>,owner => <0.526.0>,r_buffer => #Ref<0.114031393.2719088642.121370>,r_ahead_size => 0}}} in gtp_u_kernel:init/1 line 53

The notice log lines above are added in code by myself. The line failing is:

    #file_descriptor{module = prim_file,
             data   = {_Port, NsFd}} = FDesc,

My bet is that #file_descriptor record changed at some point during newer OTP releases?

I think I was able to fix the issue above with the following patch:

diff --git a/src/gtp_u_kernel.erl b/src/gtp_u_kernel.erl
index ef60038..2abd772 100644
--- a/src/gtp_u_kernel.erl
+++ b/src/gtp_u_kernel.erl
@@ -47,12 +47,11 @@ delete_pdp_context(Server, Version, SGSN, MS, LocalTEI, RemoteTEI) ->

 init([Device, FD0, FD1u, Opts]) ->
     VrfOpts = proplists:get_value(vrf, Opts, []),
-    {ok, FDesc} = get_ns_fd(VrfOpts),
+    {ok, FDesc} = get_ns_fdesc(VrfOpts),
+    NsFd = get_ns_fd(FDesc),
     lager:notice("VrfOpts: ~p~n", [VrfOpts]),
     lager:notice("FDesc: ~p~n", [FDesc]),
-    #file_descriptor{module = prim_file,
-                    data   = {_Port, NsFd}} = FDesc,
-
+    lager:notice("NsFd: ~p~n", [NsFd]),
     {RtNl, RtNlNs} = netlink_sockets(VrfOpts),
     CreateGTPLinkInfo = [{fd0, FD0}, {fd1, FD1u}, {hashsize, 131072}],
     CreateGTPData = netlink:linkinfo_enc(inet, "gtp", CreateGTPLinkInfo),
@@ -170,7 +169,7 @@ code_change(_OldVsn, State, _Extra) ->
 -define(SELF_NET_NS, "/proc/self/ns/net").
 -define(SIOCGIFINDEX, 16#8933).

-get_ns_fd(Opts) ->
+get_ns_fdesc(Opts) ->
     try
        {netns, NetNs} = lists:keyfind(netns, 1, Opts),
        {ok, _} = file:open(filename:join("/var/run/netns", NetNs), [raw, read])
@@ -179,6 +178,18 @@ get_ns_fd(Opts) ->
            {ok, _} = file:open(?SELF_NET_NS, [raw, read])
     end.

+get_ns_fd(FDesc) ->
+    lager:notice("FDesc: ~p~n", [FDesc]),
+    case FDesc of
+    #file_descriptor{module = prim_file} ->
+        #file_descriptor{data = {_, NsFd}} = FDesc,
+        NsFd;
+    #file_descriptor{module = _} ->
+        PrivFDesc = FDesc#file_descriptor.data,
+        #file_descriptor{data = #{handle := NsFd}} = PrivFDesc,
+        NsFd
+    end.
+

In summary, it seems at least my OTP-26 version is creating a tuple of type raw_file_io_list which encloses the usual file reference which is required here. This is due to the "[raw, read]" fields being passed. Maybe in older versions of OTP this didnit happen.

It seems my previous patch to take the proper FD was not good after all. The netlink code is failing at a later step because it's expecting an uint32 (unix fd) while I'm passing some other field which seems to be of another type :/

I added the err rline to log encode_huint32 params:

[error] encode_huint32: 1 28
18:22:56.094 [error] encode_huint32: 2 29
18:22:56.094 [error] encode_huint32: 3 131072
18:22:56.094 [debug] CreateGTPReq: {rtnetlink,newlink,[create,excl,ack,request],2,0,{inet,arphrd_none,0,[up],[up],[{net_ns_fd,#Ref<0.1102821846.1330774038.183476>},{ifname,"gtp0"},{linkinfo,[{kind,"gtp"},{data,<<8,0,1,0,28,0,0,0,8,0,2,0,29,0,0,0,8,0,3,0,0,0,2,0>>}]}]}}
18:22:56.094 [error] encode_huint32: 28 #Ref<0.1102821846.1330774038.183476>
18:22:56.095 [error] CRASH REPORT Process <0.526.0> with 1 neighbours crashed with reason: bad argument in netlink:encode_huint32/2 line 534

The "#Ref<0.1102821846.1330774038.183476>" is the field I'm getting with my previous patch.

I was able to apparently fix it with a new patch version:

diff --git a/src/gtp_u_kernel.erl b/src/gtp_u_kernel.erl
index ef60038..ba6785e 100644
--- a/src/gtp_u_kernel.erl
+++ b/src/gtp_u_kernel.erl
@@ -47,12 +47,11 @@ delete_pdp_context(Server, Version, SGSN, MS, LocalTEI, RemoteTEI) ->

 init([Device, FD0, FD1u, Opts]) ->
     VrfOpts = proplists:get_value(vrf, Opts, []),
-    {ok, FDesc} = get_ns_fd(VrfOpts),
+    {ok, FDesc} = get_ns_fdesc(VrfOpts),
+    NsFd = get_ns_fd(FDesc),
     lager:notice("VrfOpts: ~p~n", [VrfOpts]),
     lager:notice("FDesc: ~p~n", [FDesc]),
-    #file_descriptor{module = prim_file,
-                    data   = {_Port, NsFd}} = FDesc,
-
+    lager:notice("NsFd: ~p~n", [NsFd]),
     {RtNl, RtNlNs} = netlink_sockets(VrfOpts),
     CreateGTPLinkInfo = [{fd0, FD0}, {fd1, FD1u}, {hashsize, 131072}],
     CreateGTPData = netlink:linkinfo_enc(inet, "gtp", CreateGTPLinkInfo),
@@ -170,7 +169,7 @@ code_change(_OldVsn, State, _Extra) ->
 -define(SELF_NET_NS, "/proc/self/ns/net").
 -define(SIOCGIFINDEX, 16#8933).

-get_ns_fd(Opts) ->
+get_ns_fdesc(Opts) ->
     try
        {netns, NetNs} = lists:keyfind(netns, 1, Opts),
        {ok, _} = file:open(filename:join("/var/run/netns", NetNs), [raw, read])
@@ -179,6 +178,19 @@ get_ns_fd(Opts) ->
            {ok, _} = file:open(?SELF_NET_NS, [raw, read])
     end.

+get_ns_fd(FDesc) ->
+    lager:notice("FDesc: ~p~n", [FDesc]),
+    case FDesc of
+    #file_descriptor{module = prim_file} ->
+        #file_descriptor{data = {_, NsFd}} = FDesc,
+        NsFd;
+    #file_descriptor{module = _} ->
+        PrivFDesc = FDesc#file_descriptor.data,
+        binary:decode_unsigned(prim_file:get_handle(PrivFDesc),little)
+        %#file_descriptor{data = #{handle := NsFd}} = PrivFDesc,
+        %prim_file:get_handle(NsFd)
+    end.
+

After applying that one, the FD looks sane (30 being allocated after previous 29 one):

19:23:28.581 [error] encode_huint32: 2 29
19:23:28.581 [error] encode_huint32: 3 131072
19:23:28.581 [debug] CreateGTPReq: {rtnetlink,newlink,[create,excl,ack,request],35,0,{inet,arphrd_none,0,[up],[up],[{net_ns_fd,30},{ifname,"gtp0"},{linkinfo,[{kind,"gtp"},{data,<<8,0,1,0,28,0,0,0,8,0,2,0,29,0,0,0,8,0,3,0,0,0,2,0>>}]}]}}
19:23:28.581 [notice] nl_simple_request do_request {rtnetlink,newlink,[create,excl,ack,request],35,0,{inet,arphrd_none,0,[up],[up],[{net_ns_fd,30},{ifname,"gtp0"},{linkinfo,[{kind,"gtp"},{data,<<8,0,1,0,28,0,0,0,8,0,2,0,29,0,0,0,8,0,3,0,0,0,2,0>>}]}]}}
19:23:28.581 [error] encode_huint32: 28 30

However I'm still facing problems. Now during submitting of CreateGTPReq (rtnetlink,newlink). So some problem in netlink I still need to debug.

I created forks under github/osmocom/ for the following repos:
https://github.com/osmocom/gen_socket/
https://github.com/osmocom/gen_netlink/
https://github.com/osmocom/gtplib/
https://github.com/osmocom/gtp_u_kmod/

All those repos have the master branch tracking the travelping upstream master branch. All our patches are on top in "osmocom/master" branch. In those branches they also have rebar.config modified to pull the dependent modules from the forked repo+branch.

My WIP code in osmo-epdg.git using those modules to set up a tunnel (still not really working due to errors) can be found in branch "pespin/master".

Further investigation so far seems to indicate there's a bug in the logic handling rtnetlink.

In summary, osmo-epdg sends the following request and processes messages until receiving an ACK for it (seqnum):

Linux netlink (cooked header)
    Link-layer address type: Netlink (824)
    Family: Route (0x0000)
Linux rtnetlink (route netlink) protocol
    Netlink message header (type: Create network interface)
        Length: 92
        Message type: Create network interface (16)
        Flags: 0x0605
            .... .... .... ...1 = Request: 1
            .... .... .... ..0. = Multipart message: 0
            .... .... .... .1.. = Ack: 1
            .... .... .... 0... = Echo: 0
            .... .... ...0 .... = Dump inconsistent: 0
            .... .... ..0. .... = Dump filtered: 0
            .... ...0 .... .... = Specify tree root: 0
            .... ..1. .... .... = Return all matching: 1
            .... .1.. .... .... = Atomic: 1
        Flags: 0x0605
            .... .... .... ...1 = Request: 1
            .... .... .... ..0. = Multipart message: 0
            .... .... .... .1.. = Ack: 1
            .... .... .... 0... = Echo: 0
            .... .... ...0 .... = Dump inconsistent: 0
            .... .... ..0. .... = Dump filtered: 0
            .... ...0 .... .... = Replace: 0
            .... ..1. .... .... = Excl: 1
            .... .1.. .... .... = Create: 1
            .... 0... .... .... = Append: 0
        Sequence: 34
        Port ID: 0
    Interface family: 2
    Device type: zero header length (65534)
    Interface index: 0
    Device flags: UP (0x00000001)
    Device change flags: 1
    Attribute: NetNs fd
        Len: 8
        Type: 0x001c, NetNs fd (28)
            0... .... .... .... = Nested: False
            .0.. .... .... .... = Network byte order: False
            Attribute type: NetNs fd (28)
        Data: 1e000000
    Attribute: Device name: gtp0
        Len: 9
        Type: 0x0003, Device name (3)
            0... .... .... .... = Nested: False
            .0.. .... .... .... = Network byte order: False
            Attribute type: Device name (3)
        Device name: gtp0
    Attribute: Link info
        Len: 40
        Type: 0x0012, Link info (18)
            0... .... .... .... = Nested: False
            .0.. .... .... .... = Network byte order: False
            Attribute type: Link info (18)
        Data: 08000100677470001c000200080001001c000000080002001d0000000800030000000200

Then, after some dump messages, the ACK comes (error type but with err_code=0, so it's just confirming everything was fine):

Linux netlink (cooked header)
    Link-layer address type: Netlink (824)
    Family: Route (0x0000)
Netlink message
    Netlink message header (type: Error)
        Length: 36
        Message type: Error (0x0002)
        Flags: 0x0100
            .... .... .... ...0 = Request: 0
            .... .... .... ..0. = Multipart message: 0
            .... .... .... .0.. = Ack: 0
            .... .... .... 0... = Echo: 0
            .... .... ...0 .... = Dump inconsistent: 0
            .... .... ..0. .... = Dump filtered: 0
        Sequence: 34
        Port ID: 261026
    Error code: Success (0)
    Netlink message header (type: 0x0010)
        Length: 92
        Message type: Protocol-specific (0x0010)
        Flags: 0x0605
            .... .... .... ...1 = Request: 1
            .... .... .... ..0. = Multipart message: 0
            .... .... .... .1.. = Ack: 1
            .... .... .... 0... = Echo: 0
            .... .... ...0 .... = Dump inconsistent: 0
            .... .... ..0. .... = Dump filtered: 0
            .... ...0 .... .... = Specify tree root: 0
            .... ..1. .... .... = Return all matching: 1
            .... .1.. .... .... = Atomic: 1
        Flags: 0x0605
            .... .... .... ...1 = Request: 1
            .... .... .... ..0. = Multipart message: 0
            .... .... .... .1.. = Ack: 1
            .... .... .... 0... = Echo: 0
            .... .... ...0 .... = Dump inconsistent: 0
            .... .... ..0. .... = Dump filtered: 0
            .... ...0 .... .... = Replace: 0
            .... ..1. .... .... = Excl: 1
            .... .1.. .... .... = Create: 1
            .... 0... .... .... = Append: 0
        Sequence: 34
        Port ID: 0

This is how gen_netlink parses the ACK message:

Response: [{rtnetlink,error,[256],34,261026,[{ifinfomsg,unspec,2,92,100990992,34},{rawdata,<<0,0,0,0>>}]}]

However, that kind of message payload for an "error" type packet is not expected at all. This fails in gtp_u_kmod when matching the response from the request:

nl_simple_response(error, {0, _}, _Response) ->
    ok;
nl_simple_response(error, {Code, _}, _Response) ->
    {error, Code};
nl_simple_response(_, _, Response) ->
    Response.

nl_simple_response(_Seq, []) ->
    continue;
nl_simple_response(Seq, [Response = #rtnetlink{type = Type, seq = Seq, msg = Msg} | Next ]) ->
    nl_simple_response(-1, Next),
    lager:debug("nl_simple_response: Matching Msg=~p", [Msg]),
    nl_simple_response(Type, Msg, Response);

As one can see, it expects the "error" type messages to only have a msg payload of "{Code, whatever}", but this message has "[{ifinfomsg,unspec,2,92,100990992,34},{rawdata,<<0,0,0,0>>}]" instead.

That's because gtp_netlink is incorrectly decoding the message in nl_rt_dec() around line 983, where it calls is_rt_dump() and it wrongly returns true:

is_rt_dump(Type, Flags) ->
    (Type band 3) =:= 2 andalso Flags band ?NLM_F_DUMP =/= 0.

nl_rt_dec(Protocol, << Len:32/native-integer, Type:16/native-integer, Flags:16/native-integer, Seq:32/native-integer, Pid:32/native-integer, Data/binary >> = Msg, Acc) ->
    {DecodedMsg, Next} = case nlmsg_ok(size(Msg), Len) of
[...]
                 case is_rt_dump(Type, Flags) of
                     true ->
                     <<IfiFam:8, _Pad:8, IfiType:16/native-integer, IfiIndex:32/native-integer, IfiFlags:32/native-integer, IfiChange:32/native-integer, Filter/binary >> = PayLoad,
                     InfoMsg = #ifinfomsg{family = gen_socket:family(IfiFam),
                                  type = Type,
                                  index = IfiIndex,
                                  flags = IfiFlags,
                                  change = IfiChange},
                     {RtMsg#rtnetlink{msg = [InfoMsg | nl_dec_nla(IfiFam, fun decode_rtnetlink_link/3, Filter)]}, NextMsg};

Instead, is_rt_dump() should return false there and go through this path:

%% Error
nl_dec_payload(_Type, error, <<Error:32, Msg/binary>>) ->
    {Error, Msg};

nl_rt_dec(Protocol, << Len:32/native-integer, Type:16/native-integer, Flags:16/native-integer, Seq:32/native-integer, Pid:32/native-integer, Data/binary >> = Msg, Acc) ->
    {DecodedMsg, Next} = case nlmsg_ok(size(Msg), Len) of
[...]
                 case is_rt_dump(Type, Flags) of
[...]
                     _ ->
                     {RtMsg#rtnetlink{msg = nl_dec_payload(rtnetlink, MsgType, PayLoad)}, NextMsg}

I think the bug is due to:

%% Modifiers to GET request
-define(NLM_F_ROOT,      16#100).   %% specify tree root
-define(NLM_F_MATCH,     16#200).   %% return all matching
-define(NLM_F_ATOMIC,    16#400).   %% atomic GET
-define(NLM_F_DUMP,      (?NLM_F_ROOT bor ?NLM_F_MATCH)).

%% Modifiers to NEW request
-define(NLM_F_REPLACE,   16#100).   %% Override existing
-define(NLM_F_EXCL,      16#200).   %% Do not touch, if it exists

The received message has flags=256=0x100, so it matches NLM_F_ROOT and hence NLM_F_DUMP and finally that's why is_rt_dump() returns true.
The problem here is that afaiu, the original request which triggered the ACK is not a "GET" request, but a "NEW" request, so the flag there's is simply asking to replace the device.

So I'd say so far nl_rt_dec/is_rt_dump() is buggy in gen_netlink, but I'm not sure how to easily fix the problem yet...

https://kernel.org/doc/html/next/userspace-api/netlink/intro.html
"For GET - NLM_F_ROOT and NLM_F_MATCH are combined into NLM_F_DUMP, and not used separately. NLM_F_ATOMIC is never used."

So probably checking that both are set instead of checking any set is the way to go?

Probably something like:

diff --git a/src/netlink.erl b/src/netlink.erl
index 51efc33..08a0a95 100644
--- a/src/netlink.erl
+++ b/src/netlink.erl
@@ -974,7 +974,7 @@ nl_ct_dec(_Protocol, << >>, Acc) ->
     lists:reverse(Acc).

 is_rt_dump(Type, Flags) ->
-    (Type band 3) =:= 2 andalso Flags band ?NLM_F_DUMP =/= 0.
+    (Type band 3) =:= 2 andalso Flags band ?NLM_F_DUMP =:= ?NLM_F_DUMP.

pespin wrote in #note-11:

https://kernel.org/doc/html/next/userspace-api/netlink/intro.html
"For GET - NLM_F_ROOT and NLM_F_MATCH are combined into NLM_F_DUMP, and not used separately. NLM_F_ATOMIC is never used."

So probably checking that both are set instead of checking any set is the way to go?

ACK to myself, I gave it a try and I'm reaching way further in the gtp device setup at startup now. I pushed the commit to github/osmocom/gen_netlink.git branch osmocom/master with the other fixes.

I submitted a PR for the is_rt_dump() bug from above here:
https://github.com/travelping/gen_netlink/pull/9

I now seem to be starting everything more or less fine with the gtp tun created (doing nothing with it yet). However, this is only when the gtp0 netif was not previously created. After the first time, when I kill the process, the netdev is still kept alive, and next time I tun osmo-epdg it will fail due to netlink returning:

Netlink message
    Netlink message header (type: Error)
        Length: 112
        Message type: Error (0x0002)
        Flags: 0x0000
            .... .... .... ...0 = Request: 0
            .... .... .... ..0. = Multipart message: 0
            .... .... .... .0.. = Ack: 0
            .... .... .... 0... = Echo: 0
            .... .... ...0 .... = Dump inconsistent: 0
            .... .... ..0. .... = Dump filtered: 0
        Sequence: 34
        Port ID: 34285
    Error code: File exists (-EEXIST) (-17)
    Netlink message header (type: 0x0010)
        Length: 92
        Message type: Protocol-specific (0x0010)
        Flags: 0x0605
            .... .... .... ...1 = Request: 1
            .... .... .... ..0. = Multipart message: 0
            .... .... .... .1.. = Ack: 1
            .... .... .... 0... = Echo: 0
            .... .... ...0 .... = Dump inconsistent: 0
            .... .... ..0. .... = Dump filtered: 0
            .... ...0 .... .... = Specify tree root: 0
            .... ..1. .... .... = Return all matching: 1
            .... .1.. .... .... = Atomic: 1
        Flags: 0x0605
            .... .... .... ...1 = Request: 1
            .... .... .... ..0. = Multipart message: 0
            .... .... .... .1.. = Ack: 1
            .... .... .... 0... = Echo: 0
            .... .... ...0 .... = Dump inconsistent: 0
            .... .... ..0. .... = Dump filtered: 0
            .... ...0 .... .... = Replace: 0
            .... ..1. .... .... = Excl: 1
            .... .1.. .... .... = Create: 1
            .... 0... .... .... = Append: 0
        Sequence: 34
        Port ID: 0

So "Error code: File exists (-EEXIST) (-17)" seems acceptable given the iface is already existing, but somehow gtp_u_kmod is not contemplating that possibility in gtp_u_kmod:init/1 (line 67):

ok = nl_simple_request(RtNl, ?NETLINK_ROUTE, CreateGTPReq),

I get returned:

wait_for_response: Response: [{rtnetlink,error,[],34,41847,{-17,<<92,0,0,0,16,0,5,6,34,0,0,0,0,0,0,0,2,0,254,255,0,0,0,0,1,0,0,0,1,0,0,0,8,0,28,0,30,0,0,0,9,0,3,0,103,116,112,48,0,0,0,0,40,0,18,0,8,0,1,0,103,116,112,0,28,0,2,0,8,0,1,0,28,0,0,0,8,0,2,0,29,0,0,0,8,0,3,0,0,0,2,0>>}}]

I also fixed gen_netlink incorrectly decoding the error code field in here:
https://github.com/travelping/gen_netlink/pull/9

I tried dropping the excl flag from the new_link nl message in order to allow reusing the tundev:

diff --git a/src/gtp_u_kernel.erl b/src/gtp_u_kernel.erl
index 1918fb8..257a01d 100644
--- a/src/gtp_u_kernel.erl
+++ b/src/gtp_u_kernel.erl
@@ -58,7 +58,7 @@ init([Device, FD0, FD1u, Opts]) ->
                     {linkinfo,[{kind, "gtp"},
                                {data, CreateGTPData}]}]},
     CreateGTPReq = #rtnetlink{type  = newlink,
-                             flags = [create,excl,ack,request],
+                             flags = [create,ack,request],
                              seq   = erlang:unique_integer([positive]),
                              pid   = 0,
                              msg   = CreateGTPMsg},

But now I get "Error code: Operation not supported on transport endpoint (-EOPNOTSUPP) (-95)" instead.

I think it hits this path in the kernel in __rtnl_newlink:

        if (linkinfo[IFLA_INFO_DATA]) {
            if (!ops || ops != dev->rtnl_link_ops ||
                !ops->changelink)
                return -EOPNOTSUPP;

            err = ops->changelink(dev, tb, data, extack);
            if (err < 0)
                return err;
            status |= DO_SETLINK_NOTIFY;
        }

So we need to make sure the tun device is released before starting the app.