Project

General

Profile

Actions

Feature #5152

closed

support hnbgw co-located osmo-mgw for RTP proxying

Added by laforge almost 3 years ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
Start date:
05/13/2021
Due date:
% Done:

100%

Spec Reference:

Description

In the current osmo-hnbgw implementation, we simply configure the hNB RTP data to go directly to the MSC (or it's MSC-colocatetd MGW). This requires IP-level routing between the RAN and the CN, which is possible in most lab situations, but not possible in real operator networks.

So we'd need to add support for osmo-hnbgw to control a co-located osmo-mgw. So like osmo-bsc, osmo-hnbgw would have to manage endpoints on osmo-mgw. This way there are separaet RTP flows:
  1. between hNodeB and osmo-mgw on the RAN side
  2. between MSC/MGW and osmo-mgw on the CN side

Files

3g_call_23112021.pcapng 3g_call_23112021.pcapng 75.9 KB dexter, 11/23/2021 05:51 PM
2g_call_23112021.pcapng 2g_call_23112021.pcapng 534 KB dexter, 11/23/2021 05:51 PM
hnb_cs_mo_call_with_mgcp.png View hnb_cs_mo_call_with_mgcp.png 68.9 KB dexter, 11/23/2021 05:55 PM
hnb_cs_mo_call_with_mgcp.msc hnb_cs_mo_call_with_mgcp.msc 1.47 KB dexter, 11/23/2021 05:56 PM

Checklist

  • Finish RANAP message decoding
  • Add testsuite
  • Implement RAB Release
  • make sure fsm terminates when context map is freed (garbage collector)

Related issues

Related to OsmoBSCNAT - Feature #2545: OsmoBSCNAT misses 3GPP AoIPStalledosmith10/06/2017

Actions
Related to OsmoHNBGW - Feature #5153: support hnbgw co-located GTP-U proxy (UPF)In Progressneels05/13/2021

Actions
Actions #1

Updated by laforge over 2 years ago

Actions #2

Updated by laforge over 2 years ago

  • Assignee set to 4368
Actions #3

Updated by daniel over 2 years ago

  • Status changed from New to In Progress
  • Assignee changed from 4368 to daniel

I will be working on this with dexter.

Actions #4

Updated by dexter over 2 years ago

After some difficulties I have my testnet now up and running again. I have made two traces. One on the 3g network and one on the 2g network for reference. From that I have derived a ladder diagram (i have left all security related messages away.) I made it so that it matches the behavior we see on osmo-bsc. My idea is to reuse the FSMs from libosmo-mgcp-client.

Actions #5

Updated by laforge over 2 years ago

In case that is not obvious: I think as part of this project we also need to get towards a decent test coverage for osmo-hnbgw itself. Now that pespin has been creating Iuh related components in TTCN3 for testing osmo-hnodeb, there should be some infrastructure that can also be used to test osmo-hnbgw.

In the end, a osmo-hnbgw testsuite would have to talk Iuh on the one side and IuCS+IuPS on the other side. Once the tests include RAB assignment test cases, one can develop the RTP/MGW integration on the hnbgw step-by-step with the test suite (which would then have to include emulating a MGW towards the MGCP-speaking hnbgw).

Actions #6

Updated by dexter over 2 years ago

So far my Idea is to set up the mgcp_fsm when the RAB-AssignmentRequest is seen and terminate it when the respective IU-ReleaseCommand is seen. But before we can start implementing the FSM we need tools to decode the RAB-AssignmentRequest to extract the transport layer address and to re-encode it with the changed transport layer address. Unfortunately this is not very simple. There are 3 layers of ASN.1 we need to dig through. So far I have figured out the decoding. I get meaningful results. The next step is to make proper decoder/encoder functions out of this. So far I have done a decoder that encodes/decodes the RAB-AssignmentRequest down to its request IEs, which is the first two layers of ASN.1.

What makes me wonder a bit is that in theory it is possible to assign multiple RABs at a time. Its also possible to have multiple "protocol container pairs". I wonder if this plays a role in practice or if it is just one element there anyways. I guess multiple RABs are for conference calls or similar applications.

Actions #7

Updated by laforge over 2 years ago

On Fri, Dec 03, 2021 at 05:37:53PM +0000, dexter wrote:

Unfortunately this is not very simple. There are 3 layers of ASN.1 we need to dig through. So far I have figured out the decoding. I get meaningful results. The next step is to make proper decoder/encoder functions out of this. So far I have done a decoder that encodes/decodes the RAB-AssignmentRequest down to its request IEs, which is the first two layers of ASN.1.

we should have those RANAP-level decoders all in place in both the MSC and the SGSN,
as they both need to decode those very same messages for the exact same
purpose of accessing all those details. Please make sure we don't
duplicate code that may already exist.

What makes me wonder a bit is that in theory it is possible to assign multiple RABs at a time. Its also possible to have multiple "protocol container pairs". I wonder if this plays a role in practice or if it is just one element there anyways. I guess multiple RABs are for conference calls or similar applications.

I could imagine multiple RABs might happen in circuit-switched UMTS
video calls, I'm not sure. One would probably have to look at some
real-world traces. It would also make sense to inquire with the primary
customer whether or not UMTS video calls are something they wish to
support at all.

In the PS domain IMS over UTRAN also likely would use multiple RABs
(like different QCI in LTE) - but to the best of my knowledge, nobody
has deployed IMS over UTRAN; they all use circuit-switched telephony
on 3G and IMS only over 4G/5G.

Actions #8

Updated by dexter over 2 years ago

  • % Done changed from 0 to 30

I have now working decoders for the RAB AsssigmentRequest and for the RAB AssignmentResponse that give me the port/ip address of the RAB that is assigned. I am currently working on the functions that do the replacing. For the RAB AssignmentRequest am done for the most part. The function works fine, but has a memory leak that I still need to figure out. I guess RAB AssignmentResponse will be easier then since it is very similar to RAB AssignmentRequest. Once I am done with the ASN.1 part I can start working on the FSM part.

Actions #9

Updated by dexter over 2 years ago

  • % Done changed from 30 to 40

I am now done with the ASN.1 part. The next thing I will continue with will be the MGCP part.
See also branch: pmaier/mgw

Actions #10

Updated by dexter over 2 years ago

  • % Done changed from 40 to 50

I have now started the FSM that controls the MGW. I have made a draft implementation that does all the interaction with osmo-mgw with dummy data, while not modifying the RAB Assignment message. The FSM goes through the states just fine and I still can make a testcall. However, when I try a second call it does not work anymore. Also I need to test it a bit further to see if there are problematic corner cases. When it works stable as it is I will continue with altered RAB Assignment messages to redirect the voice streams through the MGW. I also noticed that I still need to parse the codec from the RAB Assignment so that I can signal it to the MGW as well.

Actions #11

Updated by dexter over 2 years ago

  • % Done changed from 50 to 60

The is now stable. The next logical step here is to redirect the RTP stream through the MGW.

See also branch pmaier/mgw

Actions #12

Updated by dexter over 2 years ago

  • % Done changed from 60 to 70

I have managed to redirect the RTP streams through the MGW today. Initially I had some problems because I did not create the connection that points towards the HNB in loopback mode at first but now it seems to work fine. It certainly needs some cleanup though. I also added T_defs for the FSM, but this also needs some cleanup as well.

In January we can probably begin with the TTCN3 testing.

Actions #13

Updated by dexter over 2 years ago

I have started to clean up everything. The tdefs are now correct. The call-id is now a concatenation of the rua_ctx_id and the RAB-ID, so that it can be better recognized in traces. Also replaced the printfs with LOGPFSML statements.

To try it out one would also need the following patch for osmo-mgw: https://gerrit.osmocom.org/c/osmo-mgw/+/26651. The branch pmaier/mgw on osmo-iuh.git is up to date. The configuration is similar to osmo-msc and osmo-bsc.

Actions #14

Updated by dexter about 2 years ago

I have now fixed (hopefully) all review issues that were present in gerrit. I have also split part of my patches to the new osmo-hnbgw.git

The current status can be found on osmo-hnbgw.git pmaier/mgw, and osmo-iuh.git pmaier/mgw2.

The current implementation enforces the usage of x213 nsap address format. I initially thought that it were ok to convert everything to x213 nsap, but it is not. It depends on what the HnodeB supports. The format can also be configured in osmo-msc. I did not know that, so we when rewriting the RAB AssignmentRequest we must retain the original format.

At the moment the RTP streams are terminated when an IU Release is seen, this seems to be not enough. There can also be a RAB Release, that should trigger the RTP stream termination as well.

Actions #15

Updated by neels about 2 years ago

The current implementation enforces the usage of x213 nsap address format. I initially thought that it were ok to convert everything to x213 nsap, but it is not. It depends on what the HnodeB supports. The format can also be configured in osmo-msc. I did not know that, so we when rewriting the RAB AssignmentRequest we must retain the original format.

That's correct.

At the moment the RTP streams are terminated when an IU Release is seen, this seems to be not enough. There can also be a RAB Release, that should trigger the RTP stream termination as well.

IIRC the RAB Release comes first, and the IU Release usually follows?

Actions #16

Updated by laforge about 2 years ago

On Fri, Jan 07, 2022 at 01:05:37PM +0000, neels wrote:

At the moment the RTP streams are terminated when an IU Release is seen, this seems to be not enough. There can also be a RAB Release, that should trigger the RTP stream termination as well.

IIRC the RAB Release comes first, and the IU Release usually follows?

I would also expect that both happen, and in this order.

Basically we have three layers of release

  • RAB release - releases just that RAB. This is where RTP resources should be release,
    as the RTP flows are coupled to the RAB
  • Iu release - logical release of the Iu connection at RANAP level
  • SCCP level release of the IuCS connection

Now it is of course possible that thre is an Iu release without RAB release, or even a SCCP
release without Iu or RAB release (e.g. in case of SCCP network outage, MSC restart, ...)

In both of these situations we should recover sanely and destroy the RTP connections.

Actions #17

Updated by dexter about 2 years ago

Hello Harald and Neels.

Thanks for the Input. I have had a look in the trace (3g_call_23112021.pcapng) I have made when I was starting with this ticket. What makes me wonder a bit is that in that trace I can't find a RAB Relase Request. I only see the IU release. I will look closer at this when I have resolved the remaining review issues for osmo-hnbgw.

Best regards.
Philipp

Actions #18

Updated by laforge about 2 years ago

On Mon, Jan 10, 2022 at 04:26:39PM +0000, dexter wrote:

What makes me wonder a bit is that in that trace I can't find a RAB Relase Request. I only see the IU release.

Well, the Iu release tears down any RABs within that Iu context, so it is a valid scenario.

As I wrote we need to deal with all possible scenarios (RAB release, Iu release, SCCP connection loss)

Actions #19

Updated by dexter about 2 years ago

The patches for osmo-iuh.git are all merged now. So the branch pmaier/mgw on osmo-hnbgw.git can be used with current master of osmo-iuh now.

Actions #20

Updated by neels about 2 years ago

The current implementation enforces the usage of x213 nsap address format. I initially thought that it were ok to convert everything to x213 nsap, but it is not. It depends on what the HnodeB supports. The format can also be configured in osmo-msc. I did not know that, so we when rewriting the RAB AssignmentRequest we must retain the original format.

I've just been thinking about this a bit more.
It doesn't have to be the MSC deciding about the address format.
Maybe for a sys admin, it also makes sense to be able to configure the address format at the hnb-gw?

I guess we don't need such a feature until someone asks for it, but maybe good to keep in mind.

Actions #21

Updated by dexter about 2 years ago

The patchset is currently still in review. There is currently some discussion around where and how the encoding/decoding of the ASN.1 messages should happen. There are some patches pending for osmo-iuh that intrduce a message decoder that works similar to the one in ranap_common_cn.c The problem here is that ranap_common_cn.c only decodes messages towards the core network, we also have to decode messages that are directed towards the HnodeB. (e.g. RANAP RAB Assignment Request).

The current status can be found in gerrit and on the following branches:
osmo-iuh.git: pmaier/mgw
osmo-hnbgw.git: pmaier/mgw

Actions #22

Updated by dexter about 2 years ago

Besides the architectural problems with the RANAP message decoding we also need to put the focus on testing osmo-hnbgw with TTCN3.

A minimal testsuite should test the following:
  • Normal call scenario: We should test MO and MT calls just to be complete. In any case for the scope of this ticket this is irrelevant. Regardless how the call is established, the RAB assignment procedure will work the same way. This is basically the most important test.
  • Errornous call scenarios: We might also try some errornous call establishment examples, just to see that FSMs and RTP conns on the MGW are cleared properly.
  • RAB Release: Usually the RAB gets released when the IU Release is carried out. In a simple call scenarios there will be no extra RAB Release Rquest. A RAB Release Request might happen e.g. when the HnodeB detects that the radio link has failed failed. Then a RAB Release Request is sent to the MSC, the MSC then sends another RAB Assignment Request that contains information to release the related RAB. I think for the testcase it is enough just to send an appropriate RAB Assignment Request and check that the related RTP streams are cleared properly.
  • Errornous MGW behaviour: We might also consider to try some MGW failure scenarios to make sure that osmo-hnbgw handles this properly and does not crash etc.
Actions #23

Updated by dexter about 2 years ago

  • Checklist item Finish RANAP message decoding added
  • Checklist item Add testsuite added
  • Checklist item Implement RAB Release added
Actions #24

Updated by dexter about 2 years ago

  • Checklist item make sure fsm terminates when context map is freed (garbage collector) added

I am now through with the changes to osmo-iuh and osmo-hnbgw that fix the API issues described above. We noticed that there is also an issue with the context map that is used by the FSM. The logic that creates and maintains those contexts must destroy a possibly existing FSM before it frees the context. This should be rather easy to fix.

Actions #25

Updated by dexter about 2 years ago

  • Checklist item Finish RANAP message decoding set to Done
  • Checklist item make sure fsm terminates when context map is freed (garbage collector) set to Done
Actions #26

Updated by dexter about 2 years ago

At the moment it is unclear what should happen when the RAB AssignmentResponse contains an unsuccessful outcome. At the moment the ranap_message container will decode, but as far as I can see it will not populate the procedure code. This means the message would fall through. It will not go through the FSM, it will just be passed on to the MSC. So the MSC will most likely tear down the call.

(see also: https://git.osmocom.org/osmo-iuh/tree/src/ranap_common_cn.c#n493)

However, I think the FSM should be informed about the problem, so that it can clear the half open RTP stream. For that we would need to know the procedure code and the type of the outcome. As far as I understand we have this direction field in ranap_message, that could be checked for RANAP_RANAP_PDU_PR_unsuccessfulOutcome. If it is so, the FSM should be terminated. That should be enough and this is also already present in osmo-iuh.

Actions #27

Updated by dexter about 2 years ago

I could make the testsuite that daniel running. Currently I am using TC_RAB_Assignment as a starting point to work out a testcase to verify what happens when the HnodeB responds with a Failure List instead of a Setup or Modify list.

Actions #28

Updated by dexter about 2 years ago

The FSM is now able to react to RANAP RAB AssignmentResponse messages that contain a FailedList. When a FailedList with the related RAB-ID comes back, then the FSM will terminate the RTP streams and forward the message to the MSC so that the CN is also informed about the problem.

I have started working on the RANAP RAB AssignmentResponse that contains a ReleaseList. In this case we must also terminate the RTP streams and also forward the message so that the HNB is informed abot the release as well.

At the moment all this supports only a single RAB.

Actions #29

Updated by dexter about 2 years ago

  • Checklist item Add testsuite set to Done
  • Checklist item Implement RAB Release set to Done
Actions #30

Updated by dexter about 2 years ago

  • % Done changed from 70 to 90

The relevant patches are merged. osmo-hnbgw now has support for a co-located osmo-mgw instance. The implementation is now basically ready to be tried out by actual users.

The following patches are still in review, but they address only minor problems:

https://gerrit.osmocom.org/c/osmo-hnbgw/+/27320 mgw_fsm: release call when FSM is not created
https://gerrit.osmocom.org/c/osmo-hnbgw/+/27321 ranap_rab_ass: check for more than one RAB assignment req

Actions #31

Updated by daniel about 2 years ago

The tests are now enabled on Jenkins and so far everything looks good:
https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-hnbgw-test/

Actions #32

Updated by dexter about 2 years ago

  • % Done changed from 90 to 100

The last two fixup patches are merged. At lest from my perspective, this ticket can be closed. The implementation is tested and working in the lab setup. If there are problems in the real world, they should be addressed with a new ticket.

Actions #33

Updated by dexter about 2 years ago

  • Status changed from In Progress to Resolved

The jenkins testsuite still passes all tests and all pending patches are merged. I will set this to resolved now. I case of problems we will open a dedicated ticket.

Actions #34

Updated by laforge over 1 year ago

  • Related to Feature #5153: support hnbgw co-located GTP-U proxy (UPF) added
Actions #35

Updated by laforge over 1 year ago

  • Status changed from Resolved to In Progress
  • % Done changed from 100 to 90

I just noticed that OsmoHNBGW doesn't say anything yet about a co-located mgw. Please update the wiki accordingly, including a statement what the mgw is used for.

Please coordinate with neels to avoid "merge conflicts" as he has to do a similar task for the UPF side, see my recent update to #5153

Actions #36

Updated by daniel over 1 year ago

  • Status changed from In Progress to Resolved
  • % Done changed from 90 to 100

laforge wrote in #note-35:

I just noticed that OsmoHNBGW doesn't say anything yet about a co-located mgw. Please update the wiki accordingly, including a statement what the mgw is used for.

It looks like neels has already taken care of that, thanks!

Both OsmoHNBGW and the manual mention the co-located mgw.

Actions

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)