Project

General

Profile

Feature #4472

Intra-domain connection of OsmoGBPROXY to multiple SGSNs (pooling)

Added by laforge over 1 year ago. Updated 3 months ago.

Status:
Resolved
Priority:
High
Assignee:
Target version:
-
Start date:
03/29/2020
Due date:
% Done:

100%

Spec Reference:
TS 23.236 Section 5.3.2

Description

SGSN pooling is based on splitting the P-TMSI into two parts: A few bits that identify the SGSN in the pool (the so called NRI), and the remainder, which identifies the subscriber within the SGSN.

At time of first contact (random/foreign TLLI), there is a node selection function which chooses the SGSN to use for this request. All follow-up messages are then handled via this SGSN.

This feature would have to be implemented in OsmoPCU, alongside with corresponding test suite in TTCN-3.

Each PCU then has multiple Gb links; at least one to each of the SGSNs in the pool.

sgsn-pool.png View sgsn-pool.png 153 KB laforge, 11/04/2020 11:48 AM
bssgp-pdu-overview.gnumeric bssgp-pdu-overview.gnumeric 57.2 KB laforge, 12/06/2020 11:21 AM
bssgp-pdu-overview.png View bssgp-pdu-overview.png 278 KB laforge, 12/06/2020 11:22 AM
4353
4446

Checklist

  • per-BVC FSM for RESET/BLOCK/UNBLOCK
  • ability to configure multiple SGSN-side NSE in gbproxy
  • TTCN3 test case for RESET/BLOCK/UNBLOCK with two SGSNs in pool
  • NRI configuration in VTY
  • uplink message routing based on NRI extracted from TLLI
  • RADIO-STATUS
  • BVC FLOW CONTROL
  • RIM
  • MS-REGISTRATION-ENQUIRY
  • PS-PAGING-REJECT (requires ISMI cache)

Related issues

Related to OsmoBSC - Feature #3682: Intra-domain connection of OsmoBSC to multiple MSCsResolved11/06/2018

Related to osmo-gbproxy - Bug #4897: gbproxy2: Re-introduce handling of NS_AFF_CAUSE_FAILURENew12/08/2020

Related to osmo-gbproxy - Feature #4896: gbproxy2: Routing of RIM messagesResolved12/08/2020

Related to osmo-gbproxy - Bug #4895: gbproxy2: Implement small TLLI cache for SUSPEND/RESUME procedureResolved12/08/2020

Related to osmo-gbproxy - Bug #4894: gbproxy2: Revisit information storageNew12/08/2020

Related to osmo-gbproxy - Feature #4893: gbproxy2: Make feature bitmap configurableNew12/08/2020

Related to osmo-gbproxy - Bug #4892: gbproxy2: Route BSSGP-STATUS based on "Erroneous PDU IE"New12/08/2020

Related to osmo-gbproxy - Bug #4891: gbproxy2: Implement processing of BVC-Flow-Control via BVC FSMIn Progress12/08/2020

Related to osmo-gbproxy - Bug #4890: gbproxy2: VTY configuration of NRI / pool related bitsResolved12/08/2020

Related to osmo-gbproxy - Bug #4889: implement truncating of BSSGP STATUS when exceeding the FR MTUResolved12/06/2020

Related to osmo-gbproxy - Feature #4951: more TTCN3 tests for SGSN poolingIn Progress01/15/2021

Related to osmo-gbproxy - Bug #4954: Fix routing of RADIO-STATUS by TMSIResolved01/17/2021

Associated revisions

Revision 2b7b1bbb (diff)
Added by laforge 10 months ago

gbproxy: Rename gbproxy_cfg.nses to gbproxy_cfg.bss_nses

We will soon also have a list of sgsn-side NSEs, and we need to
differentiate those.

Change-Id: If5accec0c70c01b88927ea07beba6f6488bd9d5a
Related: OS#4472

Revision e520964d (diff)
Added by laforge 10 months ago

gbproxy major rewrite for SGSN pool support

Rewrite of a large part of osmo-gbproxy in order to prepare
for SGSN pool support. The amount of changes are of such fundamental
nature that it doesn't make sense to try to split this into hundreds
of individual changesets.

Related: OS#4472
Change-Id: Ie0746f17927a9509c3806cc80dc1a31d25df7937

Revision 4bd3e49e (diff)
Added by laforge 10 months ago

gbproxy: Use "(nsei << 16) | bvci" as rate_ctr_group index

As we now have gbproxy_bvc on both the SGSN and the BSS side
with the same BVCI, using the BVCI alone will no longer render
unique indexes.

Related: OS#4472
Change-Id: I13f3c9e69562a56ad7d3742fdeb2ba48f134fdaa

Revision cfc7e8e8 (diff)
Added by laforge 10 months ago

gbproxy: Introduce new DOBJ log category; log object allocation/release

Related: OS#4472
Change-Id: I43bcbcda8667d193e7a17fd8e8e9109597b01484

Revision dbef0aa2 (diff)
Added by laforge 10 months ago

gbproxy: Don't create an extra msgb copy for SGSN DL SIG

That copy may have made sense while we were doing patching/buffering,
but we're not doing any of that anymore.

Related: OS#4472
Change-Id: I207a869ffac8bf60104f80f9ed58faf0021e5e95

Revision 06331ac8 (diff)
Added by daniel 10 months ago

gbproxy: Fix bvci check in gbprox_rx_ptp_from_*

The check for bvci in rx_ptp_from* was always false.

Change-Id: I16a0284ba3201c146c307db6997a416589d7e693
Related: OS#4472

Revision 1e7be5d1 (diff)
Added by daniel 10 months ago

osmo-gbproxy: Initialize all hash_maps

Change-Id: I9578af77a7b2f61b57c918a703768ca20221c294
Related: OS#4472

Revision 98b1b45b (diff)
Added by daniel 10 months ago

gbproxy: Fix confusing log message in gbprox_relay2nse

This function is now used to transmit messages in both directions,
BSS->SGSN and SGSN->BSS.
Print the actual direction in the logs

Change-Id: I31682156dfe88f7ca121a711968e625caed8bd5e
Related: OS#4472

Revision d4ab1f98 (diff)
Added by daniel 10 months ago

gbproxy: Add SGSN pooling support

Change-Id: I58b9f55065f6bd43450e4b07cffe7ba132b1fd9b
Related: OS#4472

Revision 8613c9df (diff)
Added by daniel 9 months ago

gbproxy: Use IMSI cache to handle PAGING_PS_REJECT

Change-Id: I7d91d9ecfba757dc81edcf05efb7a2158348099d
Related: OS#4472, OS#4951

Revision 04918c05 (diff)
Added by daniel 3 months ago

Fix TC_ms_reg_enq

In the pooling case it is not clear where the message will be forwarded
to so allow reception on any SGSN.

Change-Id: I6669ed0b49e9f278f16a67dc05ae208884ee535e
Related: OS#4472

Revision 361d0b56 (diff)
Added by daniel 3 months ago

gbproxy: Add usage flag to the imsi_cache

This is only used for the imsi cache entries for now. Further uses will
be added in subsequent commits.

Related: OS#4472
Change-Id: I4a4b8c99eb97f6bb5387d0f26aecd861e07d9914

Revision f024eeb9 (diff)
Added by daniel 3 months ago

gbproxy: Forward MS_REGISTR_ENQ/_RESP correctly

We need to save the BSS NSE <-> IMSI mapping to correctly route the
answer.

Related: OS#4472
Change-Id: I1908bbe8db11271dbd3f45b0d9f1bc0bfe48f239

History

#1 Updated by laforge over 1 year ago

  • Related to Feature #3682: Intra-domain connection of OsmoBSC to multiple MSCs added

#2 Updated by ipse over 1 year ago

It would be really nice if this feature could be implemented both in OsmoPCU and OsmoGbProxy at the same time - OsmoGbProxy is very helpful in practical installations.

#3 Updated by laforge over 1 year ago

On Sun, Mar 29, 2020 at 07:15:37PM +0000, ipse [REDMINE] wrote:

It would be really nice if this feature could be implemented both in OsmoPCU and OsmoGbProxy at the same time - OsmoGbProxy is very helpful in practical installations.

I understand that, and obviously I want that more than anyone else.
The problem with pretty much all 'feature' tickets here is: Who is going to contribute related code /
developer time or funds :) It's not like we're seeing a lot of resources thrown at Osmocom...

#4 Updated by laforge 12 months ago

  • Subject changed from Intra-domain connection of OsmoPCU to multiple SGSNs to Intra-domain connection of OsmoPCU to multiple SGSNs (pooling)

#5 Updated by laforge 12 months ago

  • Priority changed from Low to High

#6 Updated by laforge 12 months ago

4353

It has become clear that this feature will be implemented in a generic nature so it can be used from both osmo-pcu and osmo-gbproxy.

What we need to make this work is:
  • NAS Node Selection Function inside BSSGP code on BSS side
  • multiple NSE inside the BSS (one for each SGSN), each with it's own set of NS-VCs
  • signaling BVC from each SGSN
  • separate PTP BVC from each cell to each SGSN

This diagram from TS 48.016 serves as a good illustration:

#7 Updated by laforge 11 months ago

  • Status changed from New to In Progress
  • Assignee set to laforge

WE could implement this "only" in OsmoPCU, but we also need it in osmo-gbproxy. Unfortuantely the proxy as a "MITM" has quite some different implementation requirements as the PCU, so there would be relatively separate implementation in both PCU and gbproxy, using some common library/infrastructure.

The easier approach is probably to focus on gbproxy for now. If somebody needs this for a single osmo-pcu, they can always run osmo-gbproxy in front in order to get the pooling feature. And for people running multiple omso-pcu, they will want osmo-gbproxy anyway for aggregation needs.

osmo-gbproxy

I did an analysis for osmo-gbproxy in terms of what kind of processing we need to introduce at each BSSGP message.

The general assumption is that we will have
  • multiple PCU/BSS side Gb interfaces, each
    • with one NSE/NS-VCG
    • one or multiple NS-VC
    • one or multiple PTP BVC
    • no support for pooling in the PCU required or enabled
  • multiple SSN side Gb interfaces, each
    • with one NSE/NS-VCG
    • one or multiple NS-VC
    • support for pooling in the SGSN enabled

For each PTP BVC, we need to "duplicate" it, i.e. establish one per-SGSN BVC for each BSS side BVC. This is achieved by "terminating" the BVC-RESET locally in the Gbproxy on both sides, and having related interconnects, i.e. a BVC-RESET for a PTP BVC on the BSS side will trigger Nx BVC-RESET on the SGSN side, one for each SGSN/NSE.

As for the individual messages:

Downlink-Only messages with no routing requirements

Downlink-only messages are accepted from any SGSN and routed baesd on BVCI, just like it is done prior to this feature.

  • RA-CAPABILITY
  • DL-UNITDATA
  • PAGING-PS
  • PAGING-CS
  • RA-CAPABILITY-UPDATE-ACK
  • SUSPEND-ACK
  • SUSPEND-NACK
  • RESUME-ACK
  • RESUME-NACK
  • FLUSH-LL
  • FLOW-CONTROL-MS-ACK
  • FLOW-CONTROL-PFC-ACK
  • CREATE-BSS-PFC
  • MODIFY-BSS-PFC
  • DELETE-BSS-PFC
  • PS-HANDOVER-REQUIRED-ACK
  • PS-HANDOVER-REQUIRED-NACK
  • PS-HANDOVER-REQUEST
  • PS-HANDOVER-COMPLETE-ACK
  • PERFORM-LOCATION-REQUEST
  • PERFORM-LOCATION-ABORT
  • POSITION-RESPOSNE

Messages routed by NRI extracted from TLLI

  • UL-UNITDATA
  • RA-CAPABILITY-UPDATE
  • SUSPEND
  • RESUME
  • FLUSH-LL-ACK
  • LLC-DISCARDED
  • FLOW-CONTROL-MS
  • FLOW-CONTROL-PFC
  • CREATE-BSS-PFC-ACK/NACK
  • MODIFY-BSS-PFC-ACK
  • DELETE-BSS-PFC-ACK
  • DELETE-BSS-PFC-REQ
  • PS-HANDOVER-REQUIRED
  • PS-HANDOVER-REQUEST-ACK
  • PS-HANDOVER-REQUEST-NACK
  • PS-HANDOVER-COMPLETE
  • PS-HANDOVER-CANCEL
  • PERFORM-LOCATION-RESPONSE
  • POSITION_COMMAND

Special cases

RADIO-STATUS

  • if TLLI or P-TMSI are included, route based on NRI
  • if IMSI is included, route to "randomly distributed" SGSN in pool

BVC flow control

The capacity advertised by the BSS/PCU side has to be split between all SGSNs. Separate BVC flow control instances need to be operated in gbproxy for each BVC, with coordination between them. Probably implemented by osmo_fsm.

STATUS

The STATUS message doesn't really tell us where it belongs to, unless the contained PDU would have TLLI or P-TMSI information. We could consider broadcasting it?

BVC BLOCK/UNBLOCK/RESET

Those have to be terminated for each BVC inside osmo-gbproxy: One each on the BSS/PCU side, and one per-SGSN-per-BVC on the SGSN side. Coordination must happen between them. Probably implemented by osmo_fsm.

SGSN-INVOKE-TRACE

This downlink-only message must be broadcast to all BSS, like it is done in the non-pooling case.

OVERLOAD

This downlink-only message must be broadcast to all BSS

RIM

It is assumed any SGSN can route RIM messages to any other PCU, so we can simply chose any of them. In the case of responses it might be a good idea to use the same thorugh which the request was received.

MS-REGISTRATION-ENQUIRY

The ENQUIRY is in uplink direction and contains no TlLI but only IMSI. We can route it to any SGSN. The ENQUIRY-RESPONSE is in downlink direction and must be sent back to whichever PCU/NSE requesting it. We therefore need to track some state here.

#8 Updated by laforge 11 months ago

  • Checklist item per-BVC FSM for RESET/BLOCK/UNBLOCK added
  • Checklist item ability to configure multiple SGSN-side NSE in gbproxy added
  • Checklist item TTCN3 test case for RESET/BLOCK/UNBLOCK with two SGSNs in pool added
  • Checklist item NRI configuration in VTY added
  • Checklist item uplink message routing based on NRI extracted from TLLI added
  • Checklist item RADIO-STATUS added
  • Checklist item BVC FLOW CONTROL added
  • Checklist item RIM added
  • Checklist item MS-REGISTRATION-ENQUIRY added

I've started with implementing the slightly contrived BVC-RESET handling, as this will be the logical first step we need in order to bring up a GbProxy with a pool of several SGSNs.

There will be a per-BVC osmo_fsm that takes care of reset/block/unblock. This FSM can behave in either SGSN or BSSGP role, and gb-proxy will use this FSM both towards each BSS and towards each SGSN.

state-change notification call-backs and reset-indication-callbacks will be used by gbproxy to "stitch" those FSMs together. So for example, if there's a reset-indication from a BSS-side FSM, then that will trigger "reset request FSM events" on each SGSNs BVC-FSM for the same BVCI, which iwll then in turn trigger BVC-RESET procedures towards each of those SGSNs.

The plan is to get this implemented together with a minimal "bring up all BVCs with two SGSN" TTCN3 testcase, and then hand it over to daniel for implementation of the handling of various PDU types as outlined here in this ticket.

#9 Updated by laforge 10 months ago

4446

I've created a table that shows all BSSGP PDUs, their direction UL/DL, SIG/PTP BVC, SAP as well as the Prosy handling in the context of multiple SGSN / pool support.

bssgp-pdu-overview.gnumeric

#10 Updated by laforge 10 months ago

  • Project changed from OsmoPCU to osmo-gbproxy
  • Subject changed from Intra-domain connection of OsmoPCU to multiple SGSNs (pooling) to Intra-domain connection of OsmoGBPROXY to multiple SGSNs (pooling)

#11 Updated by laforge 10 months ago

  • Checklist item per-BVC FSM for RESET/BLOCK/UNBLOCK set to Done
  • Checklist item RADIO-STATUS set to Done
  • % Done changed from 0 to 20
In the laforge/gbproxy branch, there is now the foundation for this feature.
  • all data structures have been adjusted to accomodate the pooling situation
  • BVC-RESET/BLOCK/UNBLOCK is now terminated inside the proxy for all BVCs on all NSEs (BSS, SGSN)
  • RADIO STATUS routing based on either TLLI, TMSI or IMSI is implemented

#12 Updated by laforge 10 months ago

  • Related to Bug #4897: gbproxy2: Re-introduce handling of NS_AFF_CAUSE_FAILURE added

#13 Updated by laforge 10 months ago

  • Related to Feature #4896: gbproxy2: Routing of RIM messages added

#14 Updated by laforge 10 months ago

  • Related to Bug #4895: gbproxy2: Implement small TLLI cache for SUSPEND/RESUME procedure added

#15 Updated by laforge 10 months ago

  • Related to Bug #4894: gbproxy2: Revisit information storage added

#16 Updated by laforge 10 months ago

  • Related to Feature #4893: gbproxy2: Make feature bitmap configurable added

#17 Updated by laforge 10 months ago

  • Related to Bug #4892: gbproxy2: Route BSSGP-STATUS based on "Erroneous PDU IE" added

#18 Updated by laforge 10 months ago

  • Related to Bug #4891: gbproxy2: Implement processing of BVC-Flow-Control via BVC FSM added

#19 Updated by laforge 10 months ago

  • Related to Bug #4890: gbproxy2: VTY configuration of NRI / pool related bits added

#20 Updated by laforge 10 months ago

  • Related to Bug #4889: implement truncating of BSSGP STATUS when exceeding the FR MTU added

#21 Updated by laforge 10 months ago

  • Checklist item BVC FLOW CONTROL set to Done

#22 Updated by laforge 10 months ago

  • % Done changed from 20 to 40

#23 Updated by laforge 10 months ago

https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/21698 contains a WIP patch that adds support for checking the BVC-bringup to two SGSNs. It is working fine in local tests. Patch cannot be merged yet as it is waiting for the NS2 VTY related changes to be completed by lynxis

#24 Updated by daniel 10 months ago

In addition to NRI configuration in VTY issue #4890 will also implement the ability to configure multiple SGSN-side NSE in gbproxy

#25 Updated by daniel 9 months ago

  • Checklist item ability to configure multiple SGSN-side NSE in gbproxy set to Done
  • Checklist item NRI configuration in VTY set to Done
  • Checklist item uplink message routing based on NRI extracted from TLLI set to Done
  • Assignee changed from laforge to daniel
  • % Done changed from 40 to 60

Patches in https://gerrit.osmocom.org/q/topic:%22sgsn-pooling%22+(status:open%20OR%20status:merged) add pooling support to gbproxy, but still untested (apart from the simple case of only having one SGSN in the pool that is active).

I'm now implementing a TLLI cache to forward SUSPEND/RESUME properly.

#26 Updated by laforge 9 months ago

  • Checklist item TTCN3 test case for RESET/BLOCK/UNBLOCK with two SGSNs in pool set to Done

#27 Updated by laforge 9 months ago

  • % Done changed from 60 to 70

BVC-{RESET,BLOCK,UNBLOCK} for multiple SGSNs is now tested in https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/21698 as part of the normal test case initialization function

#28 Updated by laforge 9 months ago

Just committed a series of patches with https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/22231 on top which expand testing to include SGSN pool related testing. We're already verifying
  • Suspend/Resume is routed by NRI to correct SGSN
  • BVC-RESET is duplicated to every SGSN
  • BVC-FC is duplicated to every SGSN

Most of the tests currently test only with a NRI routed to the first SGSN.

I'm creating a new ticket (#4951) to track the missing TTCN3 tests so we can use this ticket to really keep only track of the actual features (only RIM + MS-REQ-ENQ missing)

#29 Updated by laforge 9 months ago

  • Related to Feature #4951: more TTCN3 tests for SGSN pooling added

#30 Updated by laforge 9 months ago

  • Checklist item PS-PAGING-REJECT (requires ISMI cache) added

#31 Updated by daniel 9 months ago

Interesting that 3GPP TS 48.018 does not specify how PS-PAGING-REJECT or DUMMY-PAGING should be sent. I'm looking at Release 16 and Table 5.4.1 has no mention of those. Table 5.2 and 5.3 also don't. I'm guessing we can assume that it's PTP or signalling just like PAGING-PS.

#32 Updated by daniel 9 months ago

IMSI cache and support for PAGING_PS_REJECT (at lease on signalling) are in Gerrig:
https://gerrit.osmocom.org/c/osmo-sgsn/+/22251
https://gerrit.osmocom.org/c/osmo-sgsn/+/22252

#33 Updated by daniel 9 months ago

  • Checklist item PS-PAGING-REJECT (requires ISMI cache) set to Done

#34 Updated by daniel 9 months ago

  • Related to Bug #4954: Fix routing of RADIO-STATUS by TMSI added

#35 Updated by daniel 8 months ago

  • Checklist item RIM set to Done

RIM support has been merged, tests are passing

#36 Updated by laforge 4 months ago

so MS-REQ-ENQ is the only missing bit to get this issue closed?

#37 Updated by daniel 3 months ago

Apart from BSSGP STATUS routing which is in #4892 yeah. I can implement it, shouldn't take too long. We already have the IMSI cache which we need to route the response.

#39 Updated by daniel 3 months ago

  • Checklist item MS-REGISTRATION-ENQUIRY set to Done
  • Status changed from In Progress to Resolved
  • % Done changed from 90 to 100

The MS Reg enquiry patches are now merged

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)