[midwell]

Stacked on #713. This branch carries that PR's three commits. The changes for review here are the last three: Do not serve a stored session that a later session superseded and the two Answer 404 … commits. Please merge #713 first. If this one is squash-merged first, it brings #713's commits with it and that PR is left empty.

Problem

With enableDBStore, a stored session is read back lazily, by whatever request next names it: an AMF update or release, a PCF SmPolicyUpdateNotification, an N1N2 failure notification, an OAM read, or a PFCP message for its SEID. GetSMContext and GetSMContextBySEID put the stored document into the session pool as a live session. Neither checks whether the subscriber's PDU session has been established again since, and neither registers the restored session in canonicalRef.

#713 releases the stored sessions when the subscriber establishes the PDU session again, as TS 24.501 clause 6.4.1.7 c) requires. That still leaves two ways a dead session is served:

• a request names it before the subscriber re-establishes, or races the release;
• the datastore already holds several sessions for one key, written before the fix. On a test cluster one subscriber had eighteen.

A dead session served as live does harm. The SMF sends user-plane modifications for it that the UPF refuses (PFCP session not found, repeatedly). Anything that looks it up by reference or SEID also acts on it: an AMF update, a PCF notification, an OAM read, a PFCP message.

Fix

Do not serve a stored session that a later session superseded. A restore now checks the subscriber's PDU session first:

• If canonicalRef names a different session, the stored one is released and the request is answered as for an unknown session.
• If nothing is current, the first restore lists every stored session for the key, using fix: release a stored session when its PDU session ID is established again #713's index. It serves the most recently created, releases the others, and registers the one it serves in canonicalRef, so a later restore of an older copy is recognised as superseded.

Sessions now record when they were created: createdSmfCount, the smfCount of the SMF start that created them, and createdSeq, a sequence within that start. Every start with the datastore enabled increments smfCount, and since #710 nothing depends on resetting it. The order requires smfCount never to go backwards: a reset would rank an earlier start's sessions above a later one's, and the SMF would then release the live session.

Some cases are decided differently:

• A session with a creation record outranks every session without one, written before this change, and those are released.
• Several sessions without a record have no order, so none is served until the subscriber establishes the PDU session again, which releases them all (fix: release a stored session when its PDU session ID is established again #713). A lone unrecorded session is restored as before.
• A restore decides under a per-key lock, so two requests naming two copies release each copy once. The lock is held across the release, including its datastore reads and deletes, its PFCP sends, and the wait for the older session's lock, so a create for the same subscriber and PDU session ID can wait for it.
• If the listing fails, the session is served as before, unregistered. If it succeeds but does not contain the session, the session is served the same way, unless its document has been deleted since. That fails open to today's behaviour, with its limits: a session served unregistered is not checked again while it stays in the pool, so if the subscriber re-establishes and fix: release a stored session when its PDU session ID is established again #713's listing fails too, it stays there. And if it was in fact the older session, a UPF restart releases it, and the AMF releases the subscriber's newer PDU session. Both need a listing to fail or miss.
• A ReleaseSmContext is honoured whatever the order of the others. It names the session its sender holds, and refusing it would leave the AMF holding a session nothing serves.
• Releasing a session that a later one already superseded does not publish the end of the subscriber's session (SubsOpDel). Consumers such as metricfunc key subscribers by IMSI, and the event would remove the record of the live session. A release on create (fix: release a stored session when its PDU session ID is established again #713) and an in-memory replacement still publish it, before the new session's start. Consumers see the delete, then the add, only if the topic keeps order, as a single partition does; the writer sets no message key.

All callers of GetSMContext and GetSMContextBySEID already treat nil as "no such session", so a refused restore needs no change at the call sites.

A restored session on a UPF that restarts. Registering a restored session as the subscriber's current session also makes the UPF-restart sweep treat it as current. Until now such a session was skipped there, because nothing had registered it as current, and was left reporting active with no forwarding. Re-installing it would program the UPF for a UE address that an earlier start of the SMF allocated, which this start's running allocator never reserved and may have given to another UE since. So a restored session anchored on a UPF that restarts is now released through the existing path for a session that cannot be restored, with the AMF and PCF told. That holds whether or not it was registered.

• One that a later session superseded is not swept. The release command names the subscriber and the PDU session ID, so the AMF would release the later session.
• Only restored sessions in the pool are affected. Stored sessions nothing has read back are not in the pool, and are neither re-installed nor released.
• The releases run after the restoration waves, so their calls to the AMF do not hold up the sessions that can be re-installed. Each session is checked again under its lock first. One released, replaced or superseded in the meantime is not released again, because the UE may have established the PDU session again since. The same check now applies to the sessions the UPF would not accept back, which are also released after the waves. A session skipped this way is logged, not counted under any outcome. The release command is still sent after the session lock is dropped, so a re-attach that arrives while it is in flight can still be hit, as on the existing path.
• A failed release counts toward the UPF's sessions without service. A successful one is reported under a sixth restoration outcome, releasedRestoredFromStore.
• Restored sessions are remembered as anchored on the UPF, so one still busy when the UPF restarts again is counted as without service. While the sweep has found nothing to re-install it keeps looking at busy restored sessions; once it has found a session to re-install, a busy restored session is left, as a busy session this SMF created already is.

The cost is that the restored flag is never cleared. A restored session that went on carrying traffic is released, not re-installed, on every later restart of its UPF, also when its address was provided by the UPF or is static. The UE re-establishes. Reconciling a restored session's address with the running allocator is the larger fix #712 leaves out of scope, and it is not done here.

Answer 404 when an update or release finds no SM context. TxnFailure tested txn.Ctxt == nil, but the lookup stores a nil *SMContext in that interface, which is not nil. The 404 was never built, the transaction ended without a response, and the request was answered 500. This predates this PR, and any unknown reference hit it, but the AMF's handling of a refused session depends on getting 404. hasSMContext now tests the context the interface holds.

Answer 404 when a policy update notification finds no SM context. TxnFailure had no arm for SmPolicyUpdateNotification, so a refused notification was also answered 500. It is now answered 404 with a ProblemDetails body. TS 29.500 clause 5.2.7.2: "If the specified target resource does not exist, the NF shall reject the HTTP method with the HTTP status code "404 Not Found"." It also matters to the PCF. From v3.2.1, the PCF's notification of established sessions treats a 4xx from the SMF as a rejected session, and counts a 5xx towards giving up on the SMF.

Testing

Unit tests in context/stored_session_test.go and producer/restored_session_test.go, against the in-memory datastore stand-in from #713:

• superseded, newest-wins and unordered cases: TestASupersededStoredSessionIsNotServed, TestTheMostRecentlyCreatedStoredSessionIsServed (including two sessions of one start, the older stored first), TestUnorderedStoredSessionsAreNeitherServedNorReleased, TestALoneStoredSessionIsRestored, TestALaterStartsSessionOutranksAnEarlierStarts;
• the release exception: TestAReleaseOfAnUnorderedStoredSessionIsHonoured, TestARestoredSessionIsFoundMarkedForRelease, TestAReleaseRequestRacingALookupActsOnThePooledInstance;
• the fail-open paths: TestARestoreWhoseListingFailsIsServedUnregistered, TestARestoreTheListingMissesIsServedUnlessReleased;
• the per-key lock: TestConcurrentRestoresOfOneSessionReleaseEachCopyOnce;
• UPF restart:
  • TestARestoredSessionIsReleasedNotReinstalledWhenItsUserPlaneRestarts (including the releasedRestoredFromStore count)
  • TestAnUnregisteredRestoredSessionIsReleasedWhenItsUserPlaneRestarts
  • TestASupersededRestoredSessionIsNotSweptForRelease
  • TestARestoredSessionReleasedSinceTheSweepIsNotReleasedAgain (re-attach, re-attach while the listing fails, released on request: the AMF stub receives no release)
  • TestABusyRestoredSessionIsReleasedOnceItCanBeExamined
  • TestABusyRestoredSessionIsCountedWithoutService
  • TestARestoredSessionWhoseReleaseFailsIsCountedUnrestored, TestAFailedRestoredReleaseIsCountedBesideTheSessionsThatCouldNotBeExamined
  • TestARestoredSessionIsReleasedBesideSessionsToReinstall
  • TestABusySessionBesideARestoredOneIsLookedAtAgain
• event publishing: TestReleasingASupersededStoredSessionReportsNoEnd, TestReleasingTheCurrentRestoredSessionReportsItsEnd;
• 404 answers: TestARequestForASupersededSessionIsAnsweredNotFound (update, release of an unknown reference, OAM read) and TestAPolicyUpdateNotificationForASupersededSessionIsAnsweredNotFound.

Each guard was reverted on its own, and a test failed every time. The reverts were:

• the superseded and released checks before the UPF-restart release;
• the sweep's restored-session branch, and its superseded exclusion;
• remembering restored sessions as anchored, and merging that record with earlier sweeps';
• counting failed restored releases in the unrestored gauge;
• the releasedRestoredFromStore outcome;
• the LoadOrStore on the release-on-request path;
• the per-key lock (TestALookupRacingTheReleaseOfAStoredSessionGetsNoSecondCopy: a lookup waits for the release and is then handed nothing);
• the Kafka suppression;
• the sequence comparison;
• the typed-nil check.

The one exception is the LocalPurged check before the release. Both purge paths also set the release state under the same lock, so the state check already catches them. It is kept to match the re-install path.

On a lab cluster (SMF with enableDBStore, BESS UPF, UERANSIM), on an earlier build of this change before its final rebase:

• An OAM read of a superseded session's reference is answered 404. The stored session is released, metricfunc keeps the live subscriber, and the UE's live session keeps its traffic.
• After the AMF's record was made to name the old session, deregistering the UE sends its release, which is answered 204, and the AMF drops its SM context.
• I could not construct a service request for a superseded session on this setup: after a UE context release, UERANSIM's gNB rejects the UE's Service Request before it reaches the AMF.
• The UPF-restart release and the policy-notification 404 were exercised in unit tests only.

pre-commit run --all-files is clean on macOS (darwin/arm64, Go 1.26.10), with no hook skipped. That covers gitleaks, gci, staticcheck, go test -race ./... -count=20, golangci-lint v2.13.2, yamlfmt and reuse. go build and go vet are clean too. Each commit builds and vets on its own.

Trade-offs

• Several SMF instances. The order is by creating start. With two SMFs running at once on one datastore, a session created later by the SMF that started earlier carries the lower smfCount, and is ordered before an older session of the SMF that started later. The newer, live session is then the one released. The SMF can run several pods (it initialises DRSM), and a sibling pod's live session would look like a stored duplicate. fix: release a stored session when its PDU session ID is established again #713's release on create has the same exposure. The fix assumes one SMF instance per datastore, or instances whose sessions do not share keys.
• A stale newest document. The newest recorded document can be a released session whose delete failed. The marker for a failed delete is in memory only and is lost on restart. No live session is lost to it, because a live session for the key was created later and outranks it.
• A served session that will not decode. The served session is registered before it is loaded. If its document lists fine but fails the full decode, the older copies are already released and the key names nothing until the next create. This needs a corrupt document.
• Up to 30 s before the releases. When the sweep finds only restored sessions and some it cannot examine, it waits out its retry window before releasing.
• Multi-UPF data paths. The release of a restored session does not delete its PFCP sessions on the other UPFs of a multi-UPF path, as the existing path for unrestorable sessions does not.
• One extra indexed read on the first restore of a key with nothing current. Later restores are checked against canonicalRef only.
• Scope. The two 404 commits are independent of the rest and can be split into their own PR if preferred. The UPF-restart release cannot be split from the superseded restore: it exists because the restore now registers restored sessions as current, which without it would have them re-installed.

🤖 Generated with Claude Code