[midwell]

Problem

With enableDBStore, a subscriber that establishes a PDU session again after an SMF restart ends up with two sessions for the same PDU session ID, and the old one is never released. Every restart followed by a re-attach adds another. On a test cluster one subscriber had accumulated eighteen stored sessions, all for PDU session ID 1, with eighteen seidSmContext and refToSeid entries beside them. The UPF was answering the SMF's Session Modification Requests for the dead sessions with PFCP session not found; the SMF serving those sessions is the subject of the follow-up below.

TS 24.501 clause 6.4.1.7 c) says what the SMF must do: "If the SMF receives a PDU SESSION ESTABLISHMENT REQUEST message with a PDU session ID identical to the PDU session ID of an existing PDU session and with request type set to "initial request", the SMF shall locally release the existing PDU session and proceed with the PDU session establishment procedure."

The SMF does this for a session it holds in memory. TxnLoadCtxt asks ResolveRef(supi, pduSessionID) and, if it finds one, HandlePduSessionContextReplacement releases it. ResolveRef reads canonicalRef, which only NewSMContext writes. After a restart canonicalRef is empty, the previous session exists only in the datastore, and nothing looks there. The SMF reads no request type here. TS 23.502 clause 4.3.2.2.1 step 3 has the AMF send Nsmf_PDUSession_CreateSMContext only when it holds no association with an SMF for the PDU session ID ("e.g. when Request Type indicates "initial request""), so a create is that case.

This is the first of two pull requests. The second is stacked on it, and makes the SMF stop serving a stored session that a later one superseded.

Fix

Five commits.

1. Release a stored session when its PDU session ID is established again. A create whose PDU session ID resolves to nothing in memory now lists the sessions stored for the subscriber and PDU session ID, and releases every one. Each loses its documents, and its PFCP sessions are deleted at the UPFs that acknowledged them, without waiting for the answers, as the in-memory replacement does. Commits 4 and 5 limit those deletions to a UPF that has not restarted since and that the SMF is associated with. If a lookup has already read a stored session into the pool, the release acts on that instance. A lookup that misses the pool while the release is running is handed the instance being released, never a second copy (LoadOrStore). A lookup that read the document before the release, and stores it after the release completes, can still put the released session back in the pool. That window existed before this PR and is unchanged.

The listing is backed by a new non-unique index on identifier and pduSessionID. It is created in SetupSmfCollection beside the existing two through EnsureIndex, because CreateIndex is single-key and unique. It has to be non-unique: there are several sessions for a key until the older ones are released, and one whose delete failed stays stored beside its successor.

Every context read back from the datastore is now flagged (restoredFromStore, set in GetSMContextByRefInDB), and releasing one differs from releasing one this process created:

• The restore never counted it in the session gauge, so its release no longer decrements the gauge. Before, the gauge then undercounted the live sessions by one, or wrapped when none was counted.
• Its data path can name a UPF this SMF does not hold, for example after the UPF was redeployed under another node ID. That node is skipped instead of dereferenced. A UPF the SMF holds is sent the deletion subject to commit 4's check, which sends nothing until the SMF has associated with that UPF since it started.
• Its UE address needs no guard here. Since fix(context): an allocator with no pool refuses the work instead of panicking #712 the release on a restored context's detached, empty allocator is a logged no-op, and the address stays with the UE the running allocator has given it to since. TestReleasingAStoredSessionLeavesItsAddressWithTheUEHoldingItNow pins fix(context): an allocator with no pool refuses the work instead of panicking #712's behaviour on this path.

2. Delete a released session's SEID records only where they are its own. DeleteSmContextInDBBySEID deleted the seidSmContext document for a local SEID, and the context document it named, without checking whose they were. A SEID document is keyed by the SEID alone, and nothing in the SMF stops a stored session and a live session from naming the same SEID. On a test cluster, two SMF pods two days apart drew SEIDs from the same DRSM chunk. I did not see a stored session's exact SEID handed to a live one. I reproduced that by inserting a stored session carrying the live session's local SEID, and releasing it deleted the live session's documents. The SEID document is now deleted only while it maps to the released session's reference, and the delete is scoped to that reference. The in-memory SEID index drops an entry only if the released session holds it (CompareAndDelete).

3. Free no rule identifiers when releasing a stored session. Deactivating a tunnel returns the PDR, FAR, QER and BAR identifiers to the UPF's generators and removes the rules from the UPF's pools. A stored session's identifiers were allocated by an earlier start of the SMF, and this one may have given the same identifiers to a live session since. The rules of every session read back from the datastore are now removed from its own PFCP session context only, on every release path (UE release, AN release, a PFCP report, replacement). Its PFCP session on a UPF this SMF holds is still deleted, within commit 4's limit. The same applies when a policy update withdraws a rule from such a session (forgetRemovedRules). The revert of an abandoned modification, and a modification that is not sent, still free what they withdraw, because this start built those rules.

4. Delete a stored session at its UPF only while that UPF has not restarted. A stored session's RemoteSEID was issued by whichever incarnation of the UPF was running when an earlier start of the SMF established it. A UPF that has restarted since no longer holds the session, and may have given the same SEID to another subscriber's session, which the deletion would remove. The PFCP context already records the UPF's recovery timestamp when the UPF acknowledged the session (AcknowledgedAtRecovery), but the datastore did not keep it. It is now stored, and the deletion for a session read back from the datastore is sent only when the stored timestamp matches the one the SMF holds for the UPF now and the SMF is associated with that UPF (commit 5). Otherwise the session is released locally and no deletion is sent. That includes documents stored before this commit, which carry no timestamp, and a release that runs before the SMF has associated with the UPF since it started. A session this start created is not affected. On pfcpiface the recovery timestamp is set per association (NewPFCPConn), so it changes exactly when the UPF loses the association and its sessions.

5. Require the association as well as the timestamp. A heartbeat that reports a UPF restart marks the UPF NotAssociated but leaves the previous incarnation's recovery timestamp held until the re-association replaces it. A release in between found that timestamp equal to the stored one and sent the deletion. The check now also requires the UPF to be associated, and reads the status and the timestamp together under UpfLock (UPF.AssociatedIncarnation).

Commits 2 and 3 were found while testing commit 1, and commits 4 and 5 were added after review. Commit 5 closes a gap in commit 4's check. The defects fixed by commits 2, 3 and 4 are on main already, for any release of a restored session, and commit 1 makes that release common.

Testing

Unit tests in context/stored_session_test.go and producer/restored_session_test.go, against memDB, a new in-memory stand-in for the datastore that answers the filters the SMF uses:

• TestACreateAfterARestartReleasesTheStoredSession, TestACreateReleasesEveryStoredCopy: a create after a simulated restart removes the stored session's context and SEID documents, and every copy where there are several, and the key then resolves to the new session.
• TestAStoredSessionAlreadyInThePoolIsReleasedAsThatInstance, TestAStoredSessionPooledWhileItIsReadForReleaseIsTheOneReleased, TestALookupRacingTheReleaseOfAStoredSessionGetsNoSecondCopy
• TestReleasingAStoredSessionLeavesTheSessionGaugeAlone: the gauge, for stored sessions released directly. TestAStoredSessionALookupReadIsReleasedAsReadBack: the gauge and the PDR, FAR, QER and BAR identifiers, for one a lookup read first.
• TestReleasingAStoredSessionLeavesItsAddressWithTheUEHoldingItNow: the stored session's address is held by another UE in the running pool. The release reaches fix(context): an allocator with no pool refuses the work instead of panicking #712's no-op and the other UE keeps the address. This pins fix(context): an allocator with no pool refuses the work instead of panicking #712 on this path; it is not a fix of this PR's.
• TestReleasingAStoredSessionLeavesAnotherSessionsSEIDAlone: a dead and a live session share a SEID; the live one's documents and index entry survive.
• TestReleasingAStoredSessionDeletesItsPFCPSessionAndFreesNoRuleIdentifiers, TestAWithdrawnRuleOfAStoredSessionLeavesItsIdentifiersToTheLiveSession: a live session holds a PDR, FAR, QER and BAR with the same identifiers as the stored one; all four stay with the live session.
• TestAStoredSessionIsDeletedOnlyAtTheIncarnationThatAcknowledgedIt: no deletion reaches the UPF when the stored session was acknowledged before the UPF's current recovery timestamp, when the document records none, when neither is known, when the document records one but the SMF has not associated with the UPF since it started, and when a heartbeat has reported the UPF restarted and it has not re-associated yet, and while the association is still being set up; the session is still released. TestTheAcknowledgingIncarnationSurvivesMarshalJSON covers the second encoder; the datastore writes through ToBsonM, which the deletion tests go through.

Each test was shown to fail when the line it guards is reverted on its own: the gauge guard, CompareAndDelete, the SEID ownership check, the identifier skips (release and policy update), the UPF == nil guards, the LoadOrStore in both read paths, the flag in GetSMContextByRefInDB, the TxnLoadCtxt call, #712's allocator guard, and commit 4's timestamp comparison, its zero check, treating an unknown held timestamp as a match, the timestamp's encoding in both encoders and its decoding, and commit 5's association check. Dropping the condition that limits commit 4 to sessions read back from the datastore fails two existing rollback tests, which expect the deletion for a session this start created. The SEID check is two guards, the read and the scoped delete, and the test fails only with both removed. The read check is redundant with the scoped delete, and no test covers the window between them. No test covers the revert path's freeing either; the not-sent path is covered.

On a lab cluster (SMF with enableDBStore, UERANSIM) I restarted the SMF three times and re-attached the UE after each restart. smf.data.smContext and seidSmContext held exactly one document for the subscriber each time, and the first cycle released the six documents left from before the fix. That ran on an earlier build of this PR and the follow-up, which did not yet include commits 2 to 5. The document count after each re-attach is what this PR is responsible for. The listing's filter matched the stored documents on that cluster's MongoDB, and explain() showed the index in use. The unit tests cannot show this, because memDB compares numbers by value. Commit 4's stored timestamp has not been exercised against MongoDB; the unit tests round-trip it through the same JSON encoding the datastore write uses.

pre-commit run --all-files is clean on macOS (darwin/arm64, Go 1.26.10), with no hook skipped. That covers gitleaks, gci, staticcheck, go test -race ./... -count=20, golangci-lint v2.13.2, yamlfmt and reuse. go build and go vet are clean too. Each commit builds and vets on its own.

Trade-offs

• Request type is not inspected, as for the in-memory replacement; gating on it is a separate change. A create carrying "existing PDU session", as in a handover between accesses with a change of AMF, goes through the same release, as it already does for a session held in memory.
• Parity with the in-memory replacement. Neither the stored session's PCF association nor its charging session is terminated, as for the in-memory replacement, which has a TODO for the PCF association. Its end-of-session event is published before the new session's start, so Kafka consumers see the delete, then the add.
• Release before create. A create that fails after the release leaves the old session gone, as the in-memory replacement does.
• Several SMF instances on one datastore. The release removes the stored sessions for the key whichever SMF wrote them, and sends their PFCP deletions. That is safe for the AMF's record: per TS 23.502 clause 4.3.2.2.1 step 2 the AMF treats a request as a new PDU session only when the PDU session ID is not used by an existing session of the UE, so no other SMF holds the AMF's association for it. Another SMF could still hold an in-memory context for it, now without documents. That is the state an in-memory replacement on that SMF would leave on this one.
• One extra indexed read on every create whose key is not in memory, with enableDBStore only. If the listing fails, the create proceeds without releasing, and stored duplicates for that key survive. A hung datastore delays the create by up to 30 s (the listing's timeout) before it proceeds. Several stored copies are released one after another before the new session is created.
• Identifiers a restored session gains later. The flag stays on a restored context, so a later ordinary release of that session also frees no identifiers. A rule this start added to it after the restart keeps its identifiers until the SMF restarts: a few identifiers lost, against taking a live session's.
• Addresses are not reconciled. A stored session's address is not returned to, or reserved in, the running pool; fix(context): an allocator with no pool refuses the work instead of panicking #712 leaves that out of scope, and so does this PR.
• The deletion's answer can reach the live session. When the stale session's local SEID equals the live session's, the UPF's Session Deletion Response for the stale one is routed by that SEID to the live session. Today this only logs, unless the live session is itself being released at that moment. Not changed here.
• Sessions stored before commit 4 record no UPF incarnation, so their release sends no deletion. That includes the sessions still in use when the SMF is upgraded to this change: a UPF that kept one across the SMF's restart also keeps it after the session is released, with its rules and any address it holds, until the UPF drops the SMF's association or restarts. The same holds for a release that runs before the SMF has associated with the UPF since it started. The cost of unrecorded documents falls once, at the upgrade, and avoids deleting another subscriber's session.
• A UPF restart the SMF has not yet noticed, up to one heartbeat interval, is not detected, so the deletion is still sent in that window.
• A UPF that is not associated, with no restart. Commit 5 sends no deletion while the SMF holds the UPF as not associated, or as still setting up the association. That also covers a UPF marked not associated because its heartbeats went unanswered, which kept its sessions and its recovery timestamp. A stored session released in that state leaves its PFCP session on the UPF until the UPF drops the SMF's association or restarts. The alternative is a deletion addressed to a SEID the UPF may have given to another subscriber, and an unanswered heartbeat usually means the deletion would not be answered either.
• An association response without a usable Recovery Time Stamp leaves the UPF marked associated under the previous incarnation's timestamp, because both association response handlers set the status before they read the timestamp. A deletion is then sent as before commit 5. Not changed here.
• The timestamp a session is stamped with on an accepted establishment is read without the association state. A restarted UPF that answered an establishment before the SMF re-associated would stamp the previous incarnation's timestamp. Establishments go out only while the UPF is associated, so the window is narrow; not changed here.
• Only the deletion is checked. A Session Modification for a restored session still carries its stored RemoteSEID. Not changed here.
• A UPF that deleted the sessions without restarting. TS 29.244 clause 6.2.6.2.2: on an Association Setup Request for a CP Node ID that already has an association, the UPF deletes that association's PFCP sessions unless the request asks for them to be retained, and the SMF does not ask. Such a UPF drops every stored session when the restarted SMF associates, its recovery timestamp is unchanged, and commit 4 still sends the deletion, to a SEID the UPF may have given to a session since. For a UPF that follows that clause this is the usual outcome of an SMF restart, not an edge case. pfcpiface does not delete sessions on re-association. It also draws each SEID at random from 64 bits, so there a reused SEID, in this case or after a UPF restart, needs a collision; the check matters for a UPF that reuses SEIDs. Closing this needs the SMF to know whether the UPF kept them, for example by sending PFCP Session Retention Information, and is not done here.
• smf.data.refToSeid is write-only today and still accumulates one entry per release. Not touched here.

Follow-up

A second PR, stacked on this one, keeps a superseded stored session from being served at all. Today any request naming it, whether an AMF update, a PCF notification, an OAM read or a PFCP message for its SEID, restores it as live.

🤖 Generated with Claude Code

[Copilot]

🔵 Needs a closer look

Cross-restart teardown needs human validation of remote-session ownership before stale PFCP deletions can safely reach a restarted UPF.

1 open issue

• Prevent restored-session cleanup from deleting a reused UPF session · New

What changed in this PR

This PR extends SMF session replacement to datastore-only sessions after a restart, preventing stale sessions from accumulating when a subscriber re-establishes the same PDU session ID.

Changes:

• Find and release stored sessions before creating their replacement, supported by a compound index.
• Protect live sessions’ SEID records, rule identifiers, and session metrics during restored-session cleanup.
• Add regression tests for stored-session replacement, concurrent lookups, and resource ownership.

File	Description
producer/​restored_session_test.go	Tests rule ownership during restored-session policy updates.
producer/​restoration.go	Registers the restored-session release handler.
producer/​pdu_session.go	Adds restored-session teardown and skips unavailable UPFs.
producer/​callback.go	Preserves rule identifiers when withdrawing restored rules.
fsm/​txnFsm.go	Triggers stored-session cleanup before creating a replacement.
context/​stored_session.go	Adds stored-session discovery and shared-instance lookup.
context/​stored_session_test.go	Tests replacement, lookup races, metrics, and resource ownership.
context/​sm_context.go	Tracks restored contexts and protects SEID entries and metrics.
context/​export_test.go	Exposes internal state helpers for external tests.
context/​db.go	Adds the lookup index, restoration flag, and ownership-scoped deletion.
context/​datapath.go	Avoids freeing restored rule identifiers and dereferencing missing UPFs.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

The timestamp-only check can permit session release with a stale RemoteSEID after a UPF restart but before re-association, potentially deleting another subscriber's session.

1 open issue

• Reject stale UPF sessions after restart · New

1 resolved since last review

• Prevent restored-session cleanup from deleting a reused UPF session

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Destructive PFCP cleanup still requires domain review and deployment-specific validation of UPF session retention across SMF re-association, while recovery-timestamp persistence through MongoDB remains unverified.

No open issues 🎉

1 resolved since last review

• Reject stale UPF sessions after restart

_{Review effort: Balanced}