Feature benchmark

[theprodsde]

What this adds

Decision Safety Suite v2 — the decision-safety track becomes an ordered op script (write / delete / delete_scope / advance_clock / query / query_as_of / inspect) instead of seed-then-query, because deletion, supersession, and injection all depend on write order.

• Six versioned batteries in benchmarks/decision_safety/: deletion_durability, ttl_expiry, poisoned_write, state_invalidation, provenance_reassertion, point_in_time.
• A runner (runner.py), an adapter contract (adapters/base.py), and our own adapter.
• Paired-metric enforcement in the result validator: a metric that rewards blocking cannot be published without the metric that punishes overdoing it. No aggregate "safety score".
• Capability gating: a battery whose requirements aren't supported is reported unsupported, never scored as zero (partial ≠ supported); an op the adapter couldn't execute is excluded from numerator and denominator, with the count published.
• Our own results across three configurations in results/REPORT.md, plus raw per-op output.

Designed in response to review feedback on mem0ai/mem0#7453. The Mem0 adapter deliberately refuses to run until its maintainers confirm the canonical self-hosted configuration, and declares no capabilities on their behalf.

Multi-tenancy — MemoryEntry.scope is a tier (user/project/…), not a tenant id, so many users in one store had no isolation primitive.

• Memory.scoped(user_id=…, session_id=…, shared=…) → hierarchical reads (session → user → shared), writes confined to the view, forget_all() per tenant or per session.
• Memory.forget_where(...) — bulk delete by scope/type/tags/metadata/predicate, including archived and expired copies; requires an explicit filter or all=True.
• Memory.resolve(where=...) applies the filter during retrieval, before scoring.

Two read-path bug fixes. Both had the same shape — the filter lived in the listing path, not the path the decision layer uses:

• MultiAgentMemory.resolve() ignored its isolation mode, so an ISOLATED agent could have another agent's memory replayed verbatim while list() correctly showed nothing.
• A memory whose TTL was shorter than the retriever's 5s result cache could be replayed after expiry on a repeated query, even though the store's read path filtered it. This is now battery ttl_expiry.

What the suite found in our own SDK

Published before any comparison, in benchmarks/decision_safety/results/REPORT.md:

• No write-time injection defence. injection_write_rate 1.00 in all three configurations, with benign_write_rate 1.00 / false_quarantine_rate 0.00 — so the low replay rates in stricter configs are abstention, not filtering.
• No supersession. At a usable gate, a superseded value comes back as current in 86% of current-state queries; invalidation_lag_writes is n/a in every config.
• point_in_time unsupported, reported as such rather than scored.
• The semantic default column scores 0.00 on every safety metric and ~0.00 on every paired recall metric — it refused 42 queries while holding candidates scoring up to 0.697 against a 0.70 gate. That column is the argument for the pairing rule.

Grounding audit

Every externally-checkable claim in the changed docs was verified, and the following were wrong and are corrected in this PR:

Claim	Reality	Where
"the archived letta-ai/letta server"	Neither letta-ai/letta nor letta-ai/letta-code is archived; both are active	docs/competitive-benchmark-rfc.md
mem0 TTL / expiry: ❌	mem0 OSS has add(..., expiration_date=…); expired memories are hidden	docs/comparison.md
mem0 local/offline: ❌ cloud-first	Self-hostable with a local vector store and local LLM	docs/comparison.md, README.md
Trap-query protection ❌ for seven systems	We have never run them; now ❔ not measured	docs/comparison.md, README.md
Mem0 adapter declared 8 capabilities	Two were wrong (ttl, tombstones); it now declares none and records only cited observations with the source line and an open question	adapters/mem0_adapter.py
"the paged buffer is in-process only"	PagedMemory persists its buffer into the same store on page-out / flush_to_recall()	docs/decision-safety-suite.md
LongMemEval latency / CPU / RSS table	Lexical figures came from a different run than the accuracy figures beside them; the semantic ones existed in no committed artifact at all, breaking the report's own "reproducible from this directory" contract	benchmarks/longmemeval/REPORT.md
make update-perf	Called scripts/update_readme_perf.py, which doesn't exist	Makefile

Verified correct and unchanged:

• 34/36 (94.4%) adversarial eval reproduces exactly, and both misses still fail safe to VERIFY rather than a wrong REPLAY.
• LongMemEval_S 98.1% semantic / 96.0% lexical Recall@5 and 87.0% on _M all match their artifacts.
• longmemeval_s_cleaned.json SHA256 matches the value published in the result artifact.
• Star counts in docs/comparison.md are accurate and dated.
• All 45 decision-safety metric cells in REPORT.md match the JSON artifacts; battery and metric names are consistent across the spec, runner, validator, schema, and artifacts; every relative doc link resolves.

LongMemEval was re-run after the retriever changes (they touch retriever.retrieve(), so "should be inert" wasn't good enough): all 45 accuracy metrics came back identical. The lexical artifact is replaced with that complete single run — same accuracy, now with p90/p99 and the execution telemetry block, so the report's table comes from one artifact per column. A full semantic re-run is in progress to do the same for the semantic column; its p90/p99 and whole-run resource cells say so instead of carrying an unverifiable number.

Checks

• ruff check agent_memory/ mcp_server/ tests/ — clean
• mypy agent_memory/ mcp_server/ — clean (enforced in CI; caught a shadowed-list annotation in scoped.py)
• pytest tests/ — 383 passed, 17 skipped
• Redis backend tests with fakeredis — 13 passed
• All decision-safety artifacts validate against benchmarks/competitive/result.schema.json

Not in this PR

No comparative result for any other system. The Mem0 adapter is a stub by design, a mnemo adapter is welcome as a PR, and no comparative table gets published before the compared project's maintainers have reviewed their adapter.

[theprodsde]

Cross-platform implementation and verification

This PR adds Windows peak working-set telemetry via GetProcessMemoryInfo(), keeps the existing macOS/Linux ru_maxrss conversions, and documents native Windows setup and run commands in the LongMemEval report.

Local checks on macOS: pytest tests/test_api.py tests/test_scoped_views.py -q passed (46 passed); run_retrieval.py --limit 1 completed and emitted all latency/CPU/RSS fields. The one-question smoke recorded Recall@5 1.0, p50 1.19 ms, wall 0.46 s, user/system CPU 0.35/0.10 s, and peak RSS 2832.75 MiB. These are a local smoke result, not a representative performance claim.

PR checks currently show 12 successful and 3 pending. CI currently runs on Ubuntu only, so this does not yet verify native Windows or macOS execution. Follow-ups opened: #42 for deterministic platform RSS unit tests and #43 for Windows/macOS CLI smoke coverage.

Issue audit: none of the existing open issues is fully completed by this PR, so none was closed. #24 has API-key authentication but not its JWT scope; #25 has SDK scoped views but not the requested tenant API routes/backend configuration/rate limits; #5 remains unimplemented and has been claimed by a contributor. The remaining open issues (#18–23, #26–27) request unrelated backend, retrieval, versioning, or streaming work.