fast_slow_store: has_with_results consults fast store and in-flight slow writes

[erneestoc]

Summary

`FastSlowStore::has_with_results` previously only consulted the slow store. This PR adds two layers of awareness so existence checks reflect what the store actually has:

1. In-flight slow writes — a new `in_flight_slow_writes: Mutex<HashMap<StoreKey, u64>>` map tracked via a cancel-safe RAII guard (`InFlightSlowWriteGuard`) registered at the start of every slow-store write path.
2. Fast-store fallback — for any keys still missing after slow + in-flight consultation.

Order: slow store first (authoritative for downstream consumers) → in-flight map → fast store.

The bugs being fixed

Two distinct correctness gaps that both surfaced as redundant work under parallel load:

1. Fast-only / pre-slow-write blobs are invisible. A blob in the fast store but not yet pushed to slow (or in a fast-only configuration) returns `NotFound`, causing callers to trigger a redundant slow-store fetch for content that's already local.

2. Concurrent writers can't see each other. Two parallel uploaders of the same digest both see `NotFound` in the slow store and both upload — wasting bandwidth and CPU, and potentially racing in the slow-store backend. Common pattern: parallel SwiftCompile actions uploading the same shared SDK module.

Cancel safety

`InFlightSlowWriteGuard` uses `Weak` (no Arc cycle) and removes its entry on `Drop`, so a cancelled `update()` future cleans up the map even if the slow-store write never completes. There is a dedicated test for this property.

Test plan

• `cargo test -p nativelink-store --test fast_slow_store_test` — 21 passed
• Empirical regression proof: reverted `fast_slow_store.rs` to its pre-fix state while keeping the new tests; all 4 fix-related tests failed (including the two cancel-safety / mixed-key gap-closures). Re-applied the fix → 21/21 pass.

New tests (all deterministic, no sleeps — concurrency via `oneshot` channels and a `GatedSlowStore`):

• `has_does_not_consult_fast_store_when_slow_store_hits` — performance: confirms slow-hit short-circuits

• `fast_store_only_value_is_reported_by_has` — fast-only fallback

• `has_checks_fast_store_when_noop` — `NoopDownloads` slow store fall-through

• `has_sees_in_flight_slow_writes` — gated slow write blocks; concurrent `has_with_results` returns Found via the in-flight map

• `dropping_update_future_cleans_up_in_flight_entry` — uses `NoopStore` fast + `ReadOnly` direction so the in-flight map is the only presence signal; proves the Drop guard removes the entry

• `has_with_results_handles_mixed_key_sources` — single call with 4 keys spread across slow/in-flight/fast-only/missing; asserts each is independently resolved

• `cargo build -p nativelink-store` clean

• `cargo clippy -p nativelink-store --lib --tests -- -D warnings` clean

• `cargo fmt --check` clean

Provenance

Equivalent to commits `f69aaf8e8` ("Fix has_with_results to check fast store, downgrade CRITICAL to warn") and `2d770d913e` ("Fix has_with_results to check in-flight slow writes") from #2243, combined into one atomic fix because they share the same observable contract.

🤖 Generated with Claude Code

---

This change is 

[MarcusSorealheis]

Wow @erneestoc this is a pretty great PR. I can help with the Nix test path as that can be quite annoying. Maybe the clankers can help.

[MarcusSorealheis]

actually that was a Nix infra flakiness instance