test: validate fold against real session corpus

Author: jstar0

docs/design.md

@@ -4,6 +4,8 @@
 
 CodexFold is an unofficial, local-first tool for measuring, deduplicating, storing, and recovering Codex session rollouts. The public project is standalone and has no private control-plane dependency.
 
+The optimization mechanisms and lifecycle boundaries are defined in [the fold taxonomy](fold-taxonomy.md). Real-corpus storage evidence is recorded separately in [the v0.3 validation](validation-real-corpus-v0.3.md).
+
 The current release capability is `storage-engine`. Transparent normal-path session access is a separate product stage defined by `docs/superpowers/specs/2026-07-11-transparent-session-filesystem-design.md`.
 
 ## Safety

docs/fold-taxonomy.md

@@ -0,0 +1,63 @@
+# CodexFold Optimization Taxonomy
+
+CodexFold has one storage objective: preserve every rollout byte while storing exact duplicate byte content once. This applies within one long session, across unrelated sessions, and across fork branches. Cross-session or cross-client boundaries are not prerequisites for reuse.
+
+## Byte-Preserving Content Reuse
+
+| Mechanism | Scope | Maximum saving | Current boundary |
+| --- | --- | --- | --- |
+| Exact raw JSON string fields | Identical large string tokens at any position | Approaches `1 - 1/N` for `N` copies, before compression | Fold V1 stores field objects explicitly |
+| Exact JSONL records | Complete identical physical records | Approaches `1 - 1/N` for `N` copies | Scan and containment measure records explicitly; Fold V1 does not yet emit an atomic record object |
+| Content-defined chunks | Repeated byte ranges at arbitrary offsets, including non-prefix fork content | Approaches `1 - 1/N` for repeated chunks | Fold V1 stores residual CDC objects explicitly |
+| Shared fork history | Any exact fields, records, or chunks shared by useful branches | Same as the underlying reuse layers | Strict byte-prefix ancestry is optional evidence, never a storage requirement |
+
+The three measurement layers overlap. Their reported duplicate bytes must never be added together. The committed fold manifest is the authoritative union representation.
+
+## Encoding And Physical Layout
+
+| Mechanism | Purpose | Maximum saving | Current boundary |
+| --- | --- | --- | --- |
+| Zstandard object encoding | Compress each unique object | Data-dependent; near zero on incompressible bytes and near 100% on highly repetitive bytes | Exact raw SHA-256 is verified after decompression |
+| Immutable packfiles and index | Replace hundreds of thousands of loose files and support efficient range reads | Removes filesystem block/inode overhead; it is not logical deduplication | Pack doctor verifies every object and complete packed manifest reconstruction |
+| Base plus append delta | Keep an active session writable without materializing the full folded base | Stores only the new tail until compaction | Append is durable; truncate and random write transition to copy-on-write |
+| Generation compaction and GC | Fold stable deltas and remove unreferenced objects, obsolete generations, and abandoned temporary files | Limited to proven unreferenced physical bytes | Never removes a referenced object or current recoverable generation |
+
+## Session Lifecycle Operations
+
+These operations are not compression algorithms and must be reported separately.
+
+| Operation | Meaning | Physical saving |
+| --- | --- | --- |
+| Archive a branch | Move an explicitly selected useful-but-inactive session out of the active list while preserving recovery | Zero by itself |
+| Keep both fork branches | Share exact objects while each branch retains its own independent writable tail | Duplicate content is stored once; both histories remain openable |
+| Remove an exactly contained branch | Delete an already archived session only after its applicable record sequence is directly proven inside a retained session and a recovery unfold succeeds | Can approach the complete contained source size when all objects are already shared |
+| Retire native and loose fallback copies | Remove full native sources, migration snapshots, or loose objects only after platform and per-session recovery gates pass | This is the step that turns logical reuse into actual disk reclamation |
+
+Fork ancestry, age, title, size, or similarity never proves that a branch is useless or contained.
+
+## Content-Changing Cleanup
+
+Prompt removal, repeated-message cleanup, repair, reconciliation, summarization, and semantic compaction can reduce a rollout further, but they change bytes. They are separate explicit workflows with separate outputs and recovery rules. Fold, pack, migration, compaction, enrollment, and GC never run them implicitly.
+
+## Transparent Access
+
+The transparent filesystem is the delivery mechanism, not another compression layer. Its contract is:
+
+- unmodified Codex Desktop and CLI open normal JSONL paths without manual materialization;
+- a managed file exposes byte-identical reads and the file operations Codex actually uses;
+- active sessions append to durable deltas and forks remain independently writable;
+- existing and newly stable sessions can be enrolled automatically under policy;
+- compatibility quarantine and rollback preserve a verified native current copy.
+
+## Physical-Saving Claim
+
+CodexFold reports these numbers separately:
+
+1. Logical rollout bytes.
+2. Unique raw object bytes after deduplication.
+3. Compressed pack bytes and metadata.
+4. Retained native sources and fallbacks.
+5. Loose objects, old generations, recovery artifacts, and temporary files.
+6. Projected reclaimable bytes and actual reclaimed bytes.
+
+No physical-saving claim is valid while equivalent full native or loose copies still occupy disk.

docs/validation-real-corpus-v0.3.md

@@ -0,0 +1,51 @@
+# Real Corpus Fold Validation
+
+## Scope
+
+This validation used local, read-only Codex rollout sources and an isolated store. Reports contain aggregate counts only; no session content, identifiers, credentials, titles, or workspace paths are included.
+
+The representative set contained 58 sessions and 16.01 GB of source bytes. It included large active sessions, large archived sessions, multiple workspaces, and every session participating in the 23 recorded fork edges. One actively changing source was correctly rejected before manifest commit. The remaining 57 sessions contributed 15.47 GB to the fold store.
+
+## Duplicate Measurements
+
+| Layer | Observations | Unique objects | Duplicate bytes |
+| --- | ---: | ---: | ---: |
+| Exact large fields | 475,835 | 174,643 | 7,534,867,486 |
+| Exact complete records | 4,876,474 | 4,707,166 | 297,477,994 |
+| Content-defined chunks | 628,948 | 398,775 | 4,567,543,137 |
+
+These layers overlap and are not additive. The scan processed 16,007,421,613 bytes in 302.9 seconds with a 1.06 GB disposable disk index. It counted 153 JSON records that could not participate in field parsing; their bytes still participated in the exact-record and CDC layers.
+
+All 23 recorded fork parent/child pairs had explicit ancestry but no strict shared record prefix, shared exact record, or exact containment. This confirms that fork graph edges and strict prefixes cannot serve as the storage model. A separate linear digest pass over all 1,881 registered sessions found no meaningful pair with an identical applicable record stream; the only duplicate digest represented sessions with no records after `session_meta`.
+
+## Fold, Pack, And Recovery
+
+| Metric | Result |
+| --- | ---: |
+| Folded source bytes | 15,472,621,697 |
+| Manifest references | 992,953 |
+| Unique referenced objects | 644,289 |
+| Unique raw object bytes | 7,722,034,851 |
+| Packed data bytes | 3,939,856,668 |
+| Pack physical bytes, including index | 4,136,652,800 |
+| Manifest physical bytes | 225,607,680 |
+
+Raw deduplication reduced the logical corpus by 50.09%. Zstandard pack encoding reduced unique raw object bytes by a further 48.98%. If native sources and loose objects are safely retired, the pack plus manifest representation is 28.19% of the folded source bytes, a projected 71.81% physical reduction for this corpus.
+
+The best order-dependent incremental session added 2.45% of its source bytes to the existing object store; the worst added 88.59%; the median added 25.32%. These figures depend on corpus order and are not standalone compression ratios.
+
+Verification completed with:
+
+- 57 of 57 manifests reconstructed from loose objects with exact size and SHA-256.
+- 644,289 of 644,289 packed objects verified.
+- 57 of 57 complete manifests reconstructed through the pack-only reader with exact size and SHA-256.
+- 57 of 57 explicit `unfold` operations independently checked by size and SHA-256.
+- Zero retained unfold output files after the batch.
+
+One rejected active fold left 38,604 unreferenced loose objects. GC identified and removed all of them, reclaiming 249,114,624 physical bytes while preserving every referenced object.
+
+## Open Gates
+
+- Fold V1 does not represent a complete repeated JSONL record as one atomic object. Exact record duplication is measured, and its bytes can be reused through field and CDC objects, but an atomic record layer requires a compatible hierarchical or precomputed Fold V2 representation rather than weakening field/chunk reuse.
+- Packed and native-source retirement are not yet an automatic production policy. Until retirement is implemented and its gates pass, aggregate logical savings do not equal current disk reclamation.
+- Real-scale pack construction peaked at approximately 1.6 GB RSS and pack doctor at approximately 624 MB RSS. These maintenance operations need bounded-memory index construction before bulk production use.

internal/cli/fs_test.go

@@ -587,7 +587,7 @@ func TestPackBuildAndDoctorCommandsUseFoldStore(t *testing.T) {
 		t.Fatalf("pack doctor: %v", err)
 	}
 	var doctor pack.DoctorResult
-	if err := json.Unmarshal(output.Bytes(), &doctor); err != nil || doctor.IssueCount != 0 {
+	if err := json.Unmarshal(output.Bytes(), &doctor); err != nil || doctor.IssueCount != 0 || doctor.ManifestCount != 1 || doctor.VerifiedManifestCount != 1 {
 		t.Fatalf("unexpected pack doctor: %#v err=%v output=%s", doctor, err, output.String())
 	}
 }

internal/cli/pack.go

@@ -54,7 +54,7 @@ func newPackDoctorCommand() *cobra.Command {
 	var jsonOutput bool
 	command := &cobra.Command{
 		Use:   "doctor",
-		Short: "Verify the active packed object generation",
+		Short: "Verify packed objects and complete manifest reconstruction",
 		Args:  cobra.NoArgs,
 		RunE: func(command *cobra.Command, _ []string) error {
 			home, err := codex.ResolveHome(codexHome)
@@ -68,7 +68,7 @@ func newPackDoctorCommand() *cobra.Command {
 			if jsonOutput {
 				return writeJSON(command, result)
 			}
-			_, err = fmt.Fprintf(command.OutOrStdout(), "generation=%s objects=%d verified=%d issues=%d\n", result.Generation, result.ObjectCount, result.VerifiedCount, result.IssueCount)
+			_, err = fmt.Fprintf(command.OutOrStdout(), "generation=%s objects=%d verified=%d manifests=%d verified_manifests=%d issues=%d\n", result.Generation, result.ObjectCount, result.VerifiedCount, result.ManifestCount, result.VerifiedManifestCount, result.IssueCount)
 			return err
 		},
 	}

internal/fold/doctor.go

@@ -37,30 +37,33 @@ type loadedManifest struct {
 
 func Doctor(ctx context.Context, storeDir string) (DoctorResult, error) {
 	result := DoctorResult{StoreDir: storeDir, Issues: make([]DoctorIssue, 0)}
-	manifests, loadIssues, err := loadAllManifests(storeDir)
-	if err != nil {
-		return DoctorResult{}, err
-	}
-	result.ManifestCount = len(manifests) + len(loadIssues)
-	result.Issues = append(result.Issues, loadIssues...)
 	store := NewObjectStore(storeDir)
 	unique := make(map[string]ObjectRef)
-	for _, loaded := range manifests {
-		manifest := loaded.Manifest
+	err := walkManifestPaths(storeDir, func(path string) error {
 		if err := ctx.Err(); err != nil {
-			return DoctorResult{}, err
+			return err
+		}
+		result.ManifestCount++
+		manifest, err := LoadManifestPath(path)
+		if err != nil {
+			result.Issues = append(result.Issues, DoctorIssue{Scope: "manifest", Path: path, Error: err.Error()})
+			return nil
 		}
 		for _, part := range manifest.Parts {
 			result.ObjectReferenceCount++
 			unique[part.Object.SHA256] = part.Object
 		}
 		if err := verifyStoredManifest(ctx, store, manifest); err != nil {
 			result.Issues = append(result.Issues, DoctorIssue{
-				Scope: "manifest", Path: loaded.Path, Error: err.Error(),
+				Scope: "manifest", Path: path, Error: err.Error(),
 			})
 		} else {
 			result.VerifiedManifestCount++
 		}
+		return nil
+	})
+	if err != nil {
+		return DoctorResult{}, err
 	}
 	result.UniqueObjectCount = len(unique)
 	for digest, ref := range unique {
@@ -93,6 +96,26 @@ func Doctor(ctx context.Context, storeDir string) (DoctorResult, error) {
 	return result, nil
 }
 
+func walkManifestPaths(storeDir string, visit func(path string) error) error {
+	manifestDir := filepath.Join(storeDir, "manifests")
+	err := filepath.WalkDir(manifestDir, func(path string, entry os.DirEntry, walkErr error) error {
+		if walkErr != nil {
+			return walkErr
+		}
+		if entry.IsDir() || filepath.Ext(entry.Name()) != ".json" {
+			return nil
+		}
+		return visit(path)
+	})
+	if errors.Is(err, os.ErrNotExist) {
+		return nil
+	}
+	if err != nil {
+		return fmt.Errorf("read manifest directory: %w", err)
+	}
+	return nil
+}
+
 func loadAllManifests(storeDir string) ([]loadedManifest, []DoctorIssue, error) {
 	manifestDir := filepath.Join(storeDir, "manifests")
 	manifests := make([]loadedManifest, 0)

internal/fold/gc.go

@@ -25,12 +25,12 @@ type GCResult struct {
 
 func GC(ctx context.Context, storeDir string, apply bool) (GCResult, error) {
 	result := GCResult{StoreDir: storeDir, DryRun: !apply}
-	_, issues, err := loadAllManifests(storeDir)
+	_, invalidManifests, err := referencedManifestObjects(ctx, storeDir, false)
 	if err != nil {
 		return GCResult{}, err
 	}
-	if len(issues) > 0 {
-		return GCResult{}, fmt.Errorf("refusing GC with %d invalid manifest(s)", len(issues))
+	if invalidManifests > 0 {
+		return GCResult{}, fmt.Errorf("refusing GC with %d invalid manifest(s)", invalidManifests)
 	}
 	before, err := storage.Scan(ctx, storage.Options{StoreDir: storeDir, AllowMetadataIssues: true})
 	if err != nil {
@@ -42,18 +42,12 @@ func GC(ctx context.Context, storeDir string, apply bool) (GCResult, error) {
 	}
 	result.Storage = storageResult
 	result.ProjectedReclaimableBytes = storageResult.ProjectedReclaimableBytes
-	manifests, issues, err := loadAllManifests(storeDir)
+	referenced, invalidManifests, err := referencedManifestObjects(ctx, storeDir, true)
 	if err != nil {
 		return GCResult{}, err
 	}
-	if len(issues) > 0 {
-		return GCResult{}, fmt.Errorf("refusing loose-object GC with %d invalid manifest(s)", len(issues))
-	}
-	referenced := make(map[string]struct{})
-	for _, loaded := range manifests {
-		for _, part := range loaded.Manifest.Parts {
-			referenced[part.Object.SHA256] = struct{}{}
-		}
+	if invalidManifests > 0 {
+		return GCResult{}, fmt.Errorf("refusing loose-object GC with %d invalid manifest(s)", invalidManifests)
 	}
 	result.Referenced = len(referenced)
 	err = walkObjectFiles(storeDir, func(path string, info os.FileInfo) error {
@@ -88,3 +82,25 @@ func GC(ctx context.Context, storeDir string, apply bool) (GCResult, error) {
 	}
 	return result, nil
 }
+
+func referencedManifestObjects(ctx context.Context, storeDir string, collect bool) (map[string]struct{}, int, error) {
+	referenced := make(map[string]struct{})
+	invalid := 0
+	err := walkManifestPaths(storeDir, func(path string) error {
+		if err := ctx.Err(); err != nil {
+			return err
+		}
+		manifest, err := LoadManifestPath(path)
+		if err != nil {
+			invalid++
+			return nil
+		}
+		if collect {
+			for _, part := range manifest.Parts {
+				referenced[part.Object.SHA256] = struct{}{}
+			}
+		}
+		return nil
+	})
+	return referenced, invalid, err
+}

internal/jsonraw/spans.go

@@ -3,12 +3,15 @@ package jsonraw
 import (
 	"bytes"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"io"
 	"strconv"
 	"strings"
 )
 
+var ErrMultipleValues = errors.New("multiple JSON values in one record")
+
 type StringSpan struct {
 	Path  string
 	Start int
@@ -24,7 +27,7 @@ func FindStringSpans(data []byte, minRawBytes int64) ([]StringSpan, error) {
 	}
 	if _, err := decoder.Token(); err != io.EOF {
 		if err == nil {
-			return nil, fmt.Errorf("multiple JSON values in one record")
+			return nil, ErrMultipleValues
 		}
 		return nil, err
 	}

internal/jsonraw/spans_test.go

@@ -1,6 +1,16 @@
 package jsonraw
 
-import "testing"
+import (
+	"errors"
+	"testing"
+)
+
+func TestFindStringSpansRejectsMultipleJSONValues(t *testing.T) {
+	_, err := FindStringSpans([]byte(`{"a":"one"}{"b":"two"}`), 1)
+	if !errors.Is(err, ErrMultipleValues) {
+		t.Fatalf("error = %v, want ErrMultipleValues", err)
+	}
+}
 
 func TestFindStringSpansReturnsExactNestedRawToken(t *testing.T) {
 	line := []byte(`{"payload":{"items":[{"text":"small"},{"text":"large\\nvalue"}]}}`)