initial commit for whitespace code

Author: ndbroadbent

AGENTS.md

@@ -4,7 +4,7 @@ The ONLY reason you should stop is if you have a blocking question for the user.
 
 ---
 
-# TrimTrailingWhitespace - AI Agent Guide
+# Whitespace - AI Agent Guide
 
 Smart case-aware search and replace across code and files with atomic apply and undo.
 
@@ -22,7 +22,7 @@ They should be mentioned in any package files, copyright notices, etc.
 
 ## Deliverables
 
-- Rust core library and CLI (`trim-trailing-whitespace-core`, `trim-trailing-whitespace-cli`)
+- Rust core library and CLI (`whitespace-core`, `whitespace-cli`)
 - VS Code extension (TypeScript) that shells out to the CLI
 - MCP server (TypeScript) that wraps the CLI and exposes tools for Cursor or other MCP clients
 
@@ -45,16 +45,16 @@ They should be mentioned in any package files, copyright notices, etc.
 - JSON for plan and history
 - TypeScript for VS Code extension and MCP wrapper
   - `ts-rs` for automatic TypeScript type generation from Rust structs
-  - Types are generated as ambient `.d.ts` files in `trim-trailing-whitespace-core/bindings/`
+  - Types are generated as ambient `.d.ts` files in `whitespace-core/bindings/`
   - VS Code and MCP projects include these bindings directly in their tsconfig
 - Optional Node bindings in future via `napi-rs` if needed
 
 ## Functional scope
 
 - Case styles: snake_case, kebab-case, camelCase, PascalCase, SCREAMING_SNAKE_CASE, Title Case, Train-Case, dot.case
-- Plan: generate all old variants, map to new variants, create a single search program, scan once, write `.trim-trailing-whitespace/plan.json`
+- Plan: generate all old variants, map to new variants, create a single search program, scan once, write `.whitespace/plan.json`
 - Apply: update file contents, then rename files and directories, all atomically
-- Undo and redo: `.trim-trailing-whitespace/history.json` with checksums
+- Undo and redo: `.whitespace/history.json` with checksums
 - Conflicts: re-validate hunks, auto-resolve simple formatting shifts, stop on real conflicts unless forced
 - Respect ignore files by default (`.gitignore`, `.ignore`, `.rgignore`, `.rnignore`), allow include and exclude globs
 - Exclude binary files by default
@@ -68,10 +68,10 @@ They should be mentioned in any package files, copyright notices, etc.
 
 ## Repo layout
 
-- `trim-trailing-whitespace-core` - core logic
-- `trim-trailing-whitespace-cli` - CLI frontend
-- `trim-trailing-whitespace-mcp` - MCP server
-- `trim-trailing-whitespace-vscode` - VS Code extension
+- `whitespace-core` - core logic
+- `whitespace-cli` - CLI frontend
+- `whitespace-mcp` - MCP server
+- `whitespace-vscode` - VS Code extension
 - `docs` - Starlight documentation
 
 ## Agent roles and behavior
@@ -97,27 +97,27 @@ They should be mentioned in any package files, copyright notices, etc.
 - Snapshot tests for plans and diffs
 - Fuzz tests for regex generation to prevent backtracking issues
 - Cross platform tests including Windows path edge cases
-- Lock signal integration tests rely on the `wait_for_lock_state` polling helper (see `trim-trailing-whitespace-cli/src/test_lock_signals.rs`); avoid reintroducing fixed sleeps that cause macOS flakes
+- Lock signal integration tests rely on the `wait_for_lock_state` polling helper (see `whitespace-cli/src/test_lock_signals.rs`); avoid reintroducing fixed sleeps that cause macOS flakes
 
 ## CLI contract
 
-Binary: `trim-trailing-whitespace`
+Binary: `whitespace`
 
 Commands:
 
-- `trim-trailing-whitespace plan <old> <new> [opts]`
+- `whitespace plan <old> <new> [opts]`
   - `--include` `--exclude` `--respect-gitignore` (default true, respects all ignore files)
   - `--rename-files` `--rename-dirs` (default true)
   - `--styles=<list>`
   - `--preview table|diff|matches|summary|none` (human-readable preview)
   - `--output summary|json` (machine-readable output)
   - `--plan-out`
   - `-u/-uu/-uuu` (unrestricted levels to control ignore file handling)
-- `trim-trailing-whitespace apply [--plan PATH | --id ID] [--atomic true] [--commit]`
-- `trim-trailing-whitespace undo <id>`
-- `trim-trailing-whitespace redo <id>`
-- `trim-trailing-whitespace history [--limit N]`
-- `trim-trailing-whitespace status`
+- `whitespace apply [--plan PATH | --id ID] [--atomic true] [--commit]`
+- `whitespace undo <id>`
+- `whitespace redo <id>`
+- `whitespace history [--limit N]`
+- `whitespace status`
 
 Exit codes:
 
@@ -128,15 +128,15 @@ Exit codes:
 
 ## Data formats
 
-`.trim-trailing-whitespace/plan.json`
+`.whitespace/plan.json`
 
 - `{ id, created_at, old, new, styles[], includes[], excludes[], matches[], renames[], stats, version }`
 
-`.trim-trailing-whitespace/history.json`
+`.whitespace/history.json`
 
 - append only with checksums and revert info
 
-`.trim-trailing-whitespace/config.toml`
+`.whitespace/config.toml`
 
 - Project configuration including atomic identifiers
 - `atomic = ["DocSpring", "GitHub", "GitHub"]` - identifiers treated as indivisible units
@@ -151,7 +151,7 @@ Exit codes:
 6. For file and directory names, detect and schedule renames with depth ordering
 7. Emit `plan.json` and fast summary stats
 
-- `--ignore-ambiguous` now prunes ambiguous identifiers (plain words that map to multiple styles) before matching or renaming; make sure new search variants respect this toggle and keep tests in `trim-trailing-whitespace-core/tests/ignore_ambiguous_test.rs` passing.
+- `--ignore-ambiguous` now prunes ambiguous identifiers (plain words that map to multiple styles) before matching or renaming; make sure new search variants respect this toggle and keep tests in `whitespace-core/tests/ignore_ambiguous_test.rs` passing.
 
 - Implementation note: `scan_repository_multi` pre-filters candidate files with an `AhoCorasick` automaton, processes them in parallel via `rayon`, and only runs the expensive compound identifier scan on lines discovered by direct variant hits or token heuristics. When adjusting matching logic, keep the `token_line_hits` bookkeeping in sync with the `additional_lines` fed into `find_enhanced_matches`.
 
@@ -163,12 +163,12 @@ Boundary rules
 
 ## Ignore file handling
 
-TrimTrailingWhitespace respects multiple ignore file formats:
+Whitespace respects multiple ignore file formats:
 
 - `.gitignore` - Standard Git ignore patterns
 - `.ignore` - Generic ignore file (compatible with ripgrep)
 - `.rgignore` - Ripgrep-specific ignore patterns
-- `.rnignore` - TrimTrailingWhitespace-specific ignore patterns (useful for excluding files from renaming without affecting Git)
+- `.rnignore` - Whitespace-specific ignore patterns (useful for excluding files from renaming without affecting Git)
 
 The unrestricted levels (`-u` flag) control ignore behavior:
 
@@ -195,7 +195,7 @@ The unrestricted levels (`-u` flag) control ignore behavior:
 
 - Node TypeScript wrapper around the CLI
 - Tools: `plan`, `apply`, `undo`, `history`, `preview`
-- Installed via `npx`, expects `trim-trailing-whitespace` on PATH
+- Installed via `npx`, expects `whitespace` on PATH
 - Codex CLI uses `~/.codex/config.toml`; ensure docs include the TOML MCP example
 
 ## Coding standards
@@ -226,7 +226,7 @@ The unrestricted levels (`-u` flag) control ignore behavior:
 ## Package Manager Requirements
 
 - **ALWAYS use pnpm, NEVER npm** - All JavaScript/TypeScript projects in this repo use pnpm
-- This applies to all subdirectories: trim-trailing-whitespace-mcp, trim-trailing-whitespace-vscode, trim-trailing-whitespace-core, docs
+- This applies to all subdirectories: whitespace-mcp, whitespace-vscode, whitespace-core, docs
 - When showing commands, always use `pnpm` not `npm`
 - Examples: `pnpm install`, `pnpm test`, `pnpm build`
 
@@ -265,24 +265,24 @@ If the user insists after your pushback, then proceed, but always voice concerns
 
 ## Testing Guidelines
 
-- **ALWAYS use the --dry-run flag when testing the trim-trailing-whitespace CLI** to avoid creating unwanted plan files and modifications
-- When running test commands with trim-trailing-whitespace, use: `./target/debug/trim-trailing-whitespace plan ... --dry-run`
-- This prevents the creation of `.trim-trailing-whitespace/plan.json` files during testing
+- **ALWAYS use the --dry-run flag when testing the whitespace CLI** to avoid creating unwanted plan files and modifications
+- When running test commands with whitespace, use: `./target/debug/whitespace plan ... --dry-run`
+- This prevents the creation of `.whitespace/plan.json` files during testing
 
 ### CI Self-Hosting Testing
 
-- **CRITICAL: Never use "trim-trailing-whitespace" or project name patterns in test content**
-- Our CI includes e2e tests that rename the entire project (trim-trailing-whitespace → <alternative-protected-string> → trim-trailing-whitespace)
+- **CRITICAL: Never use "whitespace" or project name patterns in test content**
+- Our CI includes e2e tests that rename the entire project (whitespace → <alternative-protected-string> → whitespace)
 - Any test content containing the project name will be modified during CI, potentially breaking tests if you are not careful.
-- **Mostly use generic names like "testword", "module", "config" instead of "trim-trailing-whitespace"**
+- **Mostly use generic names like "testword", "module", "config" instead of "whitespace"**
 - **Use "renamed_renaming_tool" NOT the alternative protected string in tests**
 - The alternative protected string is only allowed in files matching `.rnignore` entries:
   - `.github/workflows/`
   - `docs/src/content/docs/index.mdx`
   - `docs/src/assets/case-studies/`
   - `docs/src/content/docs/case-studies/`
 - All other test files should use "renamed_renaming_tool" as the target replacement string
-- This prevents CI failures when trim-trailing-whitespace tests itself and ensures clean self-hosting testing
+- This prevents CI failures when whitespace tests itself and ensures clean self-hosting testing
 
 ## DO NOT REDIRECT STDERR
 

BLOG_POST.md

@@ -0,0 +1,63 @@
+# Blog Post Outline: How I Built the World’s Fastest Whitespace Trimmer
+
+## Working Title
+- How I Built the World’s Fastest Whitespace Trimmer
+
+## Hook & Motivation
+- lefthook-driven workflow; editor auto-trim usually good enough.
+- Recent surge of generated/AI-written code sneaking in trailing spaces and missing final newlines.
+- Existing solutions (bash script, Python pre-commit hook, npm package, other Rust tool) would work but feel unsatisfying.
+- Reading Bun’s performance blog post rekindled the idea: build something absurdly fast for fun and for the blog.
+
+## Background & Context
+- DocSpring focus: useful open source tools/blog posts for developers; no interest in marketing fluff.
+- Personal curiosity about algorithms/performance despite limited formal CS background.
+- Renamify boilerplate: copied repo to `trim-trailing-whitespace`, used Renamify to rewrite identifiers.
+- Initial cleanup: delete MCP server, VS Code extension, legacy docs, all original Rust code; keep Taskfiles, lefthook, CI scaffolding.
+
+## Early Ideas & Questions
+- Bloom filter cache: auto-sized, tuned on DocSpring repo; handle false positives via resizes.
+- Where to store cache? Options: temp dir, `.git`, OS cache dirs.
+- Compiling `.gitignore`: convert patterns into compact machine-friendly representation; rebuild on mtime changes/additions.
+- Per-file delta detection: subdivide files into segments; can we avoid reading every byte each run?
+- Quick research (ChatGPT): deleting arbitrary bytes without rewriting is mostly impossible; file rewrite is still best.
+
+## Defining the Scope
+- Deliverables: Rust core crate + CLI only, whitespace transformations only.
+- Respect Git’s ignore configuration (`.gitignore`, `.git/info/exclude`, global excludes).
+- No IDE/MCP integrations; project is a one-off experiment tied to the blog post.
+- Performance-first mindset: compile-time feature flags to toggle optimisations, benchmarking pipeline to show improvements.
+
+## Architecture Plan Highlights
+- `whitespace-core`: transcode/trim logic, line ending normalization, tab/space conversion, SIMD implementations.
+- `whitespace-cli`: Clap-based binary, recursive walker, binary detection, `--check` mode, feature toggles.
+- Cache design:
+  - Cache paths per OS; repo ID via blake3-128 of canonical root + volume info.
+  - Files: `keys.bin` (sorted u128 keys), `meta.bin` (header, OS journal checkpoints, repo stats), `lock` guard.
+  - Key formula: blake3-128(path, dev, ino, size, mtime_ns, ctime_ns, mode).
+  - Racy guard: rescan when `mtime_sec` == `index_write_time_sec`.
+  - Change detection: macOS FSEvents, Windows USN Journal, Linux fallback walk.
+
+## Implementation Strategy
+- Build naive baseline implementation first (byte-by-byte, no cache, always rewrite).
+- Add SIMD and other optimisations behind cargo features.
+- Introduce compile-time feature matrix (baseline → simd → parallel walk → cache → mmap) for benchmarks.
+- Use hyperfine to benchmark each tier on `/Users/ndbroadbent/code/docspring` (cold vs warm cache).
+- Document results with charts comparing stages (baseline vs optimised).
+
+## Content Sections (Rough)
+1. Introduction + why the tool exists.
+2. Scaffolding and cleaning the repo.
+3. Exploring performance ideas in the car-side brain dump (Bloom filters, gitignore compilation, cache placement).
+4. Lessons from talking to AI (file rewrite reality, SIMD suggestions, other micro-optimisations).
+5. Designing the definitive cache (keys, meta, OS journals, racy guard).
+6. Naive baseline implementation and first measurements.
+7. Iterative optimisations with feature flags + benchmark plots.
+8. Integrating with lefthook and everyday workflow.
+9. Final thoughts (was it worth it? fun factor? what’s next optional, but keep minimal until work is done).
+
+## Notes & TODOs
+- Insert references/links: Bun performance blog, API client post, lefthook, DocSpring.
+- Collect benchmark data with hyperfine once implementations land.
+- Generate diagrams for cache structure (`keys.bin`, `meta.bin`).
+- Keep tone conversational, emphasise experimentation over production dogma.

Cargo.toml

@@ -1,14 +1,14 @@
 [workspace]
-members = ["trim-trailing-whitespace-core", "trim-trailing-whitespace-cli"]
+members = ["whitespace-core", "whitespace-cli"]
 resolver = "2"
 
 [workspace.package]
 version = "0.4.1"
 edition = "2021"
 authors = ["DocSpring"]
 license = "MIT"
-repository = "https://github.com/DocSpring/trim-trailing-whitespace"
-homepage = "https://github.com/DocSpring/trim-trailing-whitespace"
+repository = "https://github.com/DocSpring/whitespace"
+homepage = "https://github.com/DocSpring/whitespace"
 
 [workspace.dependencies]
 # Core dependencies
@@ -31,6 +31,7 @@ nu-ansi-term = "0.50"
 tempfile = "3"
 sha2 = "0.10"
 clap = { version = "4.5", features = ["derive", "cargo", "env"] }
+once_cell = "1.19"
 
 # Dev dependencies
 proptest = "1.4"