Skip to content

Coverage disclosure for the external audit behind the recent issues — what was NOT checked, and the specific reason for each #3399

Open

Description

@HetCreep

Over the last day I filed a number of issues and advisories from an external audit of v0.44.2 (700bdde). Several of them carry a "not checked" note. This issue collects all of those in one place, with the reason for each, so the gaps are legible rather than scattered across a dozen threads.

The reason for posting this: an audit that reports only what it found invites the reading that everything else was examined and was fine. That is not true here. Below is what I could not examine, and why. "Not checked" means no evidence either way — not "clean".

Provenance: AI-assisted source audit, human-directed. The auditing environment was a single Windows 11 machine with Rust 1.97.1, Node 24, .NET 10 and Git. Nothing else in the list below was installed, for reasons given.

A. Not checked because the tool was not installed — and installing it would not have helped

For these, rtk's filter parses the tool's real output. I used fixtures and shims, which prove the parser's behaviour on the input I supplied but say nothing about what the tool actually emits today. Installing the binary alone would not have closed the gap, because each of these needs credentials, a cluster, a daemon, or a database that I do not have and should not have:

tool why the binary alone proves nothing
aws needs live credentials + a real account; output shape varies by service and CLI version
kubectl needs a cluster
docker needs a daemon and images
psql / mysql needs a running database
gcloud, az needs an authenticated cloud project
sops, vault, age needs real key material — I was not going to generate live secrets to test a filter
terraform, ansible needs real infrastructure state to produce representative output
mvn, gradle, sbt needs a JVM toolchain and real multi-module projects; fixture output was used instead
black, prettier, ruff, eslint not installed; their filters are source-read only
gh, glab, gt gh was installed, but exercising the filters needs live API responses from a repo with the right state (failing checks, truncated lists)

Concretely, this means: every claim I made about an aws, kubectl, psql, docker, terraform, ansible, or sops filter is a claim about rtk's parsing code, not about how that filter behaves against the real tool. Where I said a filter drops a field or mis-parses a format, I verified it in rtk's source and against a fixture I wrote. A maintainer with the real tool can confirm or refute in one run.

B. Not checked because I was on the wrong OS

The audit machine is Windows 11. Not examined:

  • #[cfg(unix)] code paths — including the file-permission hardening (set_owner_only is a no-op on non-unix), signal handling, SIGPIPE behaviour, process-group semantics, and orphan reparenting. Several findings I filed reason about Unix behaviour from source; none of it was observed.
  • macOS specifics — the Apple Silicon startup-overhead numbers in NF01: rtk-rewrite.sh hook adds ~56ms overhead on Apple Silicon (vs documented <5ms) #1375 are someone else's; my own timings are Windows, where the bare process-spawn floor is ~35 ms, so absolute values are not comparable.
  • Linux release artifacts — the musl/gnu split, the .deb/.rpm packages, and the install script's behaviour were read, not run.

C. Not checked because it needs another vendor's product

rtk integrates with several agent harnesses by writing hook configs and emitting JSON they consume. I could not verify the receiving end for:

  • GitHub Copilot CLI — whether it honours permissionDecision as rtk supplies it. (Your own rtk hook copilot emits permissionDecision "ask", forcing a permission prompt on every command in Copilot CLI >= 1.0.66 #3037 answered part of this; I have no independent view.)
  • Gemini CLI, Cursor, Cline/Roo, Windsurf, OpenCode, Droid, JetBrains Copilot plugin — the schemas in hook_cmd.rs are documented in comments and I read them, but I never ran any of these products. If a vendor changed their hook schema, I would not know.
  • Claude Code itself — I exercised rtk hook claude directly by piping JSON, which tests rtk's side. I did not run it inside a real Claude Code session.

D. Not checked because it lives outside the repository

  • Branch protection and required status checks. Issue Two CI steps cannot fail, and no lint guards &str byte-slicing — the two gaps behind several open panic/false-summary issues #3394 argues two CI steps cannot fail. Whether those jobs are even required for a merge is a GitHub repository setting, not a file in the tree — I cannot see it. If they are not required, the finding is larger than I stated; if some other gate compensates, smaller.
  • The telemetry server. Retention, deletion, IP handling, and what the /erasure endpoint actually does are all server-side. Advisory GHSA-q584 reports what the client sends; I make no claim about what happens after.
  • crates.io / release-artifact provenance. I did not download and inspect your published binaries (including not running strings on them to look for a compiled-in token — that felt inappropriate for a report you had not yet triaged).

E. Not checked because I chose not to run it

Stated so it is not mistaken for an oversight:

  • rtk init and its uninstall paths — these write into the real home directory. I read the code and never executed it.
  • rtk cc-economics — it invokes npx --yes and reads real spend data.
  • rtk telemetry forget — it deletes the real local database and fires a network request. GHSA-q584 is therefore a code-property report, not an executed one.
  • Anything that would have required generating live credentials to make a realistic test case.

F. Read but not audited in depth

Large surfaces I opened enough to navigate, but did not review line by line:

  • src/hooks/init.rs (~280 KB) — grepped, not read in full.
  • src/discover/registry.rs (~155 KB) — the 178 rewrite rules were counted and the RegexSet path traced; the individual rules were not each validated.
  • src/filters/*.toml — 63 declarative filter definitions. At the time I filed the sops advisory I had opened exactly one of them.
  • src/openclaw/, src/cmds/php/, most of src/cmds/js/ and src/cmds/dotnet/ beyond the specific files cited in my reports.
  • The test suite: I ran it (2520 tests, all passing) but only examined tests adjacent to findings.

What I would ask

If any finding of mine matters to you, the fastest way to settle it is usually the one thing I could not do: run it against the real tool. Where that is the case I said so in the report. I would rather you close something as "works fine against real kubectl" than have it sit open on my say-so.

And if you would find it useful, I am happy to re-check anything on this list against a fixture you consider representative — a captured output file is enough for most of category A.


Per-report ledger: what is still unverified in each thing I filed

The categories above are the reasons. This is the mapping — for every post I made, what specifically remains unverified in it. Read this as: if you are triaging one of these, here is the part you should not take on my word.

Legend: ✅ RAN = I executed it and observed the result · ⚠️ PARTIAL = the mechanism was executed but one step was not · 📖 SOURCE = read from code only, never executed.

# what I filed status what remains unverified in it
#2281 go vet can report "No issues found" on a broken module 📖 SOURCE I never ran go vet against a module with a go.mod error. The claim is from go_cmd.rs:117,703,715 plus the f69ad6e commit. Category A (no Go toolchain configured for a broken-module fixture).
#2001 savings baseline, threshold contradiction, grep memory ⚠️ PARTIAL The memory/wall-clock numbers were measured. I did not reproduce @vpovarna's 41-command benchmark, so nothing I said explains their end-to-end token result — only the mechanism.
PR #2401 a 4th unguarded filter path; FilterMode::Buffered may be test-only ⚠️ PARTIAL The unwind-vs-abort proof was compiled and run. I did not build your branch. The claim that FilterMode::Buffered has no production constructor is a grep result — if I missed a constructor, that half is wrong.
PR #3295 3 open PRs for one bug; cargo precedent at 39a4d28 📖 SOURCE Neither PR was built or tested. This is a git log + diff-reading observation only.
#2954 the gain.rs panics look fixed; 2 sibling sites are not ⚠️ PARTIAL The two new panic sites were reproduced with verbatim extracts. I did not run rtk gain --history with a CJK/Cyrillic history, so "the original is fixed" is "the unsafe slice is gone from the code", not "I confirmed the crash no longer happens".
PR #3368 3 sibling structs have the same defect; 6 dead keys ⚠️ PARTIAL The deserialization failures were reproduced on a replica of your struct attributes (serde 1 + toml 0.8), not on your binary. The dead-key list is a grep.
PR #2581 strip_ansi cannot match \x1b[?25l; the PHP copy fails differently 📖 SOURCE I did not run either stripper against a live spinner. The CONTROL_RE claim in particular is read from its character range — worth confirming against the actual constant before acting on it.
PR #1693 24 rustdoc examples can never compile ✅ RAN cargo test --doc → error: no library targets found. Little left unverified here.
#2317 verified inventory replacing the "(likely)" list 📖 SOURCE I did not run every filter. Each row is a code-path claim. The gradlew row is from an exhibited input, not a live Gradle run — Category A (no Android/Gradle setup).
#1375 the 178-regex COMPILED tax explains the 20 ms gap ⚠️ PARTIAL Measured on Windows, where process-spawn alone is ~35 ms. I have not profiled on Apple Silicon, so I cannot say what fraction of your 56 ms this is there — the deltas are meaningful, the absolutes are not comparable.
#3392 rtk grep 31× memory / 4.9× wall-clock ✅ RAN Measured on Windows only; Linux/macOS allocator behaviour may differ.
#3393 db never shrinks; gain slows on large history ✅ RAN Measured. The query-plan claims are from EXPLAIN QUERY PLAN on a synthetic 500k-row database, not your users' real distribution.
#3394 two CI steps cannot fail; no slicing lint; semgrep baseline ⚠️ PARTIAL The deny(warnings) clippy escalation was verified on a probe crate. Whether those jobs are required for a merge is a repository setting I cannot see — Category D. If they are not required, this is worse than I said; if another gate compensates, it is smaller.
#3395 psql parsed without -X, ~/.psqlrc changes the format 📖 SOURCE Entirely source-read. psql is not installed — Category A (needs a running database to produce representative output). Every failure mode I describe is predicted, not observed.
#3396 stale 0.28.2 / rtk 0.16 version checks ✅ RAN Read directly from the files; Cargo.toml says 0.44.2.
#3397 filtered clone no caller reads; triple-retained capture 📖 SOURCE Reachability traced by reading every call site. Not profiled — I did not measure the allocation cost, only argue it exists.

Reports sent through the private security channel

Those carry their own "Testing" section stating exactly what was and was not executed — the same discipline as above. I am not restating them here while they are in triage. In short: two were reproduced end-to-end on this machine, one had its mechanism reproduced but not its tool-specific step (the tool is not installed), and two are code-property reports I deliberately did not execute because running them would have fired a real network request or deleted real local data.

The honest summary

Of the 16 public posts above: 5 were executed and observed, 6 had part of the claim executed, and 5 are source-reading only. The five source-only ones are the ones I would most want you to check before acting — they are also, not coincidentally, the ones covering tools I could not install.

Activity

  1. pszymkowiak commented on Aug 4, 2026

    @pszymkowiak
    Collaborator

    wshm · Automated triage by AI

    🔍 Automated Triage

    ❓ Category needs-info
    🟢 Priority low
    🎯 Confidence 72%

    Summary

    Meta-issue disclosing scope limitations across the reporter's recent audit findings; no single actionable bug, primarily informational and useful for triaging linked reports.


    Triaged automatically by wshm · This is an automated analysis, not a human review.

  2. HetCreep commented on Aug 4, 2026

    @HetCreep
    Author

    Baseline note, since it affects how you triage the batch.

    Everything I filed was audited at the tag v0.44.2 (700bdde), not at develop. At the time of writing, origin/develop is 13 commits ahead of that tag, and several of them are in the areas I reported on:

    b09da2d  fix(rewrite): bail on cross-line [[ ]], share quote-state walker
    3c8892f  fix(rewrite): keep rewriting quoted multi-line strings as one command
    ac5febe  fix(rewrite): bail on swallowed newlines and cross-line arithmetic
    fc8054e  feat(rewrite): rewrite each line of multi-line Bash blocks
    0fcd6ad  fix(hooks): stop Copilot from ever asserting a permissionDecision on AskRewrite
    1ebc271  fix(hooks): recognize VS Code Copilot Chat's run_in_terminal tool name
    b754b85  fix(hooks): drop redundant camelCase preToolUse entry from Copilot hook config
    

    So if any of my reports reads as already-fixed to you, that is the likely reason — I was reading a released binary and its matching tag, on purpose, because that is what users have installed.

    I did check the two areas where a develop fix seemed most likely, and both are unchanged there:

    • src/discover/lexer.rs — split_for_permissions' seg_end handling (the truncate-at-first-redirect logic) is byte-for-byte the same on develop; the diff against the tag is +20 lines elsewhere in the file.
    • src/hooks/hook_cmd.rs — the print_allow() calls are still present, despite that file changing by ~208 lines on develop.

    I have not diffed the remaining reports against develop one by one. If it would be more useful to you, I am happy to re-check any specific issue at develop tip and close it myself if it no longer reproduces — I would rather withdraw something than leave a stale item in a queue this size.

    Also worth noting for anyone reading this thread later: develop has had no commits since 2026-08-03, so nothing has moved since I filed.

  3. HetCreep commented on Aug 4, 2026

    @HetCreep
    Author

    Closing inventory: 36 items remain unverified, each for a specific missing capability rather than for lack of effort.

    Rewriting my previous comment — it was too compressed to act on. Each row below says what the check is, so anyone picking one up knows what to run and what a pass or fail would mean.


    A. Needs Linux or macOS — 8 items

    The audit machine is Windows, so every #[cfg(unix)] path was read, not executed.

    # What is unverified How to check it Why it matters
    1 The 53 integration tests behind #![cfg(unix)] (#3422) — search_faithful, search_compress, grep_faithful_format, grep_context, search_error, pipeline_stdin cargo test --all on Linux; read the counts I established only that they compile away on Windows. If they also fail on Linux, the grep/search contract has no coverage anywhere
    2 File permission bits on history.db, tee files and the device salt On Linux, run any teeing command then stat -c '%a %n' on the created files — expect 600, dirs 700 set_owner_only (core/utils.rs:266-283) is a no-op on non-unix, so I could not confirm the hardening added for GHSA-w3rq actually applies
    3 SIGPIPE handling `rtk git log head -1` on Linux; check rtk's exit status, and whether the tee file and tracking row were written
    4 Orphaned children when rtk is killed mid-run Start rtk cargo build, kill the rtk pid, check whether the cargo child survives and is reparented to init ChildGuard::drop only wait()s, never kill()s, and cannot run at all under panic = "abort"
    5 A symlinked history.db, tee dir or config file Replace each with a symlink pointing elsewhere, run a normal command, then check whether rtk followed the link or replaced it with a regular file atomic_write canonicalises first, which looks correct — but temp-file-then-rename over a symlink is exactly where that assumption breaks. Windows refused to let me create the symlinks to test it
    6 The cleanup_old_files race Two concurrent rtk runs against one tee dir, where one reads a tee file as the other's pruning pass deletes it NTFS refuses to unlink an open file, so Windows hides this entirely. POSIX unlinks happily and the reader is left holding a deleted inode — the worst platform is the one I could not test
    7 Permission-denied directories and broken symlinks in the find/ls/grep differentials (#3432, #3434) chmod 000 a subdirectory, create a dangling symlink, then compare rtk find/ls/grep against the real tools #3432 already shows rtk swallowing a missing search root as exit 0. Whether it does the same for an unreadable directory is the more dangerous variant, and it is untested
    8 Hidden-path and NOISE_DIRS filtering with POSIX separators Repeat the #3434 comparison on Linux The filtering is path-string based; I demonstrated it only with \ separators

    B. Needs a real tool plus its credential, daemon, cluster or database — 15 items

    For these I used controlled stand-ins and said so in each issue. Installing the binary alone would not have closed the gap — every one needs state I do not have and should not have.

    # Tool Why a stand-in was not enough The check that would settle it
    9 aws (≈23 filters) Needs live credentials; output shape varies by service and CLI version Run the filters against a real account, diff against raw aws output
    10 kubectl Needs a cluster Same, against a real one
    11 docker Needs a daemon and images Also worth checking docker run -it TTY handling — container.rs:671 uses passthrough and I did not trace stdio inheritance
    12 psql Needs a running database #3395 is entirely source-derived. Every failure mode described there is predicted, not observed
    13 sops Needs real key material, and I was not going to generate live secrets to test a filter GHSA-9q8h: the tee mechanism was reproduced with a stand-in; the sops-specific step was not
    14 helm Needs a chart that renders a Secret GHSA-p56g claims helm template tees plaintext on a successful run — confirm with a real chart
    15 terraform / tofu Needs real state to produce representative plan output Whether plan bodies actually exceed the 80-line cap in practice
    16 ansible Needs a playbook and inventory Whether vault-decrypted variables really do surface in failed-task JSON
    17 gcloud Needs an authenticated project The output shape of gcloud secrets versions access
    18 ruby / rubocop Not installed, and I did not build a stand-in for this one #3429 — whether --autocorrect (hyphenless) is accepted by the RuboCop versions you support. If it is not, that finding is void
    19 php / composer Reproduced with .cmd shims GHSA-r59x — confirm the bin-dir hijack with real composer and real pint
    20 gradle / mvn Needs a JVM toolchain and real multi-module projects Fixture output was used instead
    21 golangci-lint Not installed #3428 — the exit-1-becomes-0 mapping was shown with a stand-in
    22 prettier / black / ruff Not installed #3417 — the ruff format . write-mode injection was shown by capturing argv, not by watching files change on disk
    23 playwright / jest / vitest Not installed Whether Playwright treats the orphaned --reporter value as a test-path filter

    C. Needs the other vendors' products — 5 items

    This is the group I would most like closed, and the one you can probably close fastest. I can prove what rtk emits; I cannot prove what the receiving harness does with it — and that is precisely what sets the severity of the hook advisories.

    # Harness The question Why it decides severity
    24 Copilot CLI Does it honour permissionDecision as rtk supplies it? Your own #3037 says 1.0.66+ does, which is why that issue exists — but I have no independent view of it
    25 Gemini CLI Does it honour {"decision":"allow"} as a prompt bypass, and does it invoke BeforeTool before its own argument-schema validation? GHSA-546c reports that rtk hook gemini answers allow when it cannot read the command. If Gemini validates arguments first, that path may be unreachable and the advisory should be downgraded
    26 Cursor How it treats {"continue":true,"permission":"ask","updated_input":…} Cursor is the one host that returns {} on malformed input; whether that is safe depends entirely on Cursor
    27 Droid How it treats updatedInput when no permissionDecision is present The non-string tool_name case rewrites the command with no decision emitted — harmless or not depending on Droid
    28 Claude Code itself Whether behaviour inside a real session matches what I observed I exercised rtk hook claude by piping PreToolUse JSON on stdin, which tests rtk's side only. I never ran it inside a live session

    D. Outside the repository — 4 items

    # What is unverified Why I cannot see it
    29 Whether the CI jobs in #3394 are required for a merge A GitHub repository setting, not a file in the tree. If they are not required, that finding is larger than I stated; if another gate compensates, it is smaller
    30 Telemetry server retention and erasure behaviour GHSA-q584 reports only what the client sends. The 12-month purge, IP anonymisation, and what /erasure does server-side are unobservable from the binary
    31 Release-artifact provenance I did not download your published binaries, and deliberately did not run strings on them looking for the option_env! token — that felt inappropriate for a report you had not yet triaged
    32 ccusage internals Referenced by #3418, never executed here

    E. Deliberately not run — 4 items

    Listed so they are not mistaken for oversights.

    # Command Why I did not run it
    33 rtk init and rtk init --uninstall (#3400, #3403–#3406) Writes into the real home directory. The Hermes YAML cases were instead reproduced by extracting the pure functions and running them under rustc
    34 rtk learn --write-rules (GHSA-wh6r and its follow-up) There is no environment override for the transcript root, so a faithful run would read my own agent transcripts, which contain live credentials from unrelated work. The markdown-injection half was reproduced by compiling the exact format! call in isolation
    35 rtk telemetry forget (GHSA-q584) Fires a real network request and deletes the real local database
    36 rtk cc-economics (#3418) Downloads and executes an unpinned npm package (npx --yes ccusage) and reads real spend data. Every arithmetic claim in that issue is checkable by reading instead

    The short version: claims about rtk's own code are checkable by reading, and I have kept those separate throughout. Claims about how a wrapped tool or a third-party harness behaves are not, and every one of those is labelled in its own issue.

    If any of group C is a one-liner for you, that closes more uncertainty than anything further I can do from here.

  4. HetCreep commented on Aug 30, 2026

    @HetCreep
    Author

    Refreshing one section of my coverage disclosure. Re-checked at develop @ 0de9610.

    Provenance: AI-assisted source audit, human-directed, at develop @ 0de9610. Section F was re-counted at this commit; sections A–E describe the auditing environment and are unchanged.

    Sections A–E (tools not installed, Windows-only auditing box, another vendor's harness, things outside the repo, and what I chose not to run) are statements about my environment and remain accurate as written.

    Section F's map of the code is stale. src/discover/registry.rs no longer contains the rewrite-rule table at all — it moved to src/discover/rules.rs — so my sentence pointing at registry.rs now names a file with none. The count does not reproduce either: I counted 178 at the time, and at this commit rules.rs has roughly 90 rule entries (by pattern: occurrences). I have not tried to reconcile the difference; it may be a change in how the rules are expressed as much as a change in how many exist.

    The two byte-size figures in that section are stale by roughly 8–15%.

    Nothing else in the disclosure needs editing, and none of it was ever a defect claim.

  5. HetCreep commented on Sep 16, 2026

    @HetCreep
    Author

    Re-checked Section F against develop@5e0f92c — verified by reading the code at this commit, nothing built or run (this is a citation check, not a behavior claim).

    This section already needed one correction on this thread: at develop@0de9610 we found registry.rs had already lost its rule table, and the original report's "178 rewrite rules" didn't reproduce (recount then: 92). At 5e0f92c, 468 commits past that checkpoint, the split still holds and every figure has moved further:

    • Split unchanged: registry.rs:1 is now //! Matches shell commands against known RTK rewrite rules to decide how to handle them., and registry.rs:14 imports use super::rules::{IGNORED_EXACT, IGNORED_PREFIXES, RULES, RtkRule};. The rule table itself (pub const RULES: &[RtkRule] = &[...]) is entirely in the separate src/discover/rules.rs (1178 lines) — registry.rs only consumes it, building REGEX_SET from RULES at registry.rs:52-54.
    • Rule count: grep -c 'pattern:' src/discover/rules.rs = 96 now, vs. 92 at 0de9610, vs. "178" in the original report.
    • registry.rs: 239,148 bytes / 7,381 lines now, vs. 175,712 bytes at 0de9610, vs. the original report's "~155 KB".
    • src/hooks/init.rs: 397,414 bytes / 11,109 lines now, vs. 300,248 bytes at 0de9610, vs. the original report's "~280 KB".

    Unaffected: the 63-file .toml filter count elsewhere in this section still reproduces exactly (ls src/filters/*.toml | wc -l = 63).

    Not a defect — this is a coverage-disclosure note, not a bug report, and its substance (these two files are large, actively growing surfaces the original audit didn't read line-by-line) still holds, if anything more clearly than before. Updating the numbers here so they don't keep drifting further from what's actually on develop; no action needed on your end.

  6. HetCreep commented on Oct 14, 2026

    @HetCreep
    Author

    Re-checked on v1.0.0 (stable, 2026-10-13): the official release binary plus the v1.0.0 source. Provenance: AI-assisted check, human-directed. "ran" = executed against rtk 1.0.0 with a control; "read" = verified from source only.

    Refresh of the stale cites in section F of this disclosure, checked at v1.0.0 (read only; no binary run). Sections A-E (environment and scope statements) are unaffected.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions