feat: add --toon output format for token-optimized CLI output - #215
feat: add --toon output format for token-optimized CLI output#215akb4q wants to merge 4 commits into
Conversation
Add a new global --toon flag that outputs command results in TOON format (https://toonformat.dev), a token-optimized notation designed for LLM consumption — 6-9x more compact than JSON or human-readable tables. Implementation: - toonEscapeValue / toonEscapeKey: spec-compliant quoting (§7.1-7.2) and key encoding (§7.3) - outputToon: tabular array format for list commands - outputJsonToon: tabular for uniform arrays, key-value for single objects - Module-level flag set via Commander preAction hook - outputSuccess/outputInfo suppressed in TOON mode (non-structured noise) Changes: src/index.ts — register --toon global option + preAction hook src/lib/output.ts — add TOON formatters + route outputTable/outputJson Token savings measured on insforge projects list: TOON 129B vs JSON 989B (7.7x), vs human table 1162B (9x). Tested: build (tsup), 649 unit tests (1 pre-existing cloudflare failure), integration on projects list, whoami, functions list, orgs list.
|
Thanks for the PR, @akb4q! A quick note on our workflow: we ask contributors to open an issue first, get it assigned, then submit a PR that links it (e.g. "Closes #123"). This PR isn't linked to any issue. It'll still be reviewed, but please open an issue and claim it (comment that you'd like it assigned to you) so the work is tracked. |
agent-zhang-beihai
Bot
added
the
needs-issue
WalkthroughChangesTOON output mode
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related issues
Suggested labels: Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Greptile SummaryAdds a global token-optimized TOON output mode.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains; TOON now routes credential-producing commands through their complete structured-output branches, nested data is handled by the dedicated encoder, and each result is emitted as one document.
|
| Filename | Overview |
|---|---|
| src/index.ts | Registers the global --toon option and initializes TOON mode through a pre-action hook. |
| src/lib/errors.ts | Makes --toon select commands' existing structured-output branches, ensuring complete command results are passed to the TOON encoder. |
| src/lib/output.ts | Adds TOON encoding for structured objects and tables while suppressing auxiliary human-readable messages. |
| src/lib/output.test.ts | Covers TOON scalar escaping, structured encoding, null preservation, message suppression, and tabular output. |
| package.json | Adds the TOON encoder as a runtime dependency. |
| package-lock.json | Locks the new TOON dependency and synchronizes the package version. |
Reviews (4): Last reviewed commit: "fix: suppress outputSuccess/Info in toon..." | Re-trigger Greptile
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/lib/output.ts`:
- Around line 17-32: Resolve the ESLint violations in the quote-formatting logic
around needsQuoting: replace the escaped opening bracket in the punctuation
regex with a literal bracket, and remove both control-character regexes in favor
of equivalent character-code checks and replacement handling. Preserve the
existing quoting and escaping behavior, including Unicode escapes for
unsupported control characters.
- Around line 99-124: Replace the ad hoc serialization in outputJsonToon with a
conformant recursive TOON encoder, preferably using the official
`@toon-format/toon` encode API. Preserve heterogeneous array values and nulls,
recursively encode nested objects and arrays rather than stringifying them, and
retain null fields/entries; add decode round-trip coverage for these cases.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 17e2f641-7d66-42fc-a61f-608a2ac82f7b
📒 Files selected for processing (2)
src/index.tssrc/lib/output.ts
| const needsQuoting = | ||
| s === '' || | ||
| s === 'true' || s === 'false' || s === 'null' || | ||
| /^[+-]?[0-9]+(\.[0-9]+)?([eE][+-]?[0-9]+)?$/.test(s) || | ||
| /^[ \t]|[ \t]$/.test(s) || | ||
| /^[-#]/.test(s) || | ||
| /[:,"\\{}\[\]]/.test(s) || | ||
| /[\x00-\x1f]/.test(s); | ||
| if (!needsQuoting) return s; | ||
| return '"' + s | ||
| .replace(/\\/g, '\\\\') | ||
| .replace(/"/g, '\\"') | ||
| .replace(/\n/g, '\\n') | ||
| .replace(/\r/g, '\\r') | ||
| .replace(/\t/g, '\\t') | ||
| .replace(/[\x00-\x08\x0b\x0c\x0e-\x1f]/g, c => '\\u' + c.charCodeAt(0).toString(16).padStart(4, '0')) + '"'; |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Resolve the new ESLint errors.
The escaped [ and both control-character regexes are reported as ESLint errors, leaving this changed file failing lint. Replace the control-character regex checks/replacement with character-code handling and use a literal [.
🧰 Tools
🪛 ESLint
[error] 23-23: Unnecessary escape character: [.
(no-useless-escape)
[error] 24-24: Unexpected control character(s) in regular expression: \x00, \x1f.
(no-control-regex)
[error] 32-32: Unexpected control character(s) in regular expression: \x00, \x08, \x0b, \x0c, \x0e, \x1f.
(no-control-regex)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/lib/output.ts` around lines 17 - 32, Resolve the ESLint violations in the
quote-formatting logic around needsQuoting: replace the escaped opening bracket
in the punctuation regex with a literal bracket, and remove both
control-character regexes in favor of equivalent character-code checks and
replacement handling. Preserve the existing quoting and escaping behavior,
including Unicode escapes for unsupported control characters.
Source: Linters/SAST tools
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
1 issue found across 2 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="src/index.ts">
<violation number="1" location="src/index.ts:119">
P1: Using `--toon` alone can produce an empty, human-readable, or raw response instead of TOON. The new flag is parsed by the preAction hook, but command handlers still branch only on `getRootOpts(cmd).json`: `whoami --toon` takes its `outputInfo` branch and both messages are then suppressed, while commands such as `functions invoke` and `metadata` write directly with `console.log` and bypass the TOON renderer. This makes the advertised global format inconsistent unless callers already know to add `--json`; treating TOON as an effective structured-output mode (or explicitly rejecting/documenting it without `--json`) would avoid these silent/mixed responses.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| // Global options | ||
| program | ||
| .option('--json', 'Output in JSON format') | ||
| .option('--toon', 'Output in TOON format (token-optimized notation, see toonformat.dev)') |
There was a problem hiding this comment.
P1: Using --toon alone can produce an empty, human-readable, or raw response instead of TOON. The new flag is parsed by the preAction hook, but command handlers still branch only on getRootOpts(cmd).json: whoami --toon takes its outputInfo branch and both messages are then suppressed, while commands such as functions invoke and metadata write directly with console.log and bypass the TOON renderer. This makes the advertised global format inconsistent unless callers already know to add --json; treating TOON as an effective structured-output mode (or explicitly rejecting/documenting it without --json) would avoid these silent/mixed responses.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/index.ts, line 119:
<comment>Using `--toon` alone can produce an empty, human-readable, or raw response instead of TOON. The new flag is parsed by the preAction hook, but command handlers still branch only on `getRootOpts(cmd).json`: `whoami --toon` takes its `outputInfo` branch and both messages are then suppressed, while commands such as `functions invoke` and `metadata` write directly with `console.log` and bypass the TOON renderer. This makes the advertised global format inconsistent unless callers already know to add `--json`; treating TOON as an effective structured-output mode (or explicitly rejecting/documenting it without `--json`) would avoid these silent/mixed responses.</comment>
<file context>
@@ -116,6 +116,7 @@ program
// Global options
program
.option('--json', 'Output in JSON format')
+ .option('--toon', 'Output in TOON format (token-optimized notation, see toonformat.dev)')
.option('--forger', 'Play the Forger animation (root command only) and return to the interactive menu')
.option('--api-url <url>', 'Override Platform API URL')
</file context>
|
Closes #216 |
jwfing
left a comment
jwfing
left a comment
There was a problem hiding this comment.
Review: feat: add --toon output format
Summary: Clean, well-scoped feature (module-level flag routed through outputTable/outputJson, zero handler changes), but a broken regex in the value-quoting logic means values containing the delimiter or other structural characters are emitted unquoted, producing malformed TOON — including on the PR's own headline example.
Requirements context
No matching spec/plan found — this repo keeps specs under docs/specs/ (not docs/superpowers/), and none of the three present (diagnose, db-migrations) cover output formatting. Assessed against the PR description and TOON's stated goal (compact, safely-parseable output for LLM/script consumption).
Critical
1. Value-quoting regex closes its character class early → delimiter and structural chars are never quoted (src/lib/output.ts:23)
/[:,"\\{}[\\]]/.test(s)The intent is "quote if the value contains any of : , " \ { } [ ]". But inside the class, \\ is a literal backslash and the very next ] closes the class. So the pattern actually parses as [:,"\{}[\] followed by a trailing literal ] — i.e. it only matches when one of those chars is immediately followed by a ]. Verified:
"a,b" -> match: false // comma NOT detected
"a:b" -> match: false
'a"b' -> match: false
"a{b}c" -> match: false
"Smith, John"-> match: false
":]" -> match: true // only the accidental trailing-] form matches
Consequence: any value containing a comma (the TOON row delimiter) is emitted unquoted, so the row gains phantom columns and the output no longer parses. This is not hypothetical — it hits the PR's own advertised example. functions list builds the Created At cell via new Date(f.createdAt).toLocaleString() (src/commands/functions/list.ts:45), which in en-US yields e.g. 7/30/2026, 10:00:00 AM (contains a comma). Actual output:
[3]{Slug,Name,Status,"Created At"}:
func-1,My Function,active,7/30/2026, 10:00:00 AM <- 5 fields for a 4-col header
whereas the PR body shows the timestamp correctly quoted ("2026-07-30, 10:00:00 AM") — output the current code does not produce. Since the whole point of --toon is safely machine-parseable output, this defeats the feature and silently corrupts data.
Fix: escape the closing bracket so it stays inside the class, e.g. /[:,"\\{}[\]]/ (verified: this correctly quotes Smith, John). Please also cover ", \, {, }, [, ] while you're at it — all currently slip through the same way.
Suggestion
- Software engineering — no tests added (
src/lib/output.ts). This module is regex- and edge-case-heavy, and the repo has a strong test culture (600+ tests, one*.test.tsper lib module). A one-line assertion likeexpect(toonEscapeValue('a,b')).toMatch(/^"/)would have caught the Critical above immediately. Please add asrc/lib/output.test.tscovering: delimiter/structural-char quoting, empty string, numeric-looking strings, booleans/null, control chars, leading/trailing whitespace,[N]{keys}:tabular framing, the[0]{keys}:empty case, single-object key/value, and the--json --toonpath. Theinsforge-developmentskill requires every new behavior to carry at least one test. - Functionality — heterogeneous arrays (
src/lib/output.ts:102).outputJsonToonderives columns fromObject.keys(first)only; if later array elements have different keys, extra keys are dropped and missing ones render asnull. Fine for uniform list endpoints, but worth a guard or a comment documenting the assumption.
Information
--json --toonprecedence (src/lib/output.ts:50). When both flags are passed, TOON silently wins and--jsonis ignored. The PR body useswhoami --json --toonas an intended combo, so this is presumably deliberate — worth a note in--helpso callers aren't surprised.- Nested objects (
src/lib/output.ts:120-123). In single-object mode, nested objects are rendered as rawJSON.stringify(v)(embedded JSON, not TOON). Acceptable, but not strictly spec-conformant if downstream expects pure TOON. toonEscapeKey(src/lib/output.ts:37-45) doesn't apply the control-char\uXXXXcatch-all thattoonEscapeValuedoes. Keys come from code-controlled headers so the risk is negligible today.
Security
No security-relevant changes — output-only path, no new user input reaching SQL/shell/HTTP, no new dependencies (formatting is hand-rolled), and TOON suppresses outputSuccess/outputInfo without touching secret handling.
Performance
No concerns — string concatenation over CLI-sized result lists, no DB queries, no hot path or event-loop blocking introduced.
Verdict: request_changes
One Critical correctness bug (the quoting regex) blocks merge; it corrupts the exact output the feature exists to produce and lands untested. The rest is a solid, well-contained change — fixing the regex and adding a focused output.test.ts should be enough to land it.
|
@akb4q could you please address these comments? thanks! |
- Replace ad-hoc TOON serializer with official @toon-format/toon encode() (fixes nested structure loss, heterogeneous array crashes, null handling) - Make --toon standalone: getRootOpts returns json=true when --toon set (fixes empty output and credential loss without --json) - outputSuccess/outputInfo emit structured TOON in toonMode (fixes silent credential suppression) Fixes greptile P1 (credential loss, nested structure), coderabbit (use official encode API), cubic-dev-ai P1/P2 (heterogeneous arrays, null drop, standalone mode)
|
@jwfing all review comments addressed:
Build ✅, ESLint 0 errors ✅, 648/649 tests passing ✅ (1 pre-existing Cloudflare flake) |
There was a problem hiding this comment.
All reported issues were addressed across 4 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
…tests
- outputSuccess/outputInfo: suppress in TOON mode (route through outputJson)
- Fix toonEscapeValue regex: /[]:,"\{}[]/ -> /[\]:,"\{}[]/ (\] escape fixes
premature class close — commas now detected as quote-trigger chars)
- Add output.test.ts with 13 tests covering: structural-char quoting,
empty string, boolean-likes, null handling, single-doc output,
suppress behavior, tabular format, empty-table edge case
|
@jwfing addressed your review and the new greptile re-review:
Build passes, ESLint 0 errors, 661/662 tests passing (1 pre-existing Cloudflare flake). |
3 participants
Summary
Add a global
--toonflag that outputs results in TOON (Token-Oriented Object Notation), designed for LLM consumption — 6-9x more compact than JSON or human-readable tables.Performance
Measured on
insforge projects list:Usage
Implementation
preActionhook, read byoutputTableandoutputJsonat render time — zero changes to any command handlertoonEscapeValue/toonEscapeKeyconform to TOON SPEC §7.1-7.3 (escaping, quoting rules, key encoding)outputSuccess/outputInfosilently skipped in TOON mode to keep output parseableTesting
npm run build: tsup compiles clean (601 KB ESM)npm test: 649 tests pass (1 pre-existing Cloudflare OAuth failure unrelated)projects list,whoami,functions list,orgs listwith--toonand--json --toonRoot commands (create, login, feedback, etc.) produce no TOON output — they emit interactive prompts and status messages, not structured data.
Summary by cubic
Adds a global
--toonoutput format for token‑optimized results, ~6–9× smaller than JSON or tables. Uses@toon-format/toonencode()and suppresses non‑structured messages to keep output parseable.New Features
--toonvia pre‑action hook, routingoutputJson/outputTableto TOON using@toon-format/toonencode().Bug Fixes
--toonas implying JSON ingetRootOptsto avoid empty output and credential loss.outputSuccess/outputInfoin TOON mode; add tests for quoting, nulls, single‑doc output, and empty tables.Written for commit f388046. Summary will update on new commits.
Note
Add
--toonglobal flag for token-optimized TOON format output--toonglobal CLI flag in src/index.ts that callssetToonMode()before any command runs.setToonMode()and supporting helpers in src/lib/output.ts; when enabled,outputJson,outputTable,outputSuccess, andoutputInfoall emit TOON-encoded output using the@toon-format/toonpackage.outputToon()handles tabular data with a count+schema header line followed by one escaped row per record.getRootOptsin src/lib/errors.ts to treat--toonas implying--json(i.e.opts.json ?? opts.toon ?? false).--toonsilently sets thejsonflag ingetRootOpts, which may affect error handling logic that branches on that field.Macroscope summarized 0a96fe2.
Summary by CodeRabbit
--toonoption for enabling TOON-formatted command output.