Token usage comparison in openrig versus other methods

[Elderx]

I think researching the token usage gets way too little attention when building orchestration systems like these. To me it seems like almost all of these systems are mostly targeted to those who can afford to run multiple maxed out subscriptions with little to no worry about the token usage. It would be very good to see indepth research put into this, that how much more token are been spent when using orchestration like this vs "more human in the loop" style of development.

There are very large number of potential users who would greatly benefit if these systems would bring some "minimize token usage" flags and/or other toggleable systems that would be aimed at making the development as cost efficient as possible.

[mvschwarz]

Thanks, @Elderx, for raising cost for people who can't run several maxed-out subscriptions, and for asking for a measured comparison rather than a guess. We don't have that comparison to share. Two things OpenRig gives you today. You can measure your own team's spend: rig usage top --window 2h ranks your seats (the agents in your team) by tokens used per hour over the window you choose, and rig usage series lists each seat's raw samples, from the token counts OpenRig samples from Claude Code and Codex. And you can set model and effort per agent in the rig spec, so only the agents whose work needs the strongest model and the most reasoning get them. If you measure a run of your own, we'd be glad to see the numbers here.

[VirendarRawat]

Ah! Finally someone raised the question I want to raise!!!🙌 Thank You @Elderx for raising the question. I think the most useful way is to treat token efficiency as a measurable benchmark rather than comparing the systems qualitatively. A controlled experiment could use the same repos, task, model, model version, and effort level, then compare it with both
Human-in-the-loop baseline:

• One agent
• Human reviews it at defined checkpoints

OpenRig:

• Same task and starting context
• Fixed number of agents
• Same model and or or same effort where possible

For each task, I would record:

• Input tokens or tokens consumed
• Output tokens or tokens left
• Total tokens
• Number of model calls
• Number of agents involved
• Time Taken
• Whether the task was completed successfully or reached user requirements

I would also separate "useful work" from orchestration/context overhead where the telemetry allows it. Otherwise, a system could look inefficient simply because it uses more context to achieve a substantially better result.

OpenRig already seems to expose useful data for this experiment through rig usage series and rig usage top, so this could potentially be reproduced by users rather than relying on theoretical estimates.

The really interesting follow-up would be to run the same benchmark with different configurations:

• 1 seat vs 2 seats vs more seats
• low vs high effort
• cheaper model vs stronger model
• with or without human checkpoints

That would make it possible to identify where orchestration actually becomes worth the additional token cost.

If this data were collected across even 5–10 representative coding tasks, I think a table showing tokens/task, tokens/successful-task, time/task, and success/rework rate would be much more informative than a single "OpenRig uses X% more tokens" number.

[mvschwarz]

Thanks, @VirendarRawat, for laying out a controlled comparison of one agent with human checkpoints against an OpenRig team: the same repos, task, model and effort on both sides, and the measures to record for each run.

[jimy-r]

One operator's measured numbers, since the comparison asked for here does not exist yet. They come from Claude Code transcripts, so read them as a method more than a benchmark.

Cost per accepted task is the unit that held up. Of 117 delegated tasks sent to the cheaper model, none needed rework, and they cost $3.17 per accepted task. The tasks sent to the stronger model were the harder ones and cost $7.66.

Most orchestration cost is context read again, and position drives it. Every tool call sends the whole transcript back, so a token costs its size times the steps left after it arrives. A 104k token read at step 3 of one session was read back as 19.6M tokens, while a read of the same size at step 182 cost 133k. Two cheap levers follow. Subagents return a short summary and write detail to a file, and independent tool calls go out in one step.

One trap for anyone building the meter. My transcripts carried more than one usage line per message id, and summing lines inflated dollar figures by 2.5x to 6.6x.

I maintain a reference architecture with the cost model written up at https://github.com/jimy-r/agent-workspace-architecture/blob/main/PATTERNS.md#18-position-is-price--a-token-costs-more-the-earlier-you-add-it

Drafted with Claude Code and reviewed by me before posting.

[mvschwarz]

Thanks, @jimy-r, for measured numbers, and for being clear they're a method more than a benchmark. Thanks especially for these three:

• cost per accepted task: $3.17 across 117 tasks on the cheaper model with no rework, and $7.66 for the harder tasks on the stronger one;
• where context arrives: a 104k token read at step 3 of one session was read back as 19.6M tokens, while a read of the same size at step 182 cost 133k;
• the trap for anyone building a meter: more than one usage line per message id, which inflated dollar figures by 2.5x to 6.6x when summed.