-
Notifications
You must be signed in to change notification settings - Fork 57
Replace hardcoded subtraction chain with registry-driven decomposition #321
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
alexmojaki
wants to merge
83
commits into
main
Choose a base branch
from
feat/token-unit-registry
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from 17 commits
Commits
Show all changes
83 commits
Select commit
Hold shift + click to select a range
a27f990
Add token unit registry spec, reference, and implementation plan
alexmojaki 8350c58
Add token unit registry types and family definition
alexmojaki b538e70
Add containment helpers and leaf value decomposition via Mobius inver…
alexmojaki 887d11a
Rewire calc_price to use registry-driven decomposition
alexmojaki 0e3d754
Update test dataset for decomposition bugfix
alexmojaki 0db8300
Remove plan and reference doc from branch (keep only spec)
alexmojaki 6a72346
Add unit registry YAML as source of truth
alexmojaki 3e935dd
Add build step to generate units JSON from YAML
alexmojaki 6762c25
Load unit definitions from generated JSON instead of hardcoding
alexmojaki 0c7e221
Add JS unit registry types and loader
alexmojaki 3303203
Add JS decomposition engine with Mobius inversion
alexmojaki 9ce58b4
Rewire JS calcPrice to use registry-driven decomposition
alexmojaki 5ac6b8a
Validate unit definitions with Pydantic
alexmojaki afd7bd9
Symlink JS units JSON, add YAML schema, document prettier step
alexmojaki 9409945
Document ancestor coverage precondition on compute_leaf_values
alexmojaki a599386
Replace AbstractUsage Protocol with object type alias
alexmojaki 06d9710
Add ancestor coverage validation on ModelPrice construction, fix 3 Op…
alexmojaki 14aba2b
Update test fixtures for ancestor coverage validation on audio models
alexmojaki 6507696
Add ancestor coverage validation in JS calcPrice
alexmojaki 4fb8596
Rename cache_read_audio_mtok unit ID to cache_audio_read_mtok to matc…
alexmojaki d2e63a1
Remove FIELD_TO_UNIT identity mapping, use TOKENS_FAMILY.units directly
alexmojaki 3879dd2
Replace phase language with concrete descriptions, fix spec naming to…
alexmojaki 51e7957
Restore computed variable assertions in cache tokens test
alexmojaki 6ecaea0
Replace units-data.json symlink with file copy
alexmojaki b0cb6d8
Fix total_input_tokens to handle Mapping-type usage via get_usage_value
alexmojaki fb8de9f
Update spec: eliminate phases, require data-driven ModelPrice
alexmojaki 570024c
Add initial data-driven unit registry spec: axioms and derived princi…
alexmojaki 0722067
Spec: add unit distribution requirement and explicit validation rules
alexmojaki 7a7d4e4
Spec: add pricing accuracy as top-level goal, distinguish price vs us…
alexmojaki 1f298fa
Spec: add unit naming convention, acknowledge cache_audio_read incons…
alexmojaki 2497cff
Spec: replace naming convention with writability principle and schema…
alexmojaki 298ac74
Rename cache units to cache_{modality}_{op} pattern (mechanical)
alexmojaki ec247e1
Spec: add dimension-filtered aggregate counts
alexmojaki 797982d
Spec: add inconsistent usage rejection and price sanity checks
alexmojaki 501a4ea
Spec: add derived requirements — runtime mechanics, requests_kcount, …
alexmojaki 27f6bc0
Spec: add missing items from old spec — usage key default, decomposit…
alexmojaki 5cfc5a6
Remove old unit registry spec, superseded by specs/data-driven-unit-r…
alexmojaki d916fac
Spec: address review issues — fix forward ref, citations, examples, a…
alexmojaki 6c7069e
Spec: address second review — fix forward ref, join ambiguity, reques…
alexmojaki 7d27d0f
Spec: fix requests_kcount usage key — concrete value, not vague
alexmojaki a1419d6
Spec: note custom unit API as open question
alexmojaki 0516d37
Spec: add validation performance constraint — expensive checks at def…
alexmojaki 3d7b73c
Spec: clarify data.py vs data.json, validation trust is about provena…
alexmojaki 894ccaa
Spec: note JS package also has generated data file alongside Python
alexmojaki d725400
algorithm.md: clarify depth = number of dimensions, no zero-dimension…
alexmojaki 0f0894e
Spec: address third review — forward ref, citations, TieredPrices sco…
alexmojaki 619c4bb
Spec: address fourth review — unpriced family ignored, zero aggregate…
alexmojaki 31fee35
Resolve open question: custom units arrive via DataSnapshot
alexmojaki 4acaaf9
Simplify custom unit API: families live in snapshot, no separate regi…
alexmojaki 045dfc4
Add spec item: Mapping usage keys validated against registry
alexmojaki 770ad2f
Merge branch 'main' into feat/token-unit-registry
alexmojaki a6b7a1b
Spec: data.json becomes top-level dict, auto-update validation errors…
alexmojaki f623439
Spec: one global snapshot, lazy validation at set_custom_snapshot, fi…
alexmojaki db025b3
Spec review: fix forward references, stale citations, move validation…
alexmojaki c67b1ee
Spec: add explicit items for API signature preservation and calc/extr…
alexmojaki 7c1b72b
Spec: global usage key uniqueness, algorithm-validation dependency, t…
alexmojaki f7c8965
Spec: soften aggregate queries to nice-to-have, clarify as usage-only…
alexmojaki beb7b58
Spec: separate cost aggregation (required) from count aggregation (ni…
alexmojaki e1eae3e
Spec review: fix forward reference, move decomposition-correctness af…
alexmojaki e1fa5cb
Spec: address review — inference for incomplete usage, extractors in …
alexmojaki 7bf7fed
Spec: add registry join-closedness validation rule for custom families
alexmojaki 46b8f08
Spec: DataSnapshot optional unit_families param; use 'unit families' …
alexmojaki 9ccccb1
Algorithm doc: fix 'product of chains' to 'product of flat lattices'
alexmojaki 74e070a
Spec: add calc_price performance constraint — dict lookups only, no h…
alexmojaki d48bfa4
Spec: error messages must describe data problems, not algorithm inter…
alexmojaki b5d4fa8
Spec: remove auto-update retry item — existing hourly retry is fine
alexmojaki 96ed2bb
Spec: requests usage defaults to 1 per call, goes through normal deco…
alexmojaki 9d0a21c
Spec: UnitRegistry architecture — single class owns all unit state, m…
alexmojaki 3c4d6d2
Add code spec: architecture skeleton for data-driven unit registry
alexmojaki cb6037b
Code spec: address review — total_input_tokens, JSON schema, extracto…
alexmojaki 05654b1
Code spec: add explicit constraints — no algorithm jargon in errors, …
alexmojaki acaf993
Code spec: total_input_tokens from sum of input leaf values, not raw …
alexmojaki 833de9b
Code spec: use dataclasses for UnitDef/UnitFamily, drop __slots__ eve…
alexmojaki c696d5a
Spec: UnitDef and UnitFamily are dataclasses, not plain classes
alexmojaki ed73e94
Code spec: slim UnitRegistry API — public dicts, cut getter methods, …
alexmojaki 025f7c1
Code spec: use concrete type in calc_price signature, not AbstractUsage
alexmojaki 914690c
Code spec: calc_price accepts object, not AbstractUsage — getattr on …
alexmojaki 735d88f
Spec: expand reasoning for usage type being object, not AbstractUsage
alexmojaki 98904f3
Code spec: fix is_free() to check all values zero/None, not just empt…
alexmojaki 7cf7409
Spec: price sanity checks start as errors, not warnings
alexmojaki c3134ba
Spec: Usage becomes registry-aware — infers ancestors, owns decomposi…
alexmojaki 54b31d0
Code spec: clarify JS decompose.ts is standalone (no smart Usage in JS)
alexmojaki 31f90fd
Spec: leaf_value stays in calc_price, not on Usage — decomposition de…
alexmojaki File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,150 @@ | ||
| import { describe, expect, it } from 'vitest' | ||
|
|
||
| import type { Usage } from '../types' | ||
|
|
||
| import { computeLeafValues, isDescendantOrSelf } from '../decompose' | ||
| import { getUnit, TOKENS_FAMILY } from '../units' | ||
|
|
||
| describe('Containment', () => { | ||
| it('self is descendant-or-self', () => { | ||
| const unit = getUnit('input_mtok') | ||
| expect(isDescendantOrSelf(unit, unit)).toBe(true) | ||
| }) | ||
|
|
||
| it('child is descendant', () => { | ||
| expect(isDescendantOrSelf(getUnit('input_mtok'), getUnit('cache_read_mtok'))).toBe(true) | ||
| }) | ||
|
|
||
| it('parent is NOT descendant of child', () => { | ||
| expect(isDescendantOrSelf(getUnit('cache_read_mtok'), getUnit('input_mtok'))).toBe(false) | ||
| }) | ||
|
|
||
| it('grandchild is descendant', () => { | ||
| expect(isDescendantOrSelf(getUnit('input_mtok'), getUnit('cache_read_audio_mtok'))).toBe(true) | ||
| }) | ||
|
|
||
| it('sibling not descendant', () => { | ||
| expect(isDescendantOrSelf(getUnit('cache_read_mtok'), getUnit('cache_write_mtok'))).toBe(false) | ||
| }) | ||
|
|
||
| it('different direction not descendant', () => { | ||
| expect(isDescendantOrSelf(getUnit('input_mtok'), getUnit('output_mtok'))).toBe(false) | ||
| }) | ||
| }) | ||
|
|
||
| describe('Leaf Values', () => { | ||
| const family = TOKENS_FAMILY | ||
|
|
||
| it('simple text model', () => { | ||
| const priced = new Set(['input_mtok', 'output_mtok']) | ||
| const usage = { input_tokens: 1000, output_tokens: 500 } | ||
| expect(computeLeafValues(priced, usage, family)).toEqual({ input_mtok: 1000, output_mtok: 500 }) | ||
| }) | ||
|
|
||
| it('with cache', () => { | ||
| const priced = new Set(['cache_read_mtok', 'cache_write_mtok', 'input_mtok', 'output_mtok']) | ||
| const usage = { cache_read_tokens: 200, cache_write_tokens: 100, input_tokens: 1000, output_tokens: 500 } | ||
| expect(computeLeafValues(priced, usage, family)).toEqual({ | ||
| cache_read_mtok: 200, | ||
| cache_write_mtok: 100, | ||
| input_mtok: 700, | ||
| output_mtok: 500, | ||
| }) | ||
| }) | ||
|
|
||
| it('with audio', () => { | ||
| const priced = new Set(['input_audio_mtok', 'input_mtok', 'output_mtok']) | ||
| const usage = { input_audio_tokens: 300, input_tokens: 1000, output_tokens: 500 } | ||
| expect(computeLeafValues(priced, usage, family)).toEqual({ | ||
| input_audio_mtok: 300, | ||
| input_mtok: 700, | ||
| output_mtok: 500, | ||
| }) | ||
| }) | ||
|
|
||
| it('lattice: cache_read_audio carved from both', () => { | ||
| const priced = new Set(['cache_read_audio_mtok', 'cache_read_mtok', 'input_audio_mtok', 'input_mtok']) | ||
| const usage = { | ||
| cache_audio_read_tokens: 50, | ||
| cache_read_tokens: 200, | ||
| input_audio_tokens: 300, | ||
| input_tokens: 1000, | ||
| } | ||
| expect(computeLeafValues(priced, usage, family)).toEqual({ | ||
| cache_read_audio_mtok: 50, | ||
| cache_read_mtok: 150, | ||
| input_audio_mtok: 250, | ||
| input_mtok: 550, | ||
| }) | ||
| }) | ||
|
|
||
| it('unpriced audio stays in catch-all', () => { | ||
| const priced = new Set(['cache_read_mtok', 'input_mtok', 'output_mtok']) | ||
| const usage = { | ||
| cache_read_tokens: 200, | ||
| input_audio_tokens: 300, | ||
| input_tokens: 1000, | ||
| output_tokens: 500, | ||
| } | ||
| expect(computeLeafValues(priced, usage, family)).toEqual({ | ||
| cache_read_mtok: 200, | ||
| input_mtok: 800, | ||
| output_mtok: 500, | ||
| }) | ||
| }) | ||
|
|
||
| it('unpriced cache stays in catch-all', () => { | ||
| const priced = new Set(['input_mtok', 'output_mtok']) | ||
| const usage = { cache_read_tokens: 200, input_tokens: 1000, output_tokens: 500 } | ||
| expect(computeLeafValues(priced, usage, family)).toEqual({ | ||
| input_mtok: 1000, | ||
| output_mtok: 500, | ||
| }) | ||
| }) | ||
|
|
||
| it('negative leaf raises error', () => { | ||
| const priced = new Set(['cache_read_mtok', 'input_mtok']) | ||
| const usage = { cache_read_tokens: 200, input_tokens: 100 } | ||
| expect(() => computeLeafValues(priced, usage, family)).toThrow(/Negative leaf value.*input_mtok/) | ||
| }) | ||
|
|
||
| it('full 7-unit model', () => { | ||
| const priced = new Set([ | ||
| 'cache_read_audio_mtok', | ||
| 'cache_read_mtok', | ||
| 'cache_write_mtok', | ||
| 'input_audio_mtok', | ||
| 'input_mtok', | ||
| 'output_audio_mtok', | ||
| 'output_mtok', | ||
| ]) | ||
| const usage = { | ||
| cache_audio_read_tokens: 50, | ||
| cache_read_tokens: 200, | ||
| cache_write_tokens: 100, | ||
| input_audio_tokens: 300, | ||
| input_tokens: 1000, | ||
| output_audio_tokens: 150, | ||
| output_tokens: 800, | ||
| } | ||
| expect(computeLeafValues(priced, usage, family)).toEqual({ | ||
| cache_read_audio_mtok: 50, | ||
| cache_read_mtok: 150, | ||
| cache_write_mtok: 100, | ||
| input_audio_mtok: 250, | ||
| input_mtok: 450, | ||
| output_audio_mtok: 150, | ||
| output_mtok: 650, | ||
| }) | ||
| }) | ||
|
|
||
| it('accepts Usage object (optional fields)', () => { | ||
| const priced = new Set(['cache_read_mtok', 'input_mtok', 'output_mtok']) | ||
| const usage: Usage = { cache_read_tokens: 200, input_tokens: 1000, output_tokens: 500 } | ||
| expect(computeLeafValues(priced, usage as Record<string, unknown>, family)).toEqual({ | ||
| cache_read_mtok: 200, | ||
| input_mtok: 800, | ||
| output_mtok: 500, | ||
| }) | ||
| }) | ||
| }) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| import { describe, expect, it } from 'vitest' | ||
|
|
||
| import { FIELD_TO_UNIT, getFamily, getUnit, TOKENS_FAMILY } from '../units' | ||
|
|
||
| describe('Unit Registry', () => { | ||
| it('should load the tokens family', () => { | ||
| const family = getFamily('tokens') | ||
| expect(family.id).toBe('tokens') | ||
| expect(family.per).toBe(1_000_000) | ||
| }) | ||
|
|
||
| it('should have 20 token units', () => { | ||
| const family = getFamily('tokens') | ||
| expect(Object.keys(family.units)).toHaveLength(20) | ||
| }) | ||
|
|
||
| it('should look up a unit by ID', () => { | ||
| const unit = getUnit('input_mtok') | ||
| expect(unit.familyId).toBe('tokens') | ||
| expect(unit.usageKey).toBe('input_tokens') | ||
| expect(unit.dimensions).toEqual({ direction: 'input' }) | ||
| }) | ||
|
|
||
| it('should throw on unknown unit', () => { | ||
| expect(() => getUnit('nonexistent')).toThrow() | ||
| }) | ||
|
|
||
| it('should throw on unknown family', () => { | ||
| expect(() => getFamily('nonexistent')).toThrow() | ||
| }) | ||
|
|
||
| it('should have all 7 currently-used units', () => { | ||
| for (const unitId of [ | ||
| 'input_mtok', | ||
| 'output_mtok', | ||
| 'cache_read_mtok', | ||
| 'cache_write_mtok', | ||
| 'input_audio_mtok', | ||
| 'cache_read_audio_mtok', | ||
| 'output_audio_mtok', | ||
| ]) { | ||
| expect(TOKENS_FAMILY.units[unitId]).toBeDefined() | ||
| } | ||
| }) | ||
|
|
||
| it('should map cache_audio_read_mtok field to cache_read_audio_mtok unit', () => { | ||
| expect(FIELD_TO_UNIT.cache_audio_read_mtok).toBe('cache_read_audio_mtok') | ||
| }) | ||
|
|
||
| it('should have correct usage_key for cache_read_audio_mtok', () => { | ||
| const unit = getUnit('cache_read_audio_mtok') | ||
| expect(unit.usageKey).toBe('cache_audio_read_tokens') | ||
| }) | ||
|
|
||
| it('should validate all unit dimensions against family dimensions', () => { | ||
| const family = getFamily('tokens') | ||
| for (const unit of Object.values(family.units)) { | ||
| for (const [dimKey, dimVal] of Object.entries(unit.dimensions)) { | ||
| expect(family.dimensions[dimKey]).toBeDefined() | ||
| expect(family.dimensions[dimKey]).toContain(dimVal) | ||
| } | ||
| } | ||
| }) | ||
| }) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| import type { UnitDef, UnitFamily } from './units' | ||
|
|
||
| /** | ||
| * True if candidate's dimensions are a (non-strict) superset of ancestor's. | ||
| */ | ||
| export function isDescendantOrSelf(ancestor: UnitDef, candidate: UnitDef): boolean { | ||
| if (ancestor.familyId !== candidate.familyId) return false | ||
| return Object.entries(ancestor.dimensions).every(([k, v]) => candidate.dimensions[k] === v) | ||
| } | ||
|
|
||
| /** | ||
| * Get a usage value by key from a plain object. Returns 0 for missing/undefined/null. | ||
| */ | ||
| function getUsageValue(usage: Record<string, unknown>, key: string): number { | ||
| const val = usage[key] | ||
| return typeof val === 'number' ? val : 0 | ||
| } | ||
|
|
||
| /** | ||
| * Compute leaf values for each priced unit via Mobius inversion on the containment poset. | ||
| * | ||
| * Only priced units participate. Unpriced units' tokens stay in the nearest | ||
| * priced ancestor's catch-all. Throws on negative leaf values. | ||
| * | ||
| * Precondition: if a unit is priced, all its ancestors must also be priced | ||
| * (ancestor coverage rule). Violating this produces silently incorrect results. | ||
| */ | ||
| export function computeLeafValues(pricedUnitIds: Set<string>, usage: Record<string, unknown>, family: UnitFamily): Record<string, number> { | ||
| const result: Record<string, number> = {} | ||
|
|
||
| for (const unitId of pricedUnitIds) { | ||
| // eslint-disable-next-line @typescript-eslint/no-non-null-assertion | ||
| const unit = family.units[unitId]! | ||
| const targetDepth = Object.keys(unit.dimensions).length | ||
|
|
||
| let leafValue = 0 | ||
| for (const otherId of pricedUnitIds) { | ||
| // eslint-disable-next-line @typescript-eslint/no-non-null-assertion | ||
| const other = family.units[otherId]! | ||
| if (!isDescendantOrSelf(unit, other)) continue | ||
| const depthDiff = Object.keys(other.dimensions).length - targetDepth | ||
| const coefficient = (-1) ** depthDiff | ||
| leafValue += coefficient * getUsageValue(usage, other.usageKey) | ||
| } | ||
|
|
||
| if (leafValue < 0) { | ||
| const involved = [...pricedUnitIds] | ||
| .filter((oid) => { | ||
| // eslint-disable-next-line @typescript-eslint/no-non-null-assertion | ||
| const o = family.units[oid]! | ||
| return isDescendantOrSelf(unit, o) && getUsageValue(usage, o.usageKey) !== 0 | ||
| }) | ||
| .map((oid) => { | ||
| // eslint-disable-next-line @typescript-eslint/no-non-null-assertion | ||
| const o = family.units[oid]! | ||
| return `${o.usageKey}=${String(getUsageValue(usage, o.usageKey))}` | ||
| }) | ||
| throw new Error(`Negative leaf value (${String(leafValue)}) for ${unitId}: inconsistent usage values: ${involved.join(', ')}`) | ||
| } | ||
|
|
||
| result[unitId] = leafValue | ||
| } | ||
|
|
||
| return result | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.