Repository navigation
Kimi K3 vision: cap patch capacities at 16,384 and fit MoonViT resizing to the admitted capacity - #20
Open
tunamitom wants to merge 2 commits into
Conversation
PATCH_CAPACITIES 65,536 -> 16,384 (drop the top tier only): VisionInputs keeps PATCH_CAPACITIES[-1] x 588 f32 pixels and VisionScratch sizes its projected and attention slabs from the same maximum, so the dropped tier frees 320,274,432 B per rank (pixels 115,605,504 + projected 100,663,296 + attention_storage 100,663,296 + interpolation/merge indices 3,342,336) with no decode or prefill cost (vision graphs run only on image requests). The slabs are permanently reserved per rank on every deployment, so the top tier costs that memory even for text-only serving; what it buys is native resolution above about 1,790 x 1,790 px. The runtime derives max_patches_per_image from patch_capacities.last() (helios/src/server/config.rs image_limits). This commit is deliberately separable: alone, it refuses images between 16,385 and 65,536 patches (check_image_patches against the admitted tiers). The follow-up commit fits the frontend resizing to the admitted capacity so those images are downscaled instead of refused. Taking this commit alone keeps explicit refusal; taking both enables downscaling. The refuse-versus-downscale choice is presented to the maintainer in the PR body. The pixel buffer stays float32: the MoonViT frontend stages raw f32 bytes into vision_pixel_values (helios/src/frontend/images/packing.rs tensor_layout and spatial_merge), so a bf16 buffer would need a cross-language staging-ABI change. At this cap bf16 would save only 19,267,584 B per rank. Tests bind the new cap and its byte accounting (tests/recipes/test_kimi3_vision_tp.py). Validation: tests/recipes/test_kimi3_vision_tp.py on this tree (CPU). Not run in this pass: the describe-photo acceptance gate on the largest supported photo (no serving session), so the reduced tier is not validated against that gate. Material limitation: this reduces capability for every Kimi K3/K3.1 vision package, including packages already exported at 65,536. Design-Signed-Off-By: Hermes Agent (hermes-agent); design-review=a38f
extent() now takes the admitted max_patches_per_image (the image_limits bound derived from patch_capacities.last() in helios/src/server/config.rs) instead of a hard-coded 65,536 fit, so the frontend never resizes into a patch tier that was dropped. Rounding the fitted extent up to 28-alignment can cross the admission cap; in that case the extent shrinks to an aligned pair that fits, leaving already-admissible images untouched (the shrink loop terminates at 28 x 28). geometry() and resized_rgb() pass the same limit. Engine-wide behaviour change, applied to packages already exported: an image whose fitted geometry exceeded the admitted limit was refused explicitly (check_image_patches) and is now downscaled to fit, for every MoonViT package including ones whose live-feature bound is below 65,536. The model then sees a lower resolution than the 65,536-patch fit would have produced. This commit is kept separate from the capacity commit so that taking the capacity commit alone retains explicit refusal; the refuse-to-downscale choice is put to the maintainer in the PR body. Validation: cargo test --lib --features server,frontend (moonvit and image_limits tests) and tests/recipes/test_kimi3_vision_tp.py on this tree. Not run in this pass: a serving describe-photo request (the acceptance gate). Material limitation: images above the cap now get different (downscaled) model inputs than the 65,536-patch fit, so text and admissible images are numerics unchanged while larger images change. Design-Signed-Off-By: Hermes Agent (hermes-agent); design-review=a38f
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Maintainer decision requested first: refuse vs downscale
The second commit changes refuse -> downscale, engine-wide, for packages already exported. Before this change, an image whose fitted geometry exceeded the admitted patch limit was refused explicitly (
check_image_patches); after it, such an image is silently downscaled to fit — for every MoonViT package, including existing packages whose live-feature bound is below 65,536. The model then sees a lower resolution than the 65,536-patch fit would give it. That is exactly the "silently substitute / model semantics" class AGENTS.md reserves for an explicit maintainer decision. Please choose:What / why
PATCH_CAPACITIES 65,536 -> 16,384 (drop the top tier only) frees 320,274,432 B per rank (pixels + projected + attention slabs sized from the same maximum, permanently reserved on every deployment including text-only serving) with no decode/prefill cost — vision graphs run only on image requests. What it buys: native resolution above about 1,790 x 1,790 px. The second commit fits MoonViT resizing to the admitted capacity (
extent()takesmax_patches_per_imageinstead of a hard-coded 65,536; already-admissible images are untouched). The pixel buffer stays float32 (the MoonViT frontend stages raw f32 bytes; bf16 would need a cross-language staging-ABI change and would save only 19,267,584 B/rank here).Commits:
Cap Kimi K3 vision patch capacities at 16,384+Fit MoonViT resizing to admitted patch capacity.Numerics class
NUMERICS-NEUTRAL for text and admissible images (native geometry unchanged, asserted by
fitted_geometry_never_exceeds_admission_cap); images above the cap are now downscaled instead of refused / served at 65,536, so their model input differs from the old fit.Tests
tests/recipes/test_kimi3_vision_tp.py— 45 passed (this tree).cargo test --lib --features server,frontend—moonvit5 passed +image_limits1 passed (this tree; the plaincargo test --libsilently runs none of these).tests/recipes/test_kimi31.py25 andtests/cli/test_models.py40 passed;tests/cli641 passed, same 2 pre-existing failures as the base.Dependencies: none on the companion FlashInfer PRs.