Skip to content

WIP: plan decode CUDA graph integration - #182

Draft
drunkcoding wants to merge 1 commit into
mainfrom
plan/decode-cuda-graphs
Draft

WIP: plan decode CUDA graph integration#182
drunkcoding wants to merge 1 commit into
mainfrom
plan/decode-cuda-graphs

Conversation

@drunkcoding

Copy link
Copy Markdown
Contributor

Description

Plan-only design for opt-in decode CUDA graph replay using one authoritative paged-KV store.

Motivation

CudaGraphRunner exists but is not integrated and currently bypasses required serving metadata and lifecycle behavior.

Scope

  • Resident Qwen3 GQA with native paged attention only
  • Explicit graph-safety capability and eager fallback
  • Allocation-free per-layer decode KV writes
  • Capture/replay lifecycle, shutdown, and benchmark fixture

Dependencies / limitations

Offloaded MoE, FlashInfer planning, MLA, prefill, and sampling remain eager/unsupported in the first rollout.

Type of Change

  • Documentation update
  • Production implementation

Performance / support evidence

  • Not applicable — raw launch/latency qualification is planned without a speedup requirement.

Checklist

  • Graph-unsafe paths fail closed
  • Memory accounting and rollback specified
  • No production files changed

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant