Skip to content

Fix DFlash sglang target backend for sglang 0.5.13+ scheduler/batch API#664

Open
violet73 wants to merge 2 commits into
sgl-project:mainfrom
violet73:main
Open

Fix DFlash sglang target backend for sglang 0.5.13+ scheduler/batch API#664
violet73 wants to merge 2 commits into
sgl-project:mainfrom
violet73:main

Conversation

@violet73

@violet73 violet73 commented Jul 8, 2026

Copy link
Copy Markdown

Updated the dflash_target_model.py to replace deprecated methods and adjust input handling for batch processing.

Motivation

The DFlash sglang target backend (SGLangDFlashTargetModel in
specforge/modeling/target/dflash_target_model.py) still uses the pre-0.5.13
Req/ScheduleBatch scheduler API. On current sglang (0.5.13/0.5.14) an online
DFlash run crashes at the first generate_dflash_data call, before any training
step. This PR updates the request/batch construction to the new API, mirroring
the changes already made to the EAGLE3 target backend (eagle3_target_model.py).

Modifications

All in specforge/modeling/target/dflash_target_model.py
(SGLangDFlashTargetModel only; the hf backend is untouched):

  • Imports: add from array import array; import the module-level
    prepare_mlp_sync_batch_raw from
    sglang.srt.managers.scheduler_components.dp_attn; drop the now-unused
    from sglang.srt.managers.scheduler import Scheduler.
  • generate_dflash_data — Req construction: set
    full_untruncated_fill_ids / fill_len / extend_input_len /
    logprob_start_len instead of the removed fill_ids.
  • _extend — batch prep: per req call init_next_round_input(tree_cache)
    then set_extend_range(len(prefix_indices), len(full_untruncated_fill_ids))
    (no prefix-cache reuse); capture input_lens before prepare_for_extend.
  • _extend — mlp sync: call the module-level prepare_mlp_sync_batch_raw
    with attn_cp_size, without the removed spec_algorithm /
    speculative_num_draft_tokens.
  • _extend — forward: materialize prefill_input_ids_cpu to device, set
    batch.capture_hidden_mode, and call ForwardBatch.init_new(batch, runner)
    directly (drop get_model_worker_batch()).

These mirror the equivalent, already-merged handling in eagle3_target_model.py,
so the two backends stay consistent.

Related Issues

Accuracy Test

Benchmark & Profiling

Checklist

violet73 added 2 commits July 8, 2026 16:59
Updated the dflash_target_model.py to replace deprecated methods and adjust input handling for batch processing.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the dflash_target_model.py file to maintain compatibility with sglang 0.5.13+. Key changes include bypassing the scheduler's PrefillAdder to manually initialize inputs and extend ranges per request, migrating to the module-level prepare_mlp_sync_batch_raw function, handling host-to-device copying of prefill input IDs, and passing ScheduleBatch directly to ForwardBatch.init_new. Additionally, Req.fill_ids has been replaced with full_untruncated_fill_ids using an array. The reviewer recommends wrapping origin_input_ids in an array("q", ...) to prevent a potential runtime TypeError during execution.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

@@ -204,8 +226,12 @@ def generate_dflash_data(
origin_input_ids=curr_ids.view(-1).tolist(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

In sglang 0.5.13+, Req.origin_input_ids is expected to be an array of type 'q' (64-bit signed integers) [5.1.1]. Passing a standard Python list of integers can cause a TypeError during execution, specifically in Req._refresh_fill_ids when concatenating self.origin_input_ids + self.output_ids (since self.output_ids is an array("q") and Python does not allow concatenating a list and an array). Wrapping curr_ids.view(-1).tolist() in array("q", ...) ensures type safety and prevents runtime crashes.

Suggested change
origin_input_ids=curr_ids.view(-1).tolist(),
origin_input_ids=array("q", curr_ids.view(-1).tolist()),

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant