Fix DFlash sglang target backend for sglang 0.5.13+ scheduler/batch API#664
Fix DFlash sglang target backend for sglang 0.5.13+ scheduler/batch API#664violet73 wants to merge 2 commits into
Conversation
Updated the dflash_target_model.py to replace deprecated methods and adjust input handling for batch processing.
There was a problem hiding this comment.
Code Review
This pull request updates the dflash_target_model.py file to maintain compatibility with sglang 0.5.13+. Key changes include bypassing the scheduler's PrefillAdder to manually initialize inputs and extend ranges per request, migrating to the module-level prepare_mlp_sync_batch_raw function, handling host-to-device copying of prefill input IDs, and passing ScheduleBatch directly to ForwardBatch.init_new. Additionally, Req.fill_ids has been replaced with full_untruncated_fill_ids using an array. The reviewer recommends wrapping origin_input_ids in an array("q", ...) to prevent a potential runtime TypeError during execution.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| @@ -204,8 +226,12 @@ def generate_dflash_data( | |||
| origin_input_ids=curr_ids.view(-1).tolist(), | |||
There was a problem hiding this comment.
In sglang 0.5.13+, Req.origin_input_ids is expected to be an array of type 'q' (64-bit signed integers) [5.1.1]. Passing a standard Python list of integers can cause a TypeError during execution, specifically in Req._refresh_fill_ids when concatenating self.origin_input_ids + self.output_ids (since self.output_ids is an array("q") and Python does not allow concatenating a list and an array). Wrapping curr_ids.view(-1).tolist() in array("q", ...) ensures type safety and prevents runtime crashes.
| origin_input_ids=curr_ids.view(-1).tolist(), | |
| origin_input_ids=array("q", curr_ids.view(-1).tolist()), |
Updated the dflash_target_model.py to replace deprecated methods and adjust input handling for batch processing.
Motivation
The DFlash sglang target backend (
SGLangDFlashTargetModelinspecforge/modeling/target/dflash_target_model.py) still uses the pre-0.5.13Req/ScheduleBatchscheduler API. On current sglang (0.5.13/0.5.14) an onlineDFlash run crashes at the first
generate_dflash_datacall, before any trainingstep. This PR updates the request/batch construction to the new API, mirroring
the changes already made to the EAGLE3 target backend (
eagle3_target_model.py).Modifications
All in
specforge/modeling/target/dflash_target_model.py(
SGLangDFlashTargetModelonly; thehfbackend is untouched):from array import array; import the module-levelprepare_mlp_sync_batch_rawfromsglang.srt.managers.scheduler_components.dp_attn; drop the now-unusedfrom sglang.srt.managers.scheduler import Scheduler.generate_dflash_data— Req construction: setfull_untruncated_fill_ids/fill_len/extend_input_len/logprob_start_leninstead of the removedfill_ids._extend— batch prep: per req callinit_next_round_input(tree_cache)then
set_extend_range(len(prefix_indices), len(full_untruncated_fill_ids))(no prefix-cache reuse); capture
input_lensbeforeprepare_for_extend._extend— mlp sync: call the module-levelprepare_mlp_sync_batch_rawwith
attn_cp_size, without the removedspec_algorithm/speculative_num_draft_tokens._extend— forward: materializeprefill_input_ids_cputo device, setbatch.capture_hidden_mode, and callForwardBatch.init_new(batch, runner)directly (drop
get_model_worker_batch()).These mirror the equivalent, already-merged handling in
eagle3_target_model.py,so the two backends stay consistent.
Related Issues
Accuracy Test
Benchmark & Profiling
Checklist