Qwen3-Coder-Next long-context LoRA fine-tuning example

[rcano-baseten]

Summary

• Adds qwen3-80b-msswift example for fine-tuning Qwen3-Coder-Next (80B MoE) with LoRA using MS-Swift + MegatronLM
• Default config: 1 node, 8x H200, 32K sequence length, ep=8, LoRA rank 8
• README includes tested configurations table covering 16K–128K sequence lengths across 1–4 node setups
• Uses flash-linear-attention, optimizer CPU offload, and activation recomputation for memory efficiency

Tested Configurations

Nodes	GPUs	Seq Length	TP	PP	EP	Peak Memory	Status
1	8	16K	—	—	8	105 GiB	✅
1	8	32K	—	—	8	121 GiB	✅
2	16	64K	2	2	4	106 GiB	✅
4	32	48K	2	—	16	98 GiB	✅
4	32	64K	2	2	4	118 GiB	✅
4	32	128K	2	4	4	134 GiB	✅

Supersedes #86.

Test plan

• All configurations above verified end-to-end (training + checkpoint save)
• Verify truss train init --examples qwen3-80b-msswift works after merge

🤖 Generated with Claude Code

[ervinwang-jpg]

lgtm