Skip to content

docs: add Halo fine-tuning guide for LFM MoE - #131

Open
advpropsys wants to merge 2 commits into
Liquid4All:mainfrom
advpropsys:docs/halo-lfm-fine-tuning
Open

advpropsys wants to merge 2 commits into
Liquid4All:mainfrom
advpropsys:docs/halo-lfm-fine-tuning

Conversation

@advpropsys

@advpropsys advpropsys commented Sep 20, 2026

Copy link
Copy Markdown

Summary

Adds lfm/fine-tuning/halo.mdx, a guide to training LFM MoE models with Halo, White Circle's open-source distributed training framework. Halo adds multi-GPU and multi-node parallelism to the native Hugging Face workflow while models remain standard transformers modules and checkpoints remain loadable with from_pretrained.

The guide:

  • explains how one LFM integration and YAML structure carry across SFT, LoRA, GRPO, and environmental GRPO
  • documents FSDP, EP, TP, ETP, and EP+TP support for LFM MoE models
  • includes the published single-B300 LFM2.5 comparison: 31,262 vs 28,330 tok/s/GPU at 8K and 35,517 vs 29,520 at 16K, against Axolotl
  • provides a runnable EP2 SFT path for LFM2.5-8B-A1B
  • links to the complete Halo LFM2 cookbook for larger models, LoRA, inference, and RL
  • adds Halo to the fine-tuning overview and navigation

Validation

  • npm run snapshot:check
  • verified the external Halo, White Circle, and model links

@advpropsys
advpropsys force-pushed the docs/halo-lfm-fine-tuning branch from a791c97 to afc85c0 Compare September 20, 2026 14:03
@advpropsys
advpropsys force-pushed the docs/halo-lfm-fine-tuning branch from afc85c0 to 546b2e1 Compare September 20, 2026 14:13
@iamleonie
iamleonie requested a review from mlabonne as a code owner September 21, 2026 09:39

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants