Skip to content

feat(stt): Add a Nemotron 3.5 ASR Finetuning Recipe - #122

Open
iancarrasco-b10 wants to merge 5 commits into
mainfrom
ian/nemotron-asr
Open

feat(stt): Add a Nemotron 3.5 ASR Finetuning Recipe#122
iancarrasco-b10 wants to merge 5 commits into
mainfrom
ian/nemotron-asr

Conversation

@iancarrasco-b10

@iancarrasco-b10 iancarrasco-b10 commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

What

Adds a new training example, examples/nemotron-asr-nemo/, that fine-tunes NVIDIA's Nemotron 3.5 ASR streaming model (600M-param, multilingual, Cache-Aware FastConformer-RNNT) on Baseten.

Ported from NVIDIA's official fine-tuning recipe, following the structure/Truss conventions of the existing whisper-transformers example. Unlike Whisper, this is a NeMo recipe: it fine-tunes via NeMo's speech_to_text_finetune.py (init_from_nemo_model) using the streaming-prompt FastConformer config, driven by target_lang-tagged manifests.

Files

  • config.py — Truss training job (NeMo NGC base image, 1x H100, hf_access_token secret, cache + checkpointing)
  • run.sh — installs deps, clones NeMo, downloads the base .nemo checkpoint, preps data, runs the fine-tune into $BT_CHECKPOINT_DIR
  • prepare_data.py — builds NeMo JSON-lines manifests (AN4 as a quick smoke test) with target_lang tags; template for bringing your own data
  • requirements.txt, README.md

Notes

  • Uses AN4 as an end-to-end smoke test; README documents the real path (own target_lang-tagged, punctuated manifests, tarred Lhotse shards, step budgets, replay for multilingual protection).
  • Evaluation guidance uses NeMo's streaming inference at deployment latency (att_context_size).
  • NEMO_BRANCH pinned to main since the streaming-prompt config is recent — pin to a release tag once available.

@iancarrasco-b10 iancarrasco-b10 changed the title feature(stt): Add a Nemotron 3.5 ASR Finetuning Recipe feat(stt): Add a Nemotron 3.5 ASR Finetuning Recipe Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant