Skip to content

fix: use ltx_t2v_pure.json workflow for T2V — builder produces still images#2

Open
NotMainstream wants to merge 5 commits into
AEON-7:mainfrom
NotMainstream:fix/t2v-workflow
Open

fix: use ltx_t2v_pure.json workflow for T2V — builder produces still images#2
NotMainstream wants to merge 5 commits into
AEON-7:mainfrom
NotMainstream:fix/t2v-workflow

Conversation

@NotMainstream

Copy link
Copy Markdown

ROOT CAUSE: The builder (build_ltx_i2v_workflow) generates a fundamentally different pipeline than the working ltx_t2v_pure.json workflow.

Builder (broken T2V)

EmptyLatentVideo → RandomNoise → KSamplerSelect → BasicScheduler → CFGGuider → SamplerCustomAdvanced → VAEDecode → Video

The builder has NO I2V conditioning infrastructure. At very low denoising with no conditioning, diffusion produces a static photograph.

Working ltx_t2v_pure.json

LoadImage → ImageScale → LTXVPreprocess → LTXVImgToVideoConditionOnly(bypass=True) → Sampler → LTXVLatentUpsampler → VAEDecode → Video

The working workflow has full I2V infrastructure (nodes 3159, 4970) with bypass=True. When bypass=True, the node participates in the graph but with zero image influence — properly initializing temporal evolution.

Changes

  • Add _load_t2v_workflow() that loads ltx_t2v_pure.json and patches it
  • clip command now uses _load_t2v_workflow for T2V mode instead of builder
  • Add workflows/api/ directory with ltx_t2v_pure.json, ltx_i2v_api.json, ltx_t2v_vbvr.json
  • Also includes prior fixes: _SEP=os.sep, MAX_CLIP_SECONDS=10.0, text_encoder path, quality LoRA path

AEON-7 and others added 5 commits May 1, 2026 15:42
- .github/FUNDING.yml -> Sponsor button at top of repo page
- README header: ☕ Tips badge linking to AEON-7 profile tip section
- README footer: full tip jar block (BTC/ETH/SOL/XMR with QR codes)

QR images sourced from AEON-7/AEON-7 profile repo (single source of truth).
- .env.example: every variable now has a labeled 'how to use' section
  with where to get the value (HF tokens, Civitai tokens, etc.)
- README.md: new 'Configuration' section explicitly walking through
  Mode A (local) vs Mode B (remote ComfyUI), with sub-options for
  direct HTTP vs SSH tunnel vs SSH-invoked CLI
- AGENTS.md: 'Step 0 — Determine execution mode' lets agents figure
  out whether to invoke directly or wrap commands in ssh
- aeon-movie-maker specifically documents the I2V/screenplay frame-
  staging constraint: pure-HTTP remote mode doesn't work for those,
  needs SSH-invoked CLI or shared-filesystem mount
sync.sh now:
- Detects local uncommitted changes and offers to stash + re-apply
- Shows a diff preview (commit list + files-changed) before pulling
- Asks for confirmation unless --yes
- Supports --dry-run, --yes, --no-models, --help
- cd's to script dir so it works from any cwd
- Resolves origin/main vs origin/master automatically
- Reports n commits + n files changed; pulls --ff-only

README adds 'Updating an existing install' section that documents:
- The basic upgrade flow (cd repo && ./sync.sh)
- Each flag's purpose
- The fact that .env / output / __pycache__ / models are gitignored
  and never touched by sync

For users who cloned at v1, this is the canonical upgrade path.
…images

ROOT CAUSE: The builder (build_ltx_i2v_workflow) generates a fundamentally
different pipeline than the working ltx_t2v_pure.json workflow.

Builder pipeline (BROKEN T2V):
  EmptyLatentVideo → RandomNoise → KSamplerSelect → BasicScheduler
  → CFGGuider → SamplerCustomAdvanced → VAEDecode → Video

The builder has NO I2V conditioning infrastructure. At very low denoising
with no conditioning, diffusion produces a static photograph.

Working ltx_t2v_pure.json pipeline:
  LoadImage → ImageScale → LTXVPreprocess → LTXVImgToVideoConditionOnly(bypass=True)
  → Sampler → LTXVLatentUpsampler → VAEDecode → Video

The working workflow has full I2V infrastructure (nodes 3159, 4970) with
bypass=True. When bypass=True, the node participates in the graph but
with zero image influence — properly initializing temporal evolution.

Changes:
- Add _load_t2v_workflow() that loads ltx_t2v_pure.json and patches it
- clip command now uses _load_t2v_workflow for T2V mode instead of builder
- Add workflows/api/ directory with ltx_t2v_pure.json, ltx_i2v_api.json,
  ltx_t2v_vbvr.json (proven working workflow JSONs)
- Fix duration print statement to use correct node ID per workflow type
- Also includes prior fixes: _SEP=os.sep, MAX_CLIP_SECONDS=10.0,
  text_encoder path, quality LoRA path (from fix/model-paths-vbvr-ic-lora)
@AEON-7

AEON-7 commented May 4, 2026

Copy link
Copy Markdown
Owner

Thanks for the deep dive on this — the diagnosis that the builder's T2V path produces still images is plausible and worth fixing. But this PR mixes the architectural fix with several unrelated, concerning changes, and the workflow JSONs need scrubbing before they can ship in this repo. I'm going to leave it open without merging while we sort these out, and would love a focused follow-up PR on just the T2V fix once these are addressed.

Blockers (must fix before merge)

1. Workflow JSONs ship embedded prompts + non-zero seeds + wrong checkpoint name

Auditing workflows/api/ltx_t2v_pure.json:

Issue Found
Embedded positive prompt (node 2483 + 2612) "Starting from the spiraling mahogany walkway. 0-6s: The bookish adventurer winds the key o..." — looks like a personal project's narrative
Embedded negative prompt (node 4981) "pc game, console game, video game, cartoon, childish, ugly"
Non-zero noise_seed nodes 4832 and 4967 set noise_seed=43 and 42 (would override --seed from the CLI)
Hardcoded checkpoint that doesn't match repo defaults ckpt_name: "ltx-2.3-22b-dev.safetensors" referenced 3 times — but our MODES["fast"]["checkpoint"] is ltx-2.3-22b-distilled-fp8.safetensors. Inconsistency between the JSON and the new _load_t2v_workflow() caller.

Per the repo conventions (_tools/scrub.py in the staging tree — same pattern used to sanitize all our workflow JSONs), prompts must be empty strings, seeds must be 0, and filename_prefix must be "output" so user-supplied values fully control runtime behavior.

2. Text encoder filename change loses abliteration silently

The diff renames the text encoder reference:

- "text_encoder": "gemma-3-12b-abliterated-text-encoder.safetensors",
+ "text_encoder": "comfy_gemma_3_12B_it.safetensors",

Two problems:

  • It loses abliteration. The original config used the abliterated single-file encoder. The new filename is the standard non-abliterated Gemma. If you want to switch to the Comfy-Org split layout, the canonical equivalent is gemma_3_12B_it.safetensors (no comfy_ prefix — that's what comfyui-aeon-spark/download_models.py writes), with the abliterated LoRA (gemma-3-12b-it-abliterated_heretic_lora_rank64_bf16.safetensors) applied on top to preserve uncensored capability.
  • comfy_gemma_3_12B_it.safetensors doesn't match anything. Neither comfyui-aeon-spark nor the original aeon-movie-maker/setup.sh produces that filename. It looks like a local rename specific to your setup — it would break for anyone else running the documented workflow.

This is a non-trivial behavioral change. Should be its own PR with explicit user discussion and the abliterated LoRA wired in correctly.

3. Distill LoRA path rename has the same issue

- (f"ltx2{_SEP}ltx-2.3-22b-distilled-lora-384.safetensors", 0.5),
+ (f"ltxv{_SEP}ltx2{_SEP}ltx-2.3-22b-distilled-lora-384-1.1.safetensors", 0.5),

The added ltxv/ directory prefix and -1.1 suffix don't match comfyui-aeon-spark/download_models.py (which puts it at loras/ltx2/ltx-2.3-22b-distilled-lora-384.safetensors). Same problem — looks like your local setup has a different layout.

Concerns (please address but not strict blockers)

4. PR mixes unrelated changes — please split

This single PR contains:

  1. (main claim) T2V workflow architectural fix via _load_t2v_workflow + new JSONs
  2. File rename for text encoder (fix: use ltx_t2v_pure.json workflow for T2V — builder produces still images #2 above)
  3. File rename for distill LoRA (SEQ FAILED: 'foo.mp4' and 'foo.mp4' are the same file #3 above)
  4. _SEP = "\\"_SEP = os.sep (cross-platform, probably good — but unrelated)
  5. MAX_CLIP_SECONDS = 7.010.0 (LTX 2.3 coherence empirically degrades past ~8s — this might produce worse output)

Each of these is a separate decision. Mixing them makes the PR hard to evaluate. Splitting into focused PRs would let me merge the safe ones quickly and discuss the others independently.

5. T2V diagnosis could use a minimal repro

The claim "builder produces still images for T2V" is plausible — the t2v=True branch drops LoadImage and LTXVImgToVideoInplace but keeps the noise sampler. There may be a subtle conditioning bug (ConditioningZeroOut of an empty conditioning vs the proper zero-image-influence pattern). Could you add a minimal failing-output gif/mp4 + the exact CLI command that produced it? That makes the architectural rewrite easier to justify.

Path forward — suggested split

If you're up for re-splitting, this would let us merge most of it fast:

Fix Suggested PR
_SEP = os.sep cross-platform fix PR-A — instant merge, 2-line change
MAX_CLIP_SECONDS justification PR-B — needs evidence that 10s clips are actually coherent on LTX 2.3
Text encoder canonical naming (gemma_3_12B_it.safetensors + abliterated LoRA on top) PR-C — would also need a corresponding update to comfyui-aeon-spark/download_models.py if not already covered
Distill LoRA path canonical naming PR-D — needs to match comfyui-aeon-spark/download_models.py exactly
T2V workflow fix PR-E (the main one) — scrubbed JSON (zero prompts, zero seeds, output prefix) + minimal repro of the still-image bug + canonical filenames throughout

We have a sanitizer at _tools/scrub.py (in the aeon-media-staging/ tree) that I run on workflow JSONs before they ship — happy to run it on your JSONs and post the cleaned versions back if that's helpful.

Thanks again for the diagnosis work. T2V producing static frames if true is a real bug we want fixed properly — just want to do it without the collateral changes that would land in this same merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants