Summary
Run a time-boxed technical spike against Apolinario's upstream PyTorch port of Magenta RealTime 2 and determine whether LSDJ can consume it as a pinned external dependency on Linux and Windows.
The intended outcome is not an LSDJ-maintained fork. Prefer a versioned upstream package or tag; if one is not available, pin an immutable upstream VCS revision. LSDJ should own only a thin adapter around the upstream API.
Relationship to other work
Questions to answer
Dependency and packaging
- Can the implementation be installed reproducibly from an immutable upstream release, package, tag, or commit?
- What Python, PyTorch, CUDA, audio, and tokenizer dependencies are required on Ubuntu 22.04+ and Windows 11?
- Can it be packaged without requiring users to install Python, Git, a CUDA toolkit, or a shell?
- Are there import-time downloads or mutable/unpinned transitive assets?
- What upstream release/versioning commitment is needed before production use?
Functional parity
Map LSDJ's current MRT2 controls and state to the upstream port:
- prompt and negative prompt behavior;
- temperature, top-k, guidance/CFG, and seed semantics;
- steering/conditioning inputs;
- model selection, reset, and warm-up;
- overlap/context/chunk handling;
- deterministic behavior where the current backend exposes it;
- stereo output, sample rate, and continuity across generated chunks.
Document every semantic mismatch and whether it belongs in the adapter, requires an upstream contribution, or blocks adoption.
Service topology
Compare at least:
- one long-lived worker owning both deck models; and
- one worker per deck.
Measure startup time, steady-state memory/VRAM, failure isolation, shutdown behavior, and the cost of model duplication. Recommend one topology for production and explain the tradeoff.
Real-time performance
Exercise the actual scheduling modes used by LSDJ:
- normal generation: 25 frames / approximately 1 second;
- low-latency generation with both decks armed: 5 frames / approximately 200 ms.
Include the current approximately 1.5-second playback-ring behavior in the harness so the result reflects the real underrun budget rather than isolated model throughput.
Test cold start, warm-up, sustained two-deck generation, prompt/control changes, and at least 10 minutes of continuous operation. Record generated-audio time versus wall time, p50/p95/p99 latency, engine-reported underruns, memory/VRAM, and thermal/throttling observations.
Hardware and acceleration
- Validate Linux and Windows on representative supported NVIDIA hardware.
- Record exact GPU, VRAM, driver, PyTorch, and CUDA runtime versions.
- Confirm the port uses the intended accelerator and fails clearly when no supported accelerator exists.
- Identify the lowest defensible support floor for the production issues.
Deliverables
- A reproducible benchmark harness or fixture that can be reused by the production backend and platform smoke tests.
- A control/parity matrix against the current MLX backend.
- A dependency/provenance inventory with immutable upstream references.
- A packaging feasibility note for both target operating systems.
- A topology recommendation.
- A clear go/no-go recommendation, with upstream issues or contributions listed for any gaps.
Spike completion criteria
The spike is complete even if the answer is no-go when all of the following are published:
- reproducible environment and hardware details;
- results for both the 25-frame and 5-frame modes with two decks;
- p50/p95/p99 latency, underrun, memory, and VRAM data;
- functional parity matrix;
- packaging and dependency findings;
- recommended topology and explicit blockers.
Go criteria for production integration
- Both real-time modes sustain two decks for 10 minutes without engine-reported underruns on the proposed minimum supported hardware.
- Required LSDJ controls have matching semantics or an agreed adapter/upstream path.
- The dependency can be pinned immutably and installed without an LSDJ-maintained fork.
- The runtime can be delivered without system Python, Git, shell tooling, or a user-installed CUDA toolkit.
- Process startup, readiness, crash, and shutdown behavior can be supervised reliably on Linux and Windows.
Out of scope
- Shipping the production backend.
- Maintaining a fork of the upstream port.
- Changing the macOS MLX backend.
- Final user-facing license and acknowledgement UX.
Follow-up
Summary
Run a time-boxed technical spike against Apolinario's upstream PyTorch port of Magenta RealTime 2 and determine whether LSDJ can consume it as a pinned external dependency on Linux and Windows.
The intended outcome is not an LSDJ-maintained fork. Prefer a versioned upstream package or tag; if one is not available, pin an immutable upstream VCS revision. LSDJ should own only a thin adapter around the upstream API.
Relationship to other work
Questions to answer
Dependency and packaging
Functional parity
Map LSDJ's current MRT2 controls and state to the upstream port:
Document every semantic mismatch and whether it belongs in the adapter, requires an upstream contribution, or blocks adoption.
Service topology
Compare at least:
Measure startup time, steady-state memory/VRAM, failure isolation, shutdown behavior, and the cost of model duplication. Recommend one topology for production and explain the tradeoff.
Real-time performance
Exercise the actual scheduling modes used by LSDJ:
Include the current approximately 1.5-second playback-ring behavior in the harness so the result reflects the real underrun budget rather than isolated model throughput.
Test cold start, warm-up, sustained two-deck generation, prompt/control changes, and at least 10 minutes of continuous operation. Record generated-audio time versus wall time, p50/p95/p99 latency, engine-reported underruns, memory/VRAM, and thermal/throttling observations.
Hardware and acceleration
Deliverables
Spike completion criteria
The spike is complete even if the answer is no-go when all of the following are published:
Go criteria for production integration
Out of scope
Follow-up