Skip to content

Add optional Neural Rendering with pipeline controls and residual upscaling - #1158

Open
wilsjo2 wants to merge 19 commits into
optiscaler:masterfrom
wilsjo2:codex/upstream-neural-rendering
Open

wilsjo2 wants to merge 19 commits into
optiscaler:masterfrom
wilsjo2:codex/upstream-neural-rendering

Conversation

@wilsjo2

@wilsjo2 wilsjo2 commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

This adds optional Neural Rendering before/after the game's upscaler or on the finished picture. It defaults off and runs inside OptiScaler; users supply NVIDIA's NR runtime separately.

Features

  • A pipeline menu for placement, model passes/resolution, strength, skin/environment and HDR/exposure controls.
  • Early generation with later application: reconstruct the clean game image and upscale NR's edit separately. Source RR adds motion-vector accumulation; placement stays manual.
  • Classic, Matched residual and Matched residual + DLSS enlargement. The last enlarges the edit over the original full-resolution image.
  • Frame hold, comparisons, capture and GPU timing.
  • Shared D3D12 processing for native games and bridges; a separate native Vulkan pipeline. Private enlargement requires D3D12. Private RR needs compatible native guides; post/finished enlargement uses SR.

Fork MFG unlocks, external-FG mode, residual interpolation, NVFP4 selection and helper DLLs are removed. Official FG/MFG remains.

Review

Based on master 5ee53e38, including compatibility prerequisite #1157 until it lands. Other bridge/swapchain/lifetime changes share NR integration points; the file inventory explains their purpose. Attribution to Dagherbou, hhkbble, y4my4my4m, cmh1448 and RenoDX remains in the docs.

Commit-by-commit review guide — 17 NR commits

Each commit builds Release x64. The two prerequisite commits from #1157 are unchanged. Counts below exclude generated shader headers and include tests/docs.

Commit Focus Added lines
960ca493 Add shared shader routing and private texture scaling helpers 684
2743e413 Add NR settings, driver adapter and GPU lifetime tracking 2,170
4af597eb Add NR colour codecs, shader contracts and GPU regressions 3,308
fbf104d9 Add exposure discovery, readback and calibration anchors 1,261
0de77ac4 Add independent NR upscaler adapters and runtime smoke tests 876
c16a98e7 Implement DX12 NR resource ownership and command completion 3,883
6c52a033 Connect DX12 pre/post NR with model passes and guide restoration 1,409
30b0471e Route D3D11 DLSS and NR through the shared DX12 pipeline 184
98e7a395 Carry early NR edits through a private upscaler 551
470e8c6c Accumulate RR residuals with motion and select compatible private RR 140
bc02e0d7 Apply NR to finished DX12 and D3D11 pictures before presentation 1,202
62c528b4 Enlarge matched post-upscale NR residuals with private DLSS SR 151
9ac11d4e Add held NR inputs and gameplay/photo pipeline capture checks 244
82ff9b0f Integrate native Vulkan NR and finished-picture presentation 3,460
944c55e2 Expose NR controls through the render pipeline menu 1,541
a3363e05 Package NR and document review scope, setup and game results 737

| dc5611d6 | Serialize NR recording/reset/cleanup to fix the Starfield tracking crash | 72 |

Generated shader headers remain with their HLSL sources and are marked for GitHub to collapse. The original 16-commit NR series preserves the previous PR's runtime source, shaders and tests. The final follow-up fixes the Starfield tracking race and adds its regression.

Games checked

Windows, RTX 5090, driver 616.64. These are short runs across integration builds, with existing saves/settings preserved:

Game Observations
BG3, DX11 -> D3D12 DLSS Main placements, passes, reduced resolution, enlargement and deferred hold; clean follow-up exercised post/finished private DLSS. No crash observed.
KCD2, D3D12 SR Placements, post/finished enlargement and hold. Precision fix removed dense speckles; faint grid and deferred brightness/banding remain.
Jedi: Survivor, D3D12 SR Fixed invalid depth SRV/device removal. Main placements and reduced finished-picture modes ran without that crash.
Hogwarts Legacy, D3D12 RR Placements, enlargement, passes, hold/comparison across runs. Clean follow-up completed without a crash; possible grid unresolved.
Cyberpunk 2077 User reported the submission-fixed build working well. Clean DLL installed; no independent all-mode check this round.

All five received the clean integration DLL. This review branch additionally includes newer upstream DX11 swapchain adaptation, checked by build/regressions. Exact builds, settings and limits distinguish that coverage.

Every NR commit built Release x64; the final head also passed a clean rebuild, alongside production lifetime/capture/proxy, WARP swapchain and WARP/Vulkan precision checks. Shader binaries match; packaging and documentation links pass. The rewritten 18-commit range passed a secret scan.

Known limitations: HDR/temporal/grid quality needs work. BG3's follow-up reported FG OFF/Reflex inactive; FG/MFG was not comprehensively tested. Long-session VRAM stability, native Vulkan gameplay and broader GPU/driver/mode coverage remain unverified.

Starfield crash fix

The v0.8.0 report points to a null recording entry during command-list reset. The ordinary buffer-transition path could modify that tracker without the lock held by reset notifications, including after NR initialization failed. Tracker operations now serialize internally; the surrounding resource entry points follow the same lock order. No GPU wait or helper DLL was added.

A four-worker WARP test crashes with the old tracker and passes with the fix, with all 8,000 retirement callbacks delivered exactly once. Release x64 and the existing proxy/lifetime/capture regressions passed. Starfield gameplay still needs tester confirmation; this does not resolve the separate 0xBAD0000B runtime-initialization error. No fork MFG unlock or build-flag changes are included in this PR.

Resolve DLSSG, Reflex and PCL functions after device selection so driver-selected plugins supply initialized entry points. Preserve the depth-plane SRV format and check the Vulkan framebuffer handle during teardown.

Vulkan framebuffer guard incorporates y4my4my4m work (7b7220b / 7c3b65d). Validation: Release x64 build and installed-runtime Streamline active-plugin smoke test; Jedi depth-view diagnosis and NR gameplay retest recorded separately.
@wilsjo2
wilsjo2 marked this pull request as ready for review September 12, 2026 22:41
Validation: Release x64 solution build passed.
NGX routing and real WARP window/composition swapchain regressions passed.
Validation: Release x64 solution build passed.
NGX proxy and production WARP GPU-lifetime regressions passed.
Validation: Release x64 solution build passed.
Active-region, guide, timing, skin, HDR, signed-residual, RR reprojection and seam regressions passed on CPU/WARP as applicable.
Validation: Release x64 solution build passed.
Validation: Release x64 solution build passed.
Production-adapter GPU smoke passed for FSR 2.2, DLSS SR and Cyberpunk-profile DLSS RR with two live contexts.
Compile the owner, shared state and completion hooks before connecting game evaluations. Optional routes have inactive definitions until their implementation commits.

Validation: Release x64 solution build passed.
Validation: Release x64 solution build passed.
NGX proxy and production WARP GPU-lifetime regressions passed.
Validation: Release x64 solution build passed.
Validation: Release x64 solution build passed.
Validation: Release x64 solution build passed.
Residual seam, cold/warm/reset history, motion reprojection, composition and alpha regression passed.
Validation: Release x64 solution build passed.
RTX 5090 DX11 transfer/reuse/resize tests and WARP queue-readiness/native-FG-buffer handoff regressions passed.
Validation: Release x64 solution build passed.
Validation: Release x64 solution build passed.
Production WARP pipeline-capture and GPU-lifetime regressions passed.
Validation: Release x64 solution build passed.
Production Vulkan shader passed active-region, colour/alpha and multipass checks on RTX 5090.
Validation: Release x64 solution build passed.
Validation: Release x64 solution build passed.
Final proxy, GPU-lifetime and pipeline-capture regressions passed. Packaging, local documentation links and unchanged runtime-source checks passed.
@wilsjo2
wilsjo2 force-pushed the codex/upstream-neural-rendering branch from 5517182 to a3363e0 Compare September 13, 2026 00:22
gabrielmaialva33 added a commit to gabrielmaialva33/OptiScaler_DLSSNR that referenced this pull request Sep 13, 2026
Four fence-confirmed GPU measurements from one Cyberpunk 2077 session, plus a
three-hour session at full resolution that agrees with the matching point to
0.18 ms. Measurement only; nothing here proposes a change.

  scale  model        Mpx     n    median model_ms
  0.25   860x360     0.31    109       2.32
  0.50   1720x720    1.24    518       2.90
  0.75   2580x1080   2.79     81       4.37
  1.00   3440x1440   4.95    193       6.36   (11398 samples elsewhere: 6.54)

Fitting the top three gives 1.76 + 0.930/Mpx, which reproduces them to 0.02 ms
and then misses 0.25 -- a point it had never seen -- by +0.27. The curve
flattens faster than a line as the model shrinks, so the floor is at least
1.9 ms and is approached, not hit.

Three things this settles, and one it deliberately does not:

Host-side optimisation of this pass is not worth doing. outside_model_ms is
0.20 ms against 6.36: 96.9% of the pass is inside the NGX model. The candidates
that were declined for lack of evidence -- descriptor caching, the exposure
scan's mutex, ResTrack HUD bookkeeping, format-support caching -- are now
declined with it. The note keeps the caveat that this bounds GPU time, not CPU.

Working scale is the only lever with range, and 0.50 is where it stops paying:
it collects 78% of the available saving, and the step below it returns 0.58 ms
for a quarter of the model's area. The shipped default was already 0.5.

Upstream PR optiscaler#1158's "under 1.8 ms" is below this model's measured floor, and
its own timer stops before the private DLSS SR pass that the mode exists to
add (ngxTime->End precedes EnlargeMatchedResidual there).

What it does not settle: whether that mode is worth having. It is a quality
feature, not a speed one, and this note should not be cited against evaluating
it on images at a fixed scale.

Also records that scene content barely moves the number, and the parsing trap
that made an early pass of this analysis report 3.44 ms -- model_ms=[0-9.]+
also matches the tail of outside_model_ms=.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HHxok8GWNRCUW7r8fdf486
gabrielmaialva33 added a commit to gabrielmaialva33/OptiScaler_DLSSNR that referenced this pull request Sep 13, 2026
Re-audited nr-present-hook.md against 150b49d, fifteen commits after it was
written, with a second external survey. Direction unchanged -- the host is
worth pursuing, the patches must not ship verbatim -- but three things moved.

The colour feedback loop is fixed upstream. Four donor commits landed on it
while this was being reconsidered, and afad519 (2026-09-13) is the fix: the
pass submitted on the presenting queue without waiting, so under a
frame-generation interposer the write could retire after the flip and land on a
later frame, which the next pass then re-read. DlssNr.PresentSync waits on the
slot fence before the hook returns. It has to be in the port from the start,
and it pulls directly against 555ed6d, which had removed exactly that stall.
Two agents disagreed about whether this was solved; one had a stale fetch, and
verifying took a git fetch rather than an argument.

The first slice has to change. The question that prompted the re-audit was
emulators, which reach no upscaler at all, and the slice as written waits for
captured guides rather than falling back to zeroes -- so it would have produced
nothing in the case it was being re-read for. It splits into a technical proof
with its own zeroed guides, and a playable mode with reconstructed motion. Two
prerequisites appear with it: cold NGX init is unproven without a game's own
init, and a Present is not evidence of a new frame. The note also now says
plainly how little a DXGI D3D12 host reaches -- RPCS3 is Vulkan, and a native
Linux build never loads this DLL at all.

Transport has a number. Two full-resolution CopyResource per processed present,
39.63 MB at 3440x1440, and lowering the working scale does not shrink them.
Plus the 4x interaction, now that MFG 4x is measured working: four inferences
per real frame if every presentation traverses the hook, which is the concrete
reason to reject the donor's global-sequence dedup.

Two answers arrived from outside for questions the note had left open:
UseAutoMask is the model's own HUD protection and this tree already ships the
key, and overlay-after-pass is confirmed as a hard rule for a second reason.

Corrections to model-cost-vs-working-scale.md, both found in review: 96.9% is
the model's share at 1.00x and 93.4% at 0.50x, so the figure should not be
quoted bare; and the line "6.36 x 0.25 = 1.59 ms" used against PR optiscaler#1158 was the
linear extrapolation that same note's blind test disproved -- 0.50x measures
2.90. Arguing from a model the page exists to correct.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HHxok8GWNRCUW7r8fdf486
gabrielmaialva33 added a commit to gabrielmaialva33/OptiScaler_DLSSNR that referenced this pull request Sep 13, 2026
A second adversarial review of the 2026-09-13 update found the direction sound
and the confidence wrong in a dozen places. Fifteen corrections, no change of
conclusion.

The one that mattered: the update claimed DLSSNR.UseAutoMask is the model's own
HUD classifier, "independently confirmed" in three projects, and used it to
answer a risk the original note had left open. It is not supported. This tree
presents that key as "Auto skin mask" (DlssNr_Menu.cpp:254,262), declared next
to DlssNrSkinStructure, and Magpie and the Resolve filter both expose automatic
mask and UI correction as separate controls. Passing a parameter is not knowing
what it does. The HUD concern is unanswered again, which is where it was.

The rest, in kind:

Overstated the donor's fix. PresentSync does not block every frame -- only when
the fence still reads pending -- and the CPU time it costs is not "the price of
NR". It supersedes 555ed6d's no-wait policy rather than merely contradicting
it, and that tension belongs to integration, not to a step 1 that has no frame
generation to race. Also b24389c is not pure instrumentation; it adds a
renderSeq guard.

Overstated my own inferences. "Step 1 is cheaper" was a specific simplification
promoted to a general comparison. "A Present is not a new frame" turned a
possibility into an observation. Overlay-after-pass is a visual-preservation
decision, not a synchronisation requirement, and conflating the two misread the
barrier discussion it claimed to follow from.

Overstated the numbers. 39.63 MB is payload, not VRAM traffic, and a fixed
volume, not a floor in milliseconds. The model's share is 96.9/93.4/90.2% at
1.00/0.50/0.25x, so "93-97%" and "a slice of 3%" both needed scale attached.
The route migration quadruples pixel area; calling it "2.90 to 6.36 ms" quotes
after-RR measurements for a host nothing has measured. And the 4x arithmetic
needed three qualifications it did not have.

And in the cost note, the floor claims were circular: "at least 1.9 ms" and
"measured floor" both come from the fit whose failure that page documents. The
lowest point actually measured is 2.32 ms. Against PR optiscaler#1158 the defensible
statement is narrower -- the claim is not reproduced or established for
Ada/Proton, and a number from their timer would omit the private SR -- not that
it falls below a floor we never measured.

The second reviewer's summary is worth keeping: what changed was not the
argument but the degree of certainty attributed to the donor's fix and to
performance conclusions. The note now says so in its own verdict.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HHxok8GWNRCUW7r8fdf486
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant