From 2764609224752f4665d0a23677136795676d1a45 Mon Sep 17 00:00:00 2001 From: Rovey Date: Sun, 11 Oct 2026 19:32:14 +0200 Subject: [PATCH 01/15] fix: the memory scan takes three longer readings a step and stops at a 3 % drop On the reference RTX 4070 four runs on one day saved a memory offset of +650, +1600, +550 and +100 MHz. The card has no bandwidth peak before its first wrong result, so the scan could only end early by chance. Measured (hardware check 95), the readings move in three ways: - one timed run of 75 ms was too short: readings scattered by 0.4 to 1.3 % and now and then did not settle. A run now copies four times as much (kBwDispatches 256) and the scatter is 0.2 %; - a single reading decided a step. A step is now the median of three (kBandwidthReadings); a reading that only did not settle is left out when two others gave a number; - for up to a minute at a time every reading sits about 1.8 % higher. The drop that ends the scan (kBandwidthDrop) goes from 1 % to 3 %. The window's own drawing was suspected and measured: it adds nothing, so nothing was built for it. Co-Authored-By: Claude Opus 5.5 --- README.md | 2 +- docs/development.md | 2 +- docs/hardware-checks.md | 1 + src/core/search.cpp | 31 +++++++--- src/core/search.hpp | 12 +++- src/hw/stress.cpp | 9 ++- tests/test_search.cpp | 134 ++++++++++++++++++++++++++++++++++++++-- 7 files changed, 174 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index 25c410e..49ef16c 100644 --- a/README.md +++ b/README.md @@ -177,7 +177,7 @@ flowchart LR - **Every run starts with the card at stock for five minutes,** as long as the result is tested at the end, with the fans left to the NVIDIA driver. That is the "before" of the result: a card is not heated through after 30 s, so a shorter run would make every result look warmer and louder than stock. Some cards get hot there: the driver's own fan control lets them (a 3080 Ti reached 85 °C with its fans at 65 %). The measurement stops at 85 °C, what it measured until then is the "before", and the run goes on. From there every step keeps the fans on a curve that reaches 100 % at the profile's temperature limit. - **Stress test.** A DirectX 11 compute load in which the GPU checks every value it computes against a known answer. Each probe ends in a verdict: `STABLE`, `WRONG RESULT`, `DEVICE LOST`, `STALLED` (the values were right but the card computed almost nothing), `TOO HOT` or `NO TELEMETRY`. -- **Memory comes first and stops at the bandwidth peak,** not at the first error. GDDR6 and GDDR6X retry failed transfers, so an overclocked memory bus loses speed long before it returns a wrong result. The search raises the offset 50 MHz at a time, measures bandwidth at each step and keeps the lowest offset within 1 % of the best. Where bandwidth cannot be measured, the memory is left at stock: without that measurement nothing says where to stop, and the first wrong result comes far too late. +- **Memory comes first and stops at the bandwidth peak,** not at the first error. GDDR6 and GDDR6X retry failed transfers, so an overclocked memory bus loses speed long before it returns a wrong result. The search raises the offset 50 MHz at a time, measures bandwidth three times at each step, takes the middle reading, and keeps the lowest offset within 1 % of the best. Where bandwidth cannot be measured, the memory is left at stock: without that measurement nothing says where to stop, and the first wrong result comes far too late. - **The core search climbs from stock.** With the chosen memory offset applied, it raises the core offset 15 MHz at a time until a probe fails, and keeps the last value that passed. How far it may go comes from the range the driver reports for your card, so a card that can hold more is not stopped at a fixed limit. - **Each edge is confirmed.** The value a search ends on gets a 30 s probe, and a lower value is tried if it does not hold. The safety margin is applied to the confirmed value, and the result as a whole must pass the 300 s soak. - **A driver reset ends that clock's search.** On a card the search can push past its edge, a search resets the driver: the screen goes black for a moment. The run then reconnects to the driver, sets the card to stock, leaves it without load for 20 seconds and checks with a short probe at stock that it computes normally again. After that the search tries no new values for that clock: it keeps the last value that passed and confirms four steps below it. Each clock has its own search, so a reset in the memory search does not keep the core from being searched; a run can therefore reset the driver twice, once per clock. One reset more than that ends the run at stock with nothing saved, and so does a card that does not come back. Results on real cards are in [docs/hardware-checks.md](docs/hardware-checks.md), checks 51 and 52. diff --git a/docs/development.md b/docs/development.md index 0722533..bbb974a 100644 --- a/docs/development.md +++ b/docs/development.md @@ -79,7 +79,7 @@ The applied offset is always at least one step below the confirmed edge. Profile | Without the bandwidth measurement the memory is left at stock | The search used to fall back to a climb on stability alone when the measurement failed at +0. On a tester's 3080 Ti (0.4.3) that climb went to +2300 MHz with the score falling from +2150 on, got its first wrong result there, and the machine froze with a corrupted screen in the 30 s confirm probes just below it. The first wrong result is no edge to stand next to: the memory was correcting errors for hundreds of MHz before it. Now a measurement that does not work means no memory overclock (`left_at_stock`), the run says so, and the core is still searched. `optimize` keeps the stability climb only for a caller that gives no measurement at all; the app always gives one. Why the measurement failed on that card is not known: the log was only in the window and went with the reboot. | | A confirm probe that is too hot at once confirms nothing | A confirm probe that ends `TOO HOT` keeps the clock edge: over 30 s the card may run a little hotter than over the power step's 20 s, and that is the soak's business, which lowers power. But the stress run stops at its first reading above the limit, so a card that is above it when the probe begins gives `TOO HOT` within a second, and that counted as holding too. It happened on the tester's RTX 3080 Ti (hardware check 91): after a driver reset the health probe loads the card at stock power, which on that card means 76 C against a limit of 75, and `confirm core +225` was confirmed by a probe that had tested nothing. Now such a probe counts only when the clock was loaded for `kConfirmMinS` (10 s, clearly longer than the 3 s probe that found the edge) before it got too hot. Otherwise the value is not confirmed, and the card rests (`io.rest`, the 20 s of a recovery, with the fans where the curve last put them) before the next try one step lower: without the rest every lower value would start just as hot, and heat alone would step the edge down to nothing, which is what keeping the edge on `TOO HOT` was for. The same value is not tried again; one step is the price. Not changed: the health probe still runs at stock power, and a soak that begins too hot still lowers the power limit by a step. | | A profile that promises speed keeps no result that is slower than stock | Best of my GPU ended 3.8 % slower than stock on the tester's RTX 3080 Ti (4749 against 4937 it/s, hardware check 91), with the fans at 80 % instead of 66 %, and showed it as **Done**, applied and kept at every logon. Every step had done what it is for: the card runs 83 C at stock with its fans at 100 %, the profile allows 75, so the power limit went to 65 %, and +150 MHz of core clock did not win that back. But a profile called "the best of my GPU" that leaves the card slower has not done what it says, and a result like that is not offered as one. After the soak, `optimize` compares its score with the measurement at stock that went before the run (the "before" the result screen shows, so the rule judges by the number the user sees): below `kNotSlowerScore` (0.99: one percent is what two five-minute runs differ by) the run ends as a failed one does, at stock, with nothing saved, and says what it found, why, and that Undervolt is the profile for a card that runs warm. `Objectives::faster_than_stock` marks the two profiles this holds for, Best of my GPU and Max performance. Not Quiet and Cool & efficient, which trade speed for noise and heat and say so. Not the overclock step of All in one either: it goes on to an undervolt and a fan tune, and its result is judged as a whole; whether that whole may be slower than stock is not decided. This is a backstop, not the cure: what such a card needs from these profiles is in `docs/planned.md` ("never hotter than at stock"). | -| Open: on the reference card the memory offset is decided by noise | Three runs on one day applied +650, +1600 and +550 (hardware check 94). The memory search assumes that bandwidth peaks before the memory makes errors (`kBandwidthDrop`: stop at a reading 1 % below the best, re-measured once). On the reference RTX 4070 it does not: bandwidth rises by about 0.4 % per 50 MHz step all the way to +2500, and the first wrong result comes at +2550. The readings scatter by 0.4 to 0.7 % (standard deviation), so two readings in a row 1 % under a high one happen by chance, and where that happens is where the scan ends: at +1000 twice, not at all once. When it does not happen the search stands next to the first wrong result after all, which is what the bandwidth method exists to avoid; the tie rule and the margin still put the applied offset 950 MHz below it. Nothing is changed yet. What a fix has to decide: whether a card like this should get a memory overclock by another rule (a fixed share of the range the scan covered, or of the offset of the first error), and how a real peak is told from scatter on the cards that have one (several readings per step and their median, or a drop that holds over more than one step). | +| A step of the memory scan is the middle one of three readings over a longer copy, and the scan stops at a 3 % drop | Four runs on one day applied +650, +1600, +550 and +100 MHz on the reference RTX 4070, a card that runs every game at +1500 (hardware check 94). That card has no bandwidth peak: bandwidth rises by about 0.4 % per 50 MHz step all the way to +2500 and the first wrong result comes at +2550, so the scan could only end early by chance, and where it did decided the offset. Measured afterwards without a search and with nothing changed on the card (hardware check 95), the readings move in three ways, and each got its own answer. **(1) One timed run was too short.** A run copied 32 GB, 75 ms on this card, and a reading is the median of three runs that agree within 1 %. Such readings scattered by 0.4 to 1.3 % (standard deviation over 24 readings, six series), single ones by up to 4.8 %, and in two of the six series one reading did not settle. With 128 GB per run (`kBwDispatches` 256, 0.3 s) it is 0.2 %, and all 96 readings settled. **(2) A single reading decided a step.** A step is now the median of `kBandwidthReadings` (3) readings, taken one after the other while the candidate is still applied. A reading that only did not settle is left out when two others gave a number; any other failure ends the step at once, as before. The log shows all of them: `mem +200: bandwidth 440.9 GB/s (441.2 440.9 440.1)`. **(3) The readings change level.** For up to a minute at a time every reading is about 1.8 % higher than before and after (452 against 444 GB/s), at one and the same offset. The cause is not known. A median of three does nothing against it: all three are inside the stretch. So the drop that ends the scan (`kBandwidthDrop`) went from 1 % to 3 %, still re-measured once: above what the readings do by themselves, below what a real peak loses within a step or two. The price on a card that has a peak: the scan goes a step or two further past it, in 3 s probes, before it stops. The offset it keeps is still the lowest within 1 % of the best (`kBandwidthTie`), and confirming starts there. **Not built: a window that stands still during a measurement.** The suspicion was that the window's own drawing disturbed the readings of the run that saved +100. Measured with a window drawing a frame every 100 ms and every 33 ms next to the measurement: 0.2 %, the same as without. **What it costs:** a step lasts about 3 s longer, a scan to +2500 about two and a half minutes. **What is left open:** on a card without a peak the scan now always ends at the first wrong result, which the bandwidth method exists to stay away from; the tie rule and the margin are all that keep the applied offset away from it (+1600 against +2550 in the one run of check 94 that got there). Whether such a card should get its memory offset by another rule (a fixed share of the range the scan covered) is not decided. And a change of level larger than 3 % would end a scan by chance again: the answer to that is to compare a low step with a fresh reading of the best offset taken right after it, not a still higher threshold. | | Open: why the bandwidth measurement fails on some cards, and what then | On the tester's RTX 3080 Ti it failed at +0, and since 0.4.4 that means no memory overclock at all on such a card, though it computed right far beyond +2000 MHz. The measurement gives a number only when three runs in a row agree within 1 % (`kBwSettle`), and it throws away every run the driver's timer calls disjoint (the clock changed under it). That card was held at a power limit of 65 % and ran at 83 C when it was measured; the reference card has only ever been measured at 105 %, where the measurement itself draws 179 of 210 W. So the suspicion is the power limit, and hardware check 90 tests exactly that on the reference card. What it shows decides what is built, and nothing is built before it: **(1) runs are thrown away as disjoint, and the ones that are kept agree:** the timer is what fails, not the card. Time the copy differently (wall clock around a flush, or count a disjoint run when its neighbours agree) and keep the method. **(2) the readings themselves scatter by more than 1 %:** a peak that is found by a 1 % drop (`kBandwidthDrop`) cannot be found in that noise. Then the memory has to be measured where the card is not held back: search it before the power step, at stock power, where a probe and its measurement last five seconds; or take more readings and compare medians. **(3) it does not fail at 65 % on the reference card:** the power limit is not the cause. Then it was the heat (a card throttling at its temperature limit changes clock as well), or another program, and only the tester's `run.log` can say: since this change a measurement that never settled writes how many runs gave a reading, between which values, and how many were thrown away. In no case does the climb on stability alone come back. **Outcome of check 90: (3).** On the reference card the measurement gave a number fifteen times out of fifteen at 100, 80 and 65 %, in three or four runs, with nothing thrown away, and the bandwidth was the same at every limit. The power limit is ruled out; (1) and (2) are not built. What is left is the heat of that card, another program, or a timer query that failed, and the tester's `run.log` is the next thing that can tell them apart. **The tester's run (check 91): it is (2), but only inside a run.** Twenty readings between 763.9 and 821.2 GB/s, none thrown away, three seconds after the load began; after thirty seconds of load the same card read within 0.2 %. The suspicion now is the moment, not the card: the measurement is taken while the power limiter is still catching up. If `gao --bandwidth --after-load 3` scatters on that card and `--after-load 30` does not, that is it, and the fix is to measure when the card has settled (keep taking runs for some seconds instead of twenty runs), not to change the method. | | Every line of a run is also in `run.log` | `logged()` wraps the log hook of both front ends: a line is appended and flushed before it is shown, so the step a machine freezes in is the last line on disk. One file in the state folder, moved aside at 1 MB (`run.log.old`). Only elevated and only after `ensure_app_dir` has verified the folder: an elevated process appends to nothing a user could have put there. | | A run does not start while the NVIDIA app's Automatic Tuning is on | Switched on, a service of the driver scans the card when it sees fit: it loads it, changes its clocks, and starts again by itself (hardware check 89: a scan that was stopped started again 14 ms later, and after a restart of the service it resumed within two seconds). A run next to that measures nothing it can trust, so it is refused, like a run next to Afterburner, and with the same weight: a courtesy, not a guard. Read from where that service keeps the switch, `%ProgramData%\NVIDIA Corporation\nvtopps\nvtopps.db3`, table `ocScannerSettings`, fields `backgroundScanEnabled` and `grdOcEnabled`, joined to `TelemetryGpuCfg` so that only a card that is in the PC counts (a row of an earlier card can never be switched off again). That this is the source and not a copy of it was measured, not assumed: with the field set the service took the scan up again after every restart, with it cleared it did not, while the NVIDIA app itself showed the switch as off. Nothing documents the file. So: opened read-only and immutable (no lock, the service never waits for us), through `winsqlite3.dll` from System32 (no dependency added), and everything that is not exactly as expected reads as "unknown", which starts the run as before. The registry value the app sets next to it (`HKLM\SOFTWARE\NVIDIA Corporation\Global\NvApp\OC\`) changes at the same moment but belongs to the app, not to the service, and is not used. **The overlay is not checked:** no setting for it was found, its processes run whether it is open or not, and the one value that moved (`IsShadowPlayEnabled`) followed the service being up, not a switch. NVAPI was not tried: an undocumented call is no better than an undocumented file, and it cannot be compared with anything. | diff --git a/docs/hardware-checks.md b/docs/hardware-checks.md index 056f866..8f4e437 100644 --- a/docs/hardware-checks.md +++ b/docs/hardware-checks.md @@ -122,6 +122,7 @@ root; adjust the path for a Debug build. | 92 | The undervolt profile starts with the measurement at stock every run uses, and a hot card is not refused | Elevated, the tray app exited, a copy of `gao.json` kept. (a) `gao --undervolt` to the end; then `gao --status`. (b) The same with a test build whose stock limit is lowered to a temperature the reference card passes under load (63 C; it reaches 67). Put the copy of `gao.json` back. (c) Only on a card that is above the profile's limit at stock with the fans at 100 % (the tester's RTX 3080 Ti: 82 to 83 C): the **Undervolt** profile from the window, to the end, and its `run.log` | yes | (a) `the card at stock, five minutes`, a `stock: STABLE ...` line with the driver's fan speed, then `baseline: the measurement at stock`, and no second five minutes at stock: the first candidate follows at once; no `candidates are stopped at` line; `RESULT:`; `before:` shows 300 s and the fan speed the driver chose. (b) `stock: TOO HOT ... peak=64 C`, the line that the measurement stopped there, the cool-down, then `baseline: the measurement at stock` and a search that runs to `RESULT:` with `before:` showing the shortened time; no `candidates are stopped at` line, because 64 C is under the profile's 75. (c) no `not stable at stock`; `the card reached 86 C at stock: candidates are stopped at 85 C, not at the profile's 75 C`; candidates that end `STABLE` or `WRONG RESULT`, none `TOO HOT`; a result that is cooler and draws less than stock at the same clock | First run: **not as expected**, and that found a fault (2026-10-11, reference RTX 4070, driver 617.42, a test build of d1c99e9, started elevated by a script). The first halves held: (a) `stock: STABLE ... peak=65 C fan=70 %`, then `baseline: the measurement at stock` four seconds later and no second five minutes; (b) `stock: TOO HOT ... peak=64 C`, stopped after 34 s, the run went on, no `candidates are stopped at` line. But both runs then ended with `the card held no undervolt: the first step below its own voltage already failed`: their first candidates read `1070 mV (+15 MHz): SLOW ... clock=2323 MHz` and `1050 mV (+15 MHz): SLOW ... clock=2411 MHz`, on a card that ran 2791 MHz in the measurement just before. The search's load had not run yet when that candidate began (see the decision "With its reference handed over, the undervolt search warms its load up before the first candidate"). Second run, with the warm-up: **pass** for (a) and (b) (same day, a test build of 0733695). (a) `stock: STABLE score=5426 it/s clock=2788 MHz peak=66 C fan=70 %`, four seconds later `baseline: the measurement at stock`, `warming the load up: 5 s at stock`, `target: 2790 MHz, which the card reaches at 1090 mV by itself`, and the first candidate `1085 mV (+15 MHz): STABLE ... clock=2775 MHz`; 26 candidates, none `SLOW` or `TOO HOT`, the edge at 950 mV, `RESULT: 2790 MHz at 990 mV instead of 1090 mV`; `before:` 5426 it/s, 189 W, fan 70 % (300 s), `after:` 5325 it/s, 149 W, fan 53 % (300 s). (b) `stock: TOO HOT ... peak=64 C`, stopped after 74 s, the run went on, the same warm-up and a search to the same result (990 mV, edge 950 mV); `before:` shows 75 s; no `candidates are stopped at` line. The saved tune of before the test was put back and applied. (c) not run. | | 93 | A speed profile that would end slower than stock keeps nothing | Only on a card that is warmer at stock than the profile allows (the tester's RTX 3080 Ti; the reference card ends faster than stock with every speed profile): **Best of my GPU** from the window, to the end; the result screen, the dashboard and `run.log` | yes | the run goes through all its steps and its soak; then **Not applied** with `the result is slower than stock: N it/s against M (-X %)`, the power limit it had to go down to, and the pointer to Undervolt; `The card is back at stock.`; the dashboard shows stock and the saved tune of before the run, if there was one; `run.log` ends with `stopped: the result is slower than stock ... -- card restored to stock` | Not run yet. On the reference card: nothing to see, by design. | | 94 | All in one, with the undervolt step searching for the first time | Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize all` to the end with a build that has the warm-up of the undervolt step; then `gao --status`; put the copy back | yes | five steps; in step 3 a `warming the load up` line, a first candidate that is not `SLOW`, and a descent; steps 4 and 5 run on the tune of steps 2 and 3; `RESULT:` | **Not as expected at step 4**, and that found a fault (2026-10-11, reference RTX 4070, driver 617.42, a test build of 0733695, unattended). Step 1: `stock: STABLE score=5325 it/s peak=65 C fan=70 % power=186 W`. Step 2: power 105 %, `mem: bandwidth peak +2300, confirmed +2300, applying +1600`, `core: highest stable +210, confirmed +210, applying +135`, soak `STABLE score=5568 it/s peak=70 C power=197 W`. Step 3, as wanted: `warming the load up: 5 s with the overclock`, `target: 2865 MHz, which the card reaches at 1050 mV with the overclock`, nine candidates `STABLE` from 1045 mV down, `WRONG RESULT` at 990 mV, confirmed at 1010 mV after two tries, and `applying 1050 mV (+135 MHz at that point)`: the margin left no raise above the overclock, so the curve was cut off, as in the runs before the warm-up; soak `STABLE score=5434 it/s peak=67 C power=176 W`. Step 4 did not begin: `the tune could not be applied for the fan tune: the undervolt was not applied: the point already runs more than the target -- card at stock`. The saved tune of before the test was put back and applied. Cause and fix: the decision "A flat top goes back on a card that is cooler than when it was found". Second run, with the `gao.exe` of the released v0.4.6: **pass, all five steps** (same day, started elevated by a script). Step 1: `stock: STABLE score=5309 it/s peak=66 C fan=70 % power=185 W`. Step 2: power 105 %, `mem: bandwidth peak +850, confirmed +850, applying +550`, `core: highest stable +210, confirmed +210, applying +135`, soak `STABLE score=5568 it/s`. Step 3: `target: 2850 MHz, which the card reaches at 1045 mV with the overclock`, `undervolt: lowest stable 975 mV, confirmed 985 mV, applying 1025 mV (+165 MHz at that point)`, soak `STABLE score=5363 it/s peak=66 C power=163 W`. Step 4: the tune went back on the card, `fan 60 %: 63 C`, `fan 55 %: 65 C`, `fan 52 %: 67 C`, `fan tune: 52 % holds 67 C`. Step 5: the final test ran its 300 s. `RESULT: power 105 %, core +135 MHz, mem +550 MHz, 2850 MHz at 1025 mV, fans 52 % at 67 C`; `stock: 5309 it/s, 185 W, fan 70 %, 66 C`, `now: 5353 it/s, 166 W, fan 52 %, 70 C`: `+0.8 % score and -10.3 % power against stock`; `undervolt: applied`. The saved tune of before the test was put back. **What this run did not show:** its undervolt had a raise of its own (+165 over an offset of +135), so the anchor was raised and not merely cut off, and the case of the first run (an anchor that reads a step above the target on a cooler card) did not come up. That fix has still only its tests. **The memory offset differs a lot from run to run on this card, and the two logs say why:** +1600 in the first run (bandwidth peak +2300), +550 in the second (peak +850), +650 in the run that saved the tune in use that morning (the journal shows its scan ending at +1000 too). The card was in the same state each time (68 C, power 105 %, a reading every three to four seconds). In both logged scans the bandwidth rises in a straight line, 0.36 and 0.46 % per 50 MHz step, from 420 GB/s at +0 to 509 GB/s at +2450; the first run went on to `WRONG RESULT` at +2550 without the bandwidth ever turning down. The readings scatter around that line with a standard deviation of 0.4 and 0.7 %, single ones by more than 1 %. The scan ends at a reading more than 1 % below the best so far when a second reading agrees, and in the second run that was chance: +950 read high (462.8 GB/s, where the line is at about 458), +1000 read 456.6 twice, 1.3 % below it. In the first run the one low reading (+600) was followed by a higher one, and the scan ran to the first wrong result. So on this card there is no bandwidth peak before the errors, and where the scan stops is decided by noise. | +| 95 | The memory scan no longer ends by chance | (a) Not elevated, nothing changed on the card: a measuring build takes 24 bandwidth readings in one process, with 3 s of the stress load before each as a scan has, at 64, 128 and 256 dispatches per timed run; then the same with the app's window open next to it, drawing a frame every 100 ms and every 33 ms. (b) Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize best` twice in a row with the build of the change, then the copy put back | (a) no, (b) yes | (a) how much a reading scatters, and on what that depends. (b) in both runs every `mem +N: bandwidth` line shows three readings that agree within about 1 %; the scan does not end before the first `WRONG RESULT`; the two `mem: bandwidth peak` values are within 300 MHz of each other and the applied offset is above +1000; the soak is not slower than at stock | **(a) measured** (2026-10-11, reference RTX 4070, driver 617.42, the saved tune applied: power 105 %, core +135, memory +650; standard deviation over the 24 readings of a series, and the lowest and highest against the mean). 64 dispatches, no window: 0.44 % (-0.55 to +1.07, one reading did not settle), 0.66 % (-1.58 to +1.49, one did not settle), 1.31 % (-4.83 to +1.27), 0.86 % (-1.44 to +2.05). 128 dispatches, no window: 0.43 % (-1.10 to +0.79). 256 dispatches, no window: 0.22 % (-0.43 to +0.42) and 0.60 % (-0.80 to +1.37; the last five readings of this series were 451 to 453 GB/s after nineteen of 444 to 449). With the window drawing every 100 ms: 64 dispatches 1.13 % (the first six readings 452 to 453 GB/s, the rest 439 to 451), 256 dispatches 0.24 % (-0.47 to +0.61). With the window drawing every 33 ms: 64 dispatches 0.58 %, 256 dispatches 0.25 % (-0.35 to +0.51). A reading took 6.5, 5.3 and 3.9 timed runs to settle at 64, 128 and 256 dispatches. So: the longer timed run takes the scatter from 0.4 to 1.3 % down to 0.2 %, the window's drawing adds nothing, and apart from the scatter the readings sit about 1.8 % higher for up to a minute at a time (the stretch at the end of the second 256 series ran on into the next series; the last 64 series went back and forth between 444 and 452 GB/s). What sets that level is not known. **(b) not run yet.** | ## Notes diff --git a/src/core/search.cpp b/src/core/search.cpp index 135532b..0da6b7d 100644 --- a/src/core/search.cpp +++ b/src/core/search.cpp @@ -571,17 +571,26 @@ OptimizeResult optimize(const GpuControl& gpu, const Objectives& obj, Journal& j bool unavailable = false; const int peak = best_bandwidth_offset(0, bounds.mem_max_mhz, kMemStep, ceiling, [&](int v) { MemSample m; - std::optional gbps; + // kBandwidthReadings readings, and the step is their middle + // one. A reading that only did not settle is left out; any + // other failure ends the step at once. + std::vector readings; + bool failed = false; // The +0 sample is journaled without a clock, like a soak at // stock: if the machine froze in it, an open entry that said // memory 0 would read as "never search memory again". const std::optional mem_j = v > 0 ? std::optional(v) : std::nullopt; m.stable = explore(v, reset_at, [&] { - if (!is_stable(clock_candidate(std::nullopt, mem_j, 0, v, kClockProbeS, [&] { gbps = io.bandwidth(); }))) + if (!is_stable(clock_candidate(std::nullopt, mem_j, 0, v, kClockProbeS, [&] { + for (int i = 0; i < kBandwidthReadings && !failed; ++i) { + if (const auto gbps = io.bandwidth()) readings.push_back(*gbps); + else failed = !(io.bandwidth_unsettled && io.bandwidth_unsettled()); + } + }))) return false; - if (gbps) return true; + if (!failed && readings.size() >= 2) return true; // The candidate passed, the measurement failed: the step is unusable. - if (v > 0 && io.bandwidth_unsettled && io.bandwidth_unsettled()) { + if (v > 0 && !failed) { // The load computed and the readings disagreed: noise, // not the card giving out. The scan ends with the best // of what was measured; the core is still searched. @@ -601,10 +610,18 @@ OptimizeResult optimize(const GpuControl& gpu, const Objectives& obj, Journal& j return false; }); if (m.stable) { - m.gbps = *gbps; + std::string each; + for (const double g : readings) { + char one[32]; + std::snprintf(one, sizeof(one), "%s%.1f", each.empty() ? "" : " ", g); + each += one; + } + std::sort(readings.begin(), readings.end()); + const size_t n = readings.size(); + m.gbps = n % 2 ? readings[n / 2] : (readings[n / 2 - 1] + readings[n / 2]) / 2; char buf[64]; - std::snprintf(buf, sizeof(buf), "mem +%d: bandwidth %.1f GB/s", v, m.gbps); - log(buf); + std::snprintf(buf, sizeof(buf), "mem +%d: bandwidth %.1f GB/s (", v, m.gbps); + log(buf + each + ")"); } return m; }); diff --git a/src/core/search.hpp b/src/core/search.hpp index 959d618..1ec4133 100644 --- a/src/core/search.hpp +++ b/src/core/search.hpp @@ -67,11 +67,21 @@ int apply_margin(int max, int step, float perf_push); // Memory overclocks fail by losing bandwidth before they fail by producing // errors: GDDR6X/GDDR6 retry bad transfers. The memory search therefore looks // for the bandwidth peak, not the stability ceiling. -inline constexpr double kBandwidthDrop = 0.01; // stop once 1 % below the best +// The scan stops once a step is this far below the best. It was 1 %, and that +// is less than the readings move by themselves: on the reference RTX 4070 +// they sit about 1.8 % higher for tens of seconds at a time, at any offset, +// and such a stretch ended a scan wherever it began or ended (hardware check +// 95). A peak that is real falls further than that within a step or two. +inline constexpr double kBandwidthDrop = 0.03; // Within 1 % of the best counts as the best: measured noise on the 4070 is // ~0.4 % and one 50 MHz step is worth ~0.5 %, so a narrower band would let // noise pick the offset. inline constexpr double kBandwidthTie = 0.01; +// A step of the memory scan is the middle one of this many readings, so one +// reading that is off, or that did not settle, decides nothing. On the +// reference card single readings ended scans at +250, at +1000 and not at all +// on one day, and the runs saved +100, +650 and +1600 MHz (hardware check 94). +inline constexpr int kBandwidthReadings = 3; // A confirm probe that ends too hot keeps the clock edge (heat is the power // step's and the soak's business), but only when the clock was loaded this diff --git a/src/hw/stress.cpp b/src/hw/stress.cpp index ba46542..3e4c394 100644 --- a/src/hw/stress.cpp +++ b/src/hw/stress.cpp @@ -28,9 +28,14 @@ constexpr double kTargetBatchMs = 250.0; constexpr int kMaxDispatches = 2048; constexpr UINT kBwBytes = 256u * 1024 * 1024; // per buffer; far above the 4070's 36 MB L2 constexpr UINT kBwGroups = 1024; -constexpr int kBwDispatches = 64; +// Dispatches per timed run: 128 GB copied, about 0.3 s on the reference +// RTX 4070. A run of 64 (75 ms) was short enough for whatever else the machine +// did to land in one run and not in the next: readings scattered by 0.4 to +// 0.7 % (standard deviation) and one in 24 did not settle; with 256 it is +// 0.2 % and none (hardware check 95). +constexpr int kBwDispatches = 256; constexpr size_t kBwRuns = 3; // consecutive runs that must agree -constexpr int kBwMaxRuns = 20; // ~1.5 s at full speed; more if clocks are low +constexpr int kBwMaxRuns = 20; // ~6 s at full speed; more if clocks are low constexpr double kBwSettle = 0.01; // agree = within 1 % // Grid-stride copy of 16-byte elements between two raw buffers. diff --git a/tests/test_search.cpp b/tests/test_search.cpp index 04ac963..603701f 100644 --- a/tests/test_search.cpp +++ b/tests/test_search.cpp @@ -136,6 +136,11 @@ struct FakeCard { bool fail_set_on_first_soak = false; // the first write of the soak state fails once int bw_fails_once_at_mem = -1; // the bandwidth measurement at this offset fails once and the driver resets int bw_unsettled_at_mem = -1; // the bandwidth readings at this offset do not agree; the driver is fine + int bw_unsettled_once_at_mem = -1; // the first reading at this offset does not settle, the next ones do + // What a reading is off by: a factor on the curve, for the n-th reading + // (from 0) taken at that offset. Empty: none. + std::function bw_noise; + std::map bw_readings_at; // readings taken so far, by offset bool bw_unsettled = false; // what the last failed measurement was int lost_mem_from = -1; // at or above this memory offset a 3 s probe ends DEVICE LOST int recover_fails_from_call = -1; // recover() fails from this call on (1 = the first) @@ -321,7 +326,8 @@ struct Run { if (card.bw_curve) io.bandwidth = [this]() -> std::optional { ++card.bw_measurements; - card.bw_unsettled = card.mem == card.bw_unsettled_at_mem; + const int nth = card.bw_readings_at[card.mem]++; + card.bw_unsettled = card.mem == card.bw_unsettled_at_mem || (card.mem == card.bw_unsettled_once_at_mem && nth == 0); if (card.bw_unsettled) return std::nullopt; if (card.bw_loses_device) { card.driver_reset(); return std::nullopt; } if (card.bw_fails_at_zero && card.mem == 0) return std::nullopt; @@ -331,7 +337,7 @@ struct Run { return std::nullopt; } if (card.bw_fails) return std::nullopt; - return card.bw_curve(card.mem); + return card.bw_curve(card.mem) * (card.bw_noise ? card.bw_noise(card.mem, nth) : 1.0); }; if (card.bw_curve) io.bandwidth_unsettled = [this] { return card.bw_unsettled; }; return optimize(gpu, objectives_for(preset), j, io, stock); @@ -615,7 +621,7 @@ TEST_CASE("bandwidth scan stops at the peak of a rising-then-falling curve") { return MemSample{true, v <= 600 ? 500 + v * 0.3 : 680 - (v - 600) * 0.9}; }); CHECK(r == 600); - CHECK(max_sampled == 650); // stopped at the first step more than 1 % below the peak + CHECK(max_sampled == 650); // stopped at the first step more than kBandwidthDrop below the peak } TEST_CASE("a single low bandwidth reading is re-measured before it ends the scan") { @@ -2135,7 +2141,7 @@ TEST_CASE("a failed bandwidth measurement ends the memory scan; the core is stil CHECK(run.logged("core +0 / mem +0: STABLE")); CHECK(run.logged("mem +0: bandwidth 500.0 GB/s")); CHECK(run.card.max_mem_seen == 300); - CHECK(run.card.bw_measurements == 7); + CHECK(run.card.bw_measurements == 6 * kBandwidthReadings + 1); // +0 .. +250 in full, and the one at +300 that failed CHECK(r.mem_max_stable == 250); CHECK(r.mem_confirmed == 50); CHECK(r.mem_mhz == 0); @@ -2210,7 +2216,7 @@ TEST_CASE("no bandwidth is measured after a reset") { const auto r = run.go(Preset::BestOfMyGpu); REQUIRE(r.ok); CHECK(run.logged("core +0 / mem +300: DEVICE LOST")); - CHECK(run.card.bw_measurements == 6); // +0 .. +250, and none after the lost candidate + CHECK(run.card.bw_measurements == 6 * kBandwidthReadings); // +0 .. +250, and none after the lost candidate CHECK(run.card.max_mem_seen == 300); } @@ -2831,3 +2837,121 @@ TEST_CASE("without a measurement at stock there is nothing to be slower than") { Run unscored = warm_card(0); // a measurement that scored nothing says nothing CHECK(unscored.go(Preset::BestOfMyGpu).ok); } + +namespace { + +// A card like the reference RTX 4070 (hardware check 94): the bandwidth rises +// by 0.4 % per 50 MHz step all the way to the first wrong result. There is no +// peak to find; only scatter can end the scan before the edge. +Run rising_to_the_edge(int edge = 2500) { + Run run; + run.card.mem_edge = edge; + run.card.ranges = ClockOffsetRanges{{-500, 1000}, {-1000, 3000}}; + run.card.bw_curve = [](int m) { return 434.0 * (1 + 0.004 * m / 50); }; + return run; +} + +// Scatter with a standard deviation of about `sd`, the same for the same +// (seed, offset, reading): three uniform numbers summed, which is bell-shaped +// enough and needs no library whose numbers differ between compilers. +std::function scatter(unsigned seed, double sd) { + return [seed, sd](int mem, int nth) { + unsigned x = seed * 2654435761u + static_cast(mem) * 40503u + static_cast(nth) * 9176u + 12345u; + double sum = 0; + for (int i = 0; i < 3; ++i) { + x ^= x << 13; x ^= x >> 17; x ^= x << 5; + sum += (x % 100000) / 100000.0 - 0.5; // uniform in -0.5 .. 0.5, variance 1/12 + } + return 1.0 + sum * 2.0 * sd; // three of them: variance 1/4, so a deviation of sd + }; +} + +} + +TEST_CASE("one bandwidth reading that is too high does not end the memory scan") { + // The run that saved +100 MHz on a card that holds +1500: the re-measure + // at +200 read 449.0 GB/s where its neighbours read 435 to 441, the two + // readings at +250 were more than 1 % under it, and that was "the peak". + Run run = rising_to_the_edge(); + run.card.bw_noise = [](int mem, int nth) { return mem == 200 && nth == 0 ? 1.03 : 1.0; }; + const auto r = run.go(Preset::BestOfMyGpu); + REQUIRE(r.ok); + CHECK(run.card.max_mem_seen == 2550); // the scan went to the first wrong result + CHECK(r.mem_max_stable >= 2300); + CHECK(r.mem_mhz >= 1500); + // The step is the middle one of its three readings, and the log shows all three. + CHECK(run.logged("mem +200: bandwidth 440.9 GB/s (454.2 440.9 440.9)")); + CHECK(run.logged("mem +0: bandwidth 434.0 GB/s (434.0 434.0 434.0)")); +} + +TEST_CASE("one bandwidth reading that is too low does not end the memory scan either") { + // The run that stopped at +1000: +950 read high, +1000 read low twice. + Run run = rising_to_the_edge(); + run.card.bw_noise = [](int mem, int nth) { return mem == 950 && nth == 0 ? 1.02 : mem == 1000 && nth < 2 ? 0.985 : 1.0; }; + const auto r = run.go(Preset::BestOfMyGpu); + REQUIRE(r.ok); + CHECK(run.card.max_mem_seen == 2550); + CHECK(r.mem_max_stable >= 2300); +} + +TEST_CASE("readings that scatter as they do on the reference card leave the memory scan where the card puts it") { + // Measured there with the longer timing window: a standard deviation of + // 0.22 % per reading. Twice that here, on twenty different days. + for (unsigned seed = 1; seed <= 20; ++seed) { + Run run = rising_to_the_edge(); + run.card.bw_noise = scatter(seed, 0.0045); + const auto r = run.go(Preset::BestOfMyGpu); + CAPTURE(seed); + REQUIRE(r.ok); + CHECK(run.card.max_mem_seen == 2550); + CHECK(r.mem_max_stable >= 2150); + CHECK(r.mem_max_stable <= 2500); + } +} + +TEST_CASE("a real peak is still found where it is when the readings scatter") { + // The card the method was made for: bandwidth turns down long before the + // first wrong result. Scatter must not carry the scan past the peak. + for (unsigned seed = 1; seed <= 20; ++seed) { + Run run; + run.card.mem_edge = 1400; + run.card.bw_curve = [](int m) { return m <= 900 ? 500 + m * 0.3 : 770 - (m - 900) * 0.6; }; + run.card.bw_noise = scatter(seed, 0.0045); + const auto r = run.go(Preset::BestOfMyGpu); + CAPTURE(seed); + REQUIRE(r.ok); + CHECK(r.mem_max_stable >= 800); + CHECK(r.mem_max_stable <= 950); + CHECK(run.card.max_mem_seen <= 1050); + } +} + +TEST_CASE("a reading that does not settle is left out when the other two agree") { + // On the reference card one reading in twenty-four did not settle, with + // nothing wrong. One such reading used to end the scan. + Run run = recovering(); + run.card.bw_curve = [](int m) { return 500 + m * 0.3; }; + run.card.bw_unsettled_once_at_mem = 300; + const auto r = run.go(Preset::BestOfMyGpu); + REQUIRE(r.ok); + CHECK_FALSE(run.logged("did not settle")); + CHECK(run.logged("mem +300: bandwidth 590.0 GB/s (590.0 590.0)")); + CHECK(run.card.max_mem_seen > 300); + CHECK(r.driver_resets == 0); +} + +TEST_CASE("a stretch in which every reading is 2 % off does not end the memory scan") { + // Measured on the reference card (hardware check 95): for tens of seconds + // at a time every reading is about 1.8 % higher, whatever the offset, and + // then it is back. Three readings of one step are all inside such a + // stretch, so their middle one is too. + for (const double off : {0.98, 1.02}) { + Run run = rising_to_the_edge(); + run.card.bw_noise = [&card = run.card, off](int, int) { return card.bw_measurements > 60 && card.bw_measurements <= 110 ? off : 1.0; }; + const auto r = run.go(Preset::BestOfMyGpu); + CAPTURE(off); + REQUIRE(r.ok); + CHECK(run.card.max_mem_seen == 2550); + CHECK(r.mem_max_stable >= 2300); + } +} From 431a9ff9669ce6233d87161fc139c0e6ae256759 Mon Sep 17 00:00:00 2001 From: Rovey Date: Sun, 11 Oct 2026 20:20:39 +0200 Subject: [PATCH 02/15] docs: hardware check 95 (b), the offset repeats and the scan ends in a driver reset Two Best of my GPU runs in a row on the reference card applied a memory offset of +1400 and +1450 and were 4 % faster than stock. Both scans ended with the device lost during the bandwidth copy, at +2450 and +2500, where the build before measured a number and the stress load reported the first wrong result at +2550. Recorded as the open point that keeps the change from being merged. A bandwidth measurement that loses the device now says so; the log line ended in nothing. Co-Authored-By: Claude Opus 5.5 --- docs/development.md | 2 +- docs/hardware-checks.md | 2 +- src/hw/stress.cpp | 5 ++++- 3 files changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/development.md b/docs/development.md index bbb974a..15c74c7 100644 --- a/docs/development.md +++ b/docs/development.md @@ -79,7 +79,7 @@ The applied offset is always at least one step below the confirmed edge. Profile | Without the bandwidth measurement the memory is left at stock | The search used to fall back to a climb on stability alone when the measurement failed at +0. On a tester's 3080 Ti (0.4.3) that climb went to +2300 MHz with the score falling from +2150 on, got its first wrong result there, and the machine froze with a corrupted screen in the 30 s confirm probes just below it. The first wrong result is no edge to stand next to: the memory was correcting errors for hundreds of MHz before it. Now a measurement that does not work means no memory overclock (`left_at_stock`), the run says so, and the core is still searched. `optimize` keeps the stability climb only for a caller that gives no measurement at all; the app always gives one. Why the measurement failed on that card is not known: the log was only in the window and went with the reboot. | | A confirm probe that is too hot at once confirms nothing | A confirm probe that ends `TOO HOT` keeps the clock edge: over 30 s the card may run a little hotter than over the power step's 20 s, and that is the soak's business, which lowers power. But the stress run stops at its first reading above the limit, so a card that is above it when the probe begins gives `TOO HOT` within a second, and that counted as holding too. It happened on the tester's RTX 3080 Ti (hardware check 91): after a driver reset the health probe loads the card at stock power, which on that card means 76 C against a limit of 75, and `confirm core +225` was confirmed by a probe that had tested nothing. Now such a probe counts only when the clock was loaded for `kConfirmMinS` (10 s, clearly longer than the 3 s probe that found the edge) before it got too hot. Otherwise the value is not confirmed, and the card rests (`io.rest`, the 20 s of a recovery, with the fans where the curve last put them) before the next try one step lower: without the rest every lower value would start just as hot, and heat alone would step the edge down to nothing, which is what keeping the edge on `TOO HOT` was for. The same value is not tried again; one step is the price. Not changed: the health probe still runs at stock power, and a soak that begins too hot still lowers the power limit by a step. | | A profile that promises speed keeps no result that is slower than stock | Best of my GPU ended 3.8 % slower than stock on the tester's RTX 3080 Ti (4749 against 4937 it/s, hardware check 91), with the fans at 80 % instead of 66 %, and showed it as **Done**, applied and kept at every logon. Every step had done what it is for: the card runs 83 C at stock with its fans at 100 %, the profile allows 75, so the power limit went to 65 %, and +150 MHz of core clock did not win that back. But a profile called "the best of my GPU" that leaves the card slower has not done what it says, and a result like that is not offered as one. After the soak, `optimize` compares its score with the measurement at stock that went before the run (the "before" the result screen shows, so the rule judges by the number the user sees): below `kNotSlowerScore` (0.99: one percent is what two five-minute runs differ by) the run ends as a failed one does, at stock, with nothing saved, and says what it found, why, and that Undervolt is the profile for a card that runs warm. `Objectives::faster_than_stock` marks the two profiles this holds for, Best of my GPU and Max performance. Not Quiet and Cool & efficient, which trade speed for noise and heat and say so. Not the overclock step of All in one either: it goes on to an undervolt and a fan tune, and its result is judged as a whole; whether that whole may be slower than stock is not decided. This is a backstop, not the cure: what such a card needs from these profiles is in `docs/planned.md` ("never hotter than at stock"). | -| A step of the memory scan is the middle one of three readings over a longer copy, and the scan stops at a 3 % drop | Four runs on one day applied +650, +1600, +550 and +100 MHz on the reference RTX 4070, a card that runs every game at +1500 (hardware check 94). That card has no bandwidth peak: bandwidth rises by about 0.4 % per 50 MHz step all the way to +2500 and the first wrong result comes at +2550, so the scan could only end early by chance, and where it did decided the offset. Measured afterwards without a search and with nothing changed on the card (hardware check 95), the readings move in three ways, and each got its own answer. **(1) One timed run was too short.** A run copied 32 GB, 75 ms on this card, and a reading is the median of three runs that agree within 1 %. Such readings scattered by 0.4 to 1.3 % (standard deviation over 24 readings, six series), single ones by up to 4.8 %, and in two of the six series one reading did not settle. With 128 GB per run (`kBwDispatches` 256, 0.3 s) it is 0.2 %, and all 96 readings settled. **(2) A single reading decided a step.** A step is now the median of `kBandwidthReadings` (3) readings, taken one after the other while the candidate is still applied. A reading that only did not settle is left out when two others gave a number; any other failure ends the step at once, as before. The log shows all of them: `mem +200: bandwidth 440.9 GB/s (441.2 440.9 440.1)`. **(3) The readings change level.** For up to a minute at a time every reading is about 1.8 % higher than before and after (452 against 444 GB/s), at one and the same offset. The cause is not known. A median of three does nothing against it: all three are inside the stretch. So the drop that ends the scan (`kBandwidthDrop`) went from 1 % to 3 %, still re-measured once: above what the readings do by themselves, below what a real peak loses within a step or two. The price on a card that has a peak: the scan goes a step or two further past it, in 3 s probes, before it stops. The offset it keeps is still the lowest within 1 % of the best (`kBandwidthTie`), and confirming starts there. **Not built: a window that stands still during a measurement.** The suspicion was that the window's own drawing disturbed the readings of the run that saved +100. Measured with a window drawing a frame every 100 ms and every 33 ms next to the measurement: 0.2 %, the same as without. **What it costs:** a step lasts about 3 s longer, a scan to +2500 about two and a half minutes. **What is left open:** on a card without a peak the scan now always ends at the first wrong result, which the bandwidth method exists to stay away from; the tie rule and the margin are all that keep the applied offset away from it (+1600 against +2550 in the one run of check 94 that got there). Whether such a card should get its memory offset by another rule (a fixed share of the range the scan covered) is not decided. And a change of level larger than 3 % would end a scan by chance again: the answer to that is to compare a low step with a fresh reading of the best offset taken right after it, not a still higher threshold. | +| A step of the memory scan is the middle one of three readings over a longer copy, and the scan stops at a 3 % drop | Four runs on one day applied +650, +1600, +550 and +100 MHz on the reference RTX 4070, a card that runs every game at +1500 (hardware check 94). That card has no bandwidth peak: bandwidth rises by about 0.4 % per 50 MHz step all the way to +2500 and the first wrong result comes at +2550, so the scan could only end early by chance, and where it did decided the offset. Measured afterwards without a search and with nothing changed on the card (hardware check 95), the readings move in three ways, and each got its own answer. **(1) One timed run was too short.** A run copied 32 GB, 75 ms on this card, and a reading is the median of three runs that agree within 1 %. Such readings scattered by 0.4 to 1.3 % (standard deviation over 24 readings, six series), single ones by up to 4.8 %, and in two of the six series one reading did not settle. With 128 GB per run (`kBwDispatches` 256, 0.3 s) it is 0.2 %, and all 96 readings settled. **(2) A single reading decided a step.** A step is now the median of `kBandwidthReadings` (3) readings, taken one after the other while the candidate is still applied. A reading that only did not settle is left out when two others gave a number; any other failure ends the step at once, as before. The log shows all of them: `mem +200: bandwidth 440.9 GB/s (441.2 440.9 440.1)`. **(3) The readings change level.** For up to a minute at a time every reading is about 1.8 % higher than before and after (452 against 444 GB/s), at one and the same offset. The cause is not known. A median of three does nothing against it: all three are inside the stretch. So the drop that ends the scan (`kBandwidthDrop`) went from 1 % to 3 %, still re-measured once: above what the readings do by themselves, below what a real peak loses within a step or two. The price on a card that has a peak: the scan goes a step or two further past it, in 3 s probes, before it stops. The offset it keeps is still the lowest within 1 % of the best (`kBandwidthTie`), and confirming starts there. **Not built: a window that stands still during a measurement.** The suspicion was that the window's own drawing disturbed the readings of the run that saved +100. Measured with a window drawing a frame every 100 ms and every 33 ms next to the measurement: 0.2 %, the same as without. **What it costs:** a step lasts about 3 s longer, a scan to +2500 about two and a half minutes. **On the card (check 95 b):** two runs in a row applied +1400 and +1450, both 4 % faster than stock, and no step came within reach of ending the scan early. **What is left open, and keeps this from being merged:** on a card without a peak the scan now always runs to where the memory gives out, which the bandwidth method exists to stay away from, and on the reference card it no longer ends there with a wrong result from the stress load: both runs lost the device during the bandwidth copy, at +2450 and +2500, a driver reset each time. A step copies about seven times as much as before and the copy checks nothing. The search recovers and the result is good, but a reset in every run is not acceptable as the normal end of a scan. The ways out, none chosen: end the scan before the edge by another rule (a card whose bandwidth still rises needs none of the last steps: the applied offset is 70 % of the confirmed one); take the three readings only where one reading leaves a doubt (a step that looks like a drop) and one reading elsewhere, which brings the copying near the edge back to what it was; or have the copy check its result, so that failing memory shows as a wrong result. Whether such a card should get its memory offset by another rule (a fixed share of the range the scan covered) is not decided. And a change of level larger than 3 % would end a scan by chance again: the answer to that is to compare a low step with a fresh reading of the best offset taken right after it, not a still higher threshold. | | Open: why the bandwidth measurement fails on some cards, and what then | On the tester's RTX 3080 Ti it failed at +0, and since 0.4.4 that means no memory overclock at all on such a card, though it computed right far beyond +2000 MHz. The measurement gives a number only when three runs in a row agree within 1 % (`kBwSettle`), and it throws away every run the driver's timer calls disjoint (the clock changed under it). That card was held at a power limit of 65 % and ran at 83 C when it was measured; the reference card has only ever been measured at 105 %, where the measurement itself draws 179 of 210 W. So the suspicion is the power limit, and hardware check 90 tests exactly that on the reference card. What it shows decides what is built, and nothing is built before it: **(1) runs are thrown away as disjoint, and the ones that are kept agree:** the timer is what fails, not the card. Time the copy differently (wall clock around a flush, or count a disjoint run when its neighbours agree) and keep the method. **(2) the readings themselves scatter by more than 1 %:** a peak that is found by a 1 % drop (`kBandwidthDrop`) cannot be found in that noise. Then the memory has to be measured where the card is not held back: search it before the power step, at stock power, where a probe and its measurement last five seconds; or take more readings and compare medians. **(3) it does not fail at 65 % on the reference card:** the power limit is not the cause. Then it was the heat (a card throttling at its temperature limit changes clock as well), or another program, and only the tester's `run.log` can say: since this change a measurement that never settled writes how many runs gave a reading, between which values, and how many were thrown away. In no case does the climb on stability alone come back. **Outcome of check 90: (3).** On the reference card the measurement gave a number fifteen times out of fifteen at 100, 80 and 65 %, in three or four runs, with nothing thrown away, and the bandwidth was the same at every limit. The power limit is ruled out; (1) and (2) are not built. What is left is the heat of that card, another program, or a timer query that failed, and the tester's `run.log` is the next thing that can tell them apart. **The tester's run (check 91): it is (2), but only inside a run.** Twenty readings between 763.9 and 821.2 GB/s, none thrown away, three seconds after the load began; after thirty seconds of load the same card read within 0.2 %. The suspicion now is the moment, not the card: the measurement is taken while the power limiter is still catching up. If `gao --bandwidth --after-load 3` scatters on that card and `--after-load 30` does not, that is it, and the fix is to measure when the card has settled (keep taking runs for some seconds instead of twenty runs), not to change the method. | | Every line of a run is also in `run.log` | `logged()` wraps the log hook of both front ends: a line is appended and flushed before it is shown, so the step a machine freezes in is the last line on disk. One file in the state folder, moved aside at 1 MB (`run.log.old`). Only elevated and only after `ensure_app_dir` has verified the folder: an elevated process appends to nothing a user could have put there. | | A run does not start while the NVIDIA app's Automatic Tuning is on | Switched on, a service of the driver scans the card when it sees fit: it loads it, changes its clocks, and starts again by itself (hardware check 89: a scan that was stopped started again 14 ms later, and after a restart of the service it resumed within two seconds). A run next to that measures nothing it can trust, so it is refused, like a run next to Afterburner, and with the same weight: a courtesy, not a guard. Read from where that service keeps the switch, `%ProgramData%\NVIDIA Corporation\nvtopps\nvtopps.db3`, table `ocScannerSettings`, fields `backgroundScanEnabled` and `grdOcEnabled`, joined to `TelemetryGpuCfg` so that only a card that is in the PC counts (a row of an earlier card can never be switched off again). That this is the source and not a copy of it was measured, not assumed: with the field set the service took the scan up again after every restart, with it cleared it did not, while the NVIDIA app itself showed the switch as off. Nothing documents the file. So: opened read-only and immutable (no lock, the service never waits for us), through `winsqlite3.dll` from System32 (no dependency added), and everything that is not exactly as expected reads as "unknown", which starts the run as before. The registry value the app sets next to it (`HKLM\SOFTWARE\NVIDIA Corporation\Global\NvApp\OC\`) changes at the same moment but belongs to the app, not to the service, and is not used. **The overlay is not checked:** no setting for it was found, its processes run whether it is open or not, and the one value that moved (`IsShadowPlayEnabled`) followed the service being up, not a switch. NVAPI was not tried: an undocumented call is no better than an undocumented file, and it cannot be compared with anything. | diff --git a/docs/hardware-checks.md b/docs/hardware-checks.md index 8f4e437..fea640e 100644 --- a/docs/hardware-checks.md +++ b/docs/hardware-checks.md @@ -122,7 +122,7 @@ root; adjust the path for a Debug build. | 92 | The undervolt profile starts with the measurement at stock every run uses, and a hot card is not refused | Elevated, the tray app exited, a copy of `gao.json` kept. (a) `gao --undervolt` to the end; then `gao --status`. (b) The same with a test build whose stock limit is lowered to a temperature the reference card passes under load (63 C; it reaches 67). Put the copy of `gao.json` back. (c) Only on a card that is above the profile's limit at stock with the fans at 100 % (the tester's RTX 3080 Ti: 82 to 83 C): the **Undervolt** profile from the window, to the end, and its `run.log` | yes | (a) `the card at stock, five minutes`, a `stock: STABLE ...` line with the driver's fan speed, then `baseline: the measurement at stock`, and no second five minutes at stock: the first candidate follows at once; no `candidates are stopped at` line; `RESULT:`; `before:` shows 300 s and the fan speed the driver chose. (b) `stock: TOO HOT ... peak=64 C`, the line that the measurement stopped there, the cool-down, then `baseline: the measurement at stock` and a search that runs to `RESULT:` with `before:` showing the shortened time; no `candidates are stopped at` line, because 64 C is under the profile's 75. (c) no `not stable at stock`; `the card reached 86 C at stock: candidates are stopped at 85 C, not at the profile's 75 C`; candidates that end `STABLE` or `WRONG RESULT`, none `TOO HOT`; a result that is cooler and draws less than stock at the same clock | First run: **not as expected**, and that found a fault (2026-10-11, reference RTX 4070, driver 617.42, a test build of d1c99e9, started elevated by a script). The first halves held: (a) `stock: STABLE ... peak=65 C fan=70 %`, then `baseline: the measurement at stock` four seconds later and no second five minutes; (b) `stock: TOO HOT ... peak=64 C`, stopped after 34 s, the run went on, no `candidates are stopped at` line. But both runs then ended with `the card held no undervolt: the first step below its own voltage already failed`: their first candidates read `1070 mV (+15 MHz): SLOW ... clock=2323 MHz` and `1050 mV (+15 MHz): SLOW ... clock=2411 MHz`, on a card that ran 2791 MHz in the measurement just before. The search's load had not run yet when that candidate began (see the decision "With its reference handed over, the undervolt search warms its load up before the first candidate"). Second run, with the warm-up: **pass** for (a) and (b) (same day, a test build of 0733695). (a) `stock: STABLE score=5426 it/s clock=2788 MHz peak=66 C fan=70 %`, four seconds later `baseline: the measurement at stock`, `warming the load up: 5 s at stock`, `target: 2790 MHz, which the card reaches at 1090 mV by itself`, and the first candidate `1085 mV (+15 MHz): STABLE ... clock=2775 MHz`; 26 candidates, none `SLOW` or `TOO HOT`, the edge at 950 mV, `RESULT: 2790 MHz at 990 mV instead of 1090 mV`; `before:` 5426 it/s, 189 W, fan 70 % (300 s), `after:` 5325 it/s, 149 W, fan 53 % (300 s). (b) `stock: TOO HOT ... peak=64 C`, stopped after 74 s, the run went on, the same warm-up and a search to the same result (990 mV, edge 950 mV); `before:` shows 75 s; no `candidates are stopped at` line. The saved tune of before the test was put back and applied. (c) not run. | | 93 | A speed profile that would end slower than stock keeps nothing | Only on a card that is warmer at stock than the profile allows (the tester's RTX 3080 Ti; the reference card ends faster than stock with every speed profile): **Best of my GPU** from the window, to the end; the result screen, the dashboard and `run.log` | yes | the run goes through all its steps and its soak; then **Not applied** with `the result is slower than stock: N it/s against M (-X %)`, the power limit it had to go down to, and the pointer to Undervolt; `The card is back at stock.`; the dashboard shows stock and the saved tune of before the run, if there was one; `run.log` ends with `stopped: the result is slower than stock ... -- card restored to stock` | Not run yet. On the reference card: nothing to see, by design. | | 94 | All in one, with the undervolt step searching for the first time | Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize all` to the end with a build that has the warm-up of the undervolt step; then `gao --status`; put the copy back | yes | five steps; in step 3 a `warming the load up` line, a first candidate that is not `SLOW`, and a descent; steps 4 and 5 run on the tune of steps 2 and 3; `RESULT:` | **Not as expected at step 4**, and that found a fault (2026-10-11, reference RTX 4070, driver 617.42, a test build of 0733695, unattended). Step 1: `stock: STABLE score=5325 it/s peak=65 C fan=70 % power=186 W`. Step 2: power 105 %, `mem: bandwidth peak +2300, confirmed +2300, applying +1600`, `core: highest stable +210, confirmed +210, applying +135`, soak `STABLE score=5568 it/s peak=70 C power=197 W`. Step 3, as wanted: `warming the load up: 5 s with the overclock`, `target: 2865 MHz, which the card reaches at 1050 mV with the overclock`, nine candidates `STABLE` from 1045 mV down, `WRONG RESULT` at 990 mV, confirmed at 1010 mV after two tries, and `applying 1050 mV (+135 MHz at that point)`: the margin left no raise above the overclock, so the curve was cut off, as in the runs before the warm-up; soak `STABLE score=5434 it/s peak=67 C power=176 W`. Step 4 did not begin: `the tune could not be applied for the fan tune: the undervolt was not applied: the point already runs more than the target -- card at stock`. The saved tune of before the test was put back and applied. Cause and fix: the decision "A flat top goes back on a card that is cooler than when it was found". Second run, with the `gao.exe` of the released v0.4.6: **pass, all five steps** (same day, started elevated by a script). Step 1: `stock: STABLE score=5309 it/s peak=66 C fan=70 % power=185 W`. Step 2: power 105 %, `mem: bandwidth peak +850, confirmed +850, applying +550`, `core: highest stable +210, confirmed +210, applying +135`, soak `STABLE score=5568 it/s`. Step 3: `target: 2850 MHz, which the card reaches at 1045 mV with the overclock`, `undervolt: lowest stable 975 mV, confirmed 985 mV, applying 1025 mV (+165 MHz at that point)`, soak `STABLE score=5363 it/s peak=66 C power=163 W`. Step 4: the tune went back on the card, `fan 60 %: 63 C`, `fan 55 %: 65 C`, `fan 52 %: 67 C`, `fan tune: 52 % holds 67 C`. Step 5: the final test ran its 300 s. `RESULT: power 105 %, core +135 MHz, mem +550 MHz, 2850 MHz at 1025 mV, fans 52 % at 67 C`; `stock: 5309 it/s, 185 W, fan 70 %, 66 C`, `now: 5353 it/s, 166 W, fan 52 %, 70 C`: `+0.8 % score and -10.3 % power against stock`; `undervolt: applied`. The saved tune of before the test was put back. **What this run did not show:** its undervolt had a raise of its own (+165 over an offset of +135), so the anchor was raised and not merely cut off, and the case of the first run (an anchor that reads a step above the target on a cooler card) did not come up. That fix has still only its tests. **The memory offset differs a lot from run to run on this card, and the two logs say why:** +1600 in the first run (bandwidth peak +2300), +550 in the second (peak +850), +650 in the run that saved the tune in use that morning (the journal shows its scan ending at +1000 too). The card was in the same state each time (68 C, power 105 %, a reading every three to four seconds). In both logged scans the bandwidth rises in a straight line, 0.36 and 0.46 % per 50 MHz step, from 420 GB/s at +0 to 509 GB/s at +2450; the first run went on to `WRONG RESULT` at +2550 without the bandwidth ever turning down. The readings scatter around that line with a standard deviation of 0.4 and 0.7 %, single ones by more than 1 %. The scan ends at a reading more than 1 % below the best so far when a second reading agrees, and in the second run that was chance: +950 read high (462.8 GB/s, where the line is at about 458), +1000 read 456.6 twice, 1.3 % below it. In the first run the one low reading (+600) was followed by a higher one, and the scan ran to the first wrong result. So on this card there is no bandwidth peak before the errors, and where the scan stops is decided by noise. | -| 95 | The memory scan no longer ends by chance | (a) Not elevated, nothing changed on the card: a measuring build takes 24 bandwidth readings in one process, with 3 s of the stress load before each as a scan has, at 64, 128 and 256 dispatches per timed run; then the same with the app's window open next to it, drawing a frame every 100 ms and every 33 ms. (b) Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize best` twice in a row with the build of the change, then the copy put back | (a) no, (b) yes | (a) how much a reading scatters, and on what that depends. (b) in both runs every `mem +N: bandwidth` line shows three readings that agree within about 1 %; the scan does not end before the first `WRONG RESULT`; the two `mem: bandwidth peak` values are within 300 MHz of each other and the applied offset is above +1000; the soak is not slower than at stock | **(a) measured** (2026-10-11, reference RTX 4070, driver 617.42, the saved tune applied: power 105 %, core +135, memory +650; standard deviation over the 24 readings of a series, and the lowest and highest against the mean). 64 dispatches, no window: 0.44 % (-0.55 to +1.07, one reading did not settle), 0.66 % (-1.58 to +1.49, one did not settle), 1.31 % (-4.83 to +1.27), 0.86 % (-1.44 to +2.05). 128 dispatches, no window: 0.43 % (-1.10 to +0.79). 256 dispatches, no window: 0.22 % (-0.43 to +0.42) and 0.60 % (-0.80 to +1.37; the last five readings of this series were 451 to 453 GB/s after nineteen of 444 to 449). With the window drawing every 100 ms: 64 dispatches 1.13 % (the first six readings 452 to 453 GB/s, the rest 439 to 451), 256 dispatches 0.24 % (-0.47 to +0.61). With the window drawing every 33 ms: 64 dispatches 0.58 %, 256 dispatches 0.25 % (-0.35 to +0.51). A reading took 6.5, 5.3 and 3.9 timed runs to settle at 64, 128 and 256 dispatches. So: the longer timed run takes the scatter from 0.4 to 1.3 % down to 0.2 %, the window's drawing adds nothing, and apart from the scatter the readings sit about 1.8 % higher for up to a minute at a time (the stretch at the end of the second 256 series ran on into the next series; the last 64 series went back and forth between 444 and 452 GB/s). What sets that level is not known. **(b) not run yet.** | +| 95 | The memory scan no longer ends by chance | (a) Not elevated, nothing changed on the card: a measuring build takes 24 bandwidth readings in one process, with 3 s of the stress load before each as a scan has, at 64, 128 and 256 dispatches per timed run; then the same with the app's window open next to it, drawing a frame every 100 ms and every 33 ms. (b) Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize best` twice in a row with the build of the change, then the copy put back | (a) no, (b) yes | (a) how much a reading scatters, and on what that depends. (b) in both runs every `mem +N: bandwidth` line shows three readings that agree within about 1 %; the scan does not end before the first `WRONG RESULT`; the two `mem: bandwidth peak` values are within 300 MHz of each other and the applied offset is above +1000; the soak is not slower than at stock | **(a) measured** (2026-10-11, reference RTX 4070, driver 617.42, the saved tune applied: power 105 %, core +135, memory +650; standard deviation over the 24 readings of a series, and the lowest and highest against the mean). 64 dispatches, no window: 0.44 % (-0.55 to +1.07, one reading did not settle), 0.66 % (-1.58 to +1.49, one did not settle), 1.31 % (-4.83 to +1.27), 0.86 % (-1.44 to +2.05). 128 dispatches, no window: 0.43 % (-1.10 to +0.79). 256 dispatches, no window: 0.22 % (-0.43 to +0.42) and 0.60 % (-0.80 to +1.37; the last five readings of this series were 451 to 453 GB/s after nineteen of 444 to 449). With the window drawing every 100 ms: 64 dispatches 1.13 % (the first six readings 452 to 453 GB/s, the rest 439 to 451), 256 dispatches 0.24 % (-0.47 to +0.61). With the window drawing every 33 ms: 64 dispatches 0.58 %, 256 dispatches 0.25 % (-0.35 to +0.51). A reading took 6.5, 5.3 and 3.9 timed runs to settle at 64, 128 and 256 dispatches. So: the longer timed run takes the scatter from 0.4 to 1.3 % down to 0.2 %, the window's drawing adds nothing, and apart from the scatter the readings sit about 1.8 % higher for up to a minute at a time (the stretch at the end of the second 256 series ran on into the next series; the last 64 series went back and forth between 444 and 452 GB/s). What sets that level is not known. **(b) the offset is the same twice; the scan ends in a driver reset twice** (2026-10-11, same card and driver, a test build of 2764609, unattended, 40 minutes for the two runs). Run 1: `stock: STABLE score=5311 it/s peak=66 C fan=70 % power=185 W`; 49 steps from `mem +0: bandwidth 422.7 GB/s (422.7 422.6 422.7)` to `mem +2400: bandwidth 501.4 GB/s (501.3 502.3 501.4)`; at +2450 the candidate was `STABLE` and the measurement after it lost the device: `mem +2450: the bandwidth measurement failed; treated as a driver reset`, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, 20 s of rest, `health: STABLE`, `confirm core +0 / mem +2050: STABLE`, `mem: bandwidth peak +2250, confirmed +2050, applying +1400`; `core: highest stable +210, confirmed +210, applying +135`; soak `STABLE score=5536 it/s peak=69 C power=195 W`; `driver resets during this run: 1`. Run 2: stock 5312 it/s; 50 steps from 421.8 GB/s at +0 to 502.8 at +2450; the same failure at +2500; `mem: bandwidth peak +2300, confirmed +2100, applying +1450`; core +135; soak `STABLE score=5555 it/s peak=69 C power=196 W`; one driver reset. The three readings of a step differed by 0.4 % on average and by 1.2 % at most; no step was more than 0.7 % below the best before it, so neither scan came near ending early, by the old rule or the new; every step had its three readings. The scan took 5 min 25 s (about 6.5 s a step; about 3 min before the change). As expected: the two peaks are 50 MHz apart, the applied offsets are +1400 and +1450, and both results are 4 % faster than stock. **Not as expected: neither scan reached a `WRONG RESULT`.** Both ended one and two steps before +2550 with the device lost during the bandwidth copy; Windows logged it both times (System log, `nvlddmkm` events 14 and 153 at 19:51:22 and 20:11:27), and the screen goes black for a moment. In the run of check 94 that got this far, the measurements at +2450 and +2500 gave a number and the stress load reported the first wrong result at +2550, without a reset. A step now copies about seven times as much as it did (three readings of about four runs of 128 GB, against one reading of about six runs of 32 GB), and the copy checks nothing, so where the memory starts to fail it is the driver that gives out, not a verdict. Two runs against one: likely the cause, not proven. The search handles it as it should (rest, health probe, confirm four steps lower, the core is still searched), but a driver reset in every run on a card like this is not what the change was for. Not merged; what to do about it is open in `docs/development.md`. The saved tune of before the test was put back and applied (`applied best: power 105 %, core +135 MHz, mem +650 MHz`), the tray app restarted, Defender reported nothing. | ## Notes diff --git a/src/hw/stress.cpp b/src/hw/stress.cpp index 3e4c394..be18f37 100644 --- a/src/hw/stress.cpp +++ b/src/hw/stress.cpp @@ -370,7 +370,10 @@ std::optional Stress::MeasureBandwidth() { UINT64 t0 = 0, t1 = 0; if (!wait(d.q_disjoint.Get(), &dj, sizeof(dj)) || !wait(d.q_begin.Get(), &t0, sizeof(t0)) || !wait(d.q_end.Get(), &t1, sizeof(t1))) { - if (d.device->GetDeviceRemovedReason() != S_OK) d.device.Reset(); + // Say which: the log line used to end in nothing here. + const HRESULT removed = d.device->GetDeviceRemovedReason(); + error_ = removed != S_OK ? Hr("the device was lost during the copy", removed) : "the timing of the copy did not come back"; + if (removed != S_OK) d.device.Reset(); return std::nullopt; } if (dj.Disjoint || dj.Frequency == 0 || t1 <= t0) { // clock changed mid-run: discard From 71e7c2b1bbeb63637e571be6603a28bfd922c337 Mon Sep 17 00:00:00 2001 From: Rovey Date: Sun, 11 Oct 2026 20:25:18 +0200 Subject: [PATCH 03/15] fix: the memory scan reads a step three times only when it is in doubt Three readings at every step made the offset repeat on the reference card (+1400 and +1450 in two runs), and ended both scans in a driver reset during the bandwidth copy near the memory's edge (hardware check 95 b): a step copied about seven times as much as before, and the copy checks nothing. A step is one reading again. Only a step that is more than kBandwidthTie below the best is sampled kBandwidthReadings times, and its middle reading counts; that replaces the single re-measure of a drop. A reading that only did not settle is taken again. The longer timed run and the 3 % drop stay. Co-Authored-By: Claude Opus 5.5 --- README.md | 2 +- src/core/search.cpp | 43 +++++++++---------- src/core/search.hpp | 14 +++--- tests/test_search.cpp | 99 ++++++++++++++++++++++++++++++------------- 4 files changed, 99 insertions(+), 59 deletions(-) diff --git a/README.md b/README.md index 49ef16c..2c019b8 100644 --- a/README.md +++ b/README.md @@ -177,7 +177,7 @@ flowchart LR - **Every run starts with the card at stock for five minutes,** as long as the result is tested at the end, with the fans left to the NVIDIA driver. That is the "before" of the result: a card is not heated through after 30 s, so a shorter run would make every result look warmer and louder than stock. Some cards get hot there: the driver's own fan control lets them (a 3080 Ti reached 85 °C with its fans at 65 %). The measurement stops at 85 °C, what it measured until then is the "before", and the run goes on. From there every step keeps the fans on a curve that reaches 100 % at the profile's temperature limit. - **Stress test.** A DirectX 11 compute load in which the GPU checks every value it computes against a known answer. Each probe ends in a verdict: `STABLE`, `WRONG RESULT`, `DEVICE LOST`, `STALLED` (the values were right but the card computed almost nothing), `TOO HOT` or `NO TELEMETRY`. -- **Memory comes first and stops at the bandwidth peak,** not at the first error. GDDR6 and GDDR6X retry failed transfers, so an overclocked memory bus loses speed long before it returns a wrong result. The search raises the offset 50 MHz at a time, measures bandwidth three times at each step, takes the middle reading, and keeps the lowest offset within 1 % of the best. Where bandwidth cannot be measured, the memory is left at stock: without that measurement nothing says where to stop, and the first wrong result comes far too late. +- **Memory comes first and stops at the bandwidth peak,** not at the first error. GDDR6 and GDDR6X retry failed transfers, so an overclocked memory bus loses speed long before it returns a wrong result. The search raises the offset 50 MHz at a time, measures bandwidth at each step (three times, taking the middle reading, where a step looks like a drop) and keeps the lowest offset within 1 % of the best. Where bandwidth cannot be measured, the memory is left at stock: without that measurement nothing says where to stop, and the first wrong result comes far too late. - **The core search climbs from stock.** With the chosen memory offset applied, it raises the core offset 15 MHz at a time until a probe fails, and keeps the last value that passed. How far it may go comes from the range the driver reports for your card, so a card that can hold more is not stopped at a fixed limit. - **Each edge is confirmed.** The value a search ends on gets a 30 s probe, and a lower value is tried if it does not hold. The safety margin is applied to the confirmed value, and the result as a whole must pass the 300 s soak. - **A driver reset ends that clock's search.** On a card the search can push past its edge, a search resets the driver: the screen goes black for a moment. The run then reconnects to the driver, sets the card to stock, leaves it without load for 20 seconds and checks with a short probe at stock that it computes normally again. After that the search tries no new values for that clock: it keeps the last value that passed and confirms four steps below it. Each clock has its own search, so a reset in the memory search does not keep the core from being searched; a run can therefore reset the driver twice, once per clock. One reset more than that ends the run at stock with nothing saved, and so does a card that does not come back. Results on real cards are in [docs/hardware-checks.md](docs/hardware-checks.md), checks 51 and 52. diff --git a/src/core/search.cpp b/src/core/search.cpp index 0da6b7d..34e6d68 100644 --- a/src/core/search.cpp +++ b/src/core/search.cpp @@ -68,11 +68,16 @@ int best_bandwidth_offset(int lo, int hi, int step, int ceiling, const std::func for (int v = lo; v <= top; v += step) { MemSample s = sample(v); if (!s.stable) break; - // A drop is measured twice before it ends the scan: a single reading - // can land while the memory clock is still in a lower P-state. - if (s.gbps < best * (1 - kBandwidthDrop)) { - s = sample(v); + // A step that is further below the best than readings scatter is in + // doubt: it is read kBandwidthReadings times and the middle one counts. + // Only such a step: every reading is a copy that checks nothing, and + // near its edge the memory answers more of it with a driver reset. + if (s.gbps < best * (1 - kBandwidthTie)) { + std::vector readings{s.gbps}; + while (static_cast(readings.size()) < kBandwidthReadings && (s = sample(v)).stable) readings.push_back(s.gbps); if (!s.stable) break; + std::sort(readings.begin(), readings.end()); + s.gbps = readings[readings.size() / 2]; } seen.emplace_back(v, s.gbps); best = std::max(best, s.gbps); @@ -571,10 +576,10 @@ OptimizeResult optimize(const GpuControl& gpu, const Objectives& obj, Journal& j bool unavailable = false; const int peak = best_bandwidth_offset(0, bounds.mem_max_mhz, kMemStep, ceiling, [&](int v) { MemSample m; - // kBandwidthReadings readings, and the step is their middle - // one. A reading that only did not settle is left out; any - // other failure ends the step at once. - std::vector readings; + // One reading. One that only did not settle is taken again, + // kBandwidthReadings times at most; any other failure ends + // the step at once. + std::optional gbps; bool failed = false; // The +0 sample is journaled without a clock, like a soak at // stock: if the machine froze in it, an open entry that said @@ -582,13 +587,13 @@ OptimizeResult optimize(const GpuControl& gpu, const Objectives& obj, Journal& j const std::optional mem_j = v > 0 ? std::optional(v) : std::nullopt; m.stable = explore(v, reset_at, [&] { if (!is_stable(clock_candidate(std::nullopt, mem_j, 0, v, kClockProbeS, [&] { - for (int i = 0; i < kBandwidthReadings && !failed; ++i) { - if (const auto gbps = io.bandwidth()) readings.push_back(*gbps); - else failed = !(io.bandwidth_unsettled && io.bandwidth_unsettled()); + for (int i = 0; i < kBandwidthReadings && !gbps && !failed; ++i) { + gbps = io.bandwidth(); + failed = !gbps && !(io.bandwidth_unsettled && io.bandwidth_unsettled()); } }))) return false; - if (!failed && readings.size() >= 2) return true; + if (gbps) return true; // The candidate passed, the measurement failed: the step is unusable. if (v > 0 && !failed) { // The load computed and the readings disagreed: noise, @@ -610,18 +615,10 @@ OptimizeResult optimize(const GpuControl& gpu, const Objectives& obj, Journal& j return false; }); if (m.stable) { - std::string each; - for (const double g : readings) { - char one[32]; - std::snprintf(one, sizeof(one), "%s%.1f", each.empty() ? "" : " ", g); - each += one; - } - std::sort(readings.begin(), readings.end()); - const size_t n = readings.size(); - m.gbps = n % 2 ? readings[n / 2] : (readings[n / 2 - 1] + readings[n / 2]) / 2; + m.gbps = *gbps; char buf[64]; - std::snprintf(buf, sizeof(buf), "mem +%d: bandwidth %.1f GB/s (", v, m.gbps); - log(buf + each + ")"); + std::snprintf(buf, sizeof(buf), "mem +%d: bandwidth %.1f GB/s", v, m.gbps); + log(buf); } return m; }); diff --git a/src/core/search.hpp b/src/core/search.hpp index 1ec4133..732d9b0 100644 --- a/src/core/search.hpp +++ b/src/core/search.hpp @@ -77,10 +77,13 @@ inline constexpr double kBandwidthDrop = 0.03; // ~0.4 % and one 50 MHz step is worth ~0.5 %, so a narrower band would let // noise pick the offset. inline constexpr double kBandwidthTie = 0.01; -// A step of the memory scan is the middle one of this many readings, so one -// reading that is off, or that did not settle, decides nothing. On the -// reference card single readings ended scans at +250, at +1000 and not at all -// on one day, and the runs saved +100, +650 and +1600 MHz (hardware check 94). +// A step of the memory scan that is in doubt (more than kBandwidthTie below +// the best) is the middle one of this many readings, so one reading that is +// off decides nothing; and a reading that did not settle is taken again up to +// this many times. On the reference card single readings ended scans at +250, +// at +1000 and not at all on one day, and the runs saved +100, +650 and +1600 +// MHz (hardware check 94). Not every step: three readings at every step ended +// both scans of hardware check 95 in a driver reset near the memory's edge. inline constexpr int kBandwidthReadings = 3; // A confirm probe that ends too hot keeps the clock edge (heat is the power @@ -123,7 +126,8 @@ struct MemSample { // Scans lo, lo+step, ... (<= hi, < ceiling), sampling lo too. Stops at the // first unstable sample or once gbps falls more than kBandwidthDrop below the -// best so far (a drop is re-measured once before it counts). Returns the +// best so far (a step in doubt is sampled kBandwidthReadings times and its +// middle reading counts). Returns the // lowest stable offset within kBandwidthTie of the best, so noise on a flat // curve never drifts the result upward; lo when no sample was stable. int best_bandwidth_offset(int lo, int hi, int step, int ceiling, const std::function& sample); diff --git a/tests/test_search.cpp b/tests/test_search.cpp index 603701f..ae0f68c 100644 --- a/tests/test_search.cpp +++ b/tests/test_search.cpp @@ -624,19 +624,40 @@ TEST_CASE("bandwidth scan stops at the peak of a rising-then-falling curve") { CHECK(max_sampled == 650); // stopped at the first step more than kBandwidthDrop below the peak } -TEST_CASE("a single low bandwidth reading is re-measured before it ends the scan") { - // Measured on the 4070: an occasional reading lands at half speed when the - // memory clock is still in a lower P-state. One such dip must not end the - // scan; a drop that repeats does. +TEST_CASE("a single low bandwidth reading does not end the scan; three do") { + // A step that is further below the best than readings scatter is read + // kBandwidthReadings times and its middle reading counts. One dip (measured + // on the 4070: a reading at half speed, the memory clock still in a lower + // P-state) ends nothing; a drop that is there every time does. + const auto curve = [](int v) { return v <= 600 ? 500 + v * 0.3 : 680 - (v - 600) * 0.9; }; std::map calls; const int r = best_bandwidth_offset(0, 1500, 50, INT_MAX, [&](int v) { const bool dip = v == 300 && calls[v] == 0; ++calls[v]; - return MemSample{true, dip ? 250.0 : (v <= 600 ? 500 + v * 0.3 : 680 - (v - 600) * 0.9)}; + return MemSample{true, dip ? 250.0 : curve(v)}; }); CHECK(r == 600); + CHECK(calls[250] == 1); // nothing in doubt: one reading + CHECK(calls[300] == 3); + CHECK(calls[350] == 1); + CHECK(calls[650] == 3); // the real drop is read three times too, and stands + + // Two low readings of three are the middle one. + calls.clear(); + CHECK(best_bandwidth_offset(0, 1500, 50, INT_MAX, [&](int v) { + const int nth = calls[v]++; + return MemSample{true, v == 300 && nth < 2 ? 250.0 : 500 + v * 0.3}; + }) == 250); + CHECK(calls[300] == 3); + CHECK(calls.count(350) == 0); + + // A step that stops being stable while it is read again ends the scan there. + calls.clear(); + CHECK(best_bandwidth_offset(0, 1500, 50, INT_MAX, [&](int v) { + const int nth = calls[v]++; + return MemSample{!(v == 300 && nth == 1), v == 300 ? 250.0 : 500 + v * 0.3}; + }) == 250); CHECK(calls[300] == 2); - CHECK(calls[650] == 2); // the real drop is confirmed once too } TEST_CASE("bandwidth scan stops at the first unstable step") { @@ -2141,7 +2162,7 @@ TEST_CASE("a failed bandwidth measurement ends the memory scan; the core is stil CHECK(run.logged("core +0 / mem +0: STABLE")); CHECK(run.logged("mem +0: bandwidth 500.0 GB/s")); CHECK(run.card.max_mem_seen == 300); - CHECK(run.card.bw_measurements == 6 * kBandwidthReadings + 1); // +0 .. +250 in full, and the one at +300 that failed + CHECK(run.card.bw_measurements == 7); CHECK(r.mem_max_stable == 250); CHECK(r.mem_confirmed == 50); CHECK(r.mem_mhz == 0); @@ -2216,7 +2237,7 @@ TEST_CASE("no bandwidth is measured after a reset") { const auto r = run.go(Preset::BestOfMyGpu); REQUIRE(r.ok); CHECK(run.logged("core +0 / mem +300: DEVICE LOST")); - CHECK(run.card.bw_measurements == 6 * kBandwidthReadings); // +0 .. +250, and none after the lost candidate + CHECK(run.card.bw_measurements == 6); // +0 .. +250, and none after the lost candidate CHECK(run.card.max_mem_seen == 300); } @@ -2869,33 +2890,48 @@ std::function scatter(unsigned seed, double sd) { } TEST_CASE("one bandwidth reading that is too high does not end the memory scan") { - // The run that saved +100 MHz on a card that holds +1500: the re-measure - // at +200 read 449.0 GB/s where its neighbours read 435 to 441, the two - // readings at +250 were more than 1 % under it, and that was "the peak". + // The run that saved +100 MHz on a card that runs every game at +1500: + // one reading at +200 came out 2.5 % above its neighbours, the two at + // +250 were more than 1 % under it, and that was "the peak". Run run = rising_to_the_edge(); - run.card.bw_noise = [](int mem, int nth) { return mem == 200 && nth == 0 ? 1.03 : 1.0; }; + run.card.bw_noise = [](int mem, int nth) { return mem == 200 && nth == 0 ? 1.025 : 1.0; }; const auto r = run.go(Preset::BestOfMyGpu); REQUIRE(r.ok); CHECK(run.card.max_mem_seen == 2550); // the scan went to the first wrong result CHECK(r.mem_max_stable >= 2300); CHECK(r.mem_mhz >= 1500); - // The step is the middle one of its three readings, and the log shows all three. - CHECK(run.logged("mem +200: bandwidth 440.9 GB/s (454.2 440.9 440.9)")); - CHECK(run.logged("mem +0: bandwidth 434.0 GB/s (434.0 434.0 434.0)")); + CHECK(run.logged("mem +0: bandwidth 434.0 GB/s")); + // The steps after the high one were in doubt and read three times each. + CHECK(run.card.bw_readings_at[150] == 1); + CHECK(run.card.bw_readings_at[250] == kBandwidthReadings); } -TEST_CASE("one bandwidth reading that is too low does not end the memory scan either") { - // The run that stopped at +1000: +950 read high, +1000 read low twice. - Run run = rising_to_the_edge(); - run.card.bw_noise = [](int mem, int nth) { return mem == 950 && nth == 0 ? 1.02 : mem == 1000 && nth < 2 ? 0.985 : 1.0; }; - const auto r = run.go(Preset::BestOfMyGpu); - REQUIRE(r.ok); - CHECK(run.card.max_mem_seen == 2550); - CHECK(r.mem_max_stable >= 2300); +TEST_CASE("one low bandwidth reading does not end the memory scan; three low ones do") { + { + Run run = rising_to_the_edge(); + run.card.bw_noise = [](int mem, int nth) { return mem == 1000 && nth == 0 ? 0.95 : 1.0; }; + const auto r = run.go(Preset::BestOfMyGpu); + REQUIRE(r.ok); + CHECK(run.card.bw_readings_at[950] == 1); + CHECK(run.card.bw_readings_at[1000] == kBandwidthReadings); + CHECK(run.card.max_mem_seen == 2550); + CHECK(r.mem_max_stable >= 2300); + } + { + // From +1000 on the memory loses 5 %: that is a peak. + Run run = rising_to_the_edge(); + run.card.bw_noise = [](int mem, int) { return mem >= 1000 ? 0.95 : 1.0; }; + const auto r = run.go(Preset::BestOfMyGpu); + REQUIRE(r.ok); + CHECK(run.card.bw_readings_at[1000] == kBandwidthReadings); + CHECK(run.card.max_mem_seen == 1000); // the confirm probes are lower + CHECK(r.mem_max_stable >= 800); + CHECK(r.mem_max_stable <= 950); + } } TEST_CASE("readings that scatter as they do on the reference card leave the memory scan where the card puts it") { - // Measured there with the longer timing window: a standard deviation of + // Measured there with the longer timed run: a standard deviation of // 0.22 % per reading. Twice that here, on twenty different days. for (unsigned seed = 1; seed <= 20; ++seed) { Run run = rising_to_the_edge(); @@ -2926,7 +2962,7 @@ TEST_CASE("a real peak is still found where it is when the readings scatter") { } } -TEST_CASE("a reading that does not settle is left out when the other two agree") { +TEST_CASE("a reading that does not settle is taken again") { // On the reference card one reading in twenty-four did not settle, with // nothing wrong. One such reading used to end the scan. Run run = recovering(); @@ -2935,7 +2971,8 @@ TEST_CASE("a reading that does not settle is left out when the other two agree") const auto r = run.go(Preset::BestOfMyGpu); REQUIRE(r.ok); CHECK_FALSE(run.logged("did not settle")); - CHECK(run.logged("mem +300: bandwidth 590.0 GB/s (590.0 590.0)")); + CHECK(run.logged("mem +300: bandwidth 590.0 GB/s")); + CHECK(run.card.bw_readings_at[300] == 2); CHECK(run.card.max_mem_seen > 300); CHECK(r.driver_resets == 0); } @@ -2943,15 +2980,17 @@ TEST_CASE("a reading that does not settle is left out when the other two agree") TEST_CASE("a stretch in which every reading is 2 % off does not end the memory scan") { // Measured on the reference card (hardware check 95): for tens of seconds // at a time every reading is about 1.8 % higher, whatever the offset, and - // then it is back. Three readings of one step are all inside such a - // stretch, so their middle one is too. + // then it is back. The readings of one step are all inside such a + // stretch, so reading it three times changes nothing. for (const double off : {0.98, 1.02}) { Run run = rising_to_the_edge(); - run.card.bw_noise = [&card = run.card, off](int, int) { return card.bw_measurements > 60 && card.bw_measurements <= 110 ? off : 1.0; }; + run.card.bw_noise = [&card = run.card, off](int, int) { return card.bw_measurements > 20 && card.bw_measurements <= 45 ? off : 1.0; }; const auto r = run.go(Preset::BestOfMyGpu); CAPTURE(off); REQUIRE(r.ok); CHECK(run.card.max_mem_seen == 2550); - CHECK(r.mem_max_stable >= 2300); + // The stretch can leave a few steps that read as high as the top of + // the scan; the tie rule then keeps the lowest of them. + CHECK(r.mem_max_stable >= 2000); } } From 9f7a89107c38bf6fa155d668ead2972a209ca670 Mon Sep 17 00:00:00 2001 From: Rovey Date: Sun, 11 Oct 2026 21:06:32 +0200 Subject: [PATCH 04/15] docs: hardware check 95 (c), the reset is still there; design for a scan that does not end in one The same two runs with one reading a step applied +1400 and +1500, both 5 % faster than stock, and both scans again lost the device during the bandwidth copy at +2500. So the reset does not come from reading a step three times: four runs out of four with the longer timed run end this way. docs/development.md gets the open point and a design that is not built: the edge is found once and remembered in the journal, and the stress load, which can report a wrong result, meets the edge before the copy does. Two comments that named the ruled-out cause are corrected, and the error text of a lost device reads as a sentence. Co-Authored-By: Claude Opus 5.5 --- docs/development.md | 3 ++- docs/hardware-checks.md | 2 +- src/core/search.cpp | 4 ++-- src/core/search.hpp | 4 ++-- src/hw/stress.cpp | 2 +- 5 files changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/development.md b/docs/development.md index 15c74c7..1823e21 100644 --- a/docs/development.md +++ b/docs/development.md @@ -79,7 +79,8 @@ The applied offset is always at least one step below the confirmed edge. Profile | Without the bandwidth measurement the memory is left at stock | The search used to fall back to a climb on stability alone when the measurement failed at +0. On a tester's 3080 Ti (0.4.3) that climb went to +2300 MHz with the score falling from +2150 on, got its first wrong result there, and the machine froze with a corrupted screen in the 30 s confirm probes just below it. The first wrong result is no edge to stand next to: the memory was correcting errors for hundreds of MHz before it. Now a measurement that does not work means no memory overclock (`left_at_stock`), the run says so, and the core is still searched. `optimize` keeps the stability climb only for a caller that gives no measurement at all; the app always gives one. Why the measurement failed on that card is not known: the log was only in the window and went with the reboot. | | A confirm probe that is too hot at once confirms nothing | A confirm probe that ends `TOO HOT` keeps the clock edge: over 30 s the card may run a little hotter than over the power step's 20 s, and that is the soak's business, which lowers power. But the stress run stops at its first reading above the limit, so a card that is above it when the probe begins gives `TOO HOT` within a second, and that counted as holding too. It happened on the tester's RTX 3080 Ti (hardware check 91): after a driver reset the health probe loads the card at stock power, which on that card means 76 C against a limit of 75, and `confirm core +225` was confirmed by a probe that had tested nothing. Now such a probe counts only when the clock was loaded for `kConfirmMinS` (10 s, clearly longer than the 3 s probe that found the edge) before it got too hot. Otherwise the value is not confirmed, and the card rests (`io.rest`, the 20 s of a recovery, with the fans where the curve last put them) before the next try one step lower: without the rest every lower value would start just as hot, and heat alone would step the edge down to nothing, which is what keeping the edge on `TOO HOT` was for. The same value is not tried again; one step is the price. Not changed: the health probe still runs at stock power, and a soak that begins too hot still lowers the power limit by a step. | | A profile that promises speed keeps no result that is slower than stock | Best of my GPU ended 3.8 % slower than stock on the tester's RTX 3080 Ti (4749 against 4937 it/s, hardware check 91), with the fans at 80 % instead of 66 %, and showed it as **Done**, applied and kept at every logon. Every step had done what it is for: the card runs 83 C at stock with its fans at 100 %, the profile allows 75, so the power limit went to 65 %, and +150 MHz of core clock did not win that back. But a profile called "the best of my GPU" that leaves the card slower has not done what it says, and a result like that is not offered as one. After the soak, `optimize` compares its score with the measurement at stock that went before the run (the "before" the result screen shows, so the rule judges by the number the user sees): below `kNotSlowerScore` (0.99: one percent is what two five-minute runs differ by) the run ends as a failed one does, at stock, with nothing saved, and says what it found, why, and that Undervolt is the profile for a card that runs warm. `Objectives::faster_than_stock` marks the two profiles this holds for, Best of my GPU and Max performance. Not Quiet and Cool & efficient, which trade speed for noise and heat and say so. Not the overclock step of All in one either: it goes on to an undervolt and a fan tune, and its result is judged as a whole; whether that whole may be slower than stock is not decided. This is a backstop, not the cure: what such a card needs from these profiles is in `docs/planned.md` ("never hotter than at stock"). | -| A step of the memory scan is the middle one of three readings over a longer copy, and the scan stops at a 3 % drop | Four runs on one day applied +650, +1600, +550 and +100 MHz on the reference RTX 4070, a card that runs every game at +1500 (hardware check 94). That card has no bandwidth peak: bandwidth rises by about 0.4 % per 50 MHz step all the way to +2500 and the first wrong result comes at +2550, so the scan could only end early by chance, and where it did decided the offset. Measured afterwards without a search and with nothing changed on the card (hardware check 95), the readings move in three ways, and each got its own answer. **(1) One timed run was too short.** A run copied 32 GB, 75 ms on this card, and a reading is the median of three runs that agree within 1 %. Such readings scattered by 0.4 to 1.3 % (standard deviation over 24 readings, six series), single ones by up to 4.8 %, and in two of the six series one reading did not settle. With 128 GB per run (`kBwDispatches` 256, 0.3 s) it is 0.2 %, and all 96 readings settled. **(2) A single reading decided a step.** A step is now the median of `kBandwidthReadings` (3) readings, taken one after the other while the candidate is still applied. A reading that only did not settle is left out when two others gave a number; any other failure ends the step at once, as before. The log shows all of them: `mem +200: bandwidth 440.9 GB/s (441.2 440.9 440.1)`. **(3) The readings change level.** For up to a minute at a time every reading is about 1.8 % higher than before and after (452 against 444 GB/s), at one and the same offset. The cause is not known. A median of three does nothing against it: all three are inside the stretch. So the drop that ends the scan (`kBandwidthDrop`) went from 1 % to 3 %, still re-measured once: above what the readings do by themselves, below what a real peak loses within a step or two. The price on a card that has a peak: the scan goes a step or two further past it, in 3 s probes, before it stops. The offset it keeps is still the lowest within 1 % of the best (`kBandwidthTie`), and confirming starts there. **Not built: a window that stands still during a measurement.** The suspicion was that the window's own drawing disturbed the readings of the run that saved +100. Measured with a window drawing a frame every 100 ms and every 33 ms next to the measurement: 0.2 %, the same as without. **What it costs:** a step lasts about 3 s longer, a scan to +2500 about two and a half minutes. **On the card (check 95 b):** two runs in a row applied +1400 and +1450, both 4 % faster than stock, and no step came within reach of ending the scan early. **What is left open, and keeps this from being merged:** on a card without a peak the scan now always runs to where the memory gives out, which the bandwidth method exists to stay away from, and on the reference card it no longer ends there with a wrong result from the stress load: both runs lost the device during the bandwidth copy, at +2450 and +2500, a driver reset each time. A step copies about seven times as much as before and the copy checks nothing. The search recovers and the result is good, but a reset in every run is not acceptable as the normal end of a scan. The ways out, none chosen: end the scan before the edge by another rule (a card whose bandwidth still rises needs none of the last steps: the applied offset is 70 % of the confirmed one); take the three readings only where one reading leaves a doubt (a step that looks like a drop) and one reading elsewhere, which brings the copying near the edge back to what it was; or have the copy check its result, so that failing memory shows as a wrong result. Whether such a card should get its memory offset by another rule (a fixed share of the range the scan covered) is not decided. And a change of level larger than 3 % would end a scan by chance again: the answer to that is to compare a low step with a fresh reading of the best offset taken right after it, not a still higher threshold. | +| The memory scan times a longer copy, reads a step in doubt three times, and stops at a 3 % drop | Four runs on one day applied +650, +1600, +550 and +100 MHz on the reference RTX 4070, a card that runs every game at +1500 (hardware check 94). That card has no bandwidth peak: bandwidth rises by about 0.4 % per 50 MHz step all the way to +2500 and the first wrong result comes at +2550, so the scan could only end early by chance, and where it did decided the offset. Measured afterwards without a search and with nothing changed on the card (hardware check 95 a), the readings move in three ways, and each got its own answer. **(1) One timed run was too short.** A run copied 32 GB, 75 ms on this card, and a reading is the median of three runs that agree within 1 %. Such readings scattered by 0.4 to 1.3 % (standard deviation over 24 readings, six series), single ones by up to 4.8 %, and in two of the six series one reading did not settle. With 128 GB per run (`kBwDispatches` 256, 0.3 s) it is 0.2 %, and all 96 readings settled. **(2) A single reading decided the end.** A step that is more than `kBandwidthTie` (1 %) below the best is in doubt: it is sampled `kBandwidthReadings` (3) times and its middle reading counts, which replaces the single re-measure of a drop. A reading that only did not settle is taken again, three times at most; any other failure ends the step at once, as before. Three readings at every step were built first and taken back: see the row below. **(3) The readings change level.** For up to a minute at a time every reading is about 1.8 % higher than before and after (452 against 444 GB/s), at one and the same offset. The cause is not known. Reading a step three times does nothing against it: all three are inside the stretch. So the drop that ends the scan (`kBandwidthDrop`) went from 1 % to 3 %: above what the readings do by themselves, below what a real peak loses within a step or two. The price on a card that has a peak: the scan goes a step or two further past it, in 3 s probes, before it stops. The offset it keeps is still the lowest within 1 % of the best, and confirming starts there. **Not built: a window that stands still during a measurement.** The suspicion was that the window's own drawing disturbed the readings of the run that saved +100. Measured with a window drawing a frame every 100 ms and every 33 ms next to the measurement: 0.2 %, the same as without. **On the card (check 95 b and c):** four runs applied +1400, +1450, +1400 and +1500, each 4 to 5 % faster than stock; no step came within reach of ending a scan early (the lowest was 0.75 % below the best before it), and no step was in doubt. A scan of 50 steps takes 3 min 33 s, half a minute more than before. **What it does not solve** is in the row below, and keeps the change from being merged. A change of level larger than 3 % would end a scan by chance again: the answer to that is to compare a low step with a fresh reading of the best offset taken right after it, not a still higher threshold. | +| Open: on a card without a bandwidth peak the memory scan ends in a driver reset | With the change of the row above the scan on the reference card no longer ends by chance, so it always runs to where the memory gives out. In four runs out of four that was not a wrong result from the stress load but the bandwidth copy losing the device at +2450 or +2500 (`DXGI_ERROR_DEVICE_HUNG`, or the timing of the copy not coming back within its 5 s; `nvlddmkm` events 14 and 153 in the System log; the screen goes black for a moment), with three readings a step and with one (hardware check 95 b and c). So it is not how often a step is read. In the one run before the change that got this far (check 94), the copies at +2450 and +2500 gave a number and the stress load reported the first wrong result at +2550. What differs is the timed run: 128 GB in one go against 32. One run against four is thin, and a memory edge moves by a step from run to run; it is the likely cause, not a proven one. The search handles the reset as it should (rest, health probe, confirm four steps lower, the core still searched) and the result is good. But a reset as the normal end of a scan is not acceptable in a tool that is to be simple and stable. **Design, not built.** A card like this has nothing that announces its edge: the bandwidth rises until the memory fails. No stop rule can know where that is without having been there, so the design does not try to. It makes the meeting happen once, and with the probe that can report it. **(a) The edge is found once and remembered.** The journal already turns a candidate that never finished (the machine froze) into a ceiling for every later run. A memory candidate that ended a scan, by a wrong result, a driver reset or a lost device, becomes one too: later scans on the same card and driver stop `kResetBackoffSteps` (4) steps below it, which is where confirming starts today after a reset. The result is what it is now (confirmed four steps below the failure, 70 % of that applied) and the same in every run, without touching the edge again. It errs to the safe side: an edge found on a bad day stays, until the driver changes or the journal is cleared. Small, and certain to work; it leaves the first run as it is. **(b) The stress load meets the edge before the copy does.** The stress load checks its results and says `WRONG RESULT`; the copy checks nothing and can only hang. So the copy is taken only at offsets the stress load has already passed four steps higher: per step a 3 s probe at +200 MHz above it, then back and the timed copy. On the reference card the load would report the wrong result at +2550 while the highest copy taken is at +2350, where all five scans measured a number. The price: about a second more per step, and on a card that has a peak the probes run four steps beyond the point where the scan would stop. That is 3 s probes, as the 3 % rule already allows a step or two; it is not the climb on stability alone that took a 3080 Ti to +2300, because the copy still ends the scan at the peak. Before (b) is built it has to be measured that the stress load at the edge ends in a wrong result and not in a reset of its own: one observation says so (check 94), core candidates say so in every run, and that is not enough for memory. **Looked at and not proposed.** A fixed top for the scan (an offset, or a share of the driver's range): there is nothing to choose the number by, and a card whose memory gives out below it meets its edge all the same. A top at a bandwidth gain over +0 (say 15 %): the same objection, most cards will fail first. Going back to the short timed run: it brings back the scatter that made the scan end by chance, on the strength of one run. **Order:** (a) first; the measurement for (b) next, on the reference card, with the old and the new timed run a few times each from +2400 up. | | Open: why the bandwidth measurement fails on some cards, and what then | On the tester's RTX 3080 Ti it failed at +0, and since 0.4.4 that means no memory overclock at all on such a card, though it computed right far beyond +2000 MHz. The measurement gives a number only when three runs in a row agree within 1 % (`kBwSettle`), and it throws away every run the driver's timer calls disjoint (the clock changed under it). That card was held at a power limit of 65 % and ran at 83 C when it was measured; the reference card has only ever been measured at 105 %, where the measurement itself draws 179 of 210 W. So the suspicion is the power limit, and hardware check 90 tests exactly that on the reference card. What it shows decides what is built, and nothing is built before it: **(1) runs are thrown away as disjoint, and the ones that are kept agree:** the timer is what fails, not the card. Time the copy differently (wall clock around a flush, or count a disjoint run when its neighbours agree) and keep the method. **(2) the readings themselves scatter by more than 1 %:** a peak that is found by a 1 % drop (`kBandwidthDrop`) cannot be found in that noise. Then the memory has to be measured where the card is not held back: search it before the power step, at stock power, where a probe and its measurement last five seconds; or take more readings and compare medians. **(3) it does not fail at 65 % on the reference card:** the power limit is not the cause. Then it was the heat (a card throttling at its temperature limit changes clock as well), or another program, and only the tester's `run.log` can say: since this change a measurement that never settled writes how many runs gave a reading, between which values, and how many were thrown away. In no case does the climb on stability alone come back. **Outcome of check 90: (3).** On the reference card the measurement gave a number fifteen times out of fifteen at 100, 80 and 65 %, in three or four runs, with nothing thrown away, and the bandwidth was the same at every limit. The power limit is ruled out; (1) and (2) are not built. What is left is the heat of that card, another program, or a timer query that failed, and the tester's `run.log` is the next thing that can tell them apart. **The tester's run (check 91): it is (2), but only inside a run.** Twenty readings between 763.9 and 821.2 GB/s, none thrown away, three seconds after the load began; after thirty seconds of load the same card read within 0.2 %. The suspicion now is the moment, not the card: the measurement is taken while the power limiter is still catching up. If `gao --bandwidth --after-load 3` scatters on that card and `--after-load 30` does not, that is it, and the fix is to measure when the card has settled (keep taking runs for some seconds instead of twenty runs), not to change the method. | | Every line of a run is also in `run.log` | `logged()` wraps the log hook of both front ends: a line is appended and flushed before it is shown, so the step a machine freezes in is the last line on disk. One file in the state folder, moved aside at 1 MB (`run.log.old`). Only elevated and only after `ensure_app_dir` has verified the folder: an elevated process appends to nothing a user could have put there. | | A run does not start while the NVIDIA app's Automatic Tuning is on | Switched on, a service of the driver scans the card when it sees fit: it loads it, changes its clocks, and starts again by itself (hardware check 89: a scan that was stopped started again 14 ms later, and after a restart of the service it resumed within two seconds). A run next to that measures nothing it can trust, so it is refused, like a run next to Afterburner, and with the same weight: a courtesy, not a guard. Read from where that service keeps the switch, `%ProgramData%\NVIDIA Corporation\nvtopps\nvtopps.db3`, table `ocScannerSettings`, fields `backgroundScanEnabled` and `grdOcEnabled`, joined to `TelemetryGpuCfg` so that only a card that is in the PC counts (a row of an earlier card can never be switched off again). That this is the source and not a copy of it was measured, not assumed: with the field set the service took the scan up again after every restart, with it cleared it did not, while the NVIDIA app itself showed the switch as off. Nothing documents the file. So: opened read-only and immutable (no lock, the service never waits for us), through `winsqlite3.dll` from System32 (no dependency added), and everything that is not exactly as expected reads as "unknown", which starts the run as before. The registry value the app sets next to it (`HKLM\SOFTWARE\NVIDIA Corporation\Global\NvApp\OC\`) changes at the same moment but belongs to the app, not to the service, and is not used. **The overlay is not checked:** no setting for it was found, its processes run whether it is open or not, and the one value that moved (`IsShadowPlayEnabled`) followed the service being up, not a switch. NVAPI was not tried: an undocumented call is no better than an undocumented file, and it cannot be compared with anything. | diff --git a/docs/hardware-checks.md b/docs/hardware-checks.md index fea640e..4171b87 100644 --- a/docs/hardware-checks.md +++ b/docs/hardware-checks.md @@ -122,7 +122,7 @@ root; adjust the path for a Debug build. | 92 | The undervolt profile starts with the measurement at stock every run uses, and a hot card is not refused | Elevated, the tray app exited, a copy of `gao.json` kept. (a) `gao --undervolt` to the end; then `gao --status`. (b) The same with a test build whose stock limit is lowered to a temperature the reference card passes under load (63 C; it reaches 67). Put the copy of `gao.json` back. (c) Only on a card that is above the profile's limit at stock with the fans at 100 % (the tester's RTX 3080 Ti: 82 to 83 C): the **Undervolt** profile from the window, to the end, and its `run.log` | yes | (a) `the card at stock, five minutes`, a `stock: STABLE ...` line with the driver's fan speed, then `baseline: the measurement at stock`, and no second five minutes at stock: the first candidate follows at once; no `candidates are stopped at` line; `RESULT:`; `before:` shows 300 s and the fan speed the driver chose. (b) `stock: TOO HOT ... peak=64 C`, the line that the measurement stopped there, the cool-down, then `baseline: the measurement at stock` and a search that runs to `RESULT:` with `before:` showing the shortened time; no `candidates are stopped at` line, because 64 C is under the profile's 75. (c) no `not stable at stock`; `the card reached 86 C at stock: candidates are stopped at 85 C, not at the profile's 75 C`; candidates that end `STABLE` or `WRONG RESULT`, none `TOO HOT`; a result that is cooler and draws less than stock at the same clock | First run: **not as expected**, and that found a fault (2026-10-11, reference RTX 4070, driver 617.42, a test build of d1c99e9, started elevated by a script). The first halves held: (a) `stock: STABLE ... peak=65 C fan=70 %`, then `baseline: the measurement at stock` four seconds later and no second five minutes; (b) `stock: TOO HOT ... peak=64 C`, stopped after 34 s, the run went on, no `candidates are stopped at` line. But both runs then ended with `the card held no undervolt: the first step below its own voltage already failed`: their first candidates read `1070 mV (+15 MHz): SLOW ... clock=2323 MHz` and `1050 mV (+15 MHz): SLOW ... clock=2411 MHz`, on a card that ran 2791 MHz in the measurement just before. The search's load had not run yet when that candidate began (see the decision "With its reference handed over, the undervolt search warms its load up before the first candidate"). Second run, with the warm-up: **pass** for (a) and (b) (same day, a test build of 0733695). (a) `stock: STABLE score=5426 it/s clock=2788 MHz peak=66 C fan=70 %`, four seconds later `baseline: the measurement at stock`, `warming the load up: 5 s at stock`, `target: 2790 MHz, which the card reaches at 1090 mV by itself`, and the first candidate `1085 mV (+15 MHz): STABLE ... clock=2775 MHz`; 26 candidates, none `SLOW` or `TOO HOT`, the edge at 950 mV, `RESULT: 2790 MHz at 990 mV instead of 1090 mV`; `before:` 5426 it/s, 189 W, fan 70 % (300 s), `after:` 5325 it/s, 149 W, fan 53 % (300 s). (b) `stock: TOO HOT ... peak=64 C`, stopped after 74 s, the run went on, the same warm-up and a search to the same result (990 mV, edge 950 mV); `before:` shows 75 s; no `candidates are stopped at` line. The saved tune of before the test was put back and applied. (c) not run. | | 93 | A speed profile that would end slower than stock keeps nothing | Only on a card that is warmer at stock than the profile allows (the tester's RTX 3080 Ti; the reference card ends faster than stock with every speed profile): **Best of my GPU** from the window, to the end; the result screen, the dashboard and `run.log` | yes | the run goes through all its steps and its soak; then **Not applied** with `the result is slower than stock: N it/s against M (-X %)`, the power limit it had to go down to, and the pointer to Undervolt; `The card is back at stock.`; the dashboard shows stock and the saved tune of before the run, if there was one; `run.log` ends with `stopped: the result is slower than stock ... -- card restored to stock` | Not run yet. On the reference card: nothing to see, by design. | | 94 | All in one, with the undervolt step searching for the first time | Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize all` to the end with a build that has the warm-up of the undervolt step; then `gao --status`; put the copy back | yes | five steps; in step 3 a `warming the load up` line, a first candidate that is not `SLOW`, and a descent; steps 4 and 5 run on the tune of steps 2 and 3; `RESULT:` | **Not as expected at step 4**, and that found a fault (2026-10-11, reference RTX 4070, driver 617.42, a test build of 0733695, unattended). Step 1: `stock: STABLE score=5325 it/s peak=65 C fan=70 % power=186 W`. Step 2: power 105 %, `mem: bandwidth peak +2300, confirmed +2300, applying +1600`, `core: highest stable +210, confirmed +210, applying +135`, soak `STABLE score=5568 it/s peak=70 C power=197 W`. Step 3, as wanted: `warming the load up: 5 s with the overclock`, `target: 2865 MHz, which the card reaches at 1050 mV with the overclock`, nine candidates `STABLE` from 1045 mV down, `WRONG RESULT` at 990 mV, confirmed at 1010 mV after two tries, and `applying 1050 mV (+135 MHz at that point)`: the margin left no raise above the overclock, so the curve was cut off, as in the runs before the warm-up; soak `STABLE score=5434 it/s peak=67 C power=176 W`. Step 4 did not begin: `the tune could not be applied for the fan tune: the undervolt was not applied: the point already runs more than the target -- card at stock`. The saved tune of before the test was put back and applied. Cause and fix: the decision "A flat top goes back on a card that is cooler than when it was found". Second run, with the `gao.exe` of the released v0.4.6: **pass, all five steps** (same day, started elevated by a script). Step 1: `stock: STABLE score=5309 it/s peak=66 C fan=70 % power=185 W`. Step 2: power 105 %, `mem: bandwidth peak +850, confirmed +850, applying +550`, `core: highest stable +210, confirmed +210, applying +135`, soak `STABLE score=5568 it/s`. Step 3: `target: 2850 MHz, which the card reaches at 1045 mV with the overclock`, `undervolt: lowest stable 975 mV, confirmed 985 mV, applying 1025 mV (+165 MHz at that point)`, soak `STABLE score=5363 it/s peak=66 C power=163 W`. Step 4: the tune went back on the card, `fan 60 %: 63 C`, `fan 55 %: 65 C`, `fan 52 %: 67 C`, `fan tune: 52 % holds 67 C`. Step 5: the final test ran its 300 s. `RESULT: power 105 %, core +135 MHz, mem +550 MHz, 2850 MHz at 1025 mV, fans 52 % at 67 C`; `stock: 5309 it/s, 185 W, fan 70 %, 66 C`, `now: 5353 it/s, 166 W, fan 52 %, 70 C`: `+0.8 % score and -10.3 % power against stock`; `undervolt: applied`. The saved tune of before the test was put back. **What this run did not show:** its undervolt had a raise of its own (+165 over an offset of +135), so the anchor was raised and not merely cut off, and the case of the first run (an anchor that reads a step above the target on a cooler card) did not come up. That fix has still only its tests. **The memory offset differs a lot from run to run on this card, and the two logs say why:** +1600 in the first run (bandwidth peak +2300), +550 in the second (peak +850), +650 in the run that saved the tune in use that morning (the journal shows its scan ending at +1000 too). The card was in the same state each time (68 C, power 105 %, a reading every three to four seconds). In both logged scans the bandwidth rises in a straight line, 0.36 and 0.46 % per 50 MHz step, from 420 GB/s at +0 to 509 GB/s at +2450; the first run went on to `WRONG RESULT` at +2550 without the bandwidth ever turning down. The readings scatter around that line with a standard deviation of 0.4 and 0.7 %, single ones by more than 1 %. The scan ends at a reading more than 1 % below the best so far when a second reading agrees, and in the second run that was chance: +950 read high (462.8 GB/s, where the line is at about 458), +1000 read 456.6 twice, 1.3 % below it. In the first run the one low reading (+600) was followed by a higher one, and the scan ran to the first wrong result. So on this card there is no bandwidth peak before the errors, and where the scan stops is decided by noise. | -| 95 | The memory scan no longer ends by chance | (a) Not elevated, nothing changed on the card: a measuring build takes 24 bandwidth readings in one process, with 3 s of the stress load before each as a scan has, at 64, 128 and 256 dispatches per timed run; then the same with the app's window open next to it, drawing a frame every 100 ms and every 33 ms. (b) Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize best` twice in a row with the build of the change, then the copy put back | (a) no, (b) yes | (a) how much a reading scatters, and on what that depends. (b) in both runs every `mem +N: bandwidth` line shows three readings that agree within about 1 %; the scan does not end before the first `WRONG RESULT`; the two `mem: bandwidth peak` values are within 300 MHz of each other and the applied offset is above +1000; the soak is not slower than at stock | **(a) measured** (2026-10-11, reference RTX 4070, driver 617.42, the saved tune applied: power 105 %, core +135, memory +650; standard deviation over the 24 readings of a series, and the lowest and highest against the mean). 64 dispatches, no window: 0.44 % (-0.55 to +1.07, one reading did not settle), 0.66 % (-1.58 to +1.49, one did not settle), 1.31 % (-4.83 to +1.27), 0.86 % (-1.44 to +2.05). 128 dispatches, no window: 0.43 % (-1.10 to +0.79). 256 dispatches, no window: 0.22 % (-0.43 to +0.42) and 0.60 % (-0.80 to +1.37; the last five readings of this series were 451 to 453 GB/s after nineteen of 444 to 449). With the window drawing every 100 ms: 64 dispatches 1.13 % (the first six readings 452 to 453 GB/s, the rest 439 to 451), 256 dispatches 0.24 % (-0.47 to +0.61). With the window drawing every 33 ms: 64 dispatches 0.58 %, 256 dispatches 0.25 % (-0.35 to +0.51). A reading took 6.5, 5.3 and 3.9 timed runs to settle at 64, 128 and 256 dispatches. So: the longer timed run takes the scatter from 0.4 to 1.3 % down to 0.2 %, the window's drawing adds nothing, and apart from the scatter the readings sit about 1.8 % higher for up to a minute at a time (the stretch at the end of the second 256 series ran on into the next series; the last 64 series went back and forth between 444 and 452 GB/s). What sets that level is not known. **(b) the offset is the same twice; the scan ends in a driver reset twice** (2026-10-11, same card and driver, a test build of 2764609, unattended, 40 minutes for the two runs). Run 1: `stock: STABLE score=5311 it/s peak=66 C fan=70 % power=185 W`; 49 steps from `mem +0: bandwidth 422.7 GB/s (422.7 422.6 422.7)` to `mem +2400: bandwidth 501.4 GB/s (501.3 502.3 501.4)`; at +2450 the candidate was `STABLE` and the measurement after it lost the device: `mem +2450: the bandwidth measurement failed; treated as a driver reset`, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, 20 s of rest, `health: STABLE`, `confirm core +0 / mem +2050: STABLE`, `mem: bandwidth peak +2250, confirmed +2050, applying +1400`; `core: highest stable +210, confirmed +210, applying +135`; soak `STABLE score=5536 it/s peak=69 C power=195 W`; `driver resets during this run: 1`. Run 2: stock 5312 it/s; 50 steps from 421.8 GB/s at +0 to 502.8 at +2450; the same failure at +2500; `mem: bandwidth peak +2300, confirmed +2100, applying +1450`; core +135; soak `STABLE score=5555 it/s peak=69 C power=196 W`; one driver reset. The three readings of a step differed by 0.4 % on average and by 1.2 % at most; no step was more than 0.7 % below the best before it, so neither scan came near ending early, by the old rule or the new; every step had its three readings. The scan took 5 min 25 s (about 6.5 s a step; about 3 min before the change). As expected: the two peaks are 50 MHz apart, the applied offsets are +1400 and +1450, and both results are 4 % faster than stock. **Not as expected: neither scan reached a `WRONG RESULT`.** Both ended one and two steps before +2550 with the device lost during the bandwidth copy; Windows logged it both times (System log, `nvlddmkm` events 14 and 153 at 19:51:22 and 20:11:27), and the screen goes black for a moment. In the run of check 94 that got this far, the measurements at +2450 and +2500 gave a number and the stress load reported the first wrong result at +2550, without a reset. A step now copies about seven times as much as it did (three readings of about four runs of 128 GB, against one reading of about six runs of 32 GB), and the copy checks nothing, so where the memory starts to fail it is the driver that gives out, not a verdict. Two runs against one: likely the cause, not proven. The search handles it as it should (rest, health probe, confirm four steps lower, the core is still searched), but a driver reset in every run on a card like this is not what the change was for. Not merged; what to do about it is open in `docs/development.md`. The saved tune of before the test was put back and applied (`applied best: power 105 %, core +135 MHz, mem +650 MHz`), the tray app restarted, Defender reported nothing. | +| 95 | The memory scan no longer ends by chance | (a) Not elevated, nothing changed on the card: a measuring build takes 24 bandwidth readings in one process, with 3 s of the stress load before each as a scan has, at 64, 128 and 256 dispatches per timed run; then the same with the app's window open next to it, drawing a frame every 100 ms and every 33 ms. (b) Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize best` twice in a row with the build of the change, then the copy put back | (a) no, (b) yes | (a) how much a reading scatters, and on what that depends. (b) in both runs every `mem +N: bandwidth` line shows three readings that agree within about 1 %; the scan does not end before the first `WRONG RESULT`; the two `mem: bandwidth peak` values are within 300 MHz of each other and the applied offset is above +1000; the soak is not slower than at stock | **(a) measured** (2026-10-11, reference RTX 4070, driver 617.42, the saved tune applied: power 105 %, core +135, memory +650; standard deviation over the 24 readings of a series, and the lowest and highest against the mean). 64 dispatches, no window: 0.44 % (-0.55 to +1.07, one reading did not settle), 0.66 % (-1.58 to +1.49, one did not settle), 1.31 % (-4.83 to +1.27), 0.86 % (-1.44 to +2.05). 128 dispatches, no window: 0.43 % (-1.10 to +0.79). 256 dispatches, no window: 0.22 % (-0.43 to +0.42) and 0.60 % (-0.80 to +1.37; the last five readings of this series were 451 to 453 GB/s after nineteen of 444 to 449). With the window drawing every 100 ms: 64 dispatches 1.13 % (the first six readings 452 to 453 GB/s, the rest 439 to 451), 256 dispatches 0.24 % (-0.47 to +0.61). With the window drawing every 33 ms: 64 dispatches 0.58 %, 256 dispatches 0.25 % (-0.35 to +0.51). A reading took 6.5, 5.3 and 3.9 timed runs to settle at 64, 128 and 256 dispatches. So: the longer timed run takes the scatter from 0.4 to 1.3 % down to 0.2 %, the window's drawing adds nothing, and apart from the scatter the readings sit about 1.8 % higher for up to a minute at a time (the stretch at the end of the second 256 series ran on into the next series; the last 64 series went back and forth between 444 and 452 GB/s). What sets that level is not known. **(b) the offset is the same twice; the scan ends in a driver reset twice** (2026-10-11, same card and driver, a test build of 2764609, unattended, 40 minutes for the two runs). Run 1: `stock: STABLE score=5311 it/s peak=66 C fan=70 % power=185 W`; 49 steps from `mem +0: bandwidth 422.7 GB/s (422.7 422.6 422.7)` to `mem +2400: bandwidth 501.4 GB/s (501.3 502.3 501.4)`; at +2450 the candidate was `STABLE` and the measurement after it lost the device: `mem +2450: the bandwidth measurement failed; treated as a driver reset`, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, 20 s of rest, `health: STABLE`, `confirm core +0 / mem +2050: STABLE`, `mem: bandwidth peak +2250, confirmed +2050, applying +1400`; `core: highest stable +210, confirmed +210, applying +135`; soak `STABLE score=5536 it/s peak=69 C power=195 W`; `driver resets during this run: 1`. Run 2: stock 5312 it/s; 50 steps from 421.8 GB/s at +0 to 502.8 at +2450; the same failure at +2500; `mem: bandwidth peak +2300, confirmed +2100, applying +1450`; core +135; soak `STABLE score=5555 it/s peak=69 C power=196 W`; one driver reset. The three readings of a step differed by 0.4 % on average and by 1.2 % at most; no step was more than 0.7 % below the best before it, so neither scan came near ending early, by the old rule or the new; every step had its three readings. The scan took 5 min 25 s (about 6.5 s a step; about 3 min before the change). As expected: the two peaks are 50 MHz apart, the applied offsets are +1400 and +1450, and both results are 4 % faster than stock. **Not as expected: neither scan reached a `WRONG RESULT`.** Both ended one and two steps before +2550 with the device lost during the bandwidth copy; Windows logged it both times (System log, `nvlddmkm` events 14 and 153 at 19:51:22 and 20:11:27), and the screen goes black for a moment. In the run of check 94 that got this far, the measurements at +2450 and +2500 gave a number and the stress load reported the first wrong result at +2550, without a reset. A step now copies about seven times as much as it did (three readings of about four runs of 128 GB, against one reading of about six runs of 32 GB), and the copy checks nothing, so where the memory starts to fail it is the driver that gives out, not a verdict. Two runs against one: likely the cause, not proven. The search handles it as it should (rest, health probe, confirm four steps lower, the core is still searched), but a driver reset in every run on a card like this is not what the change was for. The saved tune of before the test was put back and applied (`applied best: power 105 %, core +135 MHz, mem +650 MHz`), the tray app restarted, Defender reported nothing. **(c) the same two runs with one reading a step, three only for a step in doubt** (2026-10-11 20:26 to 21:03, a test build of 71e7c2b, unattended): **the reset is still there.** Run 1: stock 5328 it/s; 50 steps from `mem +0: bandwidth 420.4 GB/s` to `mem +2450: bandwidth 502.4 GB/s`; at +2500 the candidate was `STABLE` and then `bandwidth measurement failed: the timing of the copy did not come back` ten seconds later (`nvlddmkm` event 14 at 20:37:07); `mem: bandwidth peak +2200, confirmed +2000, applying +1400`; core +135; soak `STABLE score=5572 it/s peak=70 C power=197 W`; one driver reset. Run 2: stock 5311 it/s; 421.9 GB/s at +0 to 508.0 at +2450; at +2500 `the device was lost during the copy failed (hr=0x887A0006)`, the device hung (event 14 at 20:55:29); `mem: bandwidth peak +2400, confirmed +2200, applying +1500`; core +135; soak `STABLE score=5571 it/s peak=70 C power=196 W`; one driver reset. No step was in doubt in either run (the lowest was 0.75 % below the best before it), so every step was read once; the scan took 3 min 33 s. The two peaks are 200 MHz apart: with one reading a step, which offsets count as within 1 % of the best moves by a few steps. Applied +1400 and +1500, both 5 % faster than stock. So the reset does not come from reading a step three times. Four runs out of four with the longer timed run ended this way, at +2450 or +2500. Not merged; the design for a scan that does not end like this is the open row in `docs/development.md`. The saved tune was put back and applied again (core +135, memory +650), the tray app restarted, Defender reported nothing. | ## Notes diff --git a/src/core/search.cpp b/src/core/search.cpp index 34e6d68..b3cb3f2 100644 --- a/src/core/search.cpp +++ b/src/core/search.cpp @@ -70,8 +70,8 @@ int best_bandwidth_offset(int lo, int hi, int step, int ceiling, const std::func if (!s.stable) break; // A step that is further below the best than readings scatter is in // doubt: it is read kBandwidthReadings times and the middle one counts. - // Only such a step: every reading is a copy that checks nothing, and - // near its edge the memory answers more of it with a driver reset. + // Only such a step: three readings at every step cost two minutes a + // scan and decided nothing a single reading did not (hardware check 95). if (s.gbps < best * (1 - kBandwidthTie)) { std::vector readings{s.gbps}; while (static_cast(readings.size()) < kBandwidthReadings && (s = sample(v)).stable) readings.push_back(s.gbps); diff --git a/src/core/search.hpp b/src/core/search.hpp index 732d9b0..ab21c00 100644 --- a/src/core/search.hpp +++ b/src/core/search.hpp @@ -82,8 +82,8 @@ inline constexpr double kBandwidthTie = 0.01; // off decides nothing; and a reading that did not settle is taken again up to // this many times. On the reference card single readings ended scans at +250, // at +1000 and not at all on one day, and the runs saved +100, +650 and +1600 -// MHz (hardware check 94). Not every step: three readings at every step ended -// both scans of hardware check 95 in a driver reset near the memory's edge. +// MHz (hardware check 94). Not every step: that cost two minutes a scan and +// changed no outcome on the card (hardware check 95). inline constexpr int kBandwidthReadings = 3; // A confirm probe that ends too hot keeps the clock edge (heat is the power diff --git a/src/hw/stress.cpp b/src/hw/stress.cpp index be18f37..9fb9716 100644 --- a/src/hw/stress.cpp +++ b/src/hw/stress.cpp @@ -372,7 +372,7 @@ std::optional Stress::MeasureBandwidth() { !wait(d.q_end.Get(), &t1, sizeof(t1))) { // Say which: the log line used to end in nothing here. const HRESULT removed = d.device->GetDeviceRemovedReason(); - error_ = removed != S_OK ? Hr("the device was lost during the copy", removed) : "the timing of the copy did not come back"; + error_ = removed != S_OK ? Hr("the copy", removed) + ": the device was lost" : "the timing of the copy did not come back"; if (removed != S_OK) d.device.Reset(); return std::nullopt; } From 7c215c4536b62035d25c4349015c408f6d732afb Mon Sep 17 00:00:00 2001 From: Rovey Date: Sun, 11 Oct 2026 21:12:15 +0200 Subject: [PATCH 05/15] feat: the journal remembers where a card's memory gave out, and later scans stay below it On a card whose bandwidth rises until its memory fails, nothing ends the memory scan before the edge, and on the reference card every scan ended there in a driver reset (hardware check 95). A memory scan that ended in a wrong result or a reset event now writes the offset to the journal, bound to the card and its driver version. A later run on the same card and driver stops its scan four steps below it (kResetBackoffSteps, where confirming starts after a reset) and says so. With another driver or another card the edge does not count. The line has no id, so builds that do not know it skip it. Co-Authored-By: Claude Opus 5.5 --- src/app/common.cpp | 5 +- src/core/journal.cpp | 20 +++++++- src/core/journal.hpp | 17 ++++++- src/core/search.cpp | 35 ++++++++++---- tests/test_journal.cpp | 67 +++++++++++++++++++++++++++ tests/test_search.cpp | 102 ++++++++++++++++++++++++++++++++++++++++- 6 files changed, 233 insertions(+), 13 deletions(-) diff --git a/src/app/common.cpp b/src/app/common.cpp index b11490e..1e15922 100644 --- a/src/app/common.cpp +++ b/src/app/common.cpp @@ -332,7 +332,10 @@ OptimizeOutcome run_optimize(Preset preset, const OptimizeHooks& hooks, const st if (path.empty()) return fail("the ProgramData folder could not be resolved; cannot keep the crash journal"); const auto lines = read_lines(path); if (!lines) return fail("the crash journal " + path.string() + " exists but cannot be read; not tuning without it"); - Journal journal(*lines, [&path](const std::string& l) { return append_line_durable(path, l); }); + // What a memory edge in the journal is bound to: this card with this driver. + const std::string rig_card = hw.GpuUuid(), rig_driver = hw.DriverVersion(); + Journal journal(*lines, [&path](const std::string& l) { return append_line_durable(path, l); }, + rig_card.empty() || rig_driver.empty() ? std::string() : rig_card + " / " + rig_driver); // Prove the journal is writable before any clock is touched; the parser // ignores lines without an id, so this one never becomes a ceiling. if (!append_line_durable(path, "{\"session\":\"" + now_text() + "\"}")) diff --git a/src/core/journal.cpp b/src/core/journal.cpp index 22a9ecc..26722d4 100644 --- a/src/core/journal.cpp +++ b/src/core/journal.cpp @@ -43,8 +43,8 @@ std::optional int_field(const nlohmann::json& j, const char* key) { } } -Journal::Journal(const std::vector& existing_lines, Append append) - : append_(std::move(append)) { +Journal::Journal(const std::vector& existing_lines, Append append, std::string rig) + : append_(std::move(append)), rig_(std::move(rig)) { std::map pending; // ordered by id = file order int max_id = 0; for (std::string line : existing_lines) { @@ -53,6 +53,14 @@ Journal::Journal(const std::vector& existing_lines, Append append) const bool parsed = !j.is_discarded() && j.is_object(); torn_tail_ = !line.empty() && !parsed; // only the last line's value survives the loop if (!parsed) continue; + if (const auto edge = j.find("mem_edge"); edge != j.end()) { + // A line of its own, without an id: builds that do not know it skip it. + const auto mhz = int_field(j, "mem_edge"); + const auto rig_of = j.find("rig"); + if (mhz && *mhz > 0 && *mhz <= kMaxClockMhz && !rig_.empty() && rig_of != j.end() && rig_of->is_string() && *rig_of == rig_) + mem_edge_ = std::min(mem_edge_, *mhz); + continue; + } const auto id = int_field(j, "id"); const auto state = j.find("state"); if (!id || *id <= 0 || *id >= kMaxJournalId || state == j.end() || !state->is_string()) continue; @@ -92,6 +100,14 @@ bool Journal::complete(int id, const std::string& verdict) { return true; } +bool Journal::note_mem_edge(int mhz) { + if (mhz <= 0 || rig_.empty()) return true; // nothing to bind it to: nothing is kept + const nlohmann::json j = {{"mem_edge", mhz}, {"rig", rig_}}; + if (!write(j.dump())) return false; + mem_edge_ = std::min(mem_edge_, mhz); + return true; +} + bool Journal::write(const std::string& line) { if (!append_(torn_tail_ ? "\n" + line : line)) return false; torn_tail_ = false; diff --git a/src/core/journal.hpp b/src/core/journal.hpp index 942fa9e..0364d92 100644 --- a/src/core/journal.hpp +++ b/src/core/journal.hpp @@ -30,11 +30,18 @@ struct Freeze { // that froze the machine, and it becomes a ceiling for every later run. An // offset of 0 is never a ceiling: that would end the search of that clock. // Pure: the file lives in hw/app_files, reached through `append`. +// +// The journal also remembers where the memory of a card gave out in a scan +// that finished (a wrong result, a driver reset, a lost device): a card whose +// bandwidth rises until its memory fails has nothing else to stop a scan, and +// every run would meet that edge again. Such an edge belongs to a card with a +// driver, `rig`; with another of either it does not count. An empty rig keeps +// and uses none. class Journal { public: using Append = std::function; - Journal(const std::vector& existing_lines, Append append); + Journal(const std::vector& existing_lines, Append append, std::string rig = {}); const Ceilings& ceilings() const { return ceilings_; } // Human-readable description of each unfinished candidate, in file order: @@ -43,6 +50,12 @@ class Journal { // The same candidates, each with whether it lowered a ceiling. const std::vector& freeze_entries() const { return freeze_entries_; } int next_id() const { return next_id_; } + // The lowest memory offset at which a scan on this rig gave out; INT_MAX + // when none is known. Not a ceiling: the search decides how far below it + // to stay. + int mem_edge() const { return mem_edge_; } + // Writes one down. False when the line could not be written. + bool note_mem_edge(int mhz); // The entry begin() opened and complete() has not closed; -1 when none. int open_id() const { return open_id_; } @@ -60,6 +73,8 @@ class Journal { // by a power cut, without its newline. The next write starts with '\n' // so it is not glued onto that fragment. bool torn_tail_ = false; + std::string rig_; + int mem_edge_ = INT_MAX; Ceilings ceilings_; std::vector freezes_; std::vector freeze_entries_; diff --git a/src/core/search.cpp b/src/core/search.cpp index b3cb3f2..b16f7f1 100644 --- a/src/core/search.cpp +++ b/src/core/search.cpp @@ -568,8 +568,20 @@ OptimizeResult optimize(const GpuControl& gpu, const Objectives& obj, Journal& j // Memory comes first, at core 0: the core climb is where a card is most // likely to give out, and it must not keep the memory from being tuned. if (obj.mem_oc) { - const int ceiling = journal.ceilings().mem_mhz; + int ceiling = journal.ceilings().mem_mhz; + // Where the memory gave out in an earlier scan on this card and + // driver, the scan does not go again: it stops as far below it as + // confirming starts below a reset. A card whose bandwidth rises until + // its memory fails would otherwise end every run in a wrong result or + // a driver reset (hardware check 95). + if (const int edge = journal.mem_edge(); edge != INT_MAX) { + const int top = std::max(0, edge - kResetBackoffSteps * kMemStep); + log("mem: the memory gave out at " + offset_text(edge) + " in an earlier run on this card and driver; the scan stops at " + + offset_text(top)); + ceiling = std::min(ceiling, top + 1); + } int reset_at = -1; // the memory offset at which a reset event ended the scan + int gave_out_at = 0; // the offset whose candidate computed a wrong result const bool by_bandwidth = static_cast(io.bandwidth); bool left_at_stock = false; // the measurement does not work: nothing was searched if (by_bandwidth) { @@ -586,13 +598,14 @@ OptimizeResult optimize(const GpuControl& gpu, const Objectives& obj, Journal& j // memory 0 would read as "never search memory again". const std::optional mem_j = v > 0 ? std::optional(v) : std::nullopt; m.stable = explore(v, reset_at, [&] { - if (!is_stable(clock_candidate(std::nullopt, mem_j, 0, v, kClockProbeS, [&] { - for (int i = 0; i < kBandwidthReadings && !gbps && !failed; ++i) { - gbps = io.bandwidth(); - failed = !gbps && !(io.bandwidth_unsettled && io.bandwidth_unsettled()); - } - }))) - return false; + const auto verdict = clock_candidate(std::nullopt, mem_j, 0, v, kClockProbeS, [&] { + for (int i = 0; i < kBandwidthReadings && !gbps && !failed; ++i) { + gbps = io.bandwidth(); + failed = !gbps && !(io.bandwidth_unsettled && io.bandwidth_unsettled()); + } + }); + if (verdict == Verdict::WrongResult) gave_out_at = v; + if (!is_stable(verdict)) return false; if (gbps) return true; // The candidate passed, the measurement failed: the step is unusable. if (v > 0 && !failed) { @@ -638,6 +651,12 @@ OptimizeResult optimize(const GpuControl& gpu, const Objectives& obj, Journal& j reset_event(why); } left_at_stock = unavailable; + // The edge of this card's memory, for the runs after this one. A + // reset event counts as much as a wrong result: the scan ended + // there because the memory did. + if (reset_at > 0) gave_out_at = reset_at; + if (gave_out_at > 0 && !journal.note_mem_edge(gave_out_at)) + log("warning: the memory's edge could not be written to the journal; the next run will find it again"); } if (!by_bandwidth) { r.mem_max_stable = climb_to_edge(0, bounds.mem_max_mhz, kMemStep, kMemStride, ceiling, [&](int v) { diff --git a/tests/test_journal.cpp b/tests/test_journal.cpp index 47269f3..ea2c6d2 100644 --- a/tests/test_journal.cpp +++ b/tests/test_journal.cpp @@ -227,3 +227,70 @@ TEST_CASE("an empty journal has no freeze entries") { Journal j({}, s.append()); CHECK(j.freeze_entries().empty()); } + +TEST_CASE("a journal without a memory edge has none") { + std::vector out; + Journal j({"{\"id\":1,\"mem\":2500,\"state\":\"begin\"}", "{\"id\":1,\"state\":\"complete\",\"verdict\":\"WRONG RESULT\"}"}, + [&](const std::string& l) { out.push_back(l); return true; }, "card-a / 617.42"); + CHECK(j.mem_edge() == INT_MAX); + CHECK(j.ceilings().mem_mhz == INT_MAX); +} + +TEST_CASE("a memory edge is written with the card and driver it was found on, and read back") { + std::vector out; + Journal j({}, [&](const std::string& l) { out.push_back(l); return true; }, "card-a / 617.42"); + CHECK(j.note_mem_edge(2500)); + REQUIRE(out.size() == 1); + CHECK(out[0].find("\"mem_edge\":2500") != std::string::npos); + CHECK(out[0].find("card-a / 617.42") != std::string::npos); + CHECK(j.mem_edge() == 2500); + + Journal again(out, [](const std::string&) { return true; }, "card-a / 617.42"); + CHECK(again.mem_edge() == 2500); + CHECK(again.ceilings().mem_mhz == INT_MAX); // not a freeze + CHECK(again.freezes().empty()); + CHECK(again.next_id() == 1); // not a candidate either +} + +TEST_CASE("the lowest memory edge of this card and driver counts") { + const std::vector lines = { + "{\"mem_edge\":2500,\"rig\":\"card-a / 617.42\"}", + "{\"mem_edge\":2450,\"rig\":\"card-a / 617.42\"}", + "{\"mem_edge\":1200,\"rig\":\"card-b / 617.42\"}", + "{\"mem_edge\":900,\"rig\":\"card-a / 620.10\"}", + }; + auto ok = [](const std::string&) { return true; }; + CHECK(Journal(lines, ok, "card-a / 617.42").mem_edge() == 2450); + CHECK(Journal(lines, ok, "card-b / 617.42").mem_edge() == 1200); +} + +TEST_CASE("a memory edge lapses with another driver or another card") { + const std::vector lines = {"{\"mem_edge\":2500,\"rig\":\"card-a / 617.42\"}"}; + auto ok = [](const std::string&) { return true; }; + CHECK(Journal(lines, ok, "card-a / 620.10").mem_edge() == INT_MAX); + CHECK(Journal(lines, ok, "card-b / 617.42").mem_edge() == INT_MAX); + // Back on the driver it was found with, it counts again. + CHECK(Journal(lines, ok, "card-a / 617.42").mem_edge() == 2500); +} + +TEST_CASE("without a card and driver to bind it to, no memory edge is kept or used") { + std::vector out; + Journal j({"{\"mem_edge\":2500,\"rig\":\"\"}", "{\"mem_edge\":2400}"}, [&](const std::string& l) { out.push_back(l); return true; }); + CHECK(j.mem_edge() == INT_MAX); + CHECK(j.note_mem_edge(2300)); + CHECK(out.empty()); + CHECK(j.mem_edge() == INT_MAX); +} + +TEST_CASE("memory edges that make no sense are ignored, and a failed write is reported") { + const std::vector lines = { + "{\"mem_edge\":0,\"rig\":\"r\"}", "{\"mem_edge\":-50,\"rig\":\"r\"}", "{\"mem_edge\":4294967305,\"rig\":\"r\"}", + "{\"mem_edge\":\"2500\",\"rig\":\"r\"}", "{\"mem_edge\":2500,\"rig\":7}", + }; + Journal j(lines, [](const std::string&) { return false; }, "r"); + CHECK(j.mem_edge() == INT_MAX); + CHECK_FALSE(j.note_mem_edge(2500)); + CHECK(j.mem_edge() == INT_MAX); + CHECK(j.note_mem_edge(0)); // nothing to note +} + diff --git a/tests/test_search.cpp b/tests/test_search.cpp index ae0f68c..7b7c62b 100644 --- a/tests/test_search.cpp +++ b/tests/test_search.cpp @@ -295,6 +295,7 @@ struct Run { int abort_after_probes = -1; bool no_rest = false; // the application gives the search no way to wait std::optional stock; // the measurement at stock that went before the run, when there was one + std::string rig = "card-a / 617.42"; // the card and driver the journal's memory edges are bound to OptimizeResult go(Preset preset, GpuControl gpu) { Journal j(card.journal, [this](const std::string& l) { if (l.find("\"complete\"") != std::string::npos && card.completes_ok >= 0 && card.completes_ok-- == 0) @@ -302,7 +303,7 @@ struct Run { if (l.find("\"begin\"") != std::string::npos) card.events.push_back("begin"); card.journal.push_back(l); return true; - }); + }, rig); OptimizeIo io; io.probe = card.probe(); io.aborted = [this] { return card.aborted_now || (abort_after_probes >= 0 && card.probes >= abort_after_probes); }; @@ -2994,3 +2995,102 @@ TEST_CASE("a stretch in which every reading is 2 % off does not end the memory s CHECK(r.mem_max_stable >= 2000); } } + +namespace { + +// The run after: the same card and journal, the counters of the first run gone. +OptimizeResult again(Run& run) { + run.log.clear(); + run.card.max_mem_seen = 0; + run.card.bw_measurements = 0; + run.card.bw_readings_at.clear(); + return run.go(Preset::BestOfMyGpu); +} + +} + +TEST_CASE("a memory scan that ran into a wrong result leaves the edge in the journal") { + Run run = rising_to_the_edge(); + REQUIRE(run.go(Preset::BestOfMyGpu).ok); + CHECK(run.card.max_mem_seen == 2550); + CHECK(run.card.count("\"mem_edge\":2550") == 1); + CHECK(run.card.count("card-a / 617.42") == 1); + check_journal_rules(run); +} + +TEST_CASE("the next run stays four steps below the memory edge of the run before") { + // On the reference card every scan ended in a driver reset at the edge + // (hardware check 95). Once is enough. + Run run = rising_to_the_edge(); + const auto first = run.go(Preset::BestOfMyGpu); + REQUIRE(first.ok); + const auto second = again(run); + REQUIRE(second.ok); + CHECK(run.card.max_mem_seen == 2550 - kResetBackoffSteps * 50); + CHECK(run.logged("mem: the memory gave out at +2550 in an earlier run on this card and driver; the scan stops at +2350")); + CHECK_FALSE(run.logged("mem +2400")); // no candidate above it, so none that could fail + CHECK(second.driver_resets == 0); + CHECK(second.mem_max_stable >= 2200); + CHECK(second.mem_mhz >= 1500); + CHECK(run.card.count("\"mem_edge\"") == 1); // nothing new was found + // And the run after that is the same. + const auto third = again(run); + REQUIRE(third.ok); + CHECK(third.mem_mhz == second.mem_mhz); +} + +TEST_CASE("a bandwidth measurement that lost the device leaves the edge too, and the next run has no reset") { + Run run = recovering(); + run.card.bw_curve = [](int m) { return 500 + m * 0.3; }; + run.card.bw_fails_once_at_mem = 500; + const auto first = run.go(Preset::BestOfMyGpu); + REQUIRE(first.ok); + CHECK(first.driver_resets == 1); + CHECK(run.card.count("\"mem_edge\":500") == 1); + const auto second = again(run); + REQUIRE(second.ok); + CHECK(second.driver_resets == 0); + CHECK(run.card.max_mem_seen == 300); + CHECK(second.mem_max_stable == 300); +} + +TEST_CASE("the memory edge lapses with another driver") { + Run run = rising_to_the_edge(); + REQUIRE(run.go(Preset::BestOfMyGpu).ok); + run.rig = "card-a / 620.10"; + REQUIRE(again(run).ok); + CHECK(run.card.max_mem_seen == 2550); // searched afresh + CHECK_FALSE(run.logged("in an earlier run")); + CHECK(run.card.count("\"mem_edge\":2550") == 2); // and found again, for the new driver +} + +TEST_CASE("a scan that ended for another reason leaves no memory edge") { + { // a bandwidth peak + Run run; + run.card.mem_edge = 1400; + run.card.bw_curve = [](int m) { return m <= 900 ? 500 + m * 0.3 : 770 - (m - 900) * 0.6; }; + REQUIRE(run.go(Preset::BestOfMyGpu).ok); + CHECK(run.card.count("mem_edge") == 0); + } + { // readings that did not settle + Run run = recovering(); + run.card.bw_curve = [](int m) { return 500 + m * 0.3; }; + run.card.bw_unsettled_at_mem = 300; + REQUIRE(run.go(Preset::BestOfMyGpu).ok); + CHECK(run.card.count("mem_edge") == 0); + } + { // a measurement that does not work on this machine + Run run = recovering(); + run.card.bw_curve = [](int m) { return 500 + m * 0.3; }; + run.card.bw_fails_at_zero = true; + REQUIRE(run.go(Preset::BestOfMyGpu).ok); + CHECK(run.card.count("mem_edge") == 0); + } + { // the driver's own range + Run run = rising_to_the_edge(5000); + REQUIRE(run.go(Preset::BestOfMyGpu).ok); + CHECK(run.card.max_mem_seen == 3000); + CHECK(run.card.count("mem_edge") == 0); + } +} + From 893d044596f7703de05a645085bfe43f276fb2d9 Mon Sep 17 00:00:00 2001 From: Rovey Date: Sun, 11 Oct 2026 21:16:18 +0200 Subject: [PATCH 06/15] docs: the remembered memory edge, and what hardware check 95 (d) and (e) have to show Co-Authored-By: Claude Opus 5.5 --- docs/development.md | 2 +- docs/hardware-checks.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/development.md b/docs/development.md index 1823e21..afd2138 100644 --- a/docs/development.md +++ b/docs/development.md @@ -80,7 +80,7 @@ The applied offset is always at least one step below the confirmed edge. Profile | A confirm probe that is too hot at once confirms nothing | A confirm probe that ends `TOO HOT` keeps the clock edge: over 30 s the card may run a little hotter than over the power step's 20 s, and that is the soak's business, which lowers power. But the stress run stops at its first reading above the limit, so a card that is above it when the probe begins gives `TOO HOT` within a second, and that counted as holding too. It happened on the tester's RTX 3080 Ti (hardware check 91): after a driver reset the health probe loads the card at stock power, which on that card means 76 C against a limit of 75, and `confirm core +225` was confirmed by a probe that had tested nothing. Now such a probe counts only when the clock was loaded for `kConfirmMinS` (10 s, clearly longer than the 3 s probe that found the edge) before it got too hot. Otherwise the value is not confirmed, and the card rests (`io.rest`, the 20 s of a recovery, with the fans where the curve last put them) before the next try one step lower: without the rest every lower value would start just as hot, and heat alone would step the edge down to nothing, which is what keeping the edge on `TOO HOT` was for. The same value is not tried again; one step is the price. Not changed: the health probe still runs at stock power, and a soak that begins too hot still lowers the power limit by a step. | | A profile that promises speed keeps no result that is slower than stock | Best of my GPU ended 3.8 % slower than stock on the tester's RTX 3080 Ti (4749 against 4937 it/s, hardware check 91), with the fans at 80 % instead of 66 %, and showed it as **Done**, applied and kept at every logon. Every step had done what it is for: the card runs 83 C at stock with its fans at 100 %, the profile allows 75, so the power limit went to 65 %, and +150 MHz of core clock did not win that back. But a profile called "the best of my GPU" that leaves the card slower has not done what it says, and a result like that is not offered as one. After the soak, `optimize` compares its score with the measurement at stock that went before the run (the "before" the result screen shows, so the rule judges by the number the user sees): below `kNotSlowerScore` (0.99: one percent is what two five-minute runs differ by) the run ends as a failed one does, at stock, with nothing saved, and says what it found, why, and that Undervolt is the profile for a card that runs warm. `Objectives::faster_than_stock` marks the two profiles this holds for, Best of my GPU and Max performance. Not Quiet and Cool & efficient, which trade speed for noise and heat and say so. Not the overclock step of All in one either: it goes on to an undervolt and a fan tune, and its result is judged as a whole; whether that whole may be slower than stock is not decided. This is a backstop, not the cure: what such a card needs from these profiles is in `docs/planned.md` ("never hotter than at stock"). | | The memory scan times a longer copy, reads a step in doubt three times, and stops at a 3 % drop | Four runs on one day applied +650, +1600, +550 and +100 MHz on the reference RTX 4070, a card that runs every game at +1500 (hardware check 94). That card has no bandwidth peak: bandwidth rises by about 0.4 % per 50 MHz step all the way to +2500 and the first wrong result comes at +2550, so the scan could only end early by chance, and where it did decided the offset. Measured afterwards without a search and with nothing changed on the card (hardware check 95 a), the readings move in three ways, and each got its own answer. **(1) One timed run was too short.** A run copied 32 GB, 75 ms on this card, and a reading is the median of three runs that agree within 1 %. Such readings scattered by 0.4 to 1.3 % (standard deviation over 24 readings, six series), single ones by up to 4.8 %, and in two of the six series one reading did not settle. With 128 GB per run (`kBwDispatches` 256, 0.3 s) it is 0.2 %, and all 96 readings settled. **(2) A single reading decided the end.** A step that is more than `kBandwidthTie` (1 %) below the best is in doubt: it is sampled `kBandwidthReadings` (3) times and its middle reading counts, which replaces the single re-measure of a drop. A reading that only did not settle is taken again, three times at most; any other failure ends the step at once, as before. Three readings at every step were built first and taken back: see the row below. **(3) The readings change level.** For up to a minute at a time every reading is about 1.8 % higher than before and after (452 against 444 GB/s), at one and the same offset. The cause is not known. Reading a step three times does nothing against it: all three are inside the stretch. So the drop that ends the scan (`kBandwidthDrop`) went from 1 % to 3 %: above what the readings do by themselves, below what a real peak loses within a step or two. The price on a card that has a peak: the scan goes a step or two further past it, in 3 s probes, before it stops. The offset it keeps is still the lowest within 1 % of the best, and confirming starts there. **Not built: a window that stands still during a measurement.** The suspicion was that the window's own drawing disturbed the readings of the run that saved +100. Measured with a window drawing a frame every 100 ms and every 33 ms next to the measurement: 0.2 %, the same as without. **On the card (check 95 b and c):** four runs applied +1400, +1450, +1400 and +1500, each 4 to 5 % faster than stock; no step came within reach of ending a scan early (the lowest was 0.75 % below the best before it), and no step was in doubt. A scan of 50 steps takes 3 min 33 s, half a minute more than before. **What it does not solve** is in the row below, and keeps the change from being merged. A change of level larger than 3 % would end a scan by chance again: the answer to that is to compare a low step with a fresh reading of the best offset taken right after it, not a still higher threshold. | -| Open: on a card without a bandwidth peak the memory scan ends in a driver reset | With the change of the row above the scan on the reference card no longer ends by chance, so it always runs to where the memory gives out. In four runs out of four that was not a wrong result from the stress load but the bandwidth copy losing the device at +2450 or +2500 (`DXGI_ERROR_DEVICE_HUNG`, or the timing of the copy not coming back within its 5 s; `nvlddmkm` events 14 and 153 in the System log; the screen goes black for a moment), with three readings a step and with one (hardware check 95 b and c). So it is not how often a step is read. In the one run before the change that got this far (check 94), the copies at +2450 and +2500 gave a number and the stress load reported the first wrong result at +2550. What differs is the timed run: 128 GB in one go against 32. One run against four is thin, and a memory edge moves by a step from run to run; it is the likely cause, not a proven one. The search handles the reset as it should (rest, health probe, confirm four steps lower, the core still searched) and the result is good. But a reset as the normal end of a scan is not acceptable in a tool that is to be simple and stable. **Design, not built.** A card like this has nothing that announces its edge: the bandwidth rises until the memory fails. No stop rule can know where that is without having been there, so the design does not try to. It makes the meeting happen once, and with the probe that can report it. **(a) The edge is found once and remembered.** The journal already turns a candidate that never finished (the machine froze) into a ceiling for every later run. A memory candidate that ended a scan, by a wrong result, a driver reset or a lost device, becomes one too: later scans on the same card and driver stop `kResetBackoffSteps` (4) steps below it, which is where confirming starts today after a reset. The result is what it is now (confirmed four steps below the failure, 70 % of that applied) and the same in every run, without touching the edge again. It errs to the safe side: an edge found on a bad day stays, until the driver changes or the journal is cleared. Small, and certain to work; it leaves the first run as it is. **(b) The stress load meets the edge before the copy does.** The stress load checks its results and says `WRONG RESULT`; the copy checks nothing and can only hang. So the copy is taken only at offsets the stress load has already passed four steps higher: per step a 3 s probe at +200 MHz above it, then back and the timed copy. On the reference card the load would report the wrong result at +2550 while the highest copy taken is at +2350, where all five scans measured a number. The price: about a second more per step, and on a card that has a peak the probes run four steps beyond the point where the scan would stop. That is 3 s probes, as the 3 % rule already allows a step or two; it is not the climb on stability alone that took a 3080 Ti to +2300, because the copy still ends the scan at the peak. Before (b) is built it has to be measured that the stress load at the edge ends in a wrong result and not in a reset of its own: one observation says so (check 94), core candidates say so in every run, and that is not enough for memory. **Looked at and not proposed.** A fixed top for the scan (an offset, or a share of the driver's range): there is nothing to choose the number by, and a card whose memory gives out below it meets its edge all the same. A top at a bandwidth gain over +0 (say 15 %): the same objection, most cards will fail first. Going back to the short timed run: it brings back the scatter that made the scan end by chance, on the strength of one run. **Order:** (a) first; the measurement for (b) next, on the reference card, with the old and the new timed run a few times each from +2400 up. | +| Open: on a card without a bandwidth peak the memory scan ends in a driver reset | With the change of the row above the scan on the reference card no longer ends by chance, so it always runs to where the memory gives out. In four runs out of four that was not a wrong result from the stress load but the bandwidth copy losing the device at +2450 or +2500 (`DXGI_ERROR_DEVICE_HUNG`, or the timing of the copy not coming back within its 5 s; `nvlddmkm` events 14 and 153 in the System log; the screen goes black for a moment), with three readings a step and with one (hardware check 95 b and c). So it is not how often a step is read. In the one run before the change that got this far (check 94), the copies at +2450 and +2500 gave a number and the stress load reported the first wrong result at +2550. What differs is the timed run: 128 GB in one go against 32. One run against four is thin, and a memory edge moves by a step from run to run; it is the likely cause, not a proven one. The search handles the reset as it should (rest, health probe, confirm four steps lower, the core still searched) and the result is good. But a reset as the normal end of a scan is not acceptable in a tool that is to be simple and stable. **Design, not built.** A card like this has nothing that announces its edge: the bandwidth rises until the memory fails. No stop rule can know where that is without having been there, so the design does not try to. It makes the meeting happen once, and with the probe that can report it. **(a) The edge is found once and remembered: built.** The journal already turns a candidate that never finished (the machine froze) into a ceiling for every later run. A memory scan that ended in a wrong result or a reset event (a driver reset, a lost device) now writes that offset to the journal as well, in a line of its own (`{"mem_edge":2500,"rig":" / "}`; it has no id, so a build that does not know it skips it). A later scan on the same card and driver stops `kResetBackoffSteps` (4) steps below the lowest such offset, which is where confirming starts after a reset, and says so: `mem: the memory gave out at +2500 in an earlier run on this card and driver; the scan stops at +2300`. With another driver version or another card the line does not count; back on the driver it was found with, it counts again. Without a GPU id or a driver version nothing is kept. A scan that ended for another reason (a bandwidth peak, readings that did not settle, a measurement that does not work, the driver's own range) writes nothing. It errs to the safe side: an edge found on a bad day stays until the driver changes or the journal is removed. It leaves the first run as it is. On the card: hardware check 95 (d). **(b) The stress load meets the edge before the copy does.** The stress load checks its results and says `WRONG RESULT`; the copy checks nothing and can only hang. So the copy is taken only at offsets the stress load has already passed four steps higher: per step a 3 s probe at +200 MHz above it, then back and the timed copy. On the reference card the load would report the wrong result at +2550 while the highest copy taken is at +2350, where all five scans measured a number. The price: about a second more per step, and on a card that has a peak the probes run four steps beyond the point where the scan would stop. That is 3 s probes, as the 3 % rule already allows a step or two; it is not the climb on stability alone that took a 3080 Ti to +2300, because the copy still ends the scan at the peak. Before (b) is built it has to be measured that the stress load at the edge ends in a wrong result and not in a reset of its own: one observation says so (check 94), core candidates say so in every run, and that is not enough for memory. **Looked at and not proposed.** A fixed top for the scan (an offset, or a share of the driver's range): there is nothing to choose the number by, and a card whose memory gives out below it meets its edge all the same. A top at a bandwidth gain over +0 (say 15 %): the same objection, most cards will fail first. Going back to the short timed run: it brings back the scatter that made the scan end by chance, on the strength of one run. **Order:** (a) first; the measurement for (b) next (hardware check 95 e): the stress load alone, with no search, in 3 s runs from +2300 up until it fails, five times. | | Open: why the bandwidth measurement fails on some cards, and what then | On the tester's RTX 3080 Ti it failed at +0, and since 0.4.4 that means no memory overclock at all on such a card, though it computed right far beyond +2000 MHz. The measurement gives a number only when three runs in a row agree within 1 % (`kBwSettle`), and it throws away every run the driver's timer calls disjoint (the clock changed under it). That card was held at a power limit of 65 % and ran at 83 C when it was measured; the reference card has only ever been measured at 105 %, where the measurement itself draws 179 of 210 W. So the suspicion is the power limit, and hardware check 90 tests exactly that on the reference card. What it shows decides what is built, and nothing is built before it: **(1) runs are thrown away as disjoint, and the ones that are kept agree:** the timer is what fails, not the card. Time the copy differently (wall clock around a flush, or count a disjoint run when its neighbours agree) and keep the method. **(2) the readings themselves scatter by more than 1 %:** a peak that is found by a 1 % drop (`kBandwidthDrop`) cannot be found in that noise. Then the memory has to be measured where the card is not held back: search it before the power step, at stock power, where a probe and its measurement last five seconds; or take more readings and compare medians. **(3) it does not fail at 65 % on the reference card:** the power limit is not the cause. Then it was the heat (a card throttling at its temperature limit changes clock as well), or another program, and only the tester's `run.log` can say: since this change a measurement that never settled writes how many runs gave a reading, between which values, and how many were thrown away. In no case does the climb on stability alone come back. **Outcome of check 90: (3).** On the reference card the measurement gave a number fifteen times out of fifteen at 100, 80 and 65 %, in three or four runs, with nothing thrown away, and the bandwidth was the same at every limit. The power limit is ruled out; (1) and (2) are not built. What is left is the heat of that card, another program, or a timer query that failed, and the tester's `run.log` is the next thing that can tell them apart. **The tester's run (check 91): it is (2), but only inside a run.** Twenty readings between 763.9 and 821.2 GB/s, none thrown away, three seconds after the load began; after thirty seconds of load the same card read within 0.2 %. The suspicion now is the moment, not the card: the measurement is taken while the power limiter is still catching up. If `gao --bandwidth --after-load 3` scatters on that card and `--after-load 30` does not, that is it, and the fix is to measure when the card has settled (keep taking runs for some seconds instead of twenty runs), not to change the method. | | Every line of a run is also in `run.log` | `logged()` wraps the log hook of both front ends: a line is appended and flushed before it is shown, so the step a machine freezes in is the last line on disk. One file in the state folder, moved aside at 1 MB (`run.log.old`). Only elevated and only after `ensure_app_dir` has verified the folder: an elevated process appends to nothing a user could have put there. | | A run does not start while the NVIDIA app's Automatic Tuning is on | Switched on, a service of the driver scans the card when it sees fit: it loads it, changes its clocks, and starts again by itself (hardware check 89: a scan that was stopped started again 14 ms later, and after a restart of the service it resumed within two seconds). A run next to that measures nothing it can trust, so it is refused, like a run next to Afterburner, and with the same weight: a courtesy, not a guard. Read from where that service keeps the switch, `%ProgramData%\NVIDIA Corporation\nvtopps\nvtopps.db3`, table `ocScannerSettings`, fields `backgroundScanEnabled` and `grdOcEnabled`, joined to `TelemetryGpuCfg` so that only a card that is in the PC counts (a row of an earlier card can never be switched off again). That this is the source and not a copy of it was measured, not assumed: with the field set the service took the scan up again after every restart, with it cleared it did not, while the NVIDIA app itself showed the switch as off. Nothing documents the file. So: opened read-only and immutable (no lock, the service never waits for us), through `winsqlite3.dll` from System32 (no dependency added), and everything that is not exactly as expected reads as "unknown", which starts the run as before. The registry value the app sets next to it (`HKLM\SOFTWARE\NVIDIA Corporation\Global\NvApp\OC\`) changes at the same moment but belongs to the app, not to the service, and is not used. **The overlay is not checked:** no setting for it was found, its processes run whether it is open or not, and the one value that moved (`IsShadowPlayEnabled`) followed the service being up, not a switch. NVAPI was not tried: an undocumented call is no better than an undocumented file, and it cannot be compared with anything. | diff --git a/docs/hardware-checks.md b/docs/hardware-checks.md index 4171b87..e8aa013 100644 --- a/docs/hardware-checks.md +++ b/docs/hardware-checks.md @@ -122,7 +122,7 @@ root; adjust the path for a Debug build. | 92 | The undervolt profile starts with the measurement at stock every run uses, and a hot card is not refused | Elevated, the tray app exited, a copy of `gao.json` kept. (a) `gao --undervolt` to the end; then `gao --status`. (b) The same with a test build whose stock limit is lowered to a temperature the reference card passes under load (63 C; it reaches 67). Put the copy of `gao.json` back. (c) Only on a card that is above the profile's limit at stock with the fans at 100 % (the tester's RTX 3080 Ti: 82 to 83 C): the **Undervolt** profile from the window, to the end, and its `run.log` | yes | (a) `the card at stock, five minutes`, a `stock: STABLE ...` line with the driver's fan speed, then `baseline: the measurement at stock`, and no second five minutes at stock: the first candidate follows at once; no `candidates are stopped at` line; `RESULT:`; `before:` shows 300 s and the fan speed the driver chose. (b) `stock: TOO HOT ... peak=64 C`, the line that the measurement stopped there, the cool-down, then `baseline: the measurement at stock` and a search that runs to `RESULT:` with `before:` showing the shortened time; no `candidates are stopped at` line, because 64 C is under the profile's 75. (c) no `not stable at stock`; `the card reached 86 C at stock: candidates are stopped at 85 C, not at the profile's 75 C`; candidates that end `STABLE` or `WRONG RESULT`, none `TOO HOT`; a result that is cooler and draws less than stock at the same clock | First run: **not as expected**, and that found a fault (2026-10-11, reference RTX 4070, driver 617.42, a test build of d1c99e9, started elevated by a script). The first halves held: (a) `stock: STABLE ... peak=65 C fan=70 %`, then `baseline: the measurement at stock` four seconds later and no second five minutes; (b) `stock: TOO HOT ... peak=64 C`, stopped after 34 s, the run went on, no `candidates are stopped at` line. But both runs then ended with `the card held no undervolt: the first step below its own voltage already failed`: their first candidates read `1070 mV (+15 MHz): SLOW ... clock=2323 MHz` and `1050 mV (+15 MHz): SLOW ... clock=2411 MHz`, on a card that ran 2791 MHz in the measurement just before. The search's load had not run yet when that candidate began (see the decision "With its reference handed over, the undervolt search warms its load up before the first candidate"). Second run, with the warm-up: **pass** for (a) and (b) (same day, a test build of 0733695). (a) `stock: STABLE score=5426 it/s clock=2788 MHz peak=66 C fan=70 %`, four seconds later `baseline: the measurement at stock`, `warming the load up: 5 s at stock`, `target: 2790 MHz, which the card reaches at 1090 mV by itself`, and the first candidate `1085 mV (+15 MHz): STABLE ... clock=2775 MHz`; 26 candidates, none `SLOW` or `TOO HOT`, the edge at 950 mV, `RESULT: 2790 MHz at 990 mV instead of 1090 mV`; `before:` 5426 it/s, 189 W, fan 70 % (300 s), `after:` 5325 it/s, 149 W, fan 53 % (300 s). (b) `stock: TOO HOT ... peak=64 C`, stopped after 74 s, the run went on, the same warm-up and a search to the same result (990 mV, edge 950 mV); `before:` shows 75 s; no `candidates are stopped at` line. The saved tune of before the test was put back and applied. (c) not run. | | 93 | A speed profile that would end slower than stock keeps nothing | Only on a card that is warmer at stock than the profile allows (the tester's RTX 3080 Ti; the reference card ends faster than stock with every speed profile): **Best of my GPU** from the window, to the end; the result screen, the dashboard and `run.log` | yes | the run goes through all its steps and its soak; then **Not applied** with `the result is slower than stock: N it/s against M (-X %)`, the power limit it had to go down to, and the pointer to Undervolt; `The card is back at stock.`; the dashboard shows stock and the saved tune of before the run, if there was one; `run.log` ends with `stopped: the result is slower than stock ... -- card restored to stock` | Not run yet. On the reference card: nothing to see, by design. | | 94 | All in one, with the undervolt step searching for the first time | Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize all` to the end with a build that has the warm-up of the undervolt step; then `gao --status`; put the copy back | yes | five steps; in step 3 a `warming the load up` line, a first candidate that is not `SLOW`, and a descent; steps 4 and 5 run on the tune of steps 2 and 3; `RESULT:` | **Not as expected at step 4**, and that found a fault (2026-10-11, reference RTX 4070, driver 617.42, a test build of 0733695, unattended). Step 1: `stock: STABLE score=5325 it/s peak=65 C fan=70 % power=186 W`. Step 2: power 105 %, `mem: bandwidth peak +2300, confirmed +2300, applying +1600`, `core: highest stable +210, confirmed +210, applying +135`, soak `STABLE score=5568 it/s peak=70 C power=197 W`. Step 3, as wanted: `warming the load up: 5 s with the overclock`, `target: 2865 MHz, which the card reaches at 1050 mV with the overclock`, nine candidates `STABLE` from 1045 mV down, `WRONG RESULT` at 990 mV, confirmed at 1010 mV after two tries, and `applying 1050 mV (+135 MHz at that point)`: the margin left no raise above the overclock, so the curve was cut off, as in the runs before the warm-up; soak `STABLE score=5434 it/s peak=67 C power=176 W`. Step 4 did not begin: `the tune could not be applied for the fan tune: the undervolt was not applied: the point already runs more than the target -- card at stock`. The saved tune of before the test was put back and applied. Cause and fix: the decision "A flat top goes back on a card that is cooler than when it was found". Second run, with the `gao.exe` of the released v0.4.6: **pass, all five steps** (same day, started elevated by a script). Step 1: `stock: STABLE score=5309 it/s peak=66 C fan=70 % power=185 W`. Step 2: power 105 %, `mem: bandwidth peak +850, confirmed +850, applying +550`, `core: highest stable +210, confirmed +210, applying +135`, soak `STABLE score=5568 it/s`. Step 3: `target: 2850 MHz, which the card reaches at 1045 mV with the overclock`, `undervolt: lowest stable 975 mV, confirmed 985 mV, applying 1025 mV (+165 MHz at that point)`, soak `STABLE score=5363 it/s peak=66 C power=163 W`. Step 4: the tune went back on the card, `fan 60 %: 63 C`, `fan 55 %: 65 C`, `fan 52 %: 67 C`, `fan tune: 52 % holds 67 C`. Step 5: the final test ran its 300 s. `RESULT: power 105 %, core +135 MHz, mem +550 MHz, 2850 MHz at 1025 mV, fans 52 % at 67 C`; `stock: 5309 it/s, 185 W, fan 70 %, 66 C`, `now: 5353 it/s, 166 W, fan 52 %, 70 C`: `+0.8 % score and -10.3 % power against stock`; `undervolt: applied`. The saved tune of before the test was put back. **What this run did not show:** its undervolt had a raise of its own (+165 over an offset of +135), so the anchor was raised and not merely cut off, and the case of the first run (an anchor that reads a step above the target on a cooler card) did not come up. That fix has still only its tests. **The memory offset differs a lot from run to run on this card, and the two logs say why:** +1600 in the first run (bandwidth peak +2300), +550 in the second (peak +850), +650 in the run that saved the tune in use that morning (the journal shows its scan ending at +1000 too). The card was in the same state each time (68 C, power 105 %, a reading every three to four seconds). In both logged scans the bandwidth rises in a straight line, 0.36 and 0.46 % per 50 MHz step, from 420 GB/s at +0 to 509 GB/s at +2450; the first run went on to `WRONG RESULT` at +2550 without the bandwidth ever turning down. The readings scatter around that line with a standard deviation of 0.4 and 0.7 %, single ones by more than 1 %. The scan ends at a reading more than 1 % below the best so far when a second reading agrees, and in the second run that was chance: +950 read high (462.8 GB/s, where the line is at about 458), +1000 read 456.6 twice, 1.3 % below it. In the first run the one low reading (+600) was followed by a higher one, and the scan ran to the first wrong result. So on this card there is no bandwidth peak before the errors, and where the scan stops is decided by noise. | -| 95 | The memory scan no longer ends by chance | (a) Not elevated, nothing changed on the card: a measuring build takes 24 bandwidth readings in one process, with 3 s of the stress load before each as a scan has, at 64, 128 and 256 dispatches per timed run; then the same with the app's window open next to it, drawing a frame every 100 ms and every 33 ms. (b) Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize best` twice in a row with the build of the change, then the copy put back | (a) no, (b) yes | (a) how much a reading scatters, and on what that depends. (b) in both runs every `mem +N: bandwidth` line shows three readings that agree within about 1 %; the scan does not end before the first `WRONG RESULT`; the two `mem: bandwidth peak` values are within 300 MHz of each other and the applied offset is above +1000; the soak is not slower than at stock | **(a) measured** (2026-10-11, reference RTX 4070, driver 617.42, the saved tune applied: power 105 %, core +135, memory +650; standard deviation over the 24 readings of a series, and the lowest and highest against the mean). 64 dispatches, no window: 0.44 % (-0.55 to +1.07, one reading did not settle), 0.66 % (-1.58 to +1.49, one did not settle), 1.31 % (-4.83 to +1.27), 0.86 % (-1.44 to +2.05). 128 dispatches, no window: 0.43 % (-1.10 to +0.79). 256 dispatches, no window: 0.22 % (-0.43 to +0.42) and 0.60 % (-0.80 to +1.37; the last five readings of this series were 451 to 453 GB/s after nineteen of 444 to 449). With the window drawing every 100 ms: 64 dispatches 1.13 % (the first six readings 452 to 453 GB/s, the rest 439 to 451), 256 dispatches 0.24 % (-0.47 to +0.61). With the window drawing every 33 ms: 64 dispatches 0.58 %, 256 dispatches 0.25 % (-0.35 to +0.51). A reading took 6.5, 5.3 and 3.9 timed runs to settle at 64, 128 and 256 dispatches. So: the longer timed run takes the scatter from 0.4 to 1.3 % down to 0.2 %, the window's drawing adds nothing, and apart from the scatter the readings sit about 1.8 % higher for up to a minute at a time (the stretch at the end of the second 256 series ran on into the next series; the last 64 series went back and forth between 444 and 452 GB/s). What sets that level is not known. **(b) the offset is the same twice; the scan ends in a driver reset twice** (2026-10-11, same card and driver, a test build of 2764609, unattended, 40 minutes for the two runs). Run 1: `stock: STABLE score=5311 it/s peak=66 C fan=70 % power=185 W`; 49 steps from `mem +0: bandwidth 422.7 GB/s (422.7 422.6 422.7)` to `mem +2400: bandwidth 501.4 GB/s (501.3 502.3 501.4)`; at +2450 the candidate was `STABLE` and the measurement after it lost the device: `mem +2450: the bandwidth measurement failed; treated as a driver reset`, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, 20 s of rest, `health: STABLE`, `confirm core +0 / mem +2050: STABLE`, `mem: bandwidth peak +2250, confirmed +2050, applying +1400`; `core: highest stable +210, confirmed +210, applying +135`; soak `STABLE score=5536 it/s peak=69 C power=195 W`; `driver resets during this run: 1`. Run 2: stock 5312 it/s; 50 steps from 421.8 GB/s at +0 to 502.8 at +2450; the same failure at +2500; `mem: bandwidth peak +2300, confirmed +2100, applying +1450`; core +135; soak `STABLE score=5555 it/s peak=69 C power=196 W`; one driver reset. The three readings of a step differed by 0.4 % on average and by 1.2 % at most; no step was more than 0.7 % below the best before it, so neither scan came near ending early, by the old rule or the new; every step had its three readings. The scan took 5 min 25 s (about 6.5 s a step; about 3 min before the change). As expected: the two peaks are 50 MHz apart, the applied offsets are +1400 and +1450, and both results are 4 % faster than stock. **Not as expected: neither scan reached a `WRONG RESULT`.** Both ended one and two steps before +2550 with the device lost during the bandwidth copy; Windows logged it both times (System log, `nvlddmkm` events 14 and 153 at 19:51:22 and 20:11:27), and the screen goes black for a moment. In the run of check 94 that got this far, the measurements at +2450 and +2500 gave a number and the stress load reported the first wrong result at +2550, without a reset. A step now copies about seven times as much as it did (three readings of about four runs of 128 GB, against one reading of about six runs of 32 GB), and the copy checks nothing, so where the memory starts to fail it is the driver that gives out, not a verdict. Two runs against one: likely the cause, not proven. The search handles it as it should (rest, health probe, confirm four steps lower, the core is still searched), but a driver reset in every run on a card like this is not what the change was for. The saved tune of before the test was put back and applied (`applied best: power 105 %, core +135 MHz, mem +650 MHz`), the tray app restarted, Defender reported nothing. **(c) the same two runs with one reading a step, three only for a step in doubt** (2026-10-11 20:26 to 21:03, a test build of 71e7c2b, unattended): **the reset is still there.** Run 1: stock 5328 it/s; 50 steps from `mem +0: bandwidth 420.4 GB/s` to `mem +2450: bandwidth 502.4 GB/s`; at +2500 the candidate was `STABLE` and then `bandwidth measurement failed: the timing of the copy did not come back` ten seconds later (`nvlddmkm` event 14 at 20:37:07); `mem: bandwidth peak +2200, confirmed +2000, applying +1400`; core +135; soak `STABLE score=5572 it/s peak=70 C power=197 W`; one driver reset. Run 2: stock 5311 it/s; 421.9 GB/s at +0 to 508.0 at +2450; at +2500 `the device was lost during the copy failed (hr=0x887A0006)`, the device hung (event 14 at 20:55:29); `mem: bandwidth peak +2400, confirmed +2200, applying +1500`; core +135; soak `STABLE score=5571 it/s peak=70 C power=196 W`; one driver reset. No step was in doubt in either run (the lowest was 0.75 % below the best before it), so every step was read once; the scan took 3 min 33 s. The two peaks are 200 MHz apart: with one reading a step, which offsets count as within 1 % of the best moves by a few steps. Applied +1400 and +1500, both 5 % faster than stock. So the reset does not come from reading a step three times. Four runs out of four with the longer timed run ended this way, at +2450 or +2500. Not merged; the design for a scan that does not end like this is the open row in `docs/development.md`. The saved tune was put back and applied again (core +135, memory +650), the tray app restarted, Defender reported nothing. | +| 95 | The memory scan no longer ends by chance | (a) Not elevated, nothing changed on the card: a measuring build takes 24 bandwidth readings in one process, with 3 s of the stress load before each as a scan has, at 64, 128 and 256 dispatches per timed run; then the same with the app's window open next to it, drawing a frame every 100 ms and every 33 ms. (b) Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize best` twice in a row with the build of the change, then the copy put back | (a) no, (b) yes | (a) how much a reading scatters, and on what that depends. (b) in both runs every `mem +N: bandwidth` line shows three readings that agree within about 1 %; the scan does not end before the first `WRONG RESULT`; the two `mem: bandwidth peak` values are within 300 MHz of each other and the applied offset is above +1000; the soak is not slower than at stock | **(a) measured** (2026-10-11, reference RTX 4070, driver 617.42, the saved tune applied: power 105 %, core +135, memory +650; standard deviation over the 24 readings of a series, and the lowest and highest against the mean). 64 dispatches, no window: 0.44 % (-0.55 to +1.07, one reading did not settle), 0.66 % (-1.58 to +1.49, one did not settle), 1.31 % (-4.83 to +1.27), 0.86 % (-1.44 to +2.05). 128 dispatches, no window: 0.43 % (-1.10 to +0.79). 256 dispatches, no window: 0.22 % (-0.43 to +0.42) and 0.60 % (-0.80 to +1.37; the last five readings of this series were 451 to 453 GB/s after nineteen of 444 to 449). With the window drawing every 100 ms: 64 dispatches 1.13 % (the first six readings 452 to 453 GB/s, the rest 439 to 451), 256 dispatches 0.24 % (-0.47 to +0.61). With the window drawing every 33 ms: 64 dispatches 0.58 %, 256 dispatches 0.25 % (-0.35 to +0.51). A reading took 6.5, 5.3 and 3.9 timed runs to settle at 64, 128 and 256 dispatches. So: the longer timed run takes the scatter from 0.4 to 1.3 % down to 0.2 %, the window's drawing adds nothing, and apart from the scatter the readings sit about 1.8 % higher for up to a minute at a time (the stretch at the end of the second 256 series ran on into the next series; the last 64 series went back and forth between 444 and 452 GB/s). What sets that level is not known. **(b) the offset is the same twice; the scan ends in a driver reset twice** (2026-10-11, same card and driver, a test build of 2764609, unattended, 40 minutes for the two runs). Run 1: `stock: STABLE score=5311 it/s peak=66 C fan=70 % power=185 W`; 49 steps from `mem +0: bandwidth 422.7 GB/s (422.7 422.6 422.7)` to `mem +2400: bandwidth 501.4 GB/s (501.3 502.3 501.4)`; at +2450 the candidate was `STABLE` and the measurement after it lost the device: `mem +2450: the bandwidth measurement failed; treated as a driver reset`, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, 20 s of rest, `health: STABLE`, `confirm core +0 / mem +2050: STABLE`, `mem: bandwidth peak +2250, confirmed +2050, applying +1400`; `core: highest stable +210, confirmed +210, applying +135`; soak `STABLE score=5536 it/s peak=69 C power=195 W`; `driver resets during this run: 1`. Run 2: stock 5312 it/s; 50 steps from 421.8 GB/s at +0 to 502.8 at +2450; the same failure at +2500; `mem: bandwidth peak +2300, confirmed +2100, applying +1450`; core +135; soak `STABLE score=5555 it/s peak=69 C power=196 W`; one driver reset. The three readings of a step differed by 0.4 % on average and by 1.2 % at most; no step was more than 0.7 % below the best before it, so neither scan came near ending early, by the old rule or the new; every step had its three readings. The scan took 5 min 25 s (about 6.5 s a step; about 3 min before the change). As expected: the two peaks are 50 MHz apart, the applied offsets are +1400 and +1450, and both results are 4 % faster than stock. **Not as expected: neither scan reached a `WRONG RESULT`.** Both ended one and two steps before +2550 with the device lost during the bandwidth copy; Windows logged it both times (System log, `nvlddmkm` events 14 and 153 at 19:51:22 and 20:11:27), and the screen goes black for a moment. In the run of check 94 that got this far, the measurements at +2450 and +2500 gave a number and the stress load reported the first wrong result at +2550, without a reset. A step now copies about seven times as much as it did (three readings of about four runs of 128 GB, against one reading of about six runs of 32 GB), and the copy checks nothing, so where the memory starts to fail it is the driver that gives out, not a verdict. Two runs against one: likely the cause, not proven. The search handles it as it should (rest, health probe, confirm four steps lower, the core is still searched), but a driver reset in every run on a card like this is not what the change was for. The saved tune of before the test was put back and applied (`applied best: power 105 %, core +135 MHz, mem +650 MHz`), the tray app restarted, Defender reported nothing. **(c) the same two runs with one reading a step, three only for a step in doubt** (2026-10-11 20:26 to 21:03, a test build of 71e7c2b, unattended): **the reset is still there.** Run 1: stock 5328 it/s; 50 steps from `mem +0: bandwidth 420.4 GB/s` to `mem +2450: bandwidth 502.4 GB/s`; at +2500 the candidate was `STABLE` and then `bandwidth measurement failed: the timing of the copy did not come back` ten seconds later (`nvlddmkm` event 14 at 20:37:07); `mem: bandwidth peak +2200, confirmed +2000, applying +1400`; core +135; soak `STABLE score=5572 it/s peak=70 C power=197 W`; one driver reset. Run 2: stock 5311 it/s; 421.9 GB/s at +0 to 508.0 at +2450; at +2500 `the device was lost during the copy failed (hr=0x887A0006)`, the device hung (event 14 at 20:55:29); `mem: bandwidth peak +2400, confirmed +2200, applying +1500`; core +135; soak `STABLE score=5571 it/s peak=70 C power=196 W`; one driver reset. No step was in doubt in either run (the lowest was 0.75 % below the best before it), so every step was read once; the scan took 3 min 33 s. The two peaks are 200 MHz apart: with one reading a step, which offsets count as within 1 % of the best moves by a few steps. Applied +1400 and +1500, both 5 % faster than stock. So the reset does not come from reading a step three times. Four runs out of four with the longer timed run ended this way, at +2450 or +2500. Not merged; the design for a scan that does not end like this is the open row in `docs/development.md`. The saved tune was put back and applied again (core +135, memory +650), the tray app restarted, Defender reported nothing. **(d) the edge is remembered** (a test build of 7c215c4; the same two runs): expected is that the first scan ends as before and logs nothing new, the journal gets one `mem_edge` line with the GPU id and the driver version, and the second run logs `mem: the memory gave out at +N in an earlier run on this card and driver; the scan stops at +(N-200)`, has no `driver resets during this run` line, applies more than +1000 and is not slower than stock. **Not run yet.** **(e) how the stress load meets the edge** (the same build, no search): `gao --reset`, then `gao --set-mem` and `gao --stress 3` at +2300, +2350, ... until the verdict is not `STABLE`, five times, at stock power and core +0; expected is one of two outcomes, `WRONG RESULT` every time (then the stress load can lead the copy to the edge) or `DEVICE LOST` (then it cannot). **Not run yet.** | ## Notes From 6537fd45741e22f171b1812c9e3e66bc1a4bc02e Mon Sep 17 00:00:00 2001 From: Rovey Date: Sun, 11 Oct 2026 21:37:51 +0200 Subject: [PATCH 07/15] docs: hardware check 95 (d), the first run met the memory's edge and the machine restarted Co-Authored-By: Claude Opus 5.5 --- docs/development.md | 2 +- docs/hardware-checks.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/development.md b/docs/development.md index afd2138..a819c99 100644 --- a/docs/development.md +++ b/docs/development.md @@ -80,7 +80,7 @@ The applied offset is always at least one step below the confirmed edge. Profile | A confirm probe that is too hot at once confirms nothing | A confirm probe that ends `TOO HOT` keeps the clock edge: over 30 s the card may run a little hotter than over the power step's 20 s, and that is the soak's business, which lowers power. But the stress run stops at its first reading above the limit, so a card that is above it when the probe begins gives `TOO HOT` within a second, and that counted as holding too. It happened on the tester's RTX 3080 Ti (hardware check 91): after a driver reset the health probe loads the card at stock power, which on that card means 76 C against a limit of 75, and `confirm core +225` was confirmed by a probe that had tested nothing. Now such a probe counts only when the clock was loaded for `kConfirmMinS` (10 s, clearly longer than the 3 s probe that found the edge) before it got too hot. Otherwise the value is not confirmed, and the card rests (`io.rest`, the 20 s of a recovery, with the fans where the curve last put them) before the next try one step lower: without the rest every lower value would start just as hot, and heat alone would step the edge down to nothing, which is what keeping the edge on `TOO HOT` was for. The same value is not tried again; one step is the price. Not changed: the health probe still runs at stock power, and a soak that begins too hot still lowers the power limit by a step. | | A profile that promises speed keeps no result that is slower than stock | Best of my GPU ended 3.8 % slower than stock on the tester's RTX 3080 Ti (4749 against 4937 it/s, hardware check 91), with the fans at 80 % instead of 66 %, and showed it as **Done**, applied and kept at every logon. Every step had done what it is for: the card runs 83 C at stock with its fans at 100 %, the profile allows 75, so the power limit went to 65 %, and +150 MHz of core clock did not win that back. But a profile called "the best of my GPU" that leaves the card slower has not done what it says, and a result like that is not offered as one. After the soak, `optimize` compares its score with the measurement at stock that went before the run (the "before" the result screen shows, so the rule judges by the number the user sees): below `kNotSlowerScore` (0.99: one percent is what two five-minute runs differ by) the run ends as a failed one does, at stock, with nothing saved, and says what it found, why, and that Undervolt is the profile for a card that runs warm. `Objectives::faster_than_stock` marks the two profiles this holds for, Best of my GPU and Max performance. Not Quiet and Cool & efficient, which trade speed for noise and heat and say so. Not the overclock step of All in one either: it goes on to an undervolt and a fan tune, and its result is judged as a whole; whether that whole may be slower than stock is not decided. This is a backstop, not the cure: what such a card needs from these profiles is in `docs/planned.md` ("never hotter than at stock"). | | The memory scan times a longer copy, reads a step in doubt three times, and stops at a 3 % drop | Four runs on one day applied +650, +1600, +550 and +100 MHz on the reference RTX 4070, a card that runs every game at +1500 (hardware check 94). That card has no bandwidth peak: bandwidth rises by about 0.4 % per 50 MHz step all the way to +2500 and the first wrong result comes at +2550, so the scan could only end early by chance, and where it did decided the offset. Measured afterwards without a search and with nothing changed on the card (hardware check 95 a), the readings move in three ways, and each got its own answer. **(1) One timed run was too short.** A run copied 32 GB, 75 ms on this card, and a reading is the median of three runs that agree within 1 %. Such readings scattered by 0.4 to 1.3 % (standard deviation over 24 readings, six series), single ones by up to 4.8 %, and in two of the six series one reading did not settle. With 128 GB per run (`kBwDispatches` 256, 0.3 s) it is 0.2 %, and all 96 readings settled. **(2) A single reading decided the end.** A step that is more than `kBandwidthTie` (1 %) below the best is in doubt: it is sampled `kBandwidthReadings` (3) times and its middle reading counts, which replaces the single re-measure of a drop. A reading that only did not settle is taken again, three times at most; any other failure ends the step at once, as before. Three readings at every step were built first and taken back: see the row below. **(3) The readings change level.** For up to a minute at a time every reading is about 1.8 % higher than before and after (452 against 444 GB/s), at one and the same offset. The cause is not known. Reading a step three times does nothing against it: all three are inside the stretch. So the drop that ends the scan (`kBandwidthDrop`) went from 1 % to 3 %: above what the readings do by themselves, below what a real peak loses within a step or two. The price on a card that has a peak: the scan goes a step or two further past it, in 3 s probes, before it stops. The offset it keeps is still the lowest within 1 % of the best, and confirming starts there. **Not built: a window that stands still during a measurement.** The suspicion was that the window's own drawing disturbed the readings of the run that saved +100. Measured with a window drawing a frame every 100 ms and every 33 ms next to the measurement: 0.2 %, the same as without. **On the card (check 95 b and c):** four runs applied +1400, +1450, +1400 and +1500, each 4 to 5 % faster than stock; no step came within reach of ending a scan early (the lowest was 0.75 % below the best before it), and no step was in doubt. A scan of 50 steps takes 3 min 33 s, half a minute more than before. **What it does not solve** is in the row below, and keeps the change from being merged. A change of level larger than 3 % would end a scan by chance again: the answer to that is to compare a low step with a fresh reading of the best offset taken right after it, not a still higher threshold. | -| Open: on a card without a bandwidth peak the memory scan ends in a driver reset | With the change of the row above the scan on the reference card no longer ends by chance, so it always runs to where the memory gives out. In four runs out of four that was not a wrong result from the stress load but the bandwidth copy losing the device at +2450 or +2500 (`DXGI_ERROR_DEVICE_HUNG`, or the timing of the copy not coming back within its 5 s; `nvlddmkm` events 14 and 153 in the System log; the screen goes black for a moment), with three readings a step and with one (hardware check 95 b and c). So it is not how often a step is read. In the one run before the change that got this far (check 94), the copies at +2450 and +2500 gave a number and the stress load reported the first wrong result at +2550. What differs is the timed run: 128 GB in one go against 32. One run against four is thin, and a memory edge moves by a step from run to run; it is the likely cause, not a proven one. The search handles the reset as it should (rest, health probe, confirm four steps lower, the core still searched) and the result is good. But a reset as the normal end of a scan is not acceptable in a tool that is to be simple and stable. **Design, not built.** A card like this has nothing that announces its edge: the bandwidth rises until the memory fails. No stop rule can know where that is without having been there, so the design does not try to. It makes the meeting happen once, and with the probe that can report it. **(a) The edge is found once and remembered: built.** The journal already turns a candidate that never finished (the machine froze) into a ceiling for every later run. A memory scan that ended in a wrong result or a reset event (a driver reset, a lost device) now writes that offset to the journal as well, in a line of its own (`{"mem_edge":2500,"rig":" / "}`; it has no id, so a build that does not know it skips it). A later scan on the same card and driver stops `kResetBackoffSteps` (4) steps below the lowest such offset, which is where confirming starts after a reset, and says so: `mem: the memory gave out at +2500 in an earlier run on this card and driver; the scan stops at +2300`. With another driver version or another card the line does not count; back on the driver it was found with, it counts again. Without a GPU id or a driver version nothing is kept. A scan that ended for another reason (a bandwidth peak, readings that did not settle, a measurement that does not work, the driver's own range) writes nothing. It errs to the safe side: an edge found on a bad day stays until the driver changes or the journal is removed. It leaves the first run as it is. On the card: hardware check 95 (d). **(b) The stress load meets the edge before the copy does.** The stress load checks its results and says `WRONG RESULT`; the copy checks nothing and can only hang. So the copy is taken only at offsets the stress load has already passed four steps higher: per step a 3 s probe at +200 MHz above it, then back and the timed copy. On the reference card the load would report the wrong result at +2550 while the highest copy taken is at +2350, where all five scans measured a number. The price: about a second more per step, and on a card that has a peak the probes run four steps beyond the point where the scan would stop. That is 3 s probes, as the 3 % rule already allows a step or two; it is not the climb on stability alone that took a 3080 Ti to +2300, because the copy still ends the scan at the peak. Before (b) is built it has to be measured that the stress load at the edge ends in a wrong result and not in a reset of its own: one observation says so (check 94), core candidates say so in every run, and that is not enough for memory. **Looked at and not proposed.** A fixed top for the scan (an offset, or a share of the driver's range): there is nothing to choose the number by, and a card whose memory gives out below it meets its edge all the same. A top at a bandwidth gain over +0 (say 15 %): the same objection, most cards will fail first. Going back to the short timed run: it brings back the scatter that made the scan end by chance, on the strength of one run. **Order:** (a) first; the measurement for (b) next (hardware check 95 e): the stress load alone, with no search, in 3 s runs from +2300 up until it fails, five times. | +| Open: on a card without a bandwidth peak the memory scan ends in a driver reset | With the change of the row above the scan on the reference card no longer ends by chance, so it always runs to where the memory gives out. In four runs out of four that was not a wrong result from the stress load but the bandwidth copy losing the device at +2450 or +2500 (`DXGI_ERROR_DEVICE_HUNG`, or the timing of the copy not coming back within its 5 s; `nvlddmkm` events 14 and 153 in the System log; the screen goes black for a moment), with three readings a step and with one (hardware check 95 b and c). So it is not how often a step is read. In the one run before the change that got this far (check 94), the copies at +2450 and +2500 gave a number and the stress load reported the first wrong result at +2550. What differs is the timed run: 128 GB in one go against 32. One run against four is thin, and a memory edge moves by a step from run to run; it is the likely cause, not a proven one. The search handles the reset as it should (rest, health probe, confirm four steps lower, the core still searched) and the result is good. But a reset as the normal end of a scan is not acceptable in a tool that is to be simple and stable. **Design, not built.** A card like this has nothing that announces its edge: the bandwidth rises until the memory fails. No stop rule can know where that is without having been there, so the design does not try to. It makes the meeting happen once, and with the probe that can report it. **(a) The edge is found once and remembered: built.** The journal already turns a candidate that never finished (the machine froze) into a ceiling for every later run. A memory scan that ended in a wrong result or a reset event (a driver reset, a lost device) now writes that offset to the journal as well, in a line of its own (`{"mem_edge":2500,"rig":" / "}`; it has no id, so a build that does not know it skips it). A later scan on the same card and driver stops `kResetBackoffSteps` (4) steps below the lowest such offset, which is where confirming starts after a reset, and says so: `mem: the memory gave out at +2500 in an earlier run on this card and driver; the scan stops at +2300`. With another driver version or another card the line does not count; back on the driver it was found with, it counts again. Without a GPU id or a driver version nothing is kept. A scan that ended for another reason (a bandwidth peak, readings that did not settle, a measurement that does not work, the driver's own range) writes nothing. It errs to the safe side: an edge found on a bad day stays until the driver changes or the journal is removed. It leaves the first run as it is. On the card (hardware check 95 d): the first run of its test met the edge at +2450 like the four before it, and this time the whole machine restarted, seconds after the device was lost. The edge was on disk by then, which is what this part is for. But it changes what the open point is: the first meeting with the edge is not a moment of black screen that the search recovers from, it can cost the machine. "Once per card" is not good enough for that, and until the scan no longer reaches the edge with the copy, this branch is less safe than what is released, where the short copy got there once in four runs and ended in a wrong result. **(b) The stress load meets the edge before the copy does.** The stress load checks its results and says `WRONG RESULT`; the copy checks nothing and can only hang. So the copy is taken only at offsets the stress load has already passed four steps higher: per step a 3 s probe at +200 MHz above it, then back and the timed copy. On the reference card the load would report the wrong result at +2550 while the highest copy taken is at +2350, where all five scans measured a number. The price: about a second more per step, and on a card that has a peak the probes run four steps beyond the point where the scan would stop. That is 3 s probes, as the 3 % rule already allows a step or two; it is not the climb on stability alone that took a 3080 Ti to +2300, because the copy still ends the scan at the peak. Before (b) is built it has to be measured that the stress load at the edge ends in a wrong result and not in a reset of its own: one observation says so (check 94), core candidates say so in every run, and that is not enough for memory. **Looked at and not proposed.** A fixed top for the scan (an offset, or a share of the driver's range): there is nothing to choose the number by, and a card whose memory gives out below it meets its edge all the same. A top at a bandwidth gain over +0 (say 15 %): the same objection, most cards will fail first. Going back to the short timed run: it brings back the scatter that made the scan end by chance, on the strength of one run. **Order:** (a) first; the measurement for (b) next (hardware check 95 e): the stress load alone, with no search, in 3 s runs from +2300 up until it fails, five times. | | Open: why the bandwidth measurement fails on some cards, and what then | On the tester's RTX 3080 Ti it failed at +0, and since 0.4.4 that means no memory overclock at all on such a card, though it computed right far beyond +2000 MHz. The measurement gives a number only when three runs in a row agree within 1 % (`kBwSettle`), and it throws away every run the driver's timer calls disjoint (the clock changed under it). That card was held at a power limit of 65 % and ran at 83 C when it was measured; the reference card has only ever been measured at 105 %, where the measurement itself draws 179 of 210 W. So the suspicion is the power limit, and hardware check 90 tests exactly that on the reference card. What it shows decides what is built, and nothing is built before it: **(1) runs are thrown away as disjoint, and the ones that are kept agree:** the timer is what fails, not the card. Time the copy differently (wall clock around a flush, or count a disjoint run when its neighbours agree) and keep the method. **(2) the readings themselves scatter by more than 1 %:** a peak that is found by a 1 % drop (`kBandwidthDrop`) cannot be found in that noise. Then the memory has to be measured where the card is not held back: search it before the power step, at stock power, where a probe and its measurement last five seconds; or take more readings and compare medians. **(3) it does not fail at 65 % on the reference card:** the power limit is not the cause. Then it was the heat (a card throttling at its temperature limit changes clock as well), or another program, and only the tester's `run.log` can say: since this change a measurement that never settled writes how many runs gave a reading, between which values, and how many were thrown away. In no case does the climb on stability alone come back. **Outcome of check 90: (3).** On the reference card the measurement gave a number fifteen times out of fifteen at 100, 80 and 65 %, in three or four runs, with nothing thrown away, and the bandwidth was the same at every limit. The power limit is ruled out; (1) and (2) are not built. What is left is the heat of that card, another program, or a timer query that failed, and the tester's `run.log` is the next thing that can tell them apart. **The tester's run (check 91): it is (2), but only inside a run.** Twenty readings between 763.9 and 821.2 GB/s, none thrown away, three seconds after the load began; after thirty seconds of load the same card read within 0.2 %. The suspicion now is the moment, not the card: the measurement is taken while the power limiter is still catching up. If `gao --bandwidth --after-load 3` scatters on that card and `--after-load 30` does not, that is it, and the fix is to measure when the card has settled (keep taking runs for some seconds instead of twenty runs), not to change the method. | | Every line of a run is also in `run.log` | `logged()` wraps the log hook of both front ends: a line is appended and flushed before it is shown, so the step a machine freezes in is the last line on disk. One file in the state folder, moved aside at 1 MB (`run.log.old`). Only elevated and only after `ensure_app_dir` has verified the folder: an elevated process appends to nothing a user could have put there. | | A run does not start while the NVIDIA app's Automatic Tuning is on | Switched on, a service of the driver scans the card when it sees fit: it loads it, changes its clocks, and starts again by itself (hardware check 89: a scan that was stopped started again 14 ms later, and after a restart of the service it resumed within two seconds). A run next to that measures nothing it can trust, so it is refused, like a run next to Afterburner, and with the same weight: a courtesy, not a guard. Read from where that service keeps the switch, `%ProgramData%\NVIDIA Corporation\nvtopps\nvtopps.db3`, table `ocScannerSettings`, fields `backgroundScanEnabled` and `grdOcEnabled`, joined to `TelemetryGpuCfg` so that only a card that is in the PC counts (a row of an earlier card can never be switched off again). That this is the source and not a copy of it was measured, not assumed: with the field set the service took the scan up again after every restart, with it cleared it did not, while the NVIDIA app itself showed the switch as off. Nothing documents the file. So: opened read-only and immutable (no lock, the service never waits for us), through `winsqlite3.dll` from System32 (no dependency added), and everything that is not exactly as expected reads as "unknown", which starts the run as before. The registry value the app sets next to it (`HKLM\SOFTWARE\NVIDIA Corporation\Global\NvApp\OC\`) changes at the same moment but belongs to the app, not to the service, and is not used. **The overlay is not checked:** no setting for it was found, its processes run whether it is open or not, and the one value that moved (`IsShadowPlayEnabled`) followed the service being up, not a switch. NVAPI was not tried: an undocumented call is no better than an undocumented file, and it cannot be compared with anything. | diff --git a/docs/hardware-checks.md b/docs/hardware-checks.md index e8aa013..21ee509 100644 --- a/docs/hardware-checks.md +++ b/docs/hardware-checks.md @@ -122,7 +122,7 @@ root; adjust the path for a Debug build. | 92 | The undervolt profile starts with the measurement at stock every run uses, and a hot card is not refused | Elevated, the tray app exited, a copy of `gao.json` kept. (a) `gao --undervolt` to the end; then `gao --status`. (b) The same with a test build whose stock limit is lowered to a temperature the reference card passes under load (63 C; it reaches 67). Put the copy of `gao.json` back. (c) Only on a card that is above the profile's limit at stock with the fans at 100 % (the tester's RTX 3080 Ti: 82 to 83 C): the **Undervolt** profile from the window, to the end, and its `run.log` | yes | (a) `the card at stock, five minutes`, a `stock: STABLE ...` line with the driver's fan speed, then `baseline: the measurement at stock`, and no second five minutes at stock: the first candidate follows at once; no `candidates are stopped at` line; `RESULT:`; `before:` shows 300 s and the fan speed the driver chose. (b) `stock: TOO HOT ... peak=64 C`, the line that the measurement stopped there, the cool-down, then `baseline: the measurement at stock` and a search that runs to `RESULT:` with `before:` showing the shortened time; no `candidates are stopped at` line, because 64 C is under the profile's 75. (c) no `not stable at stock`; `the card reached 86 C at stock: candidates are stopped at 85 C, not at the profile's 75 C`; candidates that end `STABLE` or `WRONG RESULT`, none `TOO HOT`; a result that is cooler and draws less than stock at the same clock | First run: **not as expected**, and that found a fault (2026-10-11, reference RTX 4070, driver 617.42, a test build of d1c99e9, started elevated by a script). The first halves held: (a) `stock: STABLE ... peak=65 C fan=70 %`, then `baseline: the measurement at stock` four seconds later and no second five minutes; (b) `stock: TOO HOT ... peak=64 C`, stopped after 34 s, the run went on, no `candidates are stopped at` line. But both runs then ended with `the card held no undervolt: the first step below its own voltage already failed`: their first candidates read `1070 mV (+15 MHz): SLOW ... clock=2323 MHz` and `1050 mV (+15 MHz): SLOW ... clock=2411 MHz`, on a card that ran 2791 MHz in the measurement just before. The search's load had not run yet when that candidate began (see the decision "With its reference handed over, the undervolt search warms its load up before the first candidate"). Second run, with the warm-up: **pass** for (a) and (b) (same day, a test build of 0733695). (a) `stock: STABLE score=5426 it/s clock=2788 MHz peak=66 C fan=70 %`, four seconds later `baseline: the measurement at stock`, `warming the load up: 5 s at stock`, `target: 2790 MHz, which the card reaches at 1090 mV by itself`, and the first candidate `1085 mV (+15 MHz): STABLE ... clock=2775 MHz`; 26 candidates, none `SLOW` or `TOO HOT`, the edge at 950 mV, `RESULT: 2790 MHz at 990 mV instead of 1090 mV`; `before:` 5426 it/s, 189 W, fan 70 % (300 s), `after:` 5325 it/s, 149 W, fan 53 % (300 s). (b) `stock: TOO HOT ... peak=64 C`, stopped after 74 s, the run went on, the same warm-up and a search to the same result (990 mV, edge 950 mV); `before:` shows 75 s; no `candidates are stopped at` line. The saved tune of before the test was put back and applied. (c) not run. | | 93 | A speed profile that would end slower than stock keeps nothing | Only on a card that is warmer at stock than the profile allows (the tester's RTX 3080 Ti; the reference card ends faster than stock with every speed profile): **Best of my GPU** from the window, to the end; the result screen, the dashboard and `run.log` | yes | the run goes through all its steps and its soak; then **Not applied** with `the result is slower than stock: N it/s against M (-X %)`, the power limit it had to go down to, and the pointer to Undervolt; `The card is back at stock.`; the dashboard shows stock and the saved tune of before the run, if there was one; `run.log` ends with `stopped: the result is slower than stock ... -- card restored to stock` | Not run yet. On the reference card: nothing to see, by design. | | 94 | All in one, with the undervolt step searching for the first time | Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize all` to the end with a build that has the warm-up of the undervolt step; then `gao --status`; put the copy back | yes | five steps; in step 3 a `warming the load up` line, a first candidate that is not `SLOW`, and a descent; steps 4 and 5 run on the tune of steps 2 and 3; `RESULT:` | **Not as expected at step 4**, and that found a fault (2026-10-11, reference RTX 4070, driver 617.42, a test build of 0733695, unattended). Step 1: `stock: STABLE score=5325 it/s peak=65 C fan=70 % power=186 W`. Step 2: power 105 %, `mem: bandwidth peak +2300, confirmed +2300, applying +1600`, `core: highest stable +210, confirmed +210, applying +135`, soak `STABLE score=5568 it/s peak=70 C power=197 W`. Step 3, as wanted: `warming the load up: 5 s with the overclock`, `target: 2865 MHz, which the card reaches at 1050 mV with the overclock`, nine candidates `STABLE` from 1045 mV down, `WRONG RESULT` at 990 mV, confirmed at 1010 mV after two tries, and `applying 1050 mV (+135 MHz at that point)`: the margin left no raise above the overclock, so the curve was cut off, as in the runs before the warm-up; soak `STABLE score=5434 it/s peak=67 C power=176 W`. Step 4 did not begin: `the tune could not be applied for the fan tune: the undervolt was not applied: the point already runs more than the target -- card at stock`. The saved tune of before the test was put back and applied. Cause and fix: the decision "A flat top goes back on a card that is cooler than when it was found". Second run, with the `gao.exe` of the released v0.4.6: **pass, all five steps** (same day, started elevated by a script). Step 1: `stock: STABLE score=5309 it/s peak=66 C fan=70 % power=185 W`. Step 2: power 105 %, `mem: bandwidth peak +850, confirmed +850, applying +550`, `core: highest stable +210, confirmed +210, applying +135`, soak `STABLE score=5568 it/s`. Step 3: `target: 2850 MHz, which the card reaches at 1045 mV with the overclock`, `undervolt: lowest stable 975 mV, confirmed 985 mV, applying 1025 mV (+165 MHz at that point)`, soak `STABLE score=5363 it/s peak=66 C power=163 W`. Step 4: the tune went back on the card, `fan 60 %: 63 C`, `fan 55 %: 65 C`, `fan 52 %: 67 C`, `fan tune: 52 % holds 67 C`. Step 5: the final test ran its 300 s. `RESULT: power 105 %, core +135 MHz, mem +550 MHz, 2850 MHz at 1025 mV, fans 52 % at 67 C`; `stock: 5309 it/s, 185 W, fan 70 %, 66 C`, `now: 5353 it/s, 166 W, fan 52 %, 70 C`: `+0.8 % score and -10.3 % power against stock`; `undervolt: applied`. The saved tune of before the test was put back. **What this run did not show:** its undervolt had a raise of its own (+165 over an offset of +135), so the anchor was raised and not merely cut off, and the case of the first run (an anchor that reads a step above the target on a cooler card) did not come up. That fix has still only its tests. **The memory offset differs a lot from run to run on this card, and the two logs say why:** +1600 in the first run (bandwidth peak +2300), +550 in the second (peak +850), +650 in the run that saved the tune in use that morning (the journal shows its scan ending at +1000 too). The card was in the same state each time (68 C, power 105 %, a reading every three to four seconds). In both logged scans the bandwidth rises in a straight line, 0.36 and 0.46 % per 50 MHz step, from 420 GB/s at +0 to 509 GB/s at +2450; the first run went on to `WRONG RESULT` at +2550 without the bandwidth ever turning down. The readings scatter around that line with a standard deviation of 0.4 and 0.7 %, single ones by more than 1 %. The scan ends at a reading more than 1 % below the best so far when a second reading agrees, and in the second run that was chance: +950 read high (462.8 GB/s, where the line is at about 458), +1000 read 456.6 twice, 1.3 % below it. In the first run the one low reading (+600) was followed by a higher one, and the scan ran to the first wrong result. So on this card there is no bandwidth peak before the errors, and where the scan stops is decided by noise. | -| 95 | The memory scan no longer ends by chance | (a) Not elevated, nothing changed on the card: a measuring build takes 24 bandwidth readings in one process, with 3 s of the stress load before each as a scan has, at 64, 128 and 256 dispatches per timed run; then the same with the app's window open next to it, drawing a frame every 100 ms and every 33 ms. (b) Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize best` twice in a row with the build of the change, then the copy put back | (a) no, (b) yes | (a) how much a reading scatters, and on what that depends. (b) in both runs every `mem +N: bandwidth` line shows three readings that agree within about 1 %; the scan does not end before the first `WRONG RESULT`; the two `mem: bandwidth peak` values are within 300 MHz of each other and the applied offset is above +1000; the soak is not slower than at stock | **(a) measured** (2026-10-11, reference RTX 4070, driver 617.42, the saved tune applied: power 105 %, core +135, memory +650; standard deviation over the 24 readings of a series, and the lowest and highest against the mean). 64 dispatches, no window: 0.44 % (-0.55 to +1.07, one reading did not settle), 0.66 % (-1.58 to +1.49, one did not settle), 1.31 % (-4.83 to +1.27), 0.86 % (-1.44 to +2.05). 128 dispatches, no window: 0.43 % (-1.10 to +0.79). 256 dispatches, no window: 0.22 % (-0.43 to +0.42) and 0.60 % (-0.80 to +1.37; the last five readings of this series were 451 to 453 GB/s after nineteen of 444 to 449). With the window drawing every 100 ms: 64 dispatches 1.13 % (the first six readings 452 to 453 GB/s, the rest 439 to 451), 256 dispatches 0.24 % (-0.47 to +0.61). With the window drawing every 33 ms: 64 dispatches 0.58 %, 256 dispatches 0.25 % (-0.35 to +0.51). A reading took 6.5, 5.3 and 3.9 timed runs to settle at 64, 128 and 256 dispatches. So: the longer timed run takes the scatter from 0.4 to 1.3 % down to 0.2 %, the window's drawing adds nothing, and apart from the scatter the readings sit about 1.8 % higher for up to a minute at a time (the stretch at the end of the second 256 series ran on into the next series; the last 64 series went back and forth between 444 and 452 GB/s). What sets that level is not known. **(b) the offset is the same twice; the scan ends in a driver reset twice** (2026-10-11, same card and driver, a test build of 2764609, unattended, 40 minutes for the two runs). Run 1: `stock: STABLE score=5311 it/s peak=66 C fan=70 % power=185 W`; 49 steps from `mem +0: bandwidth 422.7 GB/s (422.7 422.6 422.7)` to `mem +2400: bandwidth 501.4 GB/s (501.3 502.3 501.4)`; at +2450 the candidate was `STABLE` and the measurement after it lost the device: `mem +2450: the bandwidth measurement failed; treated as a driver reset`, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, 20 s of rest, `health: STABLE`, `confirm core +0 / mem +2050: STABLE`, `mem: bandwidth peak +2250, confirmed +2050, applying +1400`; `core: highest stable +210, confirmed +210, applying +135`; soak `STABLE score=5536 it/s peak=69 C power=195 W`; `driver resets during this run: 1`. Run 2: stock 5312 it/s; 50 steps from 421.8 GB/s at +0 to 502.8 at +2450; the same failure at +2500; `mem: bandwidth peak +2300, confirmed +2100, applying +1450`; core +135; soak `STABLE score=5555 it/s peak=69 C power=196 W`; one driver reset. The three readings of a step differed by 0.4 % on average and by 1.2 % at most; no step was more than 0.7 % below the best before it, so neither scan came near ending early, by the old rule or the new; every step had its three readings. The scan took 5 min 25 s (about 6.5 s a step; about 3 min before the change). As expected: the two peaks are 50 MHz apart, the applied offsets are +1400 and +1450, and both results are 4 % faster than stock. **Not as expected: neither scan reached a `WRONG RESULT`.** Both ended one and two steps before +2550 with the device lost during the bandwidth copy; Windows logged it both times (System log, `nvlddmkm` events 14 and 153 at 19:51:22 and 20:11:27), and the screen goes black for a moment. In the run of check 94 that got this far, the measurements at +2450 and +2500 gave a number and the stress load reported the first wrong result at +2550, without a reset. A step now copies about seven times as much as it did (three readings of about four runs of 128 GB, against one reading of about six runs of 32 GB), and the copy checks nothing, so where the memory starts to fail it is the driver that gives out, not a verdict. Two runs against one: likely the cause, not proven. The search handles it as it should (rest, health probe, confirm four steps lower, the core is still searched), but a driver reset in every run on a card like this is not what the change was for. The saved tune of before the test was put back and applied (`applied best: power 105 %, core +135 MHz, mem +650 MHz`), the tray app restarted, Defender reported nothing. **(c) the same two runs with one reading a step, three only for a step in doubt** (2026-10-11 20:26 to 21:03, a test build of 71e7c2b, unattended): **the reset is still there.** Run 1: stock 5328 it/s; 50 steps from `mem +0: bandwidth 420.4 GB/s` to `mem +2450: bandwidth 502.4 GB/s`; at +2500 the candidate was `STABLE` and then `bandwidth measurement failed: the timing of the copy did not come back` ten seconds later (`nvlddmkm` event 14 at 20:37:07); `mem: bandwidth peak +2200, confirmed +2000, applying +1400`; core +135; soak `STABLE score=5572 it/s peak=70 C power=197 W`; one driver reset. Run 2: stock 5311 it/s; 421.9 GB/s at +0 to 508.0 at +2450; at +2500 `the device was lost during the copy failed (hr=0x887A0006)`, the device hung (event 14 at 20:55:29); `mem: bandwidth peak +2400, confirmed +2200, applying +1500`; core +135; soak `STABLE score=5571 it/s peak=70 C power=196 W`; one driver reset. No step was in doubt in either run (the lowest was 0.75 % below the best before it), so every step was read once; the scan took 3 min 33 s. The two peaks are 200 MHz apart: with one reading a step, which offsets count as within 1 % of the best moves by a few steps. Applied +1400 and +1500, both 5 % faster than stock. So the reset does not come from reading a step three times. Four runs out of four with the longer timed run ended this way, at +2450 or +2500. Not merged; the design for a scan that does not end like this is the open row in `docs/development.md`. The saved tune was put back and applied again (core +135, memory +650), the tray app restarted, Defender reported nothing. **(d) the edge is remembered** (a test build of 7c215c4; the same two runs): expected is that the first scan ends as before and logs nothing new, the journal gets one `mem_edge` line with the GPU id and the driver version, and the second run logs `mem: the memory gave out at +N in an earlier run on this card and driver; the scan stops at +(N-200)`, has no `driver resets during this run` line, applies more than +1000 and is not slower than stock. **Not run yet.** **(e) how the stress load meets the edge** (the same build, no search): `gao --reset`, then `gao --set-mem` and `gao --stress 3` at +2300, +2350, ... until the verdict is not `STABLE`, five times, at stock power and core +0; expected is one of two outcomes, `WRONG RESULT` every time (then the stress load can lead the copy to the edge) or `DEVICE LOST` (then it cannot). **Not run yet.** | +| 95 | The memory scan no longer ends by chance | (a) Not elevated, nothing changed on the card: a measuring build takes 24 bandwidth readings in one process, with 3 s of the stress load before each as a scan has, at 64, 128 and 256 dispatches per timed run; then the same with the app's window open next to it, drawing a frame every 100 ms and every 33 ms. (b) Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize best` twice in a row with the build of the change, then the copy put back | (a) no, (b) yes | (a) how much a reading scatters, and on what that depends. (b) in both runs every `mem +N: bandwidth` line shows three readings that agree within about 1 %; the scan does not end before the first `WRONG RESULT`; the two `mem: bandwidth peak` values are within 300 MHz of each other and the applied offset is above +1000; the soak is not slower than at stock | **(a) measured** (2026-10-11, reference RTX 4070, driver 617.42, the saved tune applied: power 105 %, core +135, memory +650; standard deviation over the 24 readings of a series, and the lowest and highest against the mean). 64 dispatches, no window: 0.44 % (-0.55 to +1.07, one reading did not settle), 0.66 % (-1.58 to +1.49, one did not settle), 1.31 % (-4.83 to +1.27), 0.86 % (-1.44 to +2.05). 128 dispatches, no window: 0.43 % (-1.10 to +0.79). 256 dispatches, no window: 0.22 % (-0.43 to +0.42) and 0.60 % (-0.80 to +1.37; the last five readings of this series were 451 to 453 GB/s after nineteen of 444 to 449). With the window drawing every 100 ms: 64 dispatches 1.13 % (the first six readings 452 to 453 GB/s, the rest 439 to 451), 256 dispatches 0.24 % (-0.47 to +0.61). With the window drawing every 33 ms: 64 dispatches 0.58 %, 256 dispatches 0.25 % (-0.35 to +0.51). A reading took 6.5, 5.3 and 3.9 timed runs to settle at 64, 128 and 256 dispatches. So: the longer timed run takes the scatter from 0.4 to 1.3 % down to 0.2 %, the window's drawing adds nothing, and apart from the scatter the readings sit about 1.8 % higher for up to a minute at a time (the stretch at the end of the second 256 series ran on into the next series; the last 64 series went back and forth between 444 and 452 GB/s). What sets that level is not known. **(b) the offset is the same twice; the scan ends in a driver reset twice** (2026-10-11, same card and driver, a test build of 2764609, unattended, 40 minutes for the two runs). Run 1: `stock: STABLE score=5311 it/s peak=66 C fan=70 % power=185 W`; 49 steps from `mem +0: bandwidth 422.7 GB/s (422.7 422.6 422.7)` to `mem +2400: bandwidth 501.4 GB/s (501.3 502.3 501.4)`; at +2450 the candidate was `STABLE` and the measurement after it lost the device: `mem +2450: the bandwidth measurement failed; treated as a driver reset`, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, 20 s of rest, `health: STABLE`, `confirm core +0 / mem +2050: STABLE`, `mem: bandwidth peak +2250, confirmed +2050, applying +1400`; `core: highest stable +210, confirmed +210, applying +135`; soak `STABLE score=5536 it/s peak=69 C power=195 W`; `driver resets during this run: 1`. Run 2: stock 5312 it/s; 50 steps from 421.8 GB/s at +0 to 502.8 at +2450; the same failure at +2500; `mem: bandwidth peak +2300, confirmed +2100, applying +1450`; core +135; soak `STABLE score=5555 it/s peak=69 C power=196 W`; one driver reset. The three readings of a step differed by 0.4 % on average and by 1.2 % at most; no step was more than 0.7 % below the best before it, so neither scan came near ending early, by the old rule or the new; every step had its three readings. The scan took 5 min 25 s (about 6.5 s a step; about 3 min before the change). As expected: the two peaks are 50 MHz apart, the applied offsets are +1400 and +1450, and both results are 4 % faster than stock. **Not as expected: neither scan reached a `WRONG RESULT`.** Both ended one and two steps before +2550 with the device lost during the bandwidth copy; Windows logged it both times (System log, `nvlddmkm` events 14 and 153 at 19:51:22 and 20:11:27), and the screen goes black for a moment. In the run of check 94 that got this far, the measurements at +2450 and +2500 gave a number and the stress load reported the first wrong result at +2550, without a reset. A step now copies about seven times as much as it did (three readings of about four runs of 128 GB, against one reading of about six runs of 32 GB), and the copy checks nothing, so where the memory starts to fail it is the driver that gives out, not a verdict. Two runs against one: likely the cause, not proven. The search handles it as it should (rest, health probe, confirm four steps lower, the core is still searched), but a driver reset in every run on a card like this is not what the change was for. The saved tune of before the test was put back and applied (`applied best: power 105 %, core +135 MHz, mem +650 MHz`), the tray app restarted, Defender reported nothing. **(c) the same two runs with one reading a step, three only for a step in doubt** (2026-10-11 20:26 to 21:03, a test build of 71e7c2b, unattended): **the reset is still there.** Run 1: stock 5328 it/s; 50 steps from `mem +0: bandwidth 420.4 GB/s` to `mem +2450: bandwidth 502.4 GB/s`; at +2500 the candidate was `STABLE` and then `bandwidth measurement failed: the timing of the copy did not come back` ten seconds later (`nvlddmkm` event 14 at 20:37:07); `mem: bandwidth peak +2200, confirmed +2000, applying +1400`; core +135; soak `STABLE score=5572 it/s peak=70 C power=197 W`; one driver reset. Run 2: stock 5311 it/s; 421.9 GB/s at +0 to 508.0 at +2450; at +2500 `the device was lost during the copy failed (hr=0x887A0006)`, the device hung (event 14 at 20:55:29); `mem: bandwidth peak +2400, confirmed +2200, applying +1500`; core +135; soak `STABLE score=5571 it/s peak=70 C power=196 W`; one driver reset. No step was in doubt in either run (the lowest was 0.75 % below the best before it), so every step was read once; the scan took 3 min 33 s. The two peaks are 200 MHz apart: with one reading a step, which offsets count as within 1 % of the best moves by a few steps. Applied +1400 and +1500, both 5 % faster than stock. So the reset does not come from reading a step three times. Four runs out of four with the longer timed run ended this way, at +2450 or +2500. Not merged; the design for a scan that does not end like this is the open row in `docs/development.md`. The saved tune was put back and applied again (core +135, memory +650), the tray app restarted, Defender reported nothing. **(d) the edge is remembered** (a test build of 7c215c4; the same two runs): expected is that the first scan ends as before and logs nothing new, the journal gets one `mem_edge` line with the GPU id and the driver version, and the second run logs `mem: the memory gave out at +N in an earlier run on this card and driver; the scan stops at +(N-200)`, has no `driver resets during this run` line, applies more than +1000 and is not slower than stock. **Run 1 took the machine down; the test ended there** (2026-10-11 21:23, unattended). Stock 5321 it/s, power 105 %, then the scan: 49 steps from +0 to `mem +2400: bandwidth 539.3 GB/s`, two of them read three times (+1200 and +1250). At +2450 the candidate was `STABLE` and the copy after it failed as in the four runs before: `bandwidth measurement failed: the copy failed (hr=0x887A0007): the device was lost` at 21:35:05, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, `the driver was reset; reconnecting`. That is the last line. Windows restarted by itself within fifteen seconds: Kernel-Power event 41 at 21:35:23, `The previous system shutdown at 21:34:57 was unexpected`, no bugcheck, no dump, no `nvlddmkm` event written. So the fifth scan that met the edge with the longer timed run did not end in a driver reset the search recovers from, but in a reset of the whole machine, during the reconnect or right after it. What held: the candidate's journal entry was closed, so no ceiling of a freeze was left; the line `{"mem_edge":2450,"rig":"GPU-... / 617.42"}` was on disk before the machine went down, so no later run of this build goes there again; the run had saved nothing, and at logon the tray app applied the tune of before the test (`applied best: power 105 %, core +135 MHz, mem +650 MHz`). Not answered: the second run. Seen along the way and not explained: from the first step of the scan the stress load scored 5760 it/s and the bandwidth read about 7 % higher than in every run before (486.6 GB/s at +900 against about 455), while the measurement at stock five minutes earlier was the usual 5321. **(e) how the stress load meets the edge** (the same build, no search): `gao --reset`, then `gao --set-mem` and `gao --stress 3` at +2300, +2350, ... until the verdict is not `STABLE`, five times, at stock power and core +0; expected is one of two outcomes, `WRONG RESULT` every time (then the stress load can lead the copy to the edge) or `DEVICE LOST` (then it cannot). **Not run:** the test ended in (d). It goes to the edge on purpose, five times, and after (d) that is a risk for the machine that needs its owner's word first. | ## Notes From 988a22a0fc047b8b210453b4a7c2e385ae4323ea Mon Sep 17 00:00:00 2001 From: Rovey Date: Sun, 11 Oct 2026 21:51:44 +0200 Subject: [PATCH 08/15] fix: the memory search never takes the long copy near the memory's edge The long bandwidth copy lost the device at the memory's edge in five scans out of five on the reference card, and the fifth time the machine restarted (hardware check 95). The short copy got through there the one time it was taken that far. The search now has two passes. The scan goes up with the short copy and only finds where it ends: a step that is not stable, or one more than 5 % below the best, where the best is the highest middle of three steps in a row and a reading that would end the scan is taken three times. Then the long copy reads the offset to keep, starting four steps below the highest step the scan passed and going down until a reading is more than 1 % below the best of them. A scan held below an edge the journal remembers counts that margin from the edge, so the run after gives the same offset as the run that found it. Tested on a simulated card with the reference card's numbers; not run on a card. Co-Authored-By: Claude Opus 5.5 --- README.md | 2 +- docs/development.md | 4 +- docs/hardware-checks.md | 2 +- src/app/common.cpp | 6 +- src/core/search.cpp | 85 ++++++-- src/core/search.hpp | 85 +++++--- src/hw/stress.cpp | 23 ++- src/hw/stress.hpp | 7 +- tests/test_search.cpp | 431 +++++++++++++++++++++++++++++++--------- 9 files changed, 494 insertions(+), 151 deletions(-) diff --git a/README.md b/README.md index 2c019b8..713bd35 100644 --- a/README.md +++ b/README.md @@ -177,7 +177,7 @@ flowchart LR - **Every run starts with the card at stock for five minutes,** as long as the result is tested at the end, with the fans left to the NVIDIA driver. That is the "before" of the result: a card is not heated through after 30 s, so a shorter run would make every result look warmer and louder than stock. Some cards get hot there: the driver's own fan control lets them (a 3080 Ti reached 85 °C with its fans at 65 %). The measurement stops at 85 °C, what it measured until then is the "before", and the run goes on. From there every step keeps the fans on a curve that reaches 100 % at the profile's temperature limit. - **Stress test.** A DirectX 11 compute load in which the GPU checks every value it computes against a known answer. Each probe ends in a verdict: `STABLE`, `WRONG RESULT`, `DEVICE LOST`, `STALLED` (the values were right but the card computed almost nothing), `TOO HOT` or `NO TELEMETRY`. -- **Memory comes first and stops at the bandwidth peak,** not at the first error. GDDR6 and GDDR6X retry failed transfers, so an overclocked memory bus loses speed long before it returns a wrong result. The search raises the offset 50 MHz at a time, measures bandwidth at each step (three times, taking the middle reading, where a step looks like a drop) and keeps the lowest offset within 1 % of the best. Where bandwidth cannot be measured, the memory is left at stock: without that measurement nothing says where to stop, and the first wrong result comes far too late. +- **Memory comes first and stops at the bandwidth peak,** not at the first error. GDDR6 and GDDR6X retry failed transfers, so an overclocked memory bus loses speed long before it returns a wrong result. The search raises the offset 50 MHz at a time, measures bandwidth at each step until a step fails or has clearly lost bandwidth, then measures again more precisely from 200 MHz below that point down, and keeps the lowest offset within 1 % of the best. The precise measurement is heavy, and memory at its very edge does not survive it; that is why it stays below. Where bandwidth cannot be measured, the memory is left at stock: without that measurement nothing says where to stop, and the first wrong result comes far too late. - **The core search climbs from stock.** With the chosen memory offset applied, it raises the core offset 15 MHz at a time until a probe fails, and keeps the last value that passed. How far it may go comes from the range the driver reports for your card, so a card that can hold more is not stopped at a fixed limit. - **Each edge is confirmed.** The value a search ends on gets a 30 s probe, and a lower value is tried if it does not hold. The safety margin is applied to the confirmed value, and the result as a whole must pass the 300 s soak. - **A driver reset ends that clock's search.** On a card the search can push past its edge, a search resets the driver: the screen goes black for a moment. The run then reconnects to the driver, sets the card to stock, leaves it without load for 20 seconds and checks with a short probe at stock that it computes normally again. After that the search tries no new values for that clock: it keeps the last value that passed and confirms four steps below it. Each clock has its own search, so a reset in the memory search does not keep the core from being searched; a run can therefore reset the driver twice, once per clock. One reset more than that ends the run at stock with nothing saved, and so does a card that does not come back. Results on real cards are in [docs/hardware-checks.md](docs/hardware-checks.md), checks 51 and 52. diff --git a/docs/development.md b/docs/development.md index a819c99..8b3069d 100644 --- a/docs/development.md +++ b/docs/development.md @@ -79,8 +79,8 @@ The applied offset is always at least one step below the confirmed edge. Profile | Without the bandwidth measurement the memory is left at stock | The search used to fall back to a climb on stability alone when the measurement failed at +0. On a tester's 3080 Ti (0.4.3) that climb went to +2300 MHz with the score falling from +2150 on, got its first wrong result there, and the machine froze with a corrupted screen in the 30 s confirm probes just below it. The first wrong result is no edge to stand next to: the memory was correcting errors for hundreds of MHz before it. Now a measurement that does not work means no memory overclock (`left_at_stock`), the run says so, and the core is still searched. `optimize` keeps the stability climb only for a caller that gives no measurement at all; the app always gives one. Why the measurement failed on that card is not known: the log was only in the window and went with the reboot. | | A confirm probe that is too hot at once confirms nothing | A confirm probe that ends `TOO HOT` keeps the clock edge: over 30 s the card may run a little hotter than over the power step's 20 s, and that is the soak's business, which lowers power. But the stress run stops at its first reading above the limit, so a card that is above it when the probe begins gives `TOO HOT` within a second, and that counted as holding too. It happened on the tester's RTX 3080 Ti (hardware check 91): after a driver reset the health probe loads the card at stock power, which on that card means 76 C against a limit of 75, and `confirm core +225` was confirmed by a probe that had tested nothing. Now such a probe counts only when the clock was loaded for `kConfirmMinS` (10 s, clearly longer than the 3 s probe that found the edge) before it got too hot. Otherwise the value is not confirmed, and the card rests (`io.rest`, the 20 s of a recovery, with the fans where the curve last put them) before the next try one step lower: without the rest every lower value would start just as hot, and heat alone would step the edge down to nothing, which is what keeping the edge on `TOO HOT` was for. The same value is not tried again; one step is the price. Not changed: the health probe still runs at stock power, and a soak that begins too hot still lowers the power limit by a step. | | A profile that promises speed keeps no result that is slower than stock | Best of my GPU ended 3.8 % slower than stock on the tester's RTX 3080 Ti (4749 against 4937 it/s, hardware check 91), with the fans at 80 % instead of 66 %, and showed it as **Done**, applied and kept at every logon. Every step had done what it is for: the card runs 83 C at stock with its fans at 100 %, the profile allows 75, so the power limit went to 65 %, and +150 MHz of core clock did not win that back. But a profile called "the best of my GPU" that leaves the card slower has not done what it says, and a result like that is not offered as one. After the soak, `optimize` compares its score with the measurement at stock that went before the run (the "before" the result screen shows, so the rule judges by the number the user sees): below `kNotSlowerScore` (0.99: one percent is what two five-minute runs differ by) the run ends as a failed one does, at stock, with nothing saved, and says what it found, why, and that Undervolt is the profile for a card that runs warm. `Objectives::faster_than_stock` marks the two profiles this holds for, Best of my GPU and Max performance. Not Quiet and Cool & efficient, which trade speed for noise and heat and say so. Not the overclock step of All in one either: it goes on to an undervolt and a fan tune, and its result is judged as a whole; whether that whole may be slower than stock is not decided. This is a backstop, not the cure: what such a card needs from these profiles is in `docs/planned.md` ("never hotter than at stock"). | -| The memory scan times a longer copy, reads a step in doubt three times, and stops at a 3 % drop | Four runs on one day applied +650, +1600, +550 and +100 MHz on the reference RTX 4070, a card that runs every game at +1500 (hardware check 94). That card has no bandwidth peak: bandwidth rises by about 0.4 % per 50 MHz step all the way to +2500 and the first wrong result comes at +2550, so the scan could only end early by chance, and where it did decided the offset. Measured afterwards without a search and with nothing changed on the card (hardware check 95 a), the readings move in three ways, and each got its own answer. **(1) One timed run was too short.** A run copied 32 GB, 75 ms on this card, and a reading is the median of three runs that agree within 1 %. Such readings scattered by 0.4 to 1.3 % (standard deviation over 24 readings, six series), single ones by up to 4.8 %, and in two of the six series one reading did not settle. With 128 GB per run (`kBwDispatches` 256, 0.3 s) it is 0.2 %, and all 96 readings settled. **(2) A single reading decided the end.** A step that is more than `kBandwidthTie` (1 %) below the best is in doubt: it is sampled `kBandwidthReadings` (3) times and its middle reading counts, which replaces the single re-measure of a drop. A reading that only did not settle is taken again, three times at most; any other failure ends the step at once, as before. Three readings at every step were built first and taken back: see the row below. **(3) The readings change level.** For up to a minute at a time every reading is about 1.8 % higher than before and after (452 against 444 GB/s), at one and the same offset. The cause is not known. Reading a step three times does nothing against it: all three are inside the stretch. So the drop that ends the scan (`kBandwidthDrop`) went from 1 % to 3 %: above what the readings do by themselves, below what a real peak loses within a step or two. The price on a card that has a peak: the scan goes a step or two further past it, in 3 s probes, before it stops. The offset it keeps is still the lowest within 1 % of the best, and confirming starts there. **Not built: a window that stands still during a measurement.** The suspicion was that the window's own drawing disturbed the readings of the run that saved +100. Measured with a window drawing a frame every 100 ms and every 33 ms next to the measurement: 0.2 %, the same as without. **On the card (check 95 b and c):** four runs applied +1400, +1450, +1400 and +1500, each 4 to 5 % faster than stock; no step came within reach of ending a scan early (the lowest was 0.75 % below the best before it), and no step was in doubt. A scan of 50 steps takes 3 min 33 s, half a minute more than before. **What it does not solve** is in the row below, and keeps the change from being merged. A change of level larger than 3 % would end a scan by chance again: the answer to that is to compare a low step with a fresh reading of the best offset taken right after it, not a still higher threshold. | -| Open: on a card without a bandwidth peak the memory scan ends in a driver reset | With the change of the row above the scan on the reference card no longer ends by chance, so it always runs to where the memory gives out. In four runs out of four that was not a wrong result from the stress load but the bandwidth copy losing the device at +2450 or +2500 (`DXGI_ERROR_DEVICE_HUNG`, or the timing of the copy not coming back within its 5 s; `nvlddmkm` events 14 and 153 in the System log; the screen goes black for a moment), with three readings a step and with one (hardware check 95 b and c). So it is not how often a step is read. In the one run before the change that got this far (check 94), the copies at +2450 and +2500 gave a number and the stress load reported the first wrong result at +2550. What differs is the timed run: 128 GB in one go against 32. One run against four is thin, and a memory edge moves by a step from run to run; it is the likely cause, not a proven one. The search handles the reset as it should (rest, health probe, confirm four steps lower, the core still searched) and the result is good. But a reset as the normal end of a scan is not acceptable in a tool that is to be simple and stable. **Design, not built.** A card like this has nothing that announces its edge: the bandwidth rises until the memory fails. No stop rule can know where that is without having been there, so the design does not try to. It makes the meeting happen once, and with the probe that can report it. **(a) The edge is found once and remembered: built.** The journal already turns a candidate that never finished (the machine froze) into a ceiling for every later run. A memory scan that ended in a wrong result or a reset event (a driver reset, a lost device) now writes that offset to the journal as well, in a line of its own (`{"mem_edge":2500,"rig":" / "}`; it has no id, so a build that does not know it skips it). A later scan on the same card and driver stops `kResetBackoffSteps` (4) steps below the lowest such offset, which is where confirming starts after a reset, and says so: `mem: the memory gave out at +2500 in an earlier run on this card and driver; the scan stops at +2300`. With another driver version or another card the line does not count; back on the driver it was found with, it counts again. Without a GPU id or a driver version nothing is kept. A scan that ended for another reason (a bandwidth peak, readings that did not settle, a measurement that does not work, the driver's own range) writes nothing. It errs to the safe side: an edge found on a bad day stays until the driver changes or the journal is removed. It leaves the first run as it is. On the card (hardware check 95 d): the first run of its test met the edge at +2450 like the four before it, and this time the whole machine restarted, seconds after the device was lost. The edge was on disk by then, which is what this part is for. But it changes what the open point is: the first meeting with the edge is not a moment of black screen that the search recovers from, it can cost the machine. "Once per card" is not good enough for that, and until the scan no longer reaches the edge with the copy, this branch is less safe than what is released, where the short copy got there once in four runs and ended in a wrong result. **(b) The stress load meets the edge before the copy does.** The stress load checks its results and says `WRONG RESULT`; the copy checks nothing and can only hang. So the copy is taken only at offsets the stress load has already passed four steps higher: per step a 3 s probe at +200 MHz above it, then back and the timed copy. On the reference card the load would report the wrong result at +2550 while the highest copy taken is at +2350, where all five scans measured a number. The price: about a second more per step, and on a card that has a peak the probes run four steps beyond the point where the scan would stop. That is 3 s probes, as the 3 % rule already allows a step or two; it is not the climb on stability alone that took a 3080 Ti to +2300, because the copy still ends the scan at the peak. Before (b) is built it has to be measured that the stress load at the edge ends in a wrong result and not in a reset of its own: one observation says so (check 94), core candidates say so in every run, and that is not enough for memory. **Looked at and not proposed.** A fixed top for the scan (an offset, or a share of the driver's range): there is nothing to choose the number by, and a card whose memory gives out below it meets its edge all the same. A top at a bandwidth gain over +0 (say 15 %): the same objection, most cards will fail first. Going back to the short timed run: it brings back the scatter that made the scan end by chance, on the strength of one run. **Order:** (a) first; the measurement for (b) next (hardware check 95 e): the stress load alone, with no search, in 3 s runs from +2300 up until it fails, five times. | +| The memory search has two passes: a scan with the short copy finds where it ends, and the long copy reads the offset to keep, four steps below that | **What went wrong.** Four runs on one day applied +650, +1600, +550 and +100 MHz on the reference RTX 4070, a card that runs every game at +1500 (hardware check 94). That card has no bandwidth peak: bandwidth rises by about 0.4 % per 50 MHz step all the way to +2500 and the stress load computes its first wrong result at +2550, so the scan could only end early by chance, and where it did decided the offset. **What was measured** (hardware check 95 a, no search, nothing changed on the card). A reading is the median of three timed runs that agree within 1 %. With the timed run as it was, 32 GB or 75 ms, readings scatter by 0.4 to 1.3 % (standard deviation over 24 readings, six series), single ones by up to 4.8 %, and now and then one does not settle. With 128 GB per run it is 0.2 %. Apart from that the readings change level: for up to a minute every reading is about 1.8 % higher, at one and the same offset; the cause is not known. The app's own window, drawing every 100 or 33 ms next to the measurement, adds nothing, so nothing was built for it. **What was tried first, and what it did** (check 95 b to d). The long timed run at every step of the scan: the offset repeated (+1400 to +1500 in four runs, 4 to 5 % faster than stock), and every scan now ran to where the memory gives out and met it with the long copy. Five scans out of five lost the device there, at +2450 or +2500, a driver reset each time, with three readings a step and with one; and the fifth took the whole machine down within seconds (Kernel-Power 41, no bugcheck). In the one scan before all this that got to the edge, the short copy read a number at +2450 and +2500 and the stress load reported the wrong result at +2550, with no reset. One against five is thin, and it is all there is: the long copy is too heavy for memory at its edge, the short one was not. **The design.** One measurement cannot do both jobs, so there are two passes. *The scan* goes up as before and only has to find where it ends: a step that is not stable, or one that has lost more than `kBandwidthDrop` (5 %) against the best. It reads with the short copy (`kBwDispatches` 64), which is what the released versions take to the edge. Because those readings scatter, two things keep one of them from deciding: a reading that would end the scan is taken `kBandwidthReadings` (3) times and the middle one counts, and the best that readings are held against is the highest middle of three steps in a row, not the highest reading (the highest of fifty scattering readings is always one that came out high). 5 % is far more than the readings move by themselves; the scan is not where the offset is chosen any more, so it can afford to be coarse. *The second pass* chooses the offset, with the long copy (`MeasureBandwidth(true)`, `OptimizeIo::bandwidth_precise`): it starts `kPreciseMarginSteps` (4 steps, 200 MHz) below the highest step the scan passed (`precise_top`), goes down a step at a time until a reading is more than `kBandwidthTie` (1 %) below the best of them, and keeps the lowest offset within that 1 % (`bandwidth_peak_below`). On the reference card: the scan passes +2500, the long copy reads +2300, +2250, +2200 and +2150, and +2200 is kept; confirmed there, 70 % applied, +1500. The long copy lost the device at the highest step the stress load passed and one below it; four steps leave two to spare. The margin is kept whatever ended the scan, a drop and the driver's own range included: what is above was never tried. After a reset event in the scan the second pass is skipped, and confirming starts four steps below where it would have begun. **The edge is met once** (built before this, kept): a scan that ended in a wrong result or a reset event writes that offset to the journal with the GPU id and the driver version (`{"mem_edge":2550,"rig":" / "}`, a line without an id, so older builds skip it). A later scan on the same card and driver stops four steps below the lowest such offset, and its second pass starts where the first run's did: the margin counts from the edge, not a second time from where the scan was held. With another driver or card the line does not count. **What it costs.** A card that does have a peak keeps an offset up to 200 MHz below it when the scan ends close above the peak: the margin does not know a drop from an edge. A scan that ends for a harmless reason (readings that did not settle) loses those 200 MHz as well. The second pass adds about four steps, half a minute. **Not chosen.** The long copy everywhere: see above. The short copy everywhere, as released: the offset is chance (the +100 run). Three long readings a step, or only for a step in doubt: the reset came with both. Only remembering the edge: the first meeting can cost the machine, and once per card is once too often. The stress load probing four steps ahead of the copy: it needs the proof that the load at the edge ends in a wrong result and not in a reset of its own, and getting that proof means going to the edge on purpose. A fixed top for the scan (an offset, a share of the range, a gain over +0): there is nothing to choose the number by. **Tested** on a simulated card with the reference card's numbers (wrong result at +2550, the long copy losing the device from +2450): the long copy is never taken above +2300, in the first run and in the one after it, with short readings scattering by 1.3 % on two hundred seeds, and on forty more with level changes of 1.8 % thrown in; with the edge anywhere from +300 to +2900; and the tests fail when the margin is set to nothing. **Not tested on the card.** See the row below. | +| Open: the two-pass memory search has not run on a card | The last card test (hardware check 95 d) ended with the machine restarting, and nothing has been run on the card since. Before this is merged, its owner decides whether and when to run it there. What a run would show, and what it cannot: the journal of the reference card holds the edge at +2450 now, so a run of this build stops its scan at +2250 and reads with the long copy from +2200 down, at offsets every earlier scan passed. That tests the second pass and the result, with little risk. It does not test the first meeting with the edge with the short copy: that is what the released versions do, it went well the one time it was seen, and seeing it again means removing the edge from the journal and going there. **Still not known.** Why the long copy fails where the short one did not: the amount copied in one go, or only that five runs are more than one. Whether a card with a real bandwidth peak behaves as assumed; none has been measured through a scan (the tester's RTX 3080 Ti never got a reading to settle at +0). And one observation from the run that crashed: from the first step of its scan the stress load scored 5760 it/s and the bandwidth read about 7 % higher than in every run before (486.6 GB/s at +900 against about 455), while the measurement at stock five minutes earlier was the usual 5321. The suspicion, not checked: the monitor went to sleep and the desktop stopped being drawn. If that is it, the comparison of a result with stock depends on whether the screen is on, and the level changes of 1.8 % may be the same thing in small. | | Open: why the bandwidth measurement fails on some cards, and what then | On the tester's RTX 3080 Ti it failed at +0, and since 0.4.4 that means no memory overclock at all on such a card, though it computed right far beyond +2000 MHz. The measurement gives a number only when three runs in a row agree within 1 % (`kBwSettle`), and it throws away every run the driver's timer calls disjoint (the clock changed under it). That card was held at a power limit of 65 % and ran at 83 C when it was measured; the reference card has only ever been measured at 105 %, where the measurement itself draws 179 of 210 W. So the suspicion is the power limit, and hardware check 90 tests exactly that on the reference card. What it shows decides what is built, and nothing is built before it: **(1) runs are thrown away as disjoint, and the ones that are kept agree:** the timer is what fails, not the card. Time the copy differently (wall clock around a flush, or count a disjoint run when its neighbours agree) and keep the method. **(2) the readings themselves scatter by more than 1 %:** a peak that is found by a 1 % drop (`kBandwidthDrop`) cannot be found in that noise. Then the memory has to be measured where the card is not held back: search it before the power step, at stock power, where a probe and its measurement last five seconds; or take more readings and compare medians. **(3) it does not fail at 65 % on the reference card:** the power limit is not the cause. Then it was the heat (a card throttling at its temperature limit changes clock as well), or another program, and only the tester's `run.log` can say: since this change a measurement that never settled writes how many runs gave a reading, between which values, and how many were thrown away. In no case does the climb on stability alone come back. **Outcome of check 90: (3).** On the reference card the measurement gave a number fifteen times out of fifteen at 100, 80 and 65 %, in three or four runs, with nothing thrown away, and the bandwidth was the same at every limit. The power limit is ruled out; (1) and (2) are not built. What is left is the heat of that card, another program, or a timer query that failed, and the tester's `run.log` is the next thing that can tell them apart. **The tester's run (check 91): it is (2), but only inside a run.** Twenty readings between 763.9 and 821.2 GB/s, none thrown away, three seconds after the load began; after thirty seconds of load the same card read within 0.2 %. The suspicion now is the moment, not the card: the measurement is taken while the power limiter is still catching up. If `gao --bandwidth --after-load 3` scatters on that card and `--after-load 30` does not, that is it, and the fix is to measure when the card has settled (keep taking runs for some seconds instead of twenty runs), not to change the method. | | Every line of a run is also in `run.log` | `logged()` wraps the log hook of both front ends: a line is appended and flushed before it is shown, so the step a machine freezes in is the last line on disk. One file in the state folder, moved aside at 1 MB (`run.log.old`). Only elevated and only after `ensure_app_dir` has verified the folder: an elevated process appends to nothing a user could have put there. | | A run does not start while the NVIDIA app's Automatic Tuning is on | Switched on, a service of the driver scans the card when it sees fit: it loads it, changes its clocks, and starts again by itself (hardware check 89: a scan that was stopped started again 14 ms later, and after a restart of the service it resumed within two seconds). A run next to that measures nothing it can trust, so it is refused, like a run next to Afterburner, and with the same weight: a courtesy, not a guard. Read from where that service keeps the switch, `%ProgramData%\NVIDIA Corporation\nvtopps\nvtopps.db3`, table `ocScannerSettings`, fields `backgroundScanEnabled` and `grdOcEnabled`, joined to `TelemetryGpuCfg` so that only a card that is in the PC counts (a row of an earlier card can never be switched off again). That this is the source and not a copy of it was measured, not assumed: with the field set the service took the scan up again after every restart, with it cleared it did not, while the NVIDIA app itself showed the switch as off. Nothing documents the file. So: opened read-only and immutable (no lock, the service never waits for us), through `winsqlite3.dll` from System32 (no dependency added), and everything that is not exactly as expected reads as "unknown", which starts the run as before. The registry value the app sets next to it (`HKLM\SOFTWARE\NVIDIA Corporation\Global\NvApp\OC\`) changes at the same moment but belongs to the app, not to the service, and is not used. **The overlay is not checked:** no setting for it was found, its processes run whether it is open or not, and the one value that moved (`IsShadowPlayEnabled`) followed the service being up, not a switch. NVAPI was not tried: an undocumented call is no better than an undocumented file, and it cannot be compared with anything. | diff --git a/docs/hardware-checks.md b/docs/hardware-checks.md index 21ee509..1c11ed7 100644 --- a/docs/hardware-checks.md +++ b/docs/hardware-checks.md @@ -122,7 +122,7 @@ root; adjust the path for a Debug build. | 92 | The undervolt profile starts with the measurement at stock every run uses, and a hot card is not refused | Elevated, the tray app exited, a copy of `gao.json` kept. (a) `gao --undervolt` to the end; then `gao --status`. (b) The same with a test build whose stock limit is lowered to a temperature the reference card passes under load (63 C; it reaches 67). Put the copy of `gao.json` back. (c) Only on a card that is above the profile's limit at stock with the fans at 100 % (the tester's RTX 3080 Ti: 82 to 83 C): the **Undervolt** profile from the window, to the end, and its `run.log` | yes | (a) `the card at stock, five minutes`, a `stock: STABLE ...` line with the driver's fan speed, then `baseline: the measurement at stock`, and no second five minutes at stock: the first candidate follows at once; no `candidates are stopped at` line; `RESULT:`; `before:` shows 300 s and the fan speed the driver chose. (b) `stock: TOO HOT ... peak=64 C`, the line that the measurement stopped there, the cool-down, then `baseline: the measurement at stock` and a search that runs to `RESULT:` with `before:` showing the shortened time; no `candidates are stopped at` line, because 64 C is under the profile's 75. (c) no `not stable at stock`; `the card reached 86 C at stock: candidates are stopped at 85 C, not at the profile's 75 C`; candidates that end `STABLE` or `WRONG RESULT`, none `TOO HOT`; a result that is cooler and draws less than stock at the same clock | First run: **not as expected**, and that found a fault (2026-10-11, reference RTX 4070, driver 617.42, a test build of d1c99e9, started elevated by a script). The first halves held: (a) `stock: STABLE ... peak=65 C fan=70 %`, then `baseline: the measurement at stock` four seconds later and no second five minutes; (b) `stock: TOO HOT ... peak=64 C`, stopped after 34 s, the run went on, no `candidates are stopped at` line. But both runs then ended with `the card held no undervolt: the first step below its own voltage already failed`: their first candidates read `1070 mV (+15 MHz): SLOW ... clock=2323 MHz` and `1050 mV (+15 MHz): SLOW ... clock=2411 MHz`, on a card that ran 2791 MHz in the measurement just before. The search's load had not run yet when that candidate began (see the decision "With its reference handed over, the undervolt search warms its load up before the first candidate"). Second run, with the warm-up: **pass** for (a) and (b) (same day, a test build of 0733695). (a) `stock: STABLE score=5426 it/s clock=2788 MHz peak=66 C fan=70 %`, four seconds later `baseline: the measurement at stock`, `warming the load up: 5 s at stock`, `target: 2790 MHz, which the card reaches at 1090 mV by itself`, and the first candidate `1085 mV (+15 MHz): STABLE ... clock=2775 MHz`; 26 candidates, none `SLOW` or `TOO HOT`, the edge at 950 mV, `RESULT: 2790 MHz at 990 mV instead of 1090 mV`; `before:` 5426 it/s, 189 W, fan 70 % (300 s), `after:` 5325 it/s, 149 W, fan 53 % (300 s). (b) `stock: TOO HOT ... peak=64 C`, stopped after 74 s, the run went on, the same warm-up and a search to the same result (990 mV, edge 950 mV); `before:` shows 75 s; no `candidates are stopped at` line. The saved tune of before the test was put back and applied. (c) not run. | | 93 | A speed profile that would end slower than stock keeps nothing | Only on a card that is warmer at stock than the profile allows (the tester's RTX 3080 Ti; the reference card ends faster than stock with every speed profile): **Best of my GPU** from the window, to the end; the result screen, the dashboard and `run.log` | yes | the run goes through all its steps and its soak; then **Not applied** with `the result is slower than stock: N it/s against M (-X %)`, the power limit it had to go down to, and the pointer to Undervolt; `The card is back at stock.`; the dashboard shows stock and the saved tune of before the run, if there was one; `run.log` ends with `stopped: the result is slower than stock ... -- card restored to stock` | Not run yet. On the reference card: nothing to see, by design. | | 94 | All in one, with the undervolt step searching for the first time | Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize all` to the end with a build that has the warm-up of the undervolt step; then `gao --status`; put the copy back | yes | five steps; in step 3 a `warming the load up` line, a first candidate that is not `SLOW`, and a descent; steps 4 and 5 run on the tune of steps 2 and 3; `RESULT:` | **Not as expected at step 4**, and that found a fault (2026-10-11, reference RTX 4070, driver 617.42, a test build of 0733695, unattended). Step 1: `stock: STABLE score=5325 it/s peak=65 C fan=70 % power=186 W`. Step 2: power 105 %, `mem: bandwidth peak +2300, confirmed +2300, applying +1600`, `core: highest stable +210, confirmed +210, applying +135`, soak `STABLE score=5568 it/s peak=70 C power=197 W`. Step 3, as wanted: `warming the load up: 5 s with the overclock`, `target: 2865 MHz, which the card reaches at 1050 mV with the overclock`, nine candidates `STABLE` from 1045 mV down, `WRONG RESULT` at 990 mV, confirmed at 1010 mV after two tries, and `applying 1050 mV (+135 MHz at that point)`: the margin left no raise above the overclock, so the curve was cut off, as in the runs before the warm-up; soak `STABLE score=5434 it/s peak=67 C power=176 W`. Step 4 did not begin: `the tune could not be applied for the fan tune: the undervolt was not applied: the point already runs more than the target -- card at stock`. The saved tune of before the test was put back and applied. Cause and fix: the decision "A flat top goes back on a card that is cooler than when it was found". Second run, with the `gao.exe` of the released v0.4.6: **pass, all five steps** (same day, started elevated by a script). Step 1: `stock: STABLE score=5309 it/s peak=66 C fan=70 % power=185 W`. Step 2: power 105 %, `mem: bandwidth peak +850, confirmed +850, applying +550`, `core: highest stable +210, confirmed +210, applying +135`, soak `STABLE score=5568 it/s`. Step 3: `target: 2850 MHz, which the card reaches at 1045 mV with the overclock`, `undervolt: lowest stable 975 mV, confirmed 985 mV, applying 1025 mV (+165 MHz at that point)`, soak `STABLE score=5363 it/s peak=66 C power=163 W`. Step 4: the tune went back on the card, `fan 60 %: 63 C`, `fan 55 %: 65 C`, `fan 52 %: 67 C`, `fan tune: 52 % holds 67 C`. Step 5: the final test ran its 300 s. `RESULT: power 105 %, core +135 MHz, mem +550 MHz, 2850 MHz at 1025 mV, fans 52 % at 67 C`; `stock: 5309 it/s, 185 W, fan 70 %, 66 C`, `now: 5353 it/s, 166 W, fan 52 %, 70 C`: `+0.8 % score and -10.3 % power against stock`; `undervolt: applied`. The saved tune of before the test was put back. **What this run did not show:** its undervolt had a raise of its own (+165 over an offset of +135), so the anchor was raised and not merely cut off, and the case of the first run (an anchor that reads a step above the target on a cooler card) did not come up. That fix has still only its tests. **The memory offset differs a lot from run to run on this card, and the two logs say why:** +1600 in the first run (bandwidth peak +2300), +550 in the second (peak +850), +650 in the run that saved the tune in use that morning (the journal shows its scan ending at +1000 too). The card was in the same state each time (68 C, power 105 %, a reading every three to four seconds). In both logged scans the bandwidth rises in a straight line, 0.36 and 0.46 % per 50 MHz step, from 420 GB/s at +0 to 509 GB/s at +2450; the first run went on to `WRONG RESULT` at +2550 without the bandwidth ever turning down. The readings scatter around that line with a standard deviation of 0.4 and 0.7 %, single ones by more than 1 %. The scan ends at a reading more than 1 % below the best so far when a second reading agrees, and in the second run that was chance: +950 read high (462.8 GB/s, where the line is at about 458), +1000 read 456.6 twice, 1.3 % below it. In the first run the one low reading (+600) was followed by a higher one, and the scan ran to the first wrong result. So on this card there is no bandwidth peak before the errors, and where the scan stops is decided by noise. | -| 95 | The memory scan no longer ends by chance | (a) Not elevated, nothing changed on the card: a measuring build takes 24 bandwidth readings in one process, with 3 s of the stress load before each as a scan has, at 64, 128 and 256 dispatches per timed run; then the same with the app's window open next to it, drawing a frame every 100 ms and every 33 ms. (b) Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize best` twice in a row with the build of the change, then the copy put back | (a) no, (b) yes | (a) how much a reading scatters, and on what that depends. (b) in both runs every `mem +N: bandwidth` line shows three readings that agree within about 1 %; the scan does not end before the first `WRONG RESULT`; the two `mem: bandwidth peak` values are within 300 MHz of each other and the applied offset is above +1000; the soak is not slower than at stock | **(a) measured** (2026-10-11, reference RTX 4070, driver 617.42, the saved tune applied: power 105 %, core +135, memory +650; standard deviation over the 24 readings of a series, and the lowest and highest against the mean). 64 dispatches, no window: 0.44 % (-0.55 to +1.07, one reading did not settle), 0.66 % (-1.58 to +1.49, one did not settle), 1.31 % (-4.83 to +1.27), 0.86 % (-1.44 to +2.05). 128 dispatches, no window: 0.43 % (-1.10 to +0.79). 256 dispatches, no window: 0.22 % (-0.43 to +0.42) and 0.60 % (-0.80 to +1.37; the last five readings of this series were 451 to 453 GB/s after nineteen of 444 to 449). With the window drawing every 100 ms: 64 dispatches 1.13 % (the first six readings 452 to 453 GB/s, the rest 439 to 451), 256 dispatches 0.24 % (-0.47 to +0.61). With the window drawing every 33 ms: 64 dispatches 0.58 %, 256 dispatches 0.25 % (-0.35 to +0.51). A reading took 6.5, 5.3 and 3.9 timed runs to settle at 64, 128 and 256 dispatches. So: the longer timed run takes the scatter from 0.4 to 1.3 % down to 0.2 %, the window's drawing adds nothing, and apart from the scatter the readings sit about 1.8 % higher for up to a minute at a time (the stretch at the end of the second 256 series ran on into the next series; the last 64 series went back and forth between 444 and 452 GB/s). What sets that level is not known. **(b) the offset is the same twice; the scan ends in a driver reset twice** (2026-10-11, same card and driver, a test build of 2764609, unattended, 40 minutes for the two runs). Run 1: `stock: STABLE score=5311 it/s peak=66 C fan=70 % power=185 W`; 49 steps from `mem +0: bandwidth 422.7 GB/s (422.7 422.6 422.7)` to `mem +2400: bandwidth 501.4 GB/s (501.3 502.3 501.4)`; at +2450 the candidate was `STABLE` and the measurement after it lost the device: `mem +2450: the bandwidth measurement failed; treated as a driver reset`, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, 20 s of rest, `health: STABLE`, `confirm core +0 / mem +2050: STABLE`, `mem: bandwidth peak +2250, confirmed +2050, applying +1400`; `core: highest stable +210, confirmed +210, applying +135`; soak `STABLE score=5536 it/s peak=69 C power=195 W`; `driver resets during this run: 1`. Run 2: stock 5312 it/s; 50 steps from 421.8 GB/s at +0 to 502.8 at +2450; the same failure at +2500; `mem: bandwidth peak +2300, confirmed +2100, applying +1450`; core +135; soak `STABLE score=5555 it/s peak=69 C power=196 W`; one driver reset. The three readings of a step differed by 0.4 % on average and by 1.2 % at most; no step was more than 0.7 % below the best before it, so neither scan came near ending early, by the old rule or the new; every step had its three readings. The scan took 5 min 25 s (about 6.5 s a step; about 3 min before the change). As expected: the two peaks are 50 MHz apart, the applied offsets are +1400 and +1450, and both results are 4 % faster than stock. **Not as expected: neither scan reached a `WRONG RESULT`.** Both ended one and two steps before +2550 with the device lost during the bandwidth copy; Windows logged it both times (System log, `nvlddmkm` events 14 and 153 at 19:51:22 and 20:11:27), and the screen goes black for a moment. In the run of check 94 that got this far, the measurements at +2450 and +2500 gave a number and the stress load reported the first wrong result at +2550, without a reset. A step now copies about seven times as much as it did (three readings of about four runs of 128 GB, against one reading of about six runs of 32 GB), and the copy checks nothing, so where the memory starts to fail it is the driver that gives out, not a verdict. Two runs against one: likely the cause, not proven. The search handles it as it should (rest, health probe, confirm four steps lower, the core is still searched), but a driver reset in every run on a card like this is not what the change was for. The saved tune of before the test was put back and applied (`applied best: power 105 %, core +135 MHz, mem +650 MHz`), the tray app restarted, Defender reported nothing. **(c) the same two runs with one reading a step, three only for a step in doubt** (2026-10-11 20:26 to 21:03, a test build of 71e7c2b, unattended): **the reset is still there.** Run 1: stock 5328 it/s; 50 steps from `mem +0: bandwidth 420.4 GB/s` to `mem +2450: bandwidth 502.4 GB/s`; at +2500 the candidate was `STABLE` and then `bandwidth measurement failed: the timing of the copy did not come back` ten seconds later (`nvlddmkm` event 14 at 20:37:07); `mem: bandwidth peak +2200, confirmed +2000, applying +1400`; core +135; soak `STABLE score=5572 it/s peak=70 C power=197 W`; one driver reset. Run 2: stock 5311 it/s; 421.9 GB/s at +0 to 508.0 at +2450; at +2500 `the device was lost during the copy failed (hr=0x887A0006)`, the device hung (event 14 at 20:55:29); `mem: bandwidth peak +2400, confirmed +2200, applying +1500`; core +135; soak `STABLE score=5571 it/s peak=70 C power=196 W`; one driver reset. No step was in doubt in either run (the lowest was 0.75 % below the best before it), so every step was read once; the scan took 3 min 33 s. The two peaks are 200 MHz apart: with one reading a step, which offsets count as within 1 % of the best moves by a few steps. Applied +1400 and +1500, both 5 % faster than stock. So the reset does not come from reading a step three times. Four runs out of four with the longer timed run ended this way, at +2450 or +2500. Not merged; the design for a scan that does not end like this is the open row in `docs/development.md`. The saved tune was put back and applied again (core +135, memory +650), the tray app restarted, Defender reported nothing. **(d) the edge is remembered** (a test build of 7c215c4; the same two runs): expected is that the first scan ends as before and logs nothing new, the journal gets one `mem_edge` line with the GPU id and the driver version, and the second run logs `mem: the memory gave out at +N in an earlier run on this card and driver; the scan stops at +(N-200)`, has no `driver resets during this run` line, applies more than +1000 and is not slower than stock. **Run 1 took the machine down; the test ended there** (2026-10-11 21:23, unattended). Stock 5321 it/s, power 105 %, then the scan: 49 steps from +0 to `mem +2400: bandwidth 539.3 GB/s`, two of them read three times (+1200 and +1250). At +2450 the candidate was `STABLE` and the copy after it failed as in the four runs before: `bandwidth measurement failed: the copy failed (hr=0x887A0007): the device was lost` at 21:35:05, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, `the driver was reset; reconnecting`. That is the last line. Windows restarted by itself within fifteen seconds: Kernel-Power event 41 at 21:35:23, `The previous system shutdown at 21:34:57 was unexpected`, no bugcheck, no dump, no `nvlddmkm` event written. So the fifth scan that met the edge with the longer timed run did not end in a driver reset the search recovers from, but in a reset of the whole machine, during the reconnect or right after it. What held: the candidate's journal entry was closed, so no ceiling of a freeze was left; the line `{"mem_edge":2450,"rig":"GPU-... / 617.42"}` was on disk before the machine went down, so no later run of this build goes there again; the run had saved nothing, and at logon the tray app applied the tune of before the test (`applied best: power 105 %, core +135 MHz, mem +650 MHz`). Not answered: the second run. Seen along the way and not explained: from the first step of the scan the stress load scored 5760 it/s and the bandwidth read about 7 % higher than in every run before (486.6 GB/s at +900 against about 455), while the measurement at stock five minutes earlier was the usual 5321. **(e) how the stress load meets the edge** (the same build, no search): `gao --reset`, then `gao --set-mem` and `gao --stress 3` at +2300, +2350, ... until the verdict is not `STABLE`, five times, at stock power and core +0; expected is one of two outcomes, `WRONG RESULT` every time (then the stress load can lead the copy to the edge) or `DEVICE LOST` (then it cannot). **Not run:** the test ended in (d). It goes to the edge on purpose, five times, and after (d) that is a risk for the machine that needs its owner's word first. | +| 95 | The memory scan no longer ends by chance | (a) Not elevated, nothing changed on the card: a measuring build takes 24 bandwidth readings in one process, with 3 s of the stress load before each as a scan has, at 64, 128 and 256 dispatches per timed run; then the same with the app's window open next to it, drawing a frame every 100 ms and every 33 ms. (b) Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize best` twice in a row with the build of the change, then the copy put back | (a) no, (b) yes | (a) how much a reading scatters, and on what that depends. (b) in both runs every `mem +N: bandwidth` line shows three readings that agree within about 1 %; the scan does not end before the first `WRONG RESULT`; the two `mem: bandwidth peak` values are within 300 MHz of each other and the applied offset is above +1000; the soak is not slower than at stock | **(a) measured** (2026-10-11, reference RTX 4070, driver 617.42, the saved tune applied: power 105 %, core +135, memory +650; standard deviation over the 24 readings of a series, and the lowest and highest against the mean). 64 dispatches, no window: 0.44 % (-0.55 to +1.07, one reading did not settle), 0.66 % (-1.58 to +1.49, one did not settle), 1.31 % (-4.83 to +1.27), 0.86 % (-1.44 to +2.05). 128 dispatches, no window: 0.43 % (-1.10 to +0.79). 256 dispatches, no window: 0.22 % (-0.43 to +0.42) and 0.60 % (-0.80 to +1.37; the last five readings of this series were 451 to 453 GB/s after nineteen of 444 to 449). With the window drawing every 100 ms: 64 dispatches 1.13 % (the first six readings 452 to 453 GB/s, the rest 439 to 451), 256 dispatches 0.24 % (-0.47 to +0.61). With the window drawing every 33 ms: 64 dispatches 0.58 %, 256 dispatches 0.25 % (-0.35 to +0.51). A reading took 6.5, 5.3 and 3.9 timed runs to settle at 64, 128 and 256 dispatches. So: the longer timed run takes the scatter from 0.4 to 1.3 % down to 0.2 %, the window's drawing adds nothing, and apart from the scatter the readings sit about 1.8 % higher for up to a minute at a time (the stretch at the end of the second 256 series ran on into the next series; the last 64 series went back and forth between 444 and 452 GB/s). What sets that level is not known. **(b) the offset is the same twice; the scan ends in a driver reset twice** (2026-10-11, same card and driver, a test build of 2764609, unattended, 40 minutes for the two runs). Run 1: `stock: STABLE score=5311 it/s peak=66 C fan=70 % power=185 W`; 49 steps from `mem +0: bandwidth 422.7 GB/s (422.7 422.6 422.7)` to `mem +2400: bandwidth 501.4 GB/s (501.3 502.3 501.4)`; at +2450 the candidate was `STABLE` and the measurement after it lost the device: `mem +2450: the bandwidth measurement failed; treated as a driver reset`, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, 20 s of rest, `health: STABLE`, `confirm core +0 / mem +2050: STABLE`, `mem: bandwidth peak +2250, confirmed +2050, applying +1400`; `core: highest stable +210, confirmed +210, applying +135`; soak `STABLE score=5536 it/s peak=69 C power=195 W`; `driver resets during this run: 1`. Run 2: stock 5312 it/s; 50 steps from 421.8 GB/s at +0 to 502.8 at +2450; the same failure at +2500; `mem: bandwidth peak +2300, confirmed +2100, applying +1450`; core +135; soak `STABLE score=5555 it/s peak=69 C power=196 W`; one driver reset. The three readings of a step differed by 0.4 % on average and by 1.2 % at most; no step was more than 0.7 % below the best before it, so neither scan came near ending early, by the old rule or the new; every step had its three readings. The scan took 5 min 25 s (about 6.5 s a step; about 3 min before the change). As expected: the two peaks are 50 MHz apart, the applied offsets are +1400 and +1450, and both results are 4 % faster than stock. **Not as expected: neither scan reached a `WRONG RESULT`.** Both ended one and two steps before +2550 with the device lost during the bandwidth copy; Windows logged it both times (System log, `nvlddmkm` events 14 and 153 at 19:51:22 and 20:11:27), and the screen goes black for a moment. In the run of check 94 that got this far, the measurements at +2450 and +2500 gave a number and the stress load reported the first wrong result at +2550, without a reset. A step now copies about seven times as much as it did (three readings of about four runs of 128 GB, against one reading of about six runs of 32 GB), and the copy checks nothing, so where the memory starts to fail it is the driver that gives out, not a verdict. Two runs against one: likely the cause, not proven. The search handles it as it should (rest, health probe, confirm four steps lower, the core is still searched), but a driver reset in every run on a card like this is not what the change was for. The saved tune of before the test was put back and applied (`applied best: power 105 %, core +135 MHz, mem +650 MHz`), the tray app restarted, Defender reported nothing. **(c) the same two runs with one reading a step, three only for a step in doubt** (2026-10-11 20:26 to 21:03, a test build of 71e7c2b, unattended): **the reset is still there.** Run 1: stock 5328 it/s; 50 steps from `mem +0: bandwidth 420.4 GB/s` to `mem +2450: bandwidth 502.4 GB/s`; at +2500 the candidate was `STABLE` and then `bandwidth measurement failed: the timing of the copy did not come back` ten seconds later (`nvlddmkm` event 14 at 20:37:07); `mem: bandwidth peak +2200, confirmed +2000, applying +1400`; core +135; soak `STABLE score=5572 it/s peak=70 C power=197 W`; one driver reset. Run 2: stock 5311 it/s; 421.9 GB/s at +0 to 508.0 at +2450; at +2500 `the device was lost during the copy failed (hr=0x887A0006)`, the device hung (event 14 at 20:55:29); `mem: bandwidth peak +2400, confirmed +2200, applying +1500`; core +135; soak `STABLE score=5571 it/s peak=70 C power=196 W`; one driver reset. No step was in doubt in either run (the lowest was 0.75 % below the best before it), so every step was read once; the scan took 3 min 33 s. The two peaks are 200 MHz apart: with one reading a step, which offsets count as within 1 % of the best moves by a few steps. Applied +1400 and +1500, both 5 % faster than stock. So the reset does not come from reading a step three times. Four runs out of four with the longer timed run ended this way, at +2450 or +2500. Not merged; the design for a scan that does not end like this is the open row in `docs/development.md`. The saved tune was put back and applied again (core +135, memory +650), the tray app restarted, Defender reported nothing. **(d) the edge is remembered** (a test build of 7c215c4; the same two runs): expected is that the first scan ends as before and logs nothing new, the journal gets one `mem_edge` line with the GPU id and the driver version, and the second run logs `mem: the memory gave out at +N in an earlier run on this card and driver; the scan stops at +(N-200)`, has no `driver resets during this run` line, applies more than +1000 and is not slower than stock. **Run 1 took the machine down; the test ended there** (2026-10-11 21:23, unattended). Stock 5321 it/s, power 105 %, then the scan: 49 steps from +0 to `mem +2400: bandwidth 539.3 GB/s`, two of them read three times (+1200 and +1250). At +2450 the candidate was `STABLE` and the copy after it failed as in the four runs before: `bandwidth measurement failed: the copy failed (hr=0x887A0007): the device was lost` at 21:35:05, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, `the driver was reset; reconnecting`. That is the last line. Windows restarted by itself within fifteen seconds: Kernel-Power event 41 at 21:35:23, `The previous system shutdown at 21:34:57 was unexpected`, no bugcheck, no dump, no `nvlddmkm` event written. So the fifth scan that met the edge with the longer timed run did not end in a driver reset the search recovers from, but in a reset of the whole machine, during the reconnect or right after it. What held: the candidate's journal entry was closed, so no ceiling of a freeze was left; the line `{"mem_edge":2450,"rig":"GPU-... / 617.42"}` was on disk before the machine went down, so no later run of this build goes there again; the run had saved nothing, and at logon the tray app applied the tune of before the test (`applied best: power 105 %, core +135 MHz, mem +650 MHz`). Not answered: the second run. Seen along the way and not explained: from the first step of the scan the stress load scored 5760 it/s and the bandwidth read about 7 % higher than in every run before (486.6 GB/s at +900 against about 455), while the measurement at stock five minutes earlier was the usual 5321. **(e) how the stress load meets the edge** (the same build, no search): `gao --reset`, then `gao --set-mem` and `gao --stress 3` at +2300, +2350, ... until the verdict is not `STABLE`, five times, at stock power and core +0; expected is one of two outcomes, `WRONG RESULT` every time (then the stress load can lead the copy to the edge) or `DEVICE LOST` (then it cannot). **Not run:** the test ended in (d). It goes to the edge on purpose, five times, and after (d) that is a risk for the machine that needs its owner's word first. **(f) the two-pass search** (the short copy in the scan, the long copy from four steps below the highest step that passed; see `docs/development.md`): tested on a simulated card only. On the reference card, with the edge at +2450 in its journal: `gao --optimize best` once; expected `mem: the memory gave out at +2450 in an earlier run on this card and driver; the scan stops at +2250`, no memory candidate above +2250, `mem: the scan passed +2250; reading from +2200 down with the long copy`, a few `mem +N: bandwidth ... GB/s, the long copy` lines, no `driver resets during this run`, an applied offset above +1000 and a soak that is not slower than stock. **Not run: waits for the owner's word.** | ## Notes diff --git a/src/app/common.cpp b/src/app/common.cpp index 1e15922..f5ff054 100644 --- a/src/app/common.cpp +++ b/src/app/common.cpp @@ -358,11 +358,13 @@ OptimizeOutcome run_optimize(Preset preset, const OptimizeHooks& hooks, const st }; io.aborted = hooks.aborted; io.log = hooks.log; - io.bandwidth = [&] { - const auto gbps = load->MeasureBandwidth(); + const auto measure = [&](bool precise) { + const auto gbps = load->MeasureBandwidth(precise); if (!gbps) log("bandwidth measurement failed: " + load->Error()); return gbps; }; + io.bandwidth = [=] { return measure(false); }; + io.bandwidth_precise = [=] { return measure(true); }; io.bandwidth_unsettled = [&] { return load->BandwidthUnsettled(); }; // The wait between a driver reset and the next load. The card is left // alone: no load, no driver call, no fan tick, and no log line (the search diff --git a/src/core/search.cpp b/src/core/search.cpp index b16f7f1..92e24c6 100644 --- a/src/core/search.cpp +++ b/src/core/search.cpp @@ -61,31 +61,59 @@ int apply_margin(int max, int step, float perf_push) { return std::max(0, std::min(steps * step, max - step)); } -int best_bandwidth_offset(int lo, int hi, int step, int ceiling, const std::function& sample) { - const int top = std::min(hi, ceiling - 1); - std::vector> seen; +MemScanEnd bandwidth_scan(int lo, int hi, int step, int ceiling, const std::function& sample) { + MemScanEnd end{lo, MemScanEnd::Why::Limit}; + auto middle = [](std::vector of) { // of an even number: the lower one + std::sort(of.begin(), of.end()); + return of[(of.size() - 1) / 2]; + }; + std::vector steps; // the reading of every step that passed double best = 0; - for (int v = lo; v <= top; v += step) { + for (int v = lo; v <= std::min(hi, ceiling - 1); v += step) { MemSample s = sample(v); - if (!s.stable) break; - // A step that is further below the best than readings scatter is in - // doubt: it is read kBandwidthReadings times and the middle one counts. - // Only such a step: three readings at every step cost two minutes a - // scan and decided nothing a single reading did not (hardware check 95). - if (s.gbps < best * (1 - kBandwidthTie)) { + // A reading that would end the scan is in doubt: the step is read + // kBandwidthReadings times and the middle one counts. Only such a + // step: three readings at every step cost two minutes a scan and + // decided nothing a single reading did not (hardware check 95). + if (s.stable && s.gbps < best * (1 - kBandwidthDrop)) { std::vector readings{s.gbps}; while (static_cast(readings.size()) < kBandwidthReadings && (s = sample(v)).stable) readings.push_back(s.gbps); - if (!s.stable) break; - std::sort(readings.begin(), readings.end()); - s.gbps = readings[readings.size() / 2]; + s.gbps = middle(readings); } + if (!s.stable) { end.why = MemScanEnd::Why::Failed; break; } + end.top = v; + if (s.gbps < best * (1 - kBandwidthDrop)) { end.why = MemScanEnd::Why::Dropped; break; } + // The best is the highest middle of three steps in a row, not the + // highest reading: one reading that came out high would be a best + // every later step is held against, and the highest of fifty readings + // that scatter is always such a one. + steps.push_back(s.gbps); + best = std::max(best, middle({steps.end() - static_cast(std::min(steps.size(), 3)), steps.end()})); + } + return end; +} + +int precise_top(const MemScanEnd& end, int lo, int step, int known_edge) { + // The lowest offset that is known, or has to be assumed, not to hold. + int bound = end.top + step; + if (end.why == MemScanEnd::Why::Limit && known_edge != INT_MAX && known_edge > bound) bound = known_edge; + return std::max(lo, bound - step - kPreciseMarginSteps * step); +} + +int bandwidth_peak_below(int top, int lo, int step, const std::function& sample) { + std::vector> seen; + double best = 0; + for (int v = top; v >= lo; v -= step) { + const MemSample s = sample(v); + if (!s.stable) break; seen.emplace_back(v, s.gbps); best = std::max(best, s.gbps); - if (s.gbps < best * (1 - kBandwidthDrop)) break; + if (s.gbps < best * (1 - kBandwidthTie)) break; } + int peak = top; for (const auto& [v, gbps] : seen) - if (gbps >= best * (1 - kBandwidthTie)) return v; - return lo; + if (gbps >= best * (1 - kBandwidthTie)) peak = std::min(peak, v); + return peak; } int confirm_edge(int edge, int lo, int step, int tries, const std::function& holds, @@ -586,7 +614,10 @@ OptimizeResult optimize(const GpuControl& gpu, const Objectives& obj, Journal& j bool left_at_stock = false; // the measurement does not work: nothing was searched if (by_bandwidth) { bool unavailable = false; - const int peak = best_bandwidth_offset(0, bounds.mem_max_mhz, kMemStep, ceiling, [&](int v) { + // One step of either pass: the candidate, then the reading while + // it is still applied. `precise`: with the long copy. + const auto sample = [&](int v, bool precise) { + const auto& read = precise && io.bandwidth_precise ? io.bandwidth_precise : io.bandwidth; MemSample m; // One reading. One that only did not settle is taken again, // kBandwidthReadings times at most; any other failure ends @@ -600,7 +631,7 @@ OptimizeResult optimize(const GpuControl& gpu, const Objectives& obj, Journal& j m.stable = explore(v, reset_at, [&] { const auto verdict = clock_candidate(std::nullopt, mem_j, 0, v, kClockProbeS, [&] { for (int i = 0; i < kBandwidthReadings && !gbps && !failed; ++i) { - gbps = io.bandwidth(); + gbps = read(); failed = !gbps && !(io.bandwidth_unsettled && io.bandwidth_unsettled()); } }); @@ -630,11 +661,23 @@ OptimizeResult optimize(const GpuControl& gpu, const Objectives& obj, Journal& j if (m.stable) { m.gbps = *gbps; char buf[64]; - std::snprintf(buf, sizeof(buf), "mem +%d: bandwidth %.1f GB/s", v, m.gbps); + std::snprintf(buf, sizeof(buf), "mem +%d: bandwidth %.1f GB/s%s", v, m.gbps, + precise && io.bandwidth_precise ? ", the long copy" : ""); log(buf); } return m; - }); + }; + const MemScanEnd end = bandwidth_scan(0, bounds.mem_max_mhz, kMemStep, ceiling, [&](int v) { return sample(v, false); }); + int peak = precise_top(end, 0, kMemStep, journal.mem_edge()); + // The second pass, when there is something to read and the scan + // left the card in a state to read it: not after a reset event + // (nothing new is explored then, and confirming starts + // kResetBackoffSteps below `peak`), not on a stopped run. + if (!unavailable && stopped.empty() && !explored_enough && peak > 0) { + log("mem: the scan passed " + offset_text(end.top) + "; reading from " + offset_text(peak) + " down" + + (io.bandwidth_precise ? " with the long copy" : " once more")); + peak = bandwidth_peak_below(peak, 0, kMemStep, [&](int v) { return sample(v, true); }); + } // An unavailable measurement ended the scan at +0, and nothing // is searched without it. Memory that corrects its own errors // computes right far beyond the point where it stops getting @@ -666,7 +709,7 @@ OptimizeResult optimize(const GpuControl& gpu, const Objectives& obj, Journal& j if (!stopped.empty()) return finish_fail(stopped); if (reset_at >= 0) log("mem: the driver reset at " + offset_text(reset_at) + "; using " + offset_text(r.mem_max_stable) + - (by_bandwidth ? ", the best of the values that passed" : ", the last value that passed")); + (by_bandwidth ? ", well below the values that passed" : ", the last value that passed")); r.mem_confirmed = confirm_edge(confirm_from(r.mem_max_stable, kMemStep, reset_at), 0, kMemStep, kConfirmTries, [&](int v) { return holds(clock_candidate(std::nullopt, v, 0, v, kConfirmProbeS)); }, confirm_step_down); diff --git a/src/core/search.hpp b/src/core/search.hpp index ab21c00..bc68710 100644 --- a/src/core/search.hpp +++ b/src/core/search.hpp @@ -67,24 +67,38 @@ int apply_margin(int max, int step, float perf_push); // Memory overclocks fail by losing bandwidth before they fail by producing // errors: GDDR6X/GDDR6 retry bad transfers. The memory search therefore looks // for the bandwidth peak, not the stability ceiling. -// The scan stops once a step is this far below the best. It was 1 %, and that -// is less than the readings move by themselves: on the reference RTX 4070 -// they sit about 1.8 % higher for tens of seconds at a time, at any offset, -// and such a stretch ended a scan wherever it began or ended (hardware check -// 95). A peak that is real falls further than that within a step or two. -inline constexpr double kBandwidthDrop = 0.03; -// Within 1 % of the best counts as the best: measured noise on the 4070 is -// ~0.4 % and one 50 MHz step is worth ~0.5 %, so a narrower band would let -// noise pick the offset. +// +// It does so in two passes, because one measurement cannot do both jobs. +// The scan goes up and only has to find where it ends: a step that fails, or +// one that has clearly lost bandwidth. It reads with the short copy, which +// scatters (0.4 to 1.3 % on the reference RTX 4070, single readings by more) +// and is light: the one scan that took it to the memory's edge got a wrong +// result from the stress load there. Then the offset to keep is read with the +// long copy, which scatters by 0.2 % and is heavy: in five scans out of five +// it lost the device at the edge, and once the machine with it (hardware +// check 95). So the long copy is only taken kPreciseMarginSteps below the +// highest step the scan passed, going down from there. +// +// The scan ends once a step is this far below the best. Far more than the +// short readings move by themselves, level changes of 1.8 % included: where +// the scan ends by chance, the offset is chance again. +inline constexpr double kBandwidthDrop = 0.05; +// Within 1 % of the best counts as the best, in the second pass: one 50 MHz +// step is worth ~0.5 %, so a narrower band would let noise pick the offset. inline constexpr double kBandwidthTie = 0.01; -// A step of the memory scan that is in doubt (more than kBandwidthTie below -// the best) is the middle one of this many readings, so one reading that is -// off decides nothing; and a reading that did not settle is taken again up to -// this many times. On the reference card single readings ended scans at +250, -// at +1000 and not at all on one day, and the runs saved +100, +650 and +1600 -// MHz (hardware check 94). Not every step: that cost two minutes a scan and -// changed no outcome on the card (hardware check 95). +// A step of the scan whose reading would end it is the middle one of this +// many readings, so one low reading decides nothing; and the best that +// readings are held against is the highest middle of three steps in a row, so +// one high reading does not either. A reading that did not settle is taken +// again up to this many times. On the reference card single readings ended +// scans at +250, at +1000 and not at all on one day, and the runs saved +100, +// +650 and +1600 MHz (hardware check 94). inline constexpr int kBandwidthReadings = 3; +// How far below the highest step the scan passed the long copy stays, in +// steps. On the reference card the stress load passed +2500 and the long copy +// lost the device at +2450 and +2500: up to one step below the highest passed +// one. Four steps is where confirming starts after a reset, too. +inline constexpr int kPreciseMarginSteps = 4; // A confirm probe that ends too hot keeps the clock edge (heat is the power // step's and the soak's business), but only when the clock was loaded this @@ -124,13 +138,34 @@ struct MemSample { double gbps = 0; }; -// Scans lo, lo+step, ... (<= hi, < ceiling), sampling lo too. Stops at the -// first unstable sample or once gbps falls more than kBandwidthDrop below the -// best so far (a step in doubt is sampled kBandwidthReadings times and its -// middle reading counts). Returns the -// lowest stable offset within kBandwidthTie of the best, so noise on a flat -// curve never drifts the result upward; lo when no sample was stable. -int best_bandwidth_offset(int lo, int hi, int step, int ceiling, const std::function& sample); +// Where a bandwidth scan ended, and why. +struct MemScanEnd { + int top = 0; // the highest offset that passed; lo when none did + enum class Why { + Failed, // the step above `top` was not stable + Dropped, // `top` itself was stable but more than kBandwidthDrop below the best + Limit, // hi or the ceiling + } why = Why::Limit; +}; + +// The first pass: scans lo, lo+step, ... (<= hi, < ceiling), sampling lo too, +// until a sample is not stable or its gbps is more than kBandwidthDrop below +// the best so far, where the best is the highest middle value of three steps +// in a row. A step that would end the scan is sampled kBandwidthReadings +// times first, and its middle reading counts. +MemScanEnd bandwidth_scan(int lo, int hi, int step, int ceiling, const std::function& sample); + +// The highest offset the second pass may read: kPreciseMarginSteps below the +// highest offset that can have passed. `known_edge` is where the memory gave +// out in an earlier run (INT_MAX: nowhere): a scan that was held below it +// counts its margin from that edge, not a second time from where it was held. +int precise_top(const MemScanEnd& end, int lo, int step, int known_edge); + +// The second pass: samples top, top-step, ... (>= lo) until a sample is more +// than kBandwidthTie below the best of them, or not stable. Returns the +// lowest offset within kBandwidthTie of the best, so noise on a flat curve +// never drifts the result upward; `top` when not one sample was stable. +int bandwidth_peak_below(int top, int lo, int step, const std::function& sample); // "Search fast, confirm long": re-checks edge with a long probe; on failure // steps down and retries, at most `tries` probes, never probing lo (stock) or @@ -168,6 +203,10 @@ struct OptimizeIo { // getting faster. The app always gives one; when it fails at stock, the // memory is left at stock. std::function()> bandwidth; + // The same with the long copy: steadier, and too heavy for memory at its + // edge. Only called kPreciseMarginSteps below the highest step a scan + // passed. Empty: `bandwidth` is used there as well. + std::function()> bandwidth_precise; // Asked after `bandwidth` returned nothing: true when the load measured // but its readings did not agree (other programs using the card, say). // That is no sign of a driver reset. Empty: every failure may be one. diff --git a/src/hw/stress.cpp b/src/hw/stress.cpp index 9fb9716..85ac3ee 100644 --- a/src/hw/stress.cpp +++ b/src/hw/stress.cpp @@ -28,14 +28,16 @@ constexpr double kTargetBatchMs = 250.0; constexpr int kMaxDispatches = 2048; constexpr UINT kBwBytes = 256u * 1024 * 1024; // per buffer; far above the 4070's 36 MB L2 constexpr UINT kBwGroups = 1024; -// Dispatches per timed run: 128 GB copied, about 0.3 s on the reference -// RTX 4070. A run of 64 (75 ms) was short enough for whatever else the machine -// did to land in one run and not in the next: readings scattered by 0.4 to -// 0.7 % (standard deviation) and one in 24 did not settle; with 256 it is -// 0.2 % and none (hardware check 95). -constexpr int kBwDispatches = 256; +// Dispatches per timed run. The short run copies 32 GB, about 75 ms on the +// reference RTX 4070: short enough for whatever else the machine does to land +// in one run and not in the next, so readings scatter by 0.4 to 1.3 % +// (standard deviation) and now and then do not settle. The long run copies +// 128 GB in 0.3 s and scatters by 0.2 % (hardware check 95); what it costs is +// in stress.hpp. +constexpr int kBwDispatches = 64; +constexpr int kBwDispatchesPrecise = 256; constexpr size_t kBwRuns = 3; // consecutive runs that must agree -constexpr int kBwMaxRuns = 20; // ~6 s at full speed; more if clocks are low +constexpr int kBwMaxRuns = 20; // ~1.5 s at full speed (~6 s for the long run); more if clocks are low constexpr double kBwSettle = 0.01; // agree = within 1 % // Grid-stride copy of 16-byte elements between two raw buffers. @@ -289,8 +291,9 @@ StressBatch Stress::Batch() { return out; } -std::optional Stress::MeasureBandwidth() { +std::optional Stress::MeasureBandwidth(bool precise) { Impl& d = *impl_; + const int dispatches = precise ? kBwDispatchesPrecise : kBwDispatches; bw_unsettled_ = false; bw_runs_.clear(); if (!d.device && !CreateDevice()) return std::nullopt; @@ -362,7 +365,7 @@ std::optional Stress::MeasureBandwidth() { d.ctx->CSSetUnorderedAccessViews(0, 2, uavs, nullptr); d.ctx->Begin(d.q_disjoint.Get()); d.ctx->End(d.q_begin.Get()); - for (int i = 0; i < kBwDispatches; ++i) d.ctx->Dispatch(kBwGroups, 1, 1); + for (int i = 0; i < dispatches; ++i) d.ctx->Dispatch(kBwGroups, 1, 1); d.ctx->End(d.q_end.Get()); d.ctx->End(d.q_disjoint.Get()); d.ctx->CSSetUnorderedAccessViews(0, 2, none, nullptr); @@ -381,7 +384,7 @@ std::optional Stress::MeasureBandwidth() { continue; } const double seconds = double(t1 - t0) / double(dj.Frequency); - runs.push_back(2.0 * kBwBytes * kBwDispatches / seconds / 1e9); + runs.push_back(2.0 * kBwBytes * dispatches / seconds / 1e9); bw_runs_.push_back(runs.back()); if (runs.size() < kBwRuns) continue; std::vector last(runs.end() - kBwRuns, runs.end()); diff --git a/src/hw/stress.hpp b/src/hw/stress.hpp index f9a144d..21429f2 100644 --- a/src/hw/stress.hpp +++ b/src/hw/stress.hpp @@ -41,7 +41,12 @@ class Stress { // Device-memory bandwidth in GB/s (read + write) from a 256 MB buffer copy, // timed with GPU timestamps; median of 3 runs. nullopt when the device was // lost or the timing was unusable. - std::optional MeasureBandwidth(); + // precise: with the long timed run. Steadier readings (0.2 % against 0.4 + // to 1.3 % on the reference RTX 4070), and four times the copying in one + // go: memory at its edge lost the device under it in five scans out of + // five, and once the machine (hardware check 95). The search only asks + // for it well below that edge. + std::optional MeasureBandwidth(bool precise = false); // Why the last MeasureBandwidth() returned nothing: true when it measured // but the runs never agreed (other programs using the card, say); false // when the device was lost or a timing query failed. diff --git a/tests/test_search.cpp b/tests/test_search.cpp index 7b7c62b..85a605f 100644 --- a/tests/test_search.cpp +++ b/tests/test_search.cpp @@ -130,7 +130,15 @@ struct FakeCard { int flaky_power_sets = 0; // this many set_power_limit calls fail although nothing is wrong bool lost_on_first_core_confirm = false; // the first 30 s probe with a non-zero core offset ends DEVICE LOST bool lost_on_first_mem_confirm = false; // the first 30 s probe with a non-zero memory offset ends DEVICE LOST - int bw_measurements = 0; // bandwidth measurements taken + int bw_measurements = 0; // bandwidth measurements taken with the short copy + // The long copy. Quiet unless bw_precise_noise says otherwise, and from + // this memory offset on it takes the device down, as it did on the + // reference card at its edge (hardware check 95). + int bw_precise_loses_device_from = INT_MAX; + std::function bw_precise_noise; + int bw_precise_measurements = 0; + int max_precise_mem = -1; // the highest offset it was taken at + std::map bw_precise_at; bool bw_fails_at_zero = false; // the measurement at memory +0 fails; nothing is wrong with the card bool lost_on_first_soak = false; // the first 300 s probe ends DEVICE LOST bool fail_set_on_first_soak = false; // the first write of the soak state fails once @@ -296,6 +304,7 @@ struct Run { bool no_rest = false; // the application gives the search no way to wait std::optional stock; // the measurement at stock that went before the run, when there was one std::string rig = "card-a / 617.42"; // the card and driver the journal's memory edges are bound to + bool no_precise = false; // the application gives the search one way of measuring bandwidth only OptimizeResult go(Preset preset, GpuControl gpu) { Journal j(card.journal, [this](const std::string& l) { if (l.find("\"complete\"") != std::string::npos && card.completes_ok >= 0 && card.completes_ok-- == 0) @@ -340,6 +349,15 @@ struct Run { if (card.bw_fails) return std::nullopt; return card.bw_curve(card.mem) * (card.bw_noise ? card.bw_noise(card.mem, nth) : 1.0); }; + if (card.bw_curve && !no_precise) + io.bandwidth_precise = [this]() -> std::optional { + ++card.bw_precise_measurements; + const int nth = card.bw_precise_at[card.mem]++; + card.max_precise_mem = std::max(card.max_precise_mem, card.mem); + card.bw_unsettled = false; + if (card.mem >= card.bw_precise_loses_device_from) { card.driver_reset(); return std::nullopt; } + return card.bw_curve(card.mem) * (card.bw_precise_noise ? card.bw_precise_noise(card.mem, nth) : 1.0); + }; if (card.bw_curve) io.bandwidth_unsettled = [this] { return card.bw_unsettled; }; return optimize(gpu, objectives_for(preset), j, io, stock); } @@ -615,80 +633,172 @@ TEST_CASE("a soak that runs too hot lowers power, not clocks") { CHECK(r.mem_mhz == 550); } -TEST_CASE("bandwidth scan stops at the peak of a rising-then-falling curve") { +TEST_CASE("the bandwidth scan ends at the first step that has clearly lost bandwidth") { int max_sampled = -1; - const int r = best_bandwidth_offset(0, 1500, 50, INT_MAX, [&](int v) { + const MemScanEnd end = bandwidth_scan(0, 1500, 50, INT_MAX, [&](int v) { max_sampled = std::max(max_sampled, v); return MemSample{true, v <= 600 ? 500 + v * 0.3 : 680 - (v - 600) * 0.9}; }); - CHECK(r == 600); - CHECK(max_sampled == 650); // stopped at the first step more than kBandwidthDrop below the peak + CHECK(end.why == MemScanEnd::Why::Dropped); + // The best is the middle of three steps, 665 GB/s around the peak at + // +600: +650 is 4.5 % below it, +700 is 11 %. It was stable, so it passed. + CHECK(end.top == 700); + CHECK(max_sampled == 700); } TEST_CASE("a single low bandwidth reading does not end the scan; three do") { - // A step that is further below the best than readings scatter is read + // A step that is further from the best than readings scatter is read // kBandwidthReadings times and its middle reading counts. One dip (measured // on the 4070: a reading at half speed, the memory clock still in a lower // P-state) ends nothing; a drop that is there every time does. - const auto curve = [](int v) { return v <= 600 ? 500 + v * 0.3 : 680 - (v - 600) * 0.9; }; std::map calls; - const int r = best_bandwidth_offset(0, 1500, 50, INT_MAX, [&](int v) { + MemScanEnd end = bandwidth_scan(0, 1500, 50, INT_MAX, [&](int v) { const bool dip = v == 300 && calls[v] == 0; ++calls[v]; - return MemSample{true, dip ? 250.0 : curve(v)}; + return MemSample{true, dip ? 250.0 : 500 + v * 0.1}; }); - CHECK(r == 600); + CHECK(end.why == MemScanEnd::Why::Limit); + CHECK(end.top == 1500); CHECK(calls[250] == 1); // nothing in doubt: one reading CHECK(calls[300] == 3); CHECK(calls[350] == 1); - CHECK(calls[650] == 3); // the real drop is read three times too, and stands // Two low readings of three are the middle one. calls.clear(); - CHECK(best_bandwidth_offset(0, 1500, 50, INT_MAX, [&](int v) { + end = bandwidth_scan(0, 1500, 50, INT_MAX, [&](int v) { const int nth = calls[v]++; - return MemSample{true, v == 300 && nth < 2 ? 250.0 : 500 + v * 0.3}; - }) == 250); + return MemSample{true, v == 300 && nth < 2 ? 250.0 : 500 + v * 0.1}; + }); + CHECK(end.why == MemScanEnd::Why::Dropped); + CHECK(end.top == 300); CHECK(calls[300] == 3); CHECK(calls.count(350) == 0); - // A step that stops being stable while it is read again ends the scan there. + // A step that stops being stable while it is read again ends the scan below it. calls.clear(); - CHECK(best_bandwidth_offset(0, 1500, 50, INT_MAX, [&](int v) { + end = bandwidth_scan(0, 1500, 50, INT_MAX, [&](int v) { const int nth = calls[v]++; - return MemSample{!(v == 300 && nth == 1), v == 300 ? 250.0 : 500 + v * 0.3}; - }) == 250); + return MemSample{!(v == 300 && nth == 1), v == 300 ? 250.0 : 500 + v * 0.1}; + }); + CHECK(end.why == MemScanEnd::Why::Failed); + CHECK(end.top == 250); CHECK(calls[300] == 2); } -TEST_CASE("bandwidth scan stops at the first unstable step") { - int max_sampled = -1; - const int r = best_bandwidth_offset(0, 1500, 50, INT_MAX, [&](int v) { - max_sampled = std::max(max_sampled, v); - return MemSample{v <= 400, 500 + v * 0.3}; +TEST_CASE("a single high bandwidth reading does not become the best the scan is held against") { + // The run that saved +100 MHz on a card that runs every game at +1500: one + // reading came out far above its neighbours, and the steps after it were + // "a drop". + std::map calls; + const MemScanEnd end = bandwidth_scan(0, 1500, 50, INT_MAX, [&](int v) { + const bool spike = v == 200 && calls[v] == 0; + ++calls[v]; + return MemSample{true, (500 + v * 0.1) * (spike ? 1.08 : 1.0)}; }); - CHECK(r == 400); - CHECK(max_sampled == 450); + CHECK(end.why == MemScanEnd::Why::Limit); + CHECK(end.top == 1500); + for (const auto& [v, n] : calls) CHECK(n == 1); // and no step after it was in doubt + + // Two high steps in a row are the middle of three, and so a best: what + // comes after them is that far below it, read three times, and ends the scan. + const MemScanEnd two = bandwidth_scan(0, 1500, 50, INT_MAX, [&](int v) { + return MemSample{true, (500 + v * 0.1) * (v == 200 || v == 250 ? 1.08 : 1.0)}; + }); + CHECK(two.why == MemScanEnd::Why::Dropped); + CHECK(two.top == 300); } -TEST_CASE("a flat curve keeps the lowest offset") { - const int r = best_bandwidth_offset(0, 1500, 50, INT_MAX, [](int v) { - return MemSample{true, 500.0 + ((v / 50) % 2)}; // +-1 GB/s noise +TEST_CASE("the bandwidth scan ends below the first unstable step") { + int max_sampled = -1; + const MemScanEnd end = bandwidth_scan(0, 1500, 50, INT_MAX, [&](int v) { + max_sampled = std::max(max_sampled, v); + return MemSample{v <= 400, 500 + v * 0.1}; }); - CHECK(r == 0); + CHECK(end.why == MemScanEnd::Why::Failed); + CHECK(end.top == 400); + CHECK(max_sampled == 450); } -TEST_CASE("bandwidth scan samples lo and respects the ceiling") { +TEST_CASE("the bandwidth scan samples lo and respects the ceiling") { std::vector sampled; - const int r = best_bandwidth_offset(0, 1500, 50, 300, [&](int v) { + const MemScanEnd end = bandwidth_scan(0, 1500, 50, 300, [&](int v) { sampled.push_back(v); - return MemSample{true, 500 + v * 0.3}; + return MemSample{true, 500 + v * 0.1}; }); REQUIRE_FALSE(sampled.empty()); CHECK(sampled.front() == 0); CHECK(sampled.back() == 250); - CHECK(r == 250); - CHECK(best_bandwidth_offset(0, 1500, 50, 300, [](int) { return MemSample{false, 0}; }) == 0); + CHECK(end.top == 250); + CHECK(end.why == MemScanEnd::Why::Limit); + const MemScanEnd none = bandwidth_scan(0, 1500, 50, 300, [](int) { return MemSample{false, 0}; }); + CHECK(none.top == 0); + CHECK(none.why == MemScanEnd::Why::Failed); +} + +TEST_CASE("the long copy stays four steps below the highest step the scan passed") { + using Why = MemScanEnd::Why; + // The step above +2500 failed: +2500 passed, so +2300. + CHECK(precise_top({2500, Why::Failed}, 0, 50, INT_MAX) == 2300); + // +1000 passed and had lost bandwidth: +800. + CHECK(precise_top({1000, Why::Dropped}, 0, 50, INT_MAX) == 800); + // The driver's range, or the ceiling of a freeze: nothing failed, and the + // margin is kept all the same. What is above was never tried. + CHECK(precise_top({3000, Why::Limit}, 0, 50, INT_MAX) == 2800); + // Never below lo, and a scan that passed nothing reads nothing above it. + CHECK(precise_top({150, Why::Failed}, 0, 50, INT_MAX) == 0); + CHECK(precise_top({0, Why::Failed}, 0, 50, INT_MAX) == 0); +} + +TEST_CASE("a scan held below a known edge counts the margin from that edge, once") { + using Why = MemScanEnd::Why; + // The run that found the edge at +2550 read from +2300. The run after it is + // held at +2350 and reads from +2300 as well, not from +2150. + CHECK(precise_top({2500, Why::Failed}, 0, 50, INT_MAX) == 2300); + CHECK(precise_top({2350, Why::Limit}, 0, 50, 2550) == 2300); + // A step that failed below the known edge counts, not the edge. + CHECK(precise_top({2000, Why::Failed}, 0, 50, 2550) == 1800); + CHECK(precise_top({2000, Why::Dropped}, 0, 50, 2550) == 1800); + // An edge that is no higher than where the scan got changes nothing. + CHECK(precise_top({2350, Why::Limit}, 0, 50, 2400) == 2150); +} + +TEST_CASE("the second pass goes down from the top and keeps the lowest offset within the tie") { + // A curve that still rises at the top: 0.4 % a step. + std::vector sampled; + int r = bandwidth_peak_below(2300, 0, 50, [&](int v) { + sampled.push_back(v); + return MemSample{true, 434.0 * (1 + 0.004 * v / 50)}; + }); + CHECK(r == 2200); + CHECK(sampled == std::vector{2300, 2250, 2200, 2150}); + + // A peak below the top: the readings rise on the way down, then fall. + sampled.clear(); + r = bandwidth_peak_below(1200, 0, 50, [&](int v) { + sampled.push_back(v); + return MemSample{true, v <= 900 ? 500 + v * 0.3 : 770 - (v - 900) * 0.6}; + }); + CHECK(r == 900); + CHECK(sampled.front() == 1200); + CHECK(sampled.back() == 850); // 1.9 % below the peak: the pass ends + + // A flat curve keeps the lowest offset there is. + CHECK(bandwidth_peak_below(400, 0, 50, [](int v) { return MemSample{true, 500.0 + ((v / 50) % 2)}; }) == 0); + // Readings that scatter by twice what the long copy does on the 4070 leave + // a flat curve at the bottom. + CHECK(bandwidth_peak_below(400, 0, 50, [](int v) { return MemSample{true, 500.0 * (1 + (((v / 50) % 3) - 1) * 0.004)}; }) == 0); +} + +TEST_CASE("the second pass keeps what it has when a step is not stable") { + // Not one sample: the top it was given. + CHECK(bandwidth_peak_below(2300, 0, 50, [](int) { return MemSample{false, 0}; }) == 2300); + // One sample, then none: that one. + int n = 0; + CHECK(bandwidth_peak_below(2300, 0, 50, [&](int v) { return MemSample{n++ == 0, 500.0 + v * 0.1}; }) == 2300); + // Never below lo. + std::vector sampled; + CHECK(bandwidth_peak_below(100, 0, 50, [&](int v) { sampled.push_back(v); return MemSample{true, 500.0}; }) == 0); + CHECK(sampled == std::vector{100, 50, 0}); } TEST_CASE("confirm_edge keeps a holding edge and steps down otherwise") { @@ -711,10 +821,16 @@ TEST_CASE("memory search stops at the bandwidth peak, not the stability edge") { run.card.bw_curve = [](int m) { return m <= 900 ? 500 + m * 0.3 : 770 - (m - 900) * 0.6; }; const auto r = run.go(Preset::BestOfMyGpu); REQUIRE(r.ok); - CHECK(r.mem_max_stable == 900); - CHECK(r.mem_confirmed == 900); - CHECK(r.mem_mhz == 600); - CHECK(run.card.max_mem_seen <= 950); + // The scan ends at +1000, the first step that has lost 5 % (7.8 % below + // the peak at +900), far below the first wrong result at +1450. The long + // copy reads from four steps below that, +800, and keeps +800: the peak + // itself is inside the margin. That is the price of the margin on a card + // that has a peak. + CHECK(run.card.max_mem_seen == 1000); + CHECK(run.card.max_precise_mem == 800); + CHECK(r.mem_max_stable == 800); + CHECK(r.mem_confirmed == 800); + CHECK(r.mem_mhz == 550); Journal reread(run.card.journal, [](const std::string&) { return true; }); CHECK(reread.freezes().empty()); } @@ -757,16 +873,6 @@ TEST_CASE("a confirm probe that only runs too hot keeps the clock edge") { CHECK(r.mem_confirmed == 800); } -TEST_CASE("bandwidth noise wider than one step does not pick a higher offset") { - // Measured spread on the 4070 is ~0.4 %; one 50 MHz step is worth ~0.5 %. - // Noise of +-0.45 % (0.9 % spread, twice what was measured) on a flat - // curve must leave the result at the bottom. - const int r = best_bandwidth_offset(0, 1500, 50, INT_MAX, [](int v) { - return MemSample{true, 500.0 * (1 + (((v / 50) % 3) - 1) * 0.0045)}; - }); - CHECK(r == 0); -} - TEST_CASE("a journal entry that cannot be closed stops the run at stock") { // An open entry would blacklist the candidate forever as if it had frozen // the machine; carrying on would hide that the journal is failing. @@ -2153,19 +2259,21 @@ TEST_CASE("a failed bandwidth measurement ends the memory scan; the core is stil const auto r = run.go(Preset::BestOfMyGpu); REQUIRE(r.ok); CHECK(r.driver_resets == 1); - // The scan samples +0 .. +300; the confirm probe runs four steps below - // +250, the best of the values that were measured. The +0 sample names no - // clock in the journal. - CHECK(mem_begins(run.card) == climb(50, 300, 50, {50})); + // The scan samples +0 .. +300. +250 was the highest step that passed, so + // the long copy would read from +50; after a reset nothing is read, and + // confirming starts four steps below that, which is stock. The +0 sample + // names no clock in the journal. + CHECK(mem_begins(run.card) == climb(50, 300, 50)); CHECK(clockless_begins(run.card) == 1); - // +0, +50 .. +300, the memory confirm probe, core +15 .. +165, the core confirm probe, the soak - CHECK(run.card.count("\"begin\"") == 21); + // +0, +50 .. +300, core +15 .. +165, the core confirm probe, the soak + CHECK(run.card.count("\"begin\"") == 20); CHECK(run.logged("core +0 / mem +0: STABLE")); CHECK(run.logged("mem +0: bandwidth 500.0 GB/s")); CHECK(run.card.max_mem_seen == 300); CHECK(run.card.bw_measurements == 7); - CHECK(r.mem_max_stable == 250); - CHECK(r.mem_confirmed == 50); + CHECK(run.card.bw_precise_measurements == 0); + CHECK(r.mem_max_stable == 50); + CHECK(r.mem_confirmed == 0); CHECK(r.mem_mhz == 0); CHECK(run.card.max_core_seen == 165); CHECK(r.core_mhz == 105); @@ -2187,11 +2295,13 @@ TEST_CASE("bandwidth readings that do not settle end the memory scan and are no CHECK(run.card.recover_calls == 0); CHECK_FALSE(run.logged("treated as a driver reset")); CHECK(run.logged("mem +300: the bandwidth readings did not settle; the memory scan ends here")); - // The scan ends at +300 with the best of the measured values, +250, and - // the confirm probe runs there, not four steps below. + // The scan ends at +300; +250 was the highest step that passed, so the + // long copy reads from +50, and the confirm probe runs at what it kept, + // not four steps below that. CHECK(run.card.max_mem_seen == 300); - CHECK(r.mem_max_stable == 250); - CHECK(r.mem_confirmed == 250); + CHECK(run.card.max_precise_mem == 50); + CHECK(r.mem_max_stable == 50); + CHECK(r.mem_confirmed == 50); // The core is searched as in a run without any event. CHECK(run.card.max_core_seen > 0); CHECK(r.core_mhz > 0); @@ -2427,7 +2537,7 @@ TEST_CASE("the +0 sample of the bandwidth scan journals no clock") { run.card.bw_curve = [](int m) { return 500 + m * 0.3; }; const auto r = run.go(Preset::BestOfMyGpu); REQUIRE(r.ok); - CHECK(r.mem_max_stable == 800); + CHECK(r.mem_max_stable == 600); // +800 passed; the long copy read from +600 // The first entry of the run is the +0 sample: it ran and was measured, // and its `begin` line names neither clock. CHECK(run.logged("core +0 / mem +0: STABLE")); @@ -2458,7 +2568,7 @@ TEST_CASE("the +0 sample of the bandwidth scan journals no clock") { next.card.journal = frozen; const auto b = next.go(Preset::BestOfMyGpu); REQUIRE(b.ok); - CHECK(b.mem_max_stable == 800); + CHECK(b.mem_max_stable == 600); CHECK(b.core_max_stable == 150); } @@ -2890,59 +3000,65 @@ std::function scatter(unsigned seed, double sd) { } -TEST_CASE("one bandwidth reading that is too high does not end the memory scan") { +TEST_CASE("one bandwidth reading that is far too high does not end the memory scan") { // The run that saved +100 MHz on a card that runs every game at +1500: - // one reading at +200 came out 2.5 % above its neighbours, the two at - // +250 were more than 1 % under it, and that was "the peak". + // one reading came out far above its neighbours, the two after it were + // under it, and that was "the peak". Run run = rising_to_the_edge(); - run.card.bw_noise = [](int mem, int nth) { return mem == 200 && nth == 0 ? 1.025 : 1.0; }; + run.card.bw_noise = [](int mem, int nth) { return mem == 200 && nth == 0 ? 1.08 : 1.0; }; const auto r = run.go(Preset::BestOfMyGpu); REQUIRE(r.ok); CHECK(run.card.max_mem_seen == 2550); // the scan went to the first wrong result - CHECK(r.mem_max_stable >= 2300); - CHECK(r.mem_mhz >= 1500); CHECK(run.logged("mem +0: bandwidth 434.0 GB/s")); - // The steps after the high one were in doubt and read three times each. - CHECK(run.card.bw_readings_at[150] == 1); - CHECK(run.card.bw_readings_at[250] == kBandwidthReadings); + CHECK(run.card.bw_measurements == 51); // +0 .. +2500, each once: no step was in doubt + // +2500 passed; the long copy reads from +2300 down, and +2200 is the + // lowest offset within 1 % of what it read there. + CHECK(r.mem_max_stable == 2200); + CHECK(r.mem_mhz == 1500); } TEST_CASE("one low bandwidth reading does not end the memory scan; three low ones do") { { Run run = rising_to_the_edge(); - run.card.bw_noise = [](int mem, int nth) { return mem == 1000 && nth == 0 ? 0.95 : 1.0; }; + run.card.bw_noise = [](int mem, int nth) { return mem == 1000 && nth == 0 ? 0.9 : 1.0; }; const auto r = run.go(Preset::BestOfMyGpu); REQUIRE(r.ok); CHECK(run.card.bw_readings_at[950] == 1); CHECK(run.card.bw_readings_at[1000] == kBandwidthReadings); CHECK(run.card.max_mem_seen == 2550); - CHECK(r.mem_max_stable >= 2300); + CHECK(r.mem_max_stable == 2200); } { - // From +1000 on the memory loses 5 %: that is a peak. + // From +1000 on the memory has lost 8 %: that is a peak, whichever copy reads it. Run run = rising_to_the_edge(); - run.card.bw_noise = [](int mem, int) { return mem >= 1000 ? 0.95 : 1.0; }; + run.card.bw_curve = [](int m) { return 434.0 * (1 + 0.004 * m / 50) * (m >= 1000 ? 0.92 : 1.0); }; const auto r = run.go(Preset::BestOfMyGpu); REQUIRE(r.ok); CHECK(run.card.bw_readings_at[1000] == kBandwidthReadings); - CHECK(run.card.max_mem_seen == 1000); // the confirm probes are lower - CHECK(r.mem_max_stable >= 800); - CHECK(r.mem_max_stable <= 950); + CHECK(run.logged("mem: the scan passed +1000; reading from +800 down with the long copy")); + CHECK(run.card.max_mem_seen == 1000); + CHECK(run.card.max_precise_mem == 800); + CHECK(r.mem_max_stable == 700); // the lowest offset within 1 % of the reading at +800 } } TEST_CASE("readings that scatter as they do on the reference card leave the memory scan where the card puts it") { - // Measured there with the longer timed run: a standard deviation of - // 0.22 % per reading. Twice that here, on twenty different days. - for (unsigned seed = 1; seed <= 20; ++seed) { + // Measured there: the short copy scatters by 0.4 to 1.3 % (standard + // deviation), the long one by 0.2 %. The worst of the first and a little + // more than the second here, on two hundred different days. Every scan + // gets to the first wrong result, and the offset that is kept moves by + // four steps. + for (unsigned seed = 1; seed <= 200; ++seed) { Run run = rising_to_the_edge(); - run.card.bw_noise = scatter(seed, 0.0045); + run.card.bw_noise = scatter(seed, 0.013); + run.card.bw_precise_noise = scatter(seed + 1000, 0.0025); const auto r = run.go(Preset::BestOfMyGpu); CAPTURE(seed); REQUIRE(r.ok); CHECK(run.card.max_mem_seen == 2550); - CHECK(r.mem_max_stable >= 2150); - CHECK(r.mem_max_stable <= 2500); + CHECK(run.card.max_precise_mem == 2300); + CHECK(r.mem_max_stable >= 2100); + CHECK(r.mem_max_stable <= 2300); } } @@ -2953,13 +3069,18 @@ TEST_CASE("a real peak is still found where it is when the readings scatter") { Run run; run.card.mem_edge = 1400; run.card.bw_curve = [](int m) { return m <= 900 ? 500 + m * 0.3 : 770 - (m - 900) * 0.6; }; - run.card.bw_noise = scatter(seed, 0.0045); + run.card.bw_noise = scatter(seed, 0.013); + run.card.bw_precise_noise = scatter(seed + 1000, 0.0045); const auto r = run.go(Preset::BestOfMyGpu); CAPTURE(seed); REQUIRE(r.ok); - CHECK(r.mem_max_stable >= 800); - CHECK(r.mem_max_stable <= 950); + // +950 is 3.9 % below the peak and +1000 7.8 %: with this scatter the + // scan ends at one of the two, now and then one step later. + CHECK(run.card.max_mem_seen >= 950); CHECK(run.card.max_mem_seen <= 1050); + CHECK(run.card.max_precise_mem <= run.card.max_mem_seen - kPreciseMarginSteps * 50); + CHECK(r.mem_max_stable >= 700); + CHECK(r.mem_max_stable <= 900); } } @@ -3004,6 +3125,9 @@ OptimizeResult again(Run& run) { run.card.max_mem_seen = 0; run.card.bw_measurements = 0; run.card.bw_readings_at.clear(); + run.card.bw_precise_measurements = 0; + run.card.bw_precise_at.clear(); + run.card.max_precise_mem = -1; return run.go(Preset::BestOfMyGpu); } @@ -3030,8 +3154,12 @@ TEST_CASE("the next run stays four steps below the memory edge of the run before CHECK(run.logged("mem: the memory gave out at +2550 in an earlier run on this card and driver; the scan stops at +2350")); CHECK_FALSE(run.logged("mem +2400")); // no candidate above it, so none that could fail CHECK(second.driver_resets == 0); - CHECK(second.mem_max_stable >= 2200); - CHECK(second.mem_mhz >= 1500); + // The same offset as the run that found the edge: the margin counts from + // the edge, not a second time from where the scan was held. + CHECK(run.card.max_precise_mem == 2300); + CHECK(second.mem_max_stable == first.mem_max_stable); + CHECK(second.mem_max_stable == 2200); + CHECK(second.mem_mhz == 1500); CHECK(run.card.count("\"mem_edge\"") == 1); // nothing new was found // And the run after that is the same. const auto third = again(run); @@ -3051,7 +3179,7 @@ TEST_CASE("a bandwidth measurement that lost the device leaves the edge too, and REQUIRE(second.ok); CHECK(second.driver_resets == 0); CHECK(run.card.max_mem_seen == 300); - CHECK(second.mem_max_stable == 300); + CHECK(second.mem_max_stable == 250); // the margin counts from the edge at +500, once } TEST_CASE("the memory edge lapses with another driver") { @@ -3094,3 +3222,126 @@ TEST_CASE("a scan that ended for another reason leaves no memory edge") { } } +namespace { + +// The reference RTX 4070 as hardware check 95 found it: bandwidth rises by +// 0.4 % a step until the memory gives out, the stress load computes right up +// to +2500 and wrong at +2550, and the long copy takes the device down from +// +2450 on (in five scans out of five; once the machine went with it). The +// short copy got through there in the one scan that took it that far. +Run reference_card() { + Run run = recovering(); + run.card.mem_edge = 2500; + run.card.ranges = ClockOffsetRanges{{-500, 1000}, {-1000, 3000}}; + run.card.bw_curve = [](int m) { return 434.0 * (1 + 0.004 * m / 50); }; + run.card.bw_precise_loses_device_from = 2450; + return run; +} + +} + +TEST_CASE("on a card whose long copy loses the device at the edge, the first run never takes it there") { + Run run = reference_card(); + const auto r = run.go(Preset::BestOfMyGpu); + REQUIRE(r.ok); + // The scan finds the end with the short copy and the stress load. + CHECK(run.card.max_mem_seen == 2550); + CHECK(run.logged("core +0 / mem +2550: WRONG RESULT")); + CHECK(run.logged("mem: the scan passed +2500; reading from +2300 down with the long copy")); + // The long copy: four readings, none above +2300, and the device stayed. + CHECK(run.card.max_precise_mem == 2300); + CHECK(run.card.bw_precise_measurements == 4); + CHECK(run.logged("mem +2300: bandwidth 513.9 GB/s, the long copy")); + CHECK(r.driver_resets == 0); + CHECK(run.card.recover_calls == 0); + CHECK_FALSE(run.logged("treated as a driver reset")); + CHECK(run.logged("mem: bandwidth peak +2200, confirmed +2200, applying +1500")); + CHECK(r.mem_mhz == 1500); + // The edge is written down, and the next run does not even meet the wrong result. + CHECK(run.card.count("\"mem_edge\":2550") == 1); + const auto second = again(run); + REQUIRE(second.ok); + CHECK(run.card.max_mem_seen == 2350); + CHECK(run.card.max_precise_mem == 2300); + CHECK(second.driver_resets == 0); + CHECK(second.mem_mhz == 1500); + check_journal_rules(run); +} + +TEST_CASE("the long copy stays below the margin on that card whatever the readings do") { + // Short readings that scatter as the worst series measured, with a high + // and a low stretch thrown in; the long copy at twice its measured scatter. + for (unsigned seed = 1; seed <= 40; ++seed) { + Run run = reference_card(); + const auto noise = scatter(seed, 0.013); + run.card.bw_noise = [&card = run.card, noise, seed](int mem, int nth) { + const int n = card.bw_measurements; + const double level = n > 10 && n <= 20 ? 1.018 : n > 30 + static_cast(seed % 10) && n <= 45 ? 0.982 : 1.0; + return noise(mem, nth) * level; + }; + run.card.bw_precise_noise = scatter(seed + 1000, 0.0045); + const auto r = run.go(Preset::BestOfMyGpu); + CAPTURE(seed); + REQUIRE(r.ok); + CHECK(run.card.max_precise_mem <= run.card.max_mem_seen - 50 - kPreciseMarginSteps * 50); + CHECK(run.card.max_precise_mem < 2450); + CHECK(r.driver_resets == 0); + CHECK(run.card.max_mem_seen == 2550); + CHECK(r.mem_mhz >= 1400); // +2050 or higher kept, at twice the long copy's scatter + } +} + +TEST_CASE("the margin holds for an edge anywhere") { + // The same kind of card with its edge elsewhere: the long copy dies one + // step below the highest offset the stress load passes, as on the + // reference card, and is never taken there. + for (int edge = 300; edge <= 2900; edge += 150) { + Run run = reference_card(); + run.card.mem_edge = edge; + run.card.bw_precise_loses_device_from = edge - 50; + const auto r = run.go(Preset::BestOfMyGpu); + CAPTURE(edge); + REQUIRE(r.ok); + CHECK(run.card.max_mem_seen == edge + 50); + CHECK(run.card.max_precise_mem == edge - 200); + CHECK(r.driver_resets == 0); + CHECK(r.mem_max_stable <= edge - 200); + } +} + +TEST_CASE("a long copy that loses the device below the margin after all is a reset the search recovers from") { + // A card that is worse than the margin allows for. The run goes on as + // after any reset in the memory search, and the offset is remembered. + Run run = reference_card(); + run.card.bw_precise_loses_device_from = 2300; + const auto r = run.go(Preset::BestOfMyGpu); + REQUIRE(r.ok); + CHECK(r.driver_resets == 1); + CHECK(run.card.bw_precise_measurements == 1); + CHECK(run.logged("mem +2300: the bandwidth measurement failed; treated as a driver reset")); + CHECK(r.mem_max_stable == 2300); + CHECK(r.mem_confirmed == 2300 - kResetBackoffSteps * 50); + CHECK(run.card.max_core_seen > 0); // the core is still searched + CHECK(run.card.count("\"mem_edge\":2300") == 1); + // The next run stays below that as well, with both copies. + const auto second = again(run); + REQUIRE(second.ok); + CHECK(second.driver_resets == 0); + CHECK(run.card.max_mem_seen == 2100); + CHECK(run.card.max_precise_mem == 2050); + check_journal_rules(run); +} + +TEST_CASE("without a long copy the second pass reads with the short one") { + // A caller that gives only one way of measuring: the search is the same. + Run run = rising_to_the_edge(); + run.no_precise = true; + const auto r = run.go(Preset::BestOfMyGpu); + REQUIRE(r.ok); + CHECK(run.card.bw_precise_measurements == 0); + CHECK(run.card.bw_readings_at[2300] == 2); // once going up, once coming down + CHECK_FALSE(run.logged("the long copy")); + CHECK(run.logged("mem: the scan passed +2500; reading from +2300 down once more")); + CHECK(r.mem_max_stable == 2200); +} + From 6453c97a1152519dffdd42dcb72326b9bbe45c66 Mon Sep 17 00:00:00 2001 From: Rovey Date: Sun, 11 Oct 2026 22:09:20 +0200 Subject: [PATCH 09/15] docs: hardware check 95 (g), a video playing in a browser is the likely source of the scatter Co-Authored-By: Claude Opus 5.5 --- docs/development.md | 2 +- docs/hardware-checks.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/development.md b/docs/development.md index 8b3069d..af37747 100644 --- a/docs/development.md +++ b/docs/development.md @@ -80,7 +80,7 @@ The applied offset is always at least one step below the confirmed edge. Profile | A confirm probe that is too hot at once confirms nothing | A confirm probe that ends `TOO HOT` keeps the clock edge: over 30 s the card may run a little hotter than over the power step's 20 s, and that is the soak's business, which lowers power. But the stress run stops at its first reading above the limit, so a card that is above it when the probe begins gives `TOO HOT` within a second, and that counted as holding too. It happened on the tester's RTX 3080 Ti (hardware check 91): after a driver reset the health probe loads the card at stock power, which on that card means 76 C against a limit of 75, and `confirm core +225` was confirmed by a probe that had tested nothing. Now such a probe counts only when the clock was loaded for `kConfirmMinS` (10 s, clearly longer than the 3 s probe that found the edge) before it got too hot. Otherwise the value is not confirmed, and the card rests (`io.rest`, the 20 s of a recovery, with the fans where the curve last put them) before the next try one step lower: without the rest every lower value would start just as hot, and heat alone would step the edge down to nothing, which is what keeping the edge on `TOO HOT` was for. The same value is not tried again; one step is the price. Not changed: the health probe still runs at stock power, and a soak that begins too hot still lowers the power limit by a step. | | A profile that promises speed keeps no result that is slower than stock | Best of my GPU ended 3.8 % slower than stock on the tester's RTX 3080 Ti (4749 against 4937 it/s, hardware check 91), with the fans at 80 % instead of 66 %, and showed it as **Done**, applied and kept at every logon. Every step had done what it is for: the card runs 83 C at stock with its fans at 100 %, the profile allows 75, so the power limit went to 65 %, and +150 MHz of core clock did not win that back. But a profile called "the best of my GPU" that leaves the card slower has not done what it says, and a result like that is not offered as one. After the soak, `optimize` compares its score with the measurement at stock that went before the run (the "before" the result screen shows, so the rule judges by the number the user sees): below `kNotSlowerScore` (0.99: one percent is what two five-minute runs differ by) the run ends as a failed one does, at stock, with nothing saved, and says what it found, why, and that Undervolt is the profile for a card that runs warm. `Objectives::faster_than_stock` marks the two profiles this holds for, Best of my GPU and Max performance. Not Quiet and Cool & efficient, which trade speed for noise and heat and say so. Not the overclock step of All in one either: it goes on to an undervolt and a fan tune, and its result is judged as a whole; whether that whole may be slower than stock is not decided. This is a backstop, not the cure: what such a card needs from these profiles is in `docs/planned.md` ("never hotter than at stock"). | | The memory search has two passes: a scan with the short copy finds where it ends, and the long copy reads the offset to keep, four steps below that | **What went wrong.** Four runs on one day applied +650, +1600, +550 and +100 MHz on the reference RTX 4070, a card that runs every game at +1500 (hardware check 94). That card has no bandwidth peak: bandwidth rises by about 0.4 % per 50 MHz step all the way to +2500 and the stress load computes its first wrong result at +2550, so the scan could only end early by chance, and where it did decided the offset. **What was measured** (hardware check 95 a, no search, nothing changed on the card). A reading is the median of three timed runs that agree within 1 %. With the timed run as it was, 32 GB or 75 ms, readings scatter by 0.4 to 1.3 % (standard deviation over 24 readings, six series), single ones by up to 4.8 %, and now and then one does not settle. With 128 GB per run it is 0.2 %. Apart from that the readings change level: for up to a minute every reading is about 1.8 % higher, at one and the same offset; the cause is not known. The app's own window, drawing every 100 or 33 ms next to the measurement, adds nothing, so nothing was built for it. **What was tried first, and what it did** (check 95 b to d). The long timed run at every step of the scan: the offset repeated (+1400 to +1500 in four runs, 4 to 5 % faster than stock), and every scan now ran to where the memory gives out and met it with the long copy. Five scans out of five lost the device there, at +2450 or +2500, a driver reset each time, with three readings a step and with one; and the fifth took the whole machine down within seconds (Kernel-Power 41, no bugcheck). In the one scan before all this that got to the edge, the short copy read a number at +2450 and +2500 and the stress load reported the wrong result at +2550, with no reset. One against five is thin, and it is all there is: the long copy is too heavy for memory at its edge, the short one was not. **The design.** One measurement cannot do both jobs, so there are two passes. *The scan* goes up as before and only has to find where it ends: a step that is not stable, or one that has lost more than `kBandwidthDrop` (5 %) against the best. It reads with the short copy (`kBwDispatches` 64), which is what the released versions take to the edge. Because those readings scatter, two things keep one of them from deciding: a reading that would end the scan is taken `kBandwidthReadings` (3) times and the middle one counts, and the best that readings are held against is the highest middle of three steps in a row, not the highest reading (the highest of fifty scattering readings is always one that came out high). 5 % is far more than the readings move by themselves; the scan is not where the offset is chosen any more, so it can afford to be coarse. *The second pass* chooses the offset, with the long copy (`MeasureBandwidth(true)`, `OptimizeIo::bandwidth_precise`): it starts `kPreciseMarginSteps` (4 steps, 200 MHz) below the highest step the scan passed (`precise_top`), goes down a step at a time until a reading is more than `kBandwidthTie` (1 %) below the best of them, and keeps the lowest offset within that 1 % (`bandwidth_peak_below`). On the reference card: the scan passes +2500, the long copy reads +2300, +2250, +2200 and +2150, and +2200 is kept; confirmed there, 70 % applied, +1500. The long copy lost the device at the highest step the stress load passed and one below it; four steps leave two to spare. The margin is kept whatever ended the scan, a drop and the driver's own range included: what is above was never tried. After a reset event in the scan the second pass is skipped, and confirming starts four steps below where it would have begun. **The edge is met once** (built before this, kept): a scan that ended in a wrong result or a reset event writes that offset to the journal with the GPU id and the driver version (`{"mem_edge":2550,"rig":" / "}`, a line without an id, so older builds skip it). A later scan on the same card and driver stops four steps below the lowest such offset, and its second pass starts where the first run's did: the margin counts from the edge, not a second time from where the scan was held. With another driver or card the line does not count. **What it costs.** A card that does have a peak keeps an offset up to 200 MHz below it when the scan ends close above the peak: the margin does not know a drop from an edge. A scan that ends for a harmless reason (readings that did not settle) loses those 200 MHz as well. The second pass adds about four steps, half a minute. **Not chosen.** The long copy everywhere: see above. The short copy everywhere, as released: the offset is chance (the +100 run). Three long readings a step, or only for a step in doubt: the reset came with both. Only remembering the edge: the first meeting can cost the machine, and once per card is once too often. The stress load probing four steps ahead of the copy: it needs the proof that the load at the edge ends in a wrong result and not in a reset of its own, and getting that proof means going to the edge on purpose. A fixed top for the scan (an offset, a share of the range, a gain over +0): there is nothing to choose the number by. **Tested** on a simulated card with the reference card's numbers (wrong result at +2550, the long copy losing the device from +2450): the long copy is never taken above +2300, in the first run and in the one after it, with short readings scattering by 1.3 % on two hundred seeds, and on forty more with level changes of 1.8 % thrown in; with the edge anywhere from +300 to +2900; and the tests fail when the margin is set to nothing. **Not tested on the card.** See the row below. | -| Open: the two-pass memory search has not run on a card | The last card test (hardware check 95 d) ended with the machine restarting, and nothing has been run on the card since. Before this is merged, its owner decides whether and when to run it there. What a run would show, and what it cannot: the journal of the reference card holds the edge at +2450 now, so a run of this build stops its scan at +2250 and reads with the long copy from +2200 down, at offsets every earlier scan passed. That tests the second pass and the result, with little risk. It does not test the first meeting with the edge with the short copy: that is what the released versions do, it went well the one time it was seen, and seeing it again means removing the edge from the journal and going there. **Still not known.** Why the long copy fails where the short one did not: the amount copied in one go, or only that five runs are more than one. Whether a card with a real bandwidth peak behaves as assumed; none has been measured through a scan (the tester's RTX 3080 Ti never got a reading to settle at +0). And one observation from the run that crashed: from the first step of its scan the stress load scored 5760 it/s and the bandwidth read about 7 % higher than in every run before (486.6 GB/s at +900 against about 455), while the measurement at stock five minutes earlier was the usual 5321. The suspicion, not checked: the monitor went to sleep and the desktop stopped being drawn. If that is it, the comparison of a result with stock depends on whether the screen is on, and the level changes of 1.8 % may be the same thing in small. | +| Open: the two-pass memory search has not run on a card | The last card test (hardware check 95 d) ended with the machine restarting, and nothing has been run on the card since. Before this is merged, its owner decides whether and when to run it there. What a run would show, and what it cannot: the journal of the reference card holds the edge at +2450 now, so a run of this build stops its scan at +2250 and reads with the long copy from +2200 down, at offsets every earlier scan passed. That tests the second pass and the result, with little risk. It does not test the first meeting with the edge with the short copy: that is what the released versions do, it went well the one time it was seen, and seeing it again means removing the edge from the journal and going there. **Still not known.** Why the long copy fails where the short one did not: the amount copied in one go, or only that five runs are more than one. Whether a card with a real bandwidth peak behaves as assumed; none has been measured through a scan (the tester's RTX 3080 Ti never got a reading to settle at +0). And one observation from the run that crashed: from the first step of its scan the stress load scored 5760 it/s and the bandwidth read about 7 % higher than in every run before (486.6 GB/s at +900 against about 455), while the measurement at stock five minutes earlier was the usual 5321. It was not the monitor going to sleep, as first suspected: it never does on that machine, and the run's log shows the change in the middle of the scan, between +350 and +550 (430 to 457 GB/s within one step, 5260 to 5750 it/s), not at its start. **What it most likely is: another program using the card, here a video playing in a browser.** Measured after the restart, with the same tune applied and nothing changed (hardware check 95 g): the short copy read 480 GB/s with a scatter of 0.23 %, as steady as the long copy was all day; minutes later, with a YouTube video playing in Chrome (8.6 % of the card's video decoder, 2.8 % of its 3D engine), the same series read 434 to 473 GB/s with a scatter of 3 %, and the stress load scores about 5300 it/s in the one state and 5750 to 5830 in the other. The low state is what every measurement of that day shows until 21:31. Not proven: nobody paused and played the video on purpose while measuring. If it holds, three things follow, and none is built. The scatter of the short copy that this design is built around (0.4 to 1.3 %) was for the larger part the video, not the copy; the design stands, it only has more room than was thought. The level changes of 1.8 % are the same thing in small. And the comparison of a result with stock is worth nothing when such a program starts or stops between the two: 8 % is far more than the 1 % by which a speed profile is judged (`kNotSlowerScore`) and than anything the efficiency search decides on. What to do about that is open: say so before a run (the README names games, MSI Afterburner and RivaTuner, not a browser with a video), or read the card's use per program (Windows keeps it, per engine) and warn or wait. | | Open: why the bandwidth measurement fails on some cards, and what then | On the tester's RTX 3080 Ti it failed at +0, and since 0.4.4 that means no memory overclock at all on such a card, though it computed right far beyond +2000 MHz. The measurement gives a number only when three runs in a row agree within 1 % (`kBwSettle`), and it throws away every run the driver's timer calls disjoint (the clock changed under it). That card was held at a power limit of 65 % and ran at 83 C when it was measured; the reference card has only ever been measured at 105 %, where the measurement itself draws 179 of 210 W. So the suspicion is the power limit, and hardware check 90 tests exactly that on the reference card. What it shows decides what is built, and nothing is built before it: **(1) runs are thrown away as disjoint, and the ones that are kept agree:** the timer is what fails, not the card. Time the copy differently (wall clock around a flush, or count a disjoint run when its neighbours agree) and keep the method. **(2) the readings themselves scatter by more than 1 %:** a peak that is found by a 1 % drop (`kBandwidthDrop`) cannot be found in that noise. Then the memory has to be measured where the card is not held back: search it before the power step, at stock power, where a probe and its measurement last five seconds; or take more readings and compare medians. **(3) it does not fail at 65 % on the reference card:** the power limit is not the cause. Then it was the heat (a card throttling at its temperature limit changes clock as well), or another program, and only the tester's `run.log` can say: since this change a measurement that never settled writes how many runs gave a reading, between which values, and how many were thrown away. In no case does the climb on stability alone come back. **Outcome of check 90: (3).** On the reference card the measurement gave a number fifteen times out of fifteen at 100, 80 and 65 %, in three or four runs, with nothing thrown away, and the bandwidth was the same at every limit. The power limit is ruled out; (1) and (2) are not built. What is left is the heat of that card, another program, or a timer query that failed, and the tester's `run.log` is the next thing that can tell them apart. **The tester's run (check 91): it is (2), but only inside a run.** Twenty readings between 763.9 and 821.2 GB/s, none thrown away, three seconds after the load began; after thirty seconds of load the same card read within 0.2 %. The suspicion now is the moment, not the card: the measurement is taken while the power limiter is still catching up. If `gao --bandwidth --after-load 3` scatters on that card and `--after-load 30` does not, that is it, and the fix is to measure when the card has settled (keep taking runs for some seconds instead of twenty runs), not to change the method. | | Every line of a run is also in `run.log` | `logged()` wraps the log hook of both front ends: a line is appended and flushed before it is shown, so the step a machine freezes in is the last line on disk. One file in the state folder, moved aside at 1 MB (`run.log.old`). Only elevated and only after `ensure_app_dir` has verified the folder: an elevated process appends to nothing a user could have put there. | | A run does not start while the NVIDIA app's Automatic Tuning is on | Switched on, a service of the driver scans the card when it sees fit: it loads it, changes its clocks, and starts again by itself (hardware check 89: a scan that was stopped started again 14 ms later, and after a restart of the service it resumed within two seconds). A run next to that measures nothing it can trust, so it is refused, like a run next to Afterburner, and with the same weight: a courtesy, not a guard. Read from where that service keeps the switch, `%ProgramData%\NVIDIA Corporation\nvtopps\nvtopps.db3`, table `ocScannerSettings`, fields `backgroundScanEnabled` and `grdOcEnabled`, joined to `TelemetryGpuCfg` so that only a card that is in the PC counts (a row of an earlier card can never be switched off again). That this is the source and not a copy of it was measured, not assumed: with the field set the service took the scan up again after every restart, with it cleared it did not, while the NVIDIA app itself showed the switch as off. Nothing documents the file. So: opened read-only and immutable (no lock, the service never waits for us), through `winsqlite3.dll` from System32 (no dependency added), and everything that is not exactly as expected reads as "unknown", which starts the run as before. The registry value the app sets next to it (`HKLM\SOFTWARE\NVIDIA Corporation\Global\NvApp\OC\`) changes at the same moment but belongs to the app, not to the service, and is not used. **The overlay is not checked:** no setting for it was found, its processes run whether it is open or not, and the one value that moved (`IsShadowPlayEnabled`) followed the service being up, not a switch. NVAPI was not tried: an undocumented call is no better than an undocumented file, and it cannot be compared with anything. | diff --git a/docs/hardware-checks.md b/docs/hardware-checks.md index 1c11ed7..2e21dce 100644 --- a/docs/hardware-checks.md +++ b/docs/hardware-checks.md @@ -122,7 +122,7 @@ root; adjust the path for a Debug build. | 92 | The undervolt profile starts with the measurement at stock every run uses, and a hot card is not refused | Elevated, the tray app exited, a copy of `gao.json` kept. (a) `gao --undervolt` to the end; then `gao --status`. (b) The same with a test build whose stock limit is lowered to a temperature the reference card passes under load (63 C; it reaches 67). Put the copy of `gao.json` back. (c) Only on a card that is above the profile's limit at stock with the fans at 100 % (the tester's RTX 3080 Ti: 82 to 83 C): the **Undervolt** profile from the window, to the end, and its `run.log` | yes | (a) `the card at stock, five minutes`, a `stock: STABLE ...` line with the driver's fan speed, then `baseline: the measurement at stock`, and no second five minutes at stock: the first candidate follows at once; no `candidates are stopped at` line; `RESULT:`; `before:` shows 300 s and the fan speed the driver chose. (b) `stock: TOO HOT ... peak=64 C`, the line that the measurement stopped there, the cool-down, then `baseline: the measurement at stock` and a search that runs to `RESULT:` with `before:` showing the shortened time; no `candidates are stopped at` line, because 64 C is under the profile's 75. (c) no `not stable at stock`; `the card reached 86 C at stock: candidates are stopped at 85 C, not at the profile's 75 C`; candidates that end `STABLE` or `WRONG RESULT`, none `TOO HOT`; a result that is cooler and draws less than stock at the same clock | First run: **not as expected**, and that found a fault (2026-10-11, reference RTX 4070, driver 617.42, a test build of d1c99e9, started elevated by a script). The first halves held: (a) `stock: STABLE ... peak=65 C fan=70 %`, then `baseline: the measurement at stock` four seconds later and no second five minutes; (b) `stock: TOO HOT ... peak=64 C`, stopped after 34 s, the run went on, no `candidates are stopped at` line. But both runs then ended with `the card held no undervolt: the first step below its own voltage already failed`: their first candidates read `1070 mV (+15 MHz): SLOW ... clock=2323 MHz` and `1050 mV (+15 MHz): SLOW ... clock=2411 MHz`, on a card that ran 2791 MHz in the measurement just before. The search's load had not run yet when that candidate began (see the decision "With its reference handed over, the undervolt search warms its load up before the first candidate"). Second run, with the warm-up: **pass** for (a) and (b) (same day, a test build of 0733695). (a) `stock: STABLE score=5426 it/s clock=2788 MHz peak=66 C fan=70 %`, four seconds later `baseline: the measurement at stock`, `warming the load up: 5 s at stock`, `target: 2790 MHz, which the card reaches at 1090 mV by itself`, and the first candidate `1085 mV (+15 MHz): STABLE ... clock=2775 MHz`; 26 candidates, none `SLOW` or `TOO HOT`, the edge at 950 mV, `RESULT: 2790 MHz at 990 mV instead of 1090 mV`; `before:` 5426 it/s, 189 W, fan 70 % (300 s), `after:` 5325 it/s, 149 W, fan 53 % (300 s). (b) `stock: TOO HOT ... peak=64 C`, stopped after 74 s, the run went on, the same warm-up and a search to the same result (990 mV, edge 950 mV); `before:` shows 75 s; no `candidates are stopped at` line. The saved tune of before the test was put back and applied. (c) not run. | | 93 | A speed profile that would end slower than stock keeps nothing | Only on a card that is warmer at stock than the profile allows (the tester's RTX 3080 Ti; the reference card ends faster than stock with every speed profile): **Best of my GPU** from the window, to the end; the result screen, the dashboard and `run.log` | yes | the run goes through all its steps and its soak; then **Not applied** with `the result is slower than stock: N it/s against M (-X %)`, the power limit it had to go down to, and the pointer to Undervolt; `The card is back at stock.`; the dashboard shows stock and the saved tune of before the run, if there was one; `run.log` ends with `stopped: the result is slower than stock ... -- card restored to stock` | Not run yet. On the reference card: nothing to see, by design. | | 94 | All in one, with the undervolt step searching for the first time | Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize all` to the end with a build that has the warm-up of the undervolt step; then `gao --status`; put the copy back | yes | five steps; in step 3 a `warming the load up` line, a first candidate that is not `SLOW`, and a descent; steps 4 and 5 run on the tune of steps 2 and 3; `RESULT:` | **Not as expected at step 4**, and that found a fault (2026-10-11, reference RTX 4070, driver 617.42, a test build of 0733695, unattended). Step 1: `stock: STABLE score=5325 it/s peak=65 C fan=70 % power=186 W`. Step 2: power 105 %, `mem: bandwidth peak +2300, confirmed +2300, applying +1600`, `core: highest stable +210, confirmed +210, applying +135`, soak `STABLE score=5568 it/s peak=70 C power=197 W`. Step 3, as wanted: `warming the load up: 5 s with the overclock`, `target: 2865 MHz, which the card reaches at 1050 mV with the overclock`, nine candidates `STABLE` from 1045 mV down, `WRONG RESULT` at 990 mV, confirmed at 1010 mV after two tries, and `applying 1050 mV (+135 MHz at that point)`: the margin left no raise above the overclock, so the curve was cut off, as in the runs before the warm-up; soak `STABLE score=5434 it/s peak=67 C power=176 W`. Step 4 did not begin: `the tune could not be applied for the fan tune: the undervolt was not applied: the point already runs more than the target -- card at stock`. The saved tune of before the test was put back and applied. Cause and fix: the decision "A flat top goes back on a card that is cooler than when it was found". Second run, with the `gao.exe` of the released v0.4.6: **pass, all five steps** (same day, started elevated by a script). Step 1: `stock: STABLE score=5309 it/s peak=66 C fan=70 % power=185 W`. Step 2: power 105 %, `mem: bandwidth peak +850, confirmed +850, applying +550`, `core: highest stable +210, confirmed +210, applying +135`, soak `STABLE score=5568 it/s`. Step 3: `target: 2850 MHz, which the card reaches at 1045 mV with the overclock`, `undervolt: lowest stable 975 mV, confirmed 985 mV, applying 1025 mV (+165 MHz at that point)`, soak `STABLE score=5363 it/s peak=66 C power=163 W`. Step 4: the tune went back on the card, `fan 60 %: 63 C`, `fan 55 %: 65 C`, `fan 52 %: 67 C`, `fan tune: 52 % holds 67 C`. Step 5: the final test ran its 300 s. `RESULT: power 105 %, core +135 MHz, mem +550 MHz, 2850 MHz at 1025 mV, fans 52 % at 67 C`; `stock: 5309 it/s, 185 W, fan 70 %, 66 C`, `now: 5353 it/s, 166 W, fan 52 %, 70 C`: `+0.8 % score and -10.3 % power against stock`; `undervolt: applied`. The saved tune of before the test was put back. **What this run did not show:** its undervolt had a raise of its own (+165 over an offset of +135), so the anchor was raised and not merely cut off, and the case of the first run (an anchor that reads a step above the target on a cooler card) did not come up. That fix has still only its tests. **The memory offset differs a lot from run to run on this card, and the two logs say why:** +1600 in the first run (bandwidth peak +2300), +550 in the second (peak +850), +650 in the run that saved the tune in use that morning (the journal shows its scan ending at +1000 too). The card was in the same state each time (68 C, power 105 %, a reading every three to four seconds). In both logged scans the bandwidth rises in a straight line, 0.36 and 0.46 % per 50 MHz step, from 420 GB/s at +0 to 509 GB/s at +2450; the first run went on to `WRONG RESULT` at +2550 without the bandwidth ever turning down. The readings scatter around that line with a standard deviation of 0.4 and 0.7 %, single ones by more than 1 %. The scan ends at a reading more than 1 % below the best so far when a second reading agrees, and in the second run that was chance: +950 read high (462.8 GB/s, where the line is at about 458), +1000 read 456.6 twice, 1.3 % below it. In the first run the one low reading (+600) was followed by a higher one, and the scan ran to the first wrong result. So on this card there is no bandwidth peak before the errors, and where the scan stops is decided by noise. | -| 95 | The memory scan no longer ends by chance | (a) Not elevated, nothing changed on the card: a measuring build takes 24 bandwidth readings in one process, with 3 s of the stress load before each as a scan has, at 64, 128 and 256 dispatches per timed run; then the same with the app's window open next to it, drawing a frame every 100 ms and every 33 ms. (b) Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize best` twice in a row with the build of the change, then the copy put back | (a) no, (b) yes | (a) how much a reading scatters, and on what that depends. (b) in both runs every `mem +N: bandwidth` line shows three readings that agree within about 1 %; the scan does not end before the first `WRONG RESULT`; the two `mem: bandwidth peak` values are within 300 MHz of each other and the applied offset is above +1000; the soak is not slower than at stock | **(a) measured** (2026-10-11, reference RTX 4070, driver 617.42, the saved tune applied: power 105 %, core +135, memory +650; standard deviation over the 24 readings of a series, and the lowest and highest against the mean). 64 dispatches, no window: 0.44 % (-0.55 to +1.07, one reading did not settle), 0.66 % (-1.58 to +1.49, one did not settle), 1.31 % (-4.83 to +1.27), 0.86 % (-1.44 to +2.05). 128 dispatches, no window: 0.43 % (-1.10 to +0.79). 256 dispatches, no window: 0.22 % (-0.43 to +0.42) and 0.60 % (-0.80 to +1.37; the last five readings of this series were 451 to 453 GB/s after nineteen of 444 to 449). With the window drawing every 100 ms: 64 dispatches 1.13 % (the first six readings 452 to 453 GB/s, the rest 439 to 451), 256 dispatches 0.24 % (-0.47 to +0.61). With the window drawing every 33 ms: 64 dispatches 0.58 %, 256 dispatches 0.25 % (-0.35 to +0.51). A reading took 6.5, 5.3 and 3.9 timed runs to settle at 64, 128 and 256 dispatches. So: the longer timed run takes the scatter from 0.4 to 1.3 % down to 0.2 %, the window's drawing adds nothing, and apart from the scatter the readings sit about 1.8 % higher for up to a minute at a time (the stretch at the end of the second 256 series ran on into the next series; the last 64 series went back and forth between 444 and 452 GB/s). What sets that level is not known. **(b) the offset is the same twice; the scan ends in a driver reset twice** (2026-10-11, same card and driver, a test build of 2764609, unattended, 40 minutes for the two runs). Run 1: `stock: STABLE score=5311 it/s peak=66 C fan=70 % power=185 W`; 49 steps from `mem +0: bandwidth 422.7 GB/s (422.7 422.6 422.7)` to `mem +2400: bandwidth 501.4 GB/s (501.3 502.3 501.4)`; at +2450 the candidate was `STABLE` and the measurement after it lost the device: `mem +2450: the bandwidth measurement failed; treated as a driver reset`, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, 20 s of rest, `health: STABLE`, `confirm core +0 / mem +2050: STABLE`, `mem: bandwidth peak +2250, confirmed +2050, applying +1400`; `core: highest stable +210, confirmed +210, applying +135`; soak `STABLE score=5536 it/s peak=69 C power=195 W`; `driver resets during this run: 1`. Run 2: stock 5312 it/s; 50 steps from 421.8 GB/s at +0 to 502.8 at +2450; the same failure at +2500; `mem: bandwidth peak +2300, confirmed +2100, applying +1450`; core +135; soak `STABLE score=5555 it/s peak=69 C power=196 W`; one driver reset. The three readings of a step differed by 0.4 % on average and by 1.2 % at most; no step was more than 0.7 % below the best before it, so neither scan came near ending early, by the old rule or the new; every step had its three readings. The scan took 5 min 25 s (about 6.5 s a step; about 3 min before the change). As expected: the two peaks are 50 MHz apart, the applied offsets are +1400 and +1450, and both results are 4 % faster than stock. **Not as expected: neither scan reached a `WRONG RESULT`.** Both ended one and two steps before +2550 with the device lost during the bandwidth copy; Windows logged it both times (System log, `nvlddmkm` events 14 and 153 at 19:51:22 and 20:11:27), and the screen goes black for a moment. In the run of check 94 that got this far, the measurements at +2450 and +2500 gave a number and the stress load reported the first wrong result at +2550, without a reset. A step now copies about seven times as much as it did (three readings of about four runs of 128 GB, against one reading of about six runs of 32 GB), and the copy checks nothing, so where the memory starts to fail it is the driver that gives out, not a verdict. Two runs against one: likely the cause, not proven. The search handles it as it should (rest, health probe, confirm four steps lower, the core is still searched), but a driver reset in every run on a card like this is not what the change was for. The saved tune of before the test was put back and applied (`applied best: power 105 %, core +135 MHz, mem +650 MHz`), the tray app restarted, Defender reported nothing. **(c) the same two runs with one reading a step, three only for a step in doubt** (2026-10-11 20:26 to 21:03, a test build of 71e7c2b, unattended): **the reset is still there.** Run 1: stock 5328 it/s; 50 steps from `mem +0: bandwidth 420.4 GB/s` to `mem +2450: bandwidth 502.4 GB/s`; at +2500 the candidate was `STABLE` and then `bandwidth measurement failed: the timing of the copy did not come back` ten seconds later (`nvlddmkm` event 14 at 20:37:07); `mem: bandwidth peak +2200, confirmed +2000, applying +1400`; core +135; soak `STABLE score=5572 it/s peak=70 C power=197 W`; one driver reset. Run 2: stock 5311 it/s; 421.9 GB/s at +0 to 508.0 at +2450; at +2500 `the device was lost during the copy failed (hr=0x887A0006)`, the device hung (event 14 at 20:55:29); `mem: bandwidth peak +2400, confirmed +2200, applying +1500`; core +135; soak `STABLE score=5571 it/s peak=70 C power=196 W`; one driver reset. No step was in doubt in either run (the lowest was 0.75 % below the best before it), so every step was read once; the scan took 3 min 33 s. The two peaks are 200 MHz apart: with one reading a step, which offsets count as within 1 % of the best moves by a few steps. Applied +1400 and +1500, both 5 % faster than stock. So the reset does not come from reading a step three times. Four runs out of four with the longer timed run ended this way, at +2450 or +2500. Not merged; the design for a scan that does not end like this is the open row in `docs/development.md`. The saved tune was put back and applied again (core +135, memory +650), the tray app restarted, Defender reported nothing. **(d) the edge is remembered** (a test build of 7c215c4; the same two runs): expected is that the first scan ends as before and logs nothing new, the journal gets one `mem_edge` line with the GPU id and the driver version, and the second run logs `mem: the memory gave out at +N in an earlier run on this card and driver; the scan stops at +(N-200)`, has no `driver resets during this run` line, applies more than +1000 and is not slower than stock. **Run 1 took the machine down; the test ended there** (2026-10-11 21:23, unattended). Stock 5321 it/s, power 105 %, then the scan: 49 steps from +0 to `mem +2400: bandwidth 539.3 GB/s`, two of them read three times (+1200 and +1250). At +2450 the candidate was `STABLE` and the copy after it failed as in the four runs before: `bandwidth measurement failed: the copy failed (hr=0x887A0007): the device was lost` at 21:35:05, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, `the driver was reset; reconnecting`. That is the last line. Windows restarted by itself within fifteen seconds: Kernel-Power event 41 at 21:35:23, `The previous system shutdown at 21:34:57 was unexpected`, no bugcheck, no dump, no `nvlddmkm` event written. So the fifth scan that met the edge with the longer timed run did not end in a driver reset the search recovers from, but in a reset of the whole machine, during the reconnect or right after it. What held: the candidate's journal entry was closed, so no ceiling of a freeze was left; the line `{"mem_edge":2450,"rig":"GPU-... / 617.42"}` was on disk before the machine went down, so no later run of this build goes there again; the run had saved nothing, and at logon the tray app applied the tune of before the test (`applied best: power 105 %, core +135 MHz, mem +650 MHz`). Not answered: the second run. Seen along the way and not explained: from the first step of the scan the stress load scored 5760 it/s and the bandwidth read about 7 % higher than in every run before (486.6 GB/s at +900 against about 455), while the measurement at stock five minutes earlier was the usual 5321. **(e) how the stress load meets the edge** (the same build, no search): `gao --reset`, then `gao --set-mem` and `gao --stress 3` at +2300, +2350, ... until the verdict is not `STABLE`, five times, at stock power and core +0; expected is one of two outcomes, `WRONG RESULT` every time (then the stress load can lead the copy to the edge) or `DEVICE LOST` (then it cannot). **Not run:** the test ended in (d). It goes to the edge on purpose, five times, and after (d) that is a risk for the machine that needs its owner's word first. **(f) the two-pass search** (the short copy in the scan, the long copy from four steps below the highest step that passed; see `docs/development.md`): tested on a simulated card only. On the reference card, with the edge at +2450 in its journal: `gao --optimize best` once; expected `mem: the memory gave out at +2450 in an earlier run on this card and driver; the scan stops at +2250`, no memory candidate above +2250, `mem: the scan passed +2250; reading from +2200 down with the long copy`, a few `mem +N: bandwidth ... GB/s, the long copy` lines, no `driver resets during this run`, an applied offset above +1000 and a soak that is not slower than stock. **Not run: waits for the owner's word.** | +| 95 | The memory scan no longer ends by chance | (a) Not elevated, nothing changed on the card: a measuring build takes 24 bandwidth readings in one process, with 3 s of the stress load before each as a scan has, at 64, 128 and 256 dispatches per timed run; then the same with the app's window open next to it, drawing a frame every 100 ms and every 33 ms. (b) Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize best` twice in a row with the build of the change, then the copy put back | (a) no, (b) yes | (a) how much a reading scatters, and on what that depends. (b) in both runs every `mem +N: bandwidth` line shows three readings that agree within about 1 %; the scan does not end before the first `WRONG RESULT`; the two `mem: bandwidth peak` values are within 300 MHz of each other and the applied offset is above +1000; the soak is not slower than at stock | **(a) measured** (2026-10-11, reference RTX 4070, driver 617.42, the saved tune applied: power 105 %, core +135, memory +650; standard deviation over the 24 readings of a series, and the lowest and highest against the mean). 64 dispatches, no window: 0.44 % (-0.55 to +1.07, one reading did not settle), 0.66 % (-1.58 to +1.49, one did not settle), 1.31 % (-4.83 to +1.27), 0.86 % (-1.44 to +2.05). 128 dispatches, no window: 0.43 % (-1.10 to +0.79). 256 dispatches, no window: 0.22 % (-0.43 to +0.42) and 0.60 % (-0.80 to +1.37; the last five readings of this series were 451 to 453 GB/s after nineteen of 444 to 449). With the window drawing every 100 ms: 64 dispatches 1.13 % (the first six readings 452 to 453 GB/s, the rest 439 to 451), 256 dispatches 0.24 % (-0.47 to +0.61). With the window drawing every 33 ms: 64 dispatches 0.58 %, 256 dispatches 0.25 % (-0.35 to +0.51). A reading took 6.5, 5.3 and 3.9 timed runs to settle at 64, 128 and 256 dispatches. So: the longer timed run takes the scatter from 0.4 to 1.3 % down to 0.2 %, the window's drawing adds nothing, and apart from the scatter the readings sit about 1.8 % higher for up to a minute at a time (the stretch at the end of the second 256 series ran on into the next series; the last 64 series went back and forth between 444 and 452 GB/s). What sets that level is not known. **(b) the offset is the same twice; the scan ends in a driver reset twice** (2026-10-11, same card and driver, a test build of 2764609, unattended, 40 minutes for the two runs). Run 1: `stock: STABLE score=5311 it/s peak=66 C fan=70 % power=185 W`; 49 steps from `mem +0: bandwidth 422.7 GB/s (422.7 422.6 422.7)` to `mem +2400: bandwidth 501.4 GB/s (501.3 502.3 501.4)`; at +2450 the candidate was `STABLE` and the measurement after it lost the device: `mem +2450: the bandwidth measurement failed; treated as a driver reset`, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, 20 s of rest, `health: STABLE`, `confirm core +0 / mem +2050: STABLE`, `mem: bandwidth peak +2250, confirmed +2050, applying +1400`; `core: highest stable +210, confirmed +210, applying +135`; soak `STABLE score=5536 it/s peak=69 C power=195 W`; `driver resets during this run: 1`. Run 2: stock 5312 it/s; 50 steps from 421.8 GB/s at +0 to 502.8 at +2450; the same failure at +2500; `mem: bandwidth peak +2300, confirmed +2100, applying +1450`; core +135; soak `STABLE score=5555 it/s peak=69 C power=196 W`; one driver reset. The three readings of a step differed by 0.4 % on average and by 1.2 % at most; no step was more than 0.7 % below the best before it, so neither scan came near ending early, by the old rule or the new; every step had its three readings. The scan took 5 min 25 s (about 6.5 s a step; about 3 min before the change). As expected: the two peaks are 50 MHz apart, the applied offsets are +1400 and +1450, and both results are 4 % faster than stock. **Not as expected: neither scan reached a `WRONG RESULT`.** Both ended one and two steps before +2550 with the device lost during the bandwidth copy; Windows logged it both times (System log, `nvlddmkm` events 14 and 153 at 19:51:22 and 20:11:27), and the screen goes black for a moment. In the run of check 94 that got this far, the measurements at +2450 and +2500 gave a number and the stress load reported the first wrong result at +2550, without a reset. A step now copies about seven times as much as it did (three readings of about four runs of 128 GB, against one reading of about six runs of 32 GB), and the copy checks nothing, so where the memory starts to fail it is the driver that gives out, not a verdict. Two runs against one: likely the cause, not proven. The search handles it as it should (rest, health probe, confirm four steps lower, the core is still searched), but a driver reset in every run on a card like this is not what the change was for. The saved tune of before the test was put back and applied (`applied best: power 105 %, core +135 MHz, mem +650 MHz`), the tray app restarted, Defender reported nothing. **(c) the same two runs with one reading a step, three only for a step in doubt** (2026-10-11 20:26 to 21:03, a test build of 71e7c2b, unattended): **the reset is still there.** Run 1: stock 5328 it/s; 50 steps from `mem +0: bandwidth 420.4 GB/s` to `mem +2450: bandwidth 502.4 GB/s`; at +2500 the candidate was `STABLE` and then `bandwidth measurement failed: the timing of the copy did not come back` ten seconds later (`nvlddmkm` event 14 at 20:37:07); `mem: bandwidth peak +2200, confirmed +2000, applying +1400`; core +135; soak `STABLE score=5572 it/s peak=70 C power=197 W`; one driver reset. Run 2: stock 5311 it/s; 421.9 GB/s at +0 to 508.0 at +2450; at +2500 `the device was lost during the copy failed (hr=0x887A0006)`, the device hung (event 14 at 20:55:29); `mem: bandwidth peak +2400, confirmed +2200, applying +1500`; core +135; soak `STABLE score=5571 it/s peak=70 C power=196 W`; one driver reset. No step was in doubt in either run (the lowest was 0.75 % below the best before it), so every step was read once; the scan took 3 min 33 s. The two peaks are 200 MHz apart: with one reading a step, which offsets count as within 1 % of the best moves by a few steps. Applied +1400 and +1500, both 5 % faster than stock. So the reset does not come from reading a step three times. Four runs out of four with the longer timed run ended this way, at +2450 or +2500. Not merged; the design for a scan that does not end like this is the open row in `docs/development.md`. The saved tune was put back and applied again (core +135, memory +650), the tray app restarted, Defender reported nothing. **(d) the edge is remembered** (a test build of 7c215c4; the same two runs): expected is that the first scan ends as before and logs nothing new, the journal gets one `mem_edge` line with the GPU id and the driver version, and the second run logs `mem: the memory gave out at +N in an earlier run on this card and driver; the scan stops at +(N-200)`, has no `driver resets during this run` line, applies more than +1000 and is not slower than stock. **Run 1 took the machine down; the test ended there** (2026-10-11 21:23, unattended). Stock 5321 it/s, power 105 %, then the scan: 49 steps from +0 to `mem +2400: bandwidth 539.3 GB/s`, two of them read three times (+1200 and +1250). At +2450 the candidate was `STABLE` and the copy after it failed as in the four runs before: `bandwidth measurement failed: the copy failed (hr=0x887A0007): the device was lost` at 21:35:05, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, `the driver was reset; reconnecting`. That is the last line. Windows restarted by itself within fifteen seconds: Kernel-Power event 41 at 21:35:23, `The previous system shutdown at 21:34:57 was unexpected`, no bugcheck, no dump, no `nvlddmkm` event written. So the fifth scan that met the edge with the longer timed run did not end in a driver reset the search recovers from, but in a reset of the whole machine, during the reconnect or right after it. What held: the candidate's journal entry was closed, so no ceiling of a freeze was left; the line `{"mem_edge":2450,"rig":"GPU-... / 617.42"}` was on disk before the machine went down, so no later run of this build goes there again; the run had saved nothing, and at logon the tray app applied the tune of before the test (`applied best: power 105 %, core +135 MHz, mem +650 MHz`). Not answered: the second run. Seen along the way and not explained: from the first step of the scan the stress load scored 5760 it/s and the bandwidth read about 7 % higher than in every run before (486.6 GB/s at +900 against about 455), while the measurement at stock five minutes earlier was the usual 5321. **(e) how the stress load meets the edge** (the same build, no search): `gao --reset`, then `gao --set-mem` and `gao --stress 3` at +2300, +2350, ... until the verdict is not `STABLE`, five times, at stock power and core +0; expected is one of two outcomes, `WRONG RESULT` every time (then the stress load can lead the copy to the edge) or `DEVICE LOST` (then it cannot). **Not run:** the test ended in (d). It goes to the edge on purpose, five times, and after (d) that is a risk for the machine that needs its owner's word first. **(f) the two-pass search** (the short copy in the scan, the long copy from four steps below the highest step that passed; see `docs/development.md`): tested on a simulated card only. On the reference card, with the edge at +2450 in its journal: `gao --optimize best` once; expected `mem: the memory gave out at +2450 in an earlier run on this card and driver; the scan stops at +2250`, no memory candidate above +2250, `mem: the scan passed +2250; reading from +2200 down with the long copy`, a few `mem +N: bandwidth ... GB/s, the long copy` lines, no `driver resets during this run`, an applied offset above +1000 and a soak that is not slower than stock. **Not run: waits for the owner's word.** **(g) what else uses the card** (2026-10-11 22:00, after the restart of (d), not elevated, the saved tune applied, nothing changed): `gao --stress 5` scored 5828 it/s and `gao --bandwidth --after-load 3` read 484.0 GB/s with its three runs within 0.0 %, against 5226 it/s and 444 to 452 GB/s with the same tune earlier that evening. The series of (a) again, 24 readings each: 64 dispatches 480.0 GB/s, 0.23 % (-0.66 to +0.18); 256 dispatches 475.2 GB/s, 0.68 %, sliding from 480 to 471 during the series; 64 dispatches again 454.0 GB/s, 3.02 % (-4.37 to +4.22), the readings jumping between 434 and 473. Read right after it, the card's use per program: Chrome 8.6 % of the video decoder (19 % at most) and 2.8 % of the 3D engine, with a YouTube video playing; nothing else above 1.1 %. So the short copy is as steady as the long one on a card that is left alone, and a video playing next to it costs about 8 % of the bandwidth and of the stress score and makes the readings jump. That is the likely source of the scatter and of the level changes of (a), and of the higher level in the run of (d) from 21:31 on. Not a controlled test: the video was not started and stopped on purpose. | ## Notes From 290de9da1ed1f936ad6a9a9b44e81f19c467f533 Mon Sep 17 00:00:00 2001 From: Rovey Date: Sun, 11 Oct 2026 22:10:35 +0200 Subject: [PATCH 10/15] docs: what was playing during the day's tests fits the two bandwidth levels Co-Authored-By: Claude Opus 5.5 --- docs/development.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/development.md b/docs/development.md index af37747..8600b1f 100644 --- a/docs/development.md +++ b/docs/development.md @@ -80,7 +80,7 @@ The applied offset is always at least one step below the confirmed edge. Profile | A confirm probe that is too hot at once confirms nothing | A confirm probe that ends `TOO HOT` keeps the clock edge: over 30 s the card may run a little hotter than over the power step's 20 s, and that is the soak's business, which lowers power. But the stress run stops at its first reading above the limit, so a card that is above it when the probe begins gives `TOO HOT` within a second, and that counted as holding too. It happened on the tester's RTX 3080 Ti (hardware check 91): after a driver reset the health probe loads the card at stock power, which on that card means 76 C against a limit of 75, and `confirm core +225` was confirmed by a probe that had tested nothing. Now such a probe counts only when the clock was loaded for `kConfirmMinS` (10 s, clearly longer than the 3 s probe that found the edge) before it got too hot. Otherwise the value is not confirmed, and the card rests (`io.rest`, the 20 s of a recovery, with the fans where the curve last put them) before the next try one step lower: without the rest every lower value would start just as hot, and heat alone would step the edge down to nothing, which is what keeping the edge on `TOO HOT` was for. The same value is not tried again; one step is the price. Not changed: the health probe still runs at stock power, and a soak that begins too hot still lowers the power limit by a step. | | A profile that promises speed keeps no result that is slower than stock | Best of my GPU ended 3.8 % slower than stock on the tester's RTX 3080 Ti (4749 against 4937 it/s, hardware check 91), with the fans at 80 % instead of 66 %, and showed it as **Done**, applied and kept at every logon. Every step had done what it is for: the card runs 83 C at stock with its fans at 100 %, the profile allows 75, so the power limit went to 65 %, and +150 MHz of core clock did not win that back. But a profile called "the best of my GPU" that leaves the card slower has not done what it says, and a result like that is not offered as one. After the soak, `optimize` compares its score with the measurement at stock that went before the run (the "before" the result screen shows, so the rule judges by the number the user sees): below `kNotSlowerScore` (0.99: one percent is what two five-minute runs differ by) the run ends as a failed one does, at stock, with nothing saved, and says what it found, why, and that Undervolt is the profile for a card that runs warm. `Objectives::faster_than_stock` marks the two profiles this holds for, Best of my GPU and Max performance. Not Quiet and Cool & efficient, which trade speed for noise and heat and say so. Not the overclock step of All in one either: it goes on to an undervolt and a fan tune, and its result is judged as a whole; whether that whole may be slower than stock is not decided. This is a backstop, not the cure: what such a card needs from these profiles is in `docs/planned.md` ("never hotter than at stock"). | | The memory search has two passes: a scan with the short copy finds where it ends, and the long copy reads the offset to keep, four steps below that | **What went wrong.** Four runs on one day applied +650, +1600, +550 and +100 MHz on the reference RTX 4070, a card that runs every game at +1500 (hardware check 94). That card has no bandwidth peak: bandwidth rises by about 0.4 % per 50 MHz step all the way to +2500 and the stress load computes its first wrong result at +2550, so the scan could only end early by chance, and where it did decided the offset. **What was measured** (hardware check 95 a, no search, nothing changed on the card). A reading is the median of three timed runs that agree within 1 %. With the timed run as it was, 32 GB or 75 ms, readings scatter by 0.4 to 1.3 % (standard deviation over 24 readings, six series), single ones by up to 4.8 %, and now and then one does not settle. With 128 GB per run it is 0.2 %. Apart from that the readings change level: for up to a minute every reading is about 1.8 % higher, at one and the same offset; the cause is not known. The app's own window, drawing every 100 or 33 ms next to the measurement, adds nothing, so nothing was built for it. **What was tried first, and what it did** (check 95 b to d). The long timed run at every step of the scan: the offset repeated (+1400 to +1500 in four runs, 4 to 5 % faster than stock), and every scan now ran to where the memory gives out and met it with the long copy. Five scans out of five lost the device there, at +2450 or +2500, a driver reset each time, with three readings a step and with one; and the fifth took the whole machine down within seconds (Kernel-Power 41, no bugcheck). In the one scan before all this that got to the edge, the short copy read a number at +2450 and +2500 and the stress load reported the wrong result at +2550, with no reset. One against five is thin, and it is all there is: the long copy is too heavy for memory at its edge, the short one was not. **The design.** One measurement cannot do both jobs, so there are two passes. *The scan* goes up as before and only has to find where it ends: a step that is not stable, or one that has lost more than `kBandwidthDrop` (5 %) against the best. It reads with the short copy (`kBwDispatches` 64), which is what the released versions take to the edge. Because those readings scatter, two things keep one of them from deciding: a reading that would end the scan is taken `kBandwidthReadings` (3) times and the middle one counts, and the best that readings are held against is the highest middle of three steps in a row, not the highest reading (the highest of fifty scattering readings is always one that came out high). 5 % is far more than the readings move by themselves; the scan is not where the offset is chosen any more, so it can afford to be coarse. *The second pass* chooses the offset, with the long copy (`MeasureBandwidth(true)`, `OptimizeIo::bandwidth_precise`): it starts `kPreciseMarginSteps` (4 steps, 200 MHz) below the highest step the scan passed (`precise_top`), goes down a step at a time until a reading is more than `kBandwidthTie` (1 %) below the best of them, and keeps the lowest offset within that 1 % (`bandwidth_peak_below`). On the reference card: the scan passes +2500, the long copy reads +2300, +2250, +2200 and +2150, and +2200 is kept; confirmed there, 70 % applied, +1500. The long copy lost the device at the highest step the stress load passed and one below it; four steps leave two to spare. The margin is kept whatever ended the scan, a drop and the driver's own range included: what is above was never tried. After a reset event in the scan the second pass is skipped, and confirming starts four steps below where it would have begun. **The edge is met once** (built before this, kept): a scan that ended in a wrong result or a reset event writes that offset to the journal with the GPU id and the driver version (`{"mem_edge":2550,"rig":" / "}`, a line without an id, so older builds skip it). A later scan on the same card and driver stops four steps below the lowest such offset, and its second pass starts where the first run's did: the margin counts from the edge, not a second time from where the scan was held. With another driver or card the line does not count. **What it costs.** A card that does have a peak keeps an offset up to 200 MHz below it when the scan ends close above the peak: the margin does not know a drop from an edge. A scan that ends for a harmless reason (readings that did not settle) loses those 200 MHz as well. The second pass adds about four steps, half a minute. **Not chosen.** The long copy everywhere: see above. The short copy everywhere, as released: the offset is chance (the +100 run). Three long readings a step, or only for a step in doubt: the reset came with both. Only remembering the edge: the first meeting can cost the machine, and once per card is once too often. The stress load probing four steps ahead of the copy: it needs the proof that the load at the edge ends in a wrong result and not in a reset of its own, and getting that proof means going to the edge on purpose. A fixed top for the scan (an offset, a share of the range, a gain over +0): there is nothing to choose the number by. **Tested** on a simulated card with the reference card's numbers (wrong result at +2550, the long copy losing the device from +2450): the long copy is never taken above +2300, in the first run and in the one after it, with short readings scattering by 1.3 % on two hundred seeds, and on forty more with level changes of 1.8 % thrown in; with the edge anywhere from +300 to +2900; and the tests fail when the margin is set to nothing. **Not tested on the card.** See the row below. | -| Open: the two-pass memory search has not run on a card | The last card test (hardware check 95 d) ended with the machine restarting, and nothing has been run on the card since. Before this is merged, its owner decides whether and when to run it there. What a run would show, and what it cannot: the journal of the reference card holds the edge at +2450 now, so a run of this build stops its scan at +2250 and reads with the long copy from +2200 down, at offsets every earlier scan passed. That tests the second pass and the result, with little risk. It does not test the first meeting with the edge with the short copy: that is what the released versions do, it went well the one time it was seen, and seeing it again means removing the edge from the journal and going there. **Still not known.** Why the long copy fails where the short one did not: the amount copied in one go, or only that five runs are more than one. Whether a card with a real bandwidth peak behaves as assumed; none has been measured through a scan (the tester's RTX 3080 Ti never got a reading to settle at +0). And one observation from the run that crashed: from the first step of its scan the stress load scored 5760 it/s and the bandwidth read about 7 % higher than in every run before (486.6 GB/s at +900 against about 455), while the measurement at stock five minutes earlier was the usual 5321. It was not the monitor going to sleep, as first suspected: it never does on that machine, and the run's log shows the change in the middle of the scan, between +350 and +550 (430 to 457 GB/s within one step, 5260 to 5750 it/s), not at its start. **What it most likely is: another program using the card, here a video playing in a browser.** Measured after the restart, with the same tune applied and nothing changed (hardware check 95 g): the short copy read 480 GB/s with a scatter of 0.23 %, as steady as the long copy was all day; minutes later, with a YouTube video playing in Chrome (8.6 % of the card's video decoder, 2.8 % of its 3D engine), the same series read 434 to 473 GB/s with a scatter of 3 %, and the stress load scores about 5300 it/s in the one state and 5750 to 5830 in the other. The low state is what every measurement of that day shows until 21:31. Not proven: nobody paused and played the video on purpose while measuring. If it holds, three things follow, and none is built. The scatter of the short copy that this design is built around (0.4 to 1.3 %) was for the larger part the video, not the copy; the design stands, it only has more room than was thought. The level changes of 1.8 % are the same thing in small. And the comparison of a result with stock is worth nothing when such a program starts or stops between the two: 8 % is far more than the 1 % by which a speed profile is judged (`kNotSlowerScore`) and than anything the efficiency search decides on. What to do about that is open: say so before a run (the README names games, MSI Afterburner and RivaTuner, not a browser with a video), or read the card's use per program (Windows keeps it, per engine) and warn or wait. | +| Open: the two-pass memory search has not run on a card | The last card test (hardware check 95 d) ended with the machine restarting, and nothing has been run on the card since. Before this is merged, its owner decides whether and when to run it there. What a run would show, and what it cannot: the journal of the reference card holds the edge at +2450 now, so a run of this build stops its scan at +2250 and reads with the long copy from +2200 down, at offsets every earlier scan passed. That tests the second pass and the result, with little risk. It does not test the first meeting with the edge with the short copy: that is what the released versions do, it went well the one time it was seen, and seeing it again means removing the edge from the journal and going there. **Still not known.** Why the long copy fails where the short one did not: the amount copied in one go, or only that five runs are more than one. Whether a card with a real bandwidth peak behaves as assumed; none has been measured through a scan (the tester's RTX 3080 Ti never got a reading to settle at +0). And one observation from the run that crashed: from the first step of its scan the stress load scored 5760 it/s and the bandwidth read about 7 % higher than in every run before (486.6 GB/s at +900 against about 455), while the measurement at stock five minutes earlier was the usual 5321. It was not the monitor going to sleep, as first suspected: it never does on that machine, and the run's log shows the change in the middle of the scan, between +350 and +550 (430 to 457 GB/s within one step, 5260 to 5750 it/s), not at its start. **What it most likely is: another program using the card, here a video playing in a browser.** Measured after the restart, with the same tune applied and nothing changed (hardware check 95 g): the short copy read 480 GB/s with a scatter of 0.23 %, as steady as the long copy was all day; minutes later, with a YouTube video playing in Chrome (8.6 % of the card's video decoder, 2.8 % of its 3D engine), the same series read 434 to 473 GB/s with a scatter of 3 %, and the stress load scores about 5300 it/s in the one state and 5750 to 5830 in the other. The low state is what every measurement of that day shows until 21:31, and it fits what was going on at the machine: videos were watched during the day's tests, and none was playing at 21:31. Not proven: nobody paused and played a video on purpose while measuring. If it holds, three things follow, and none is built. The scatter of the short copy that this design is built around (0.4 to 1.3 %) was for the larger part the video, not the copy; the design stands, it only has more room than was thought. The level changes of 1.8 % are the same thing in small. And the comparison of a result with stock is worth nothing when such a program starts or stops between the two: 8 % is far more than the 1 % by which a speed profile is judged (`kNotSlowerScore`) and than anything the efficiency search decides on. What to do about that is open: say so before a run (the README names games, MSI Afterburner and RivaTuner, not a browser with a video), or read the card's use per program (Windows keeps it, per engine) and warn or wait. | | Open: why the bandwidth measurement fails on some cards, and what then | On the tester's RTX 3080 Ti it failed at +0, and since 0.4.4 that means no memory overclock at all on such a card, though it computed right far beyond +2000 MHz. The measurement gives a number only when three runs in a row agree within 1 % (`kBwSettle`), and it throws away every run the driver's timer calls disjoint (the clock changed under it). That card was held at a power limit of 65 % and ran at 83 C when it was measured; the reference card has only ever been measured at 105 %, where the measurement itself draws 179 of 210 W. So the suspicion is the power limit, and hardware check 90 tests exactly that on the reference card. What it shows decides what is built, and nothing is built before it: **(1) runs are thrown away as disjoint, and the ones that are kept agree:** the timer is what fails, not the card. Time the copy differently (wall clock around a flush, or count a disjoint run when its neighbours agree) and keep the method. **(2) the readings themselves scatter by more than 1 %:** a peak that is found by a 1 % drop (`kBandwidthDrop`) cannot be found in that noise. Then the memory has to be measured where the card is not held back: search it before the power step, at stock power, where a probe and its measurement last five seconds; or take more readings and compare medians. **(3) it does not fail at 65 % on the reference card:** the power limit is not the cause. Then it was the heat (a card throttling at its temperature limit changes clock as well), or another program, and only the tester's `run.log` can say: since this change a measurement that never settled writes how many runs gave a reading, between which values, and how many were thrown away. In no case does the climb on stability alone come back. **Outcome of check 90: (3).** On the reference card the measurement gave a number fifteen times out of fifteen at 100, 80 and 65 %, in three or four runs, with nothing thrown away, and the bandwidth was the same at every limit. The power limit is ruled out; (1) and (2) are not built. What is left is the heat of that card, another program, or a timer query that failed, and the tester's `run.log` is the next thing that can tell them apart. **The tester's run (check 91): it is (2), but only inside a run.** Twenty readings between 763.9 and 821.2 GB/s, none thrown away, three seconds after the load began; after thirty seconds of load the same card read within 0.2 %. The suspicion now is the moment, not the card: the measurement is taken while the power limiter is still catching up. If `gao --bandwidth --after-load 3` scatters on that card and `--after-load 30` does not, that is it, and the fix is to measure when the card has settled (keep taking runs for some seconds instead of twenty runs), not to change the method. | | Every line of a run is also in `run.log` | `logged()` wraps the log hook of both front ends: a line is appended and flushed before it is shown, so the step a machine freezes in is the last line on disk. One file in the state folder, moved aside at 1 MB (`run.log.old`). Only elevated and only after `ensure_app_dir` has verified the folder: an elevated process appends to nothing a user could have put there. | | A run does not start while the NVIDIA app's Automatic Tuning is on | Switched on, a service of the driver scans the card when it sees fit: it loads it, changes its clocks, and starts again by itself (hardware check 89: a scan that was stopped started again 14 ms later, and after a restart of the service it resumed within two seconds). A run next to that measures nothing it can trust, so it is refused, like a run next to Afterburner, and with the same weight: a courtesy, not a guard. Read from where that service keeps the switch, `%ProgramData%\NVIDIA Corporation\nvtopps\nvtopps.db3`, table `ocScannerSettings`, fields `backgroundScanEnabled` and `grdOcEnabled`, joined to `TelemetryGpuCfg` so that only a card that is in the PC counts (a row of an earlier card can never be switched off again). That this is the source and not a copy of it was measured, not assumed: with the field set the service took the scan up again after every restart, with it cleared it did not, while the NVIDIA app itself showed the switch as off. Nothing documents the file. So: opened read-only and immutable (no lock, the service never waits for us), through `winsqlite3.dll` from System32 (no dependency added), and everything that is not exactly as expected reads as "unknown", which starts the run as before. The registry value the app sets next to it (`HKLM\SOFTWARE\NVIDIA Corporation\Global\NvApp\OC\`) changes at the same moment but belongs to the app, not to the service, and is not used. **The overlay is not checked:** no setting for it was found, its processes run whether it is open or not, and the one value that moved (`IsShadowPlayEnabled`) followed the service being up, not a switch. NVAPI was not tried: an undocumented call is no better than an undocumented file, and it cannot be compared with anything. | From b65befe867189c918f9bb76bae3bbd6f62a28f3c Mon Sep 17 00:00:00 2001 From: Rovey Date: Sun, 11 Oct 2026 22:16:08 +0200 Subject: [PATCH 11/15] docs: hardware check 95 (g), the bandwidth reading against the card's use per program Co-Authored-By: Claude Opus 5.5 --- docs/development.md | 2 +- docs/hardware-checks.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/development.md b/docs/development.md index 8600b1f..e129348 100644 --- a/docs/development.md +++ b/docs/development.md @@ -80,7 +80,7 @@ The applied offset is always at least one step below the confirmed edge. Profile | A confirm probe that is too hot at once confirms nothing | A confirm probe that ends `TOO HOT` keeps the clock edge: over 30 s the card may run a little hotter than over the power step's 20 s, and that is the soak's business, which lowers power. But the stress run stops at its first reading above the limit, so a card that is above it when the probe begins gives `TOO HOT` within a second, and that counted as holding too. It happened on the tester's RTX 3080 Ti (hardware check 91): after a driver reset the health probe loads the card at stock power, which on that card means 76 C against a limit of 75, and `confirm core +225` was confirmed by a probe that had tested nothing. Now such a probe counts only when the clock was loaded for `kConfirmMinS` (10 s, clearly longer than the 3 s probe that found the edge) before it got too hot. Otherwise the value is not confirmed, and the card rests (`io.rest`, the 20 s of a recovery, with the fans where the curve last put them) before the next try one step lower: without the rest every lower value would start just as hot, and heat alone would step the edge down to nothing, which is what keeping the edge on `TOO HOT` was for. The same value is not tried again; one step is the price. Not changed: the health probe still runs at stock power, and a soak that begins too hot still lowers the power limit by a step. | | A profile that promises speed keeps no result that is slower than stock | Best of my GPU ended 3.8 % slower than stock on the tester's RTX 3080 Ti (4749 against 4937 it/s, hardware check 91), with the fans at 80 % instead of 66 %, and showed it as **Done**, applied and kept at every logon. Every step had done what it is for: the card runs 83 C at stock with its fans at 100 %, the profile allows 75, so the power limit went to 65 %, and +150 MHz of core clock did not win that back. But a profile called "the best of my GPU" that leaves the card slower has not done what it says, and a result like that is not offered as one. After the soak, `optimize` compares its score with the measurement at stock that went before the run (the "before" the result screen shows, so the rule judges by the number the user sees): below `kNotSlowerScore` (0.99: one percent is what two five-minute runs differ by) the run ends as a failed one does, at stock, with nothing saved, and says what it found, why, and that Undervolt is the profile for a card that runs warm. `Objectives::faster_than_stock` marks the two profiles this holds for, Best of my GPU and Max performance. Not Quiet and Cool & efficient, which trade speed for noise and heat and say so. Not the overclock step of All in one either: it goes on to an undervolt and a fan tune, and its result is judged as a whole; whether that whole may be slower than stock is not decided. This is a backstop, not the cure: what such a card needs from these profiles is in `docs/planned.md` ("never hotter than at stock"). | | The memory search has two passes: a scan with the short copy finds where it ends, and the long copy reads the offset to keep, four steps below that | **What went wrong.** Four runs on one day applied +650, +1600, +550 and +100 MHz on the reference RTX 4070, a card that runs every game at +1500 (hardware check 94). That card has no bandwidth peak: bandwidth rises by about 0.4 % per 50 MHz step all the way to +2500 and the stress load computes its first wrong result at +2550, so the scan could only end early by chance, and where it did decided the offset. **What was measured** (hardware check 95 a, no search, nothing changed on the card). A reading is the median of three timed runs that agree within 1 %. With the timed run as it was, 32 GB or 75 ms, readings scatter by 0.4 to 1.3 % (standard deviation over 24 readings, six series), single ones by up to 4.8 %, and now and then one does not settle. With 128 GB per run it is 0.2 %. Apart from that the readings change level: for up to a minute every reading is about 1.8 % higher, at one and the same offset; the cause is not known. The app's own window, drawing every 100 or 33 ms next to the measurement, adds nothing, so nothing was built for it. **What was tried first, and what it did** (check 95 b to d). The long timed run at every step of the scan: the offset repeated (+1400 to +1500 in four runs, 4 to 5 % faster than stock), and every scan now ran to where the memory gives out and met it with the long copy. Five scans out of five lost the device there, at +2450 or +2500, a driver reset each time, with three readings a step and with one; and the fifth took the whole machine down within seconds (Kernel-Power 41, no bugcheck). In the one scan before all this that got to the edge, the short copy read a number at +2450 and +2500 and the stress load reported the wrong result at +2550, with no reset. One against five is thin, and it is all there is: the long copy is too heavy for memory at its edge, the short one was not. **The design.** One measurement cannot do both jobs, so there are two passes. *The scan* goes up as before and only has to find where it ends: a step that is not stable, or one that has lost more than `kBandwidthDrop` (5 %) against the best. It reads with the short copy (`kBwDispatches` 64), which is what the released versions take to the edge. Because those readings scatter, two things keep one of them from deciding: a reading that would end the scan is taken `kBandwidthReadings` (3) times and the middle one counts, and the best that readings are held against is the highest middle of three steps in a row, not the highest reading (the highest of fifty scattering readings is always one that came out high). 5 % is far more than the readings move by themselves; the scan is not where the offset is chosen any more, so it can afford to be coarse. *The second pass* chooses the offset, with the long copy (`MeasureBandwidth(true)`, `OptimizeIo::bandwidth_precise`): it starts `kPreciseMarginSteps` (4 steps, 200 MHz) below the highest step the scan passed (`precise_top`), goes down a step at a time until a reading is more than `kBandwidthTie` (1 %) below the best of them, and keeps the lowest offset within that 1 % (`bandwidth_peak_below`). On the reference card: the scan passes +2500, the long copy reads +2300, +2250, +2200 and +2150, and +2200 is kept; confirmed there, 70 % applied, +1500. The long copy lost the device at the highest step the stress load passed and one below it; four steps leave two to spare. The margin is kept whatever ended the scan, a drop and the driver's own range included: what is above was never tried. After a reset event in the scan the second pass is skipped, and confirming starts four steps below where it would have begun. **The edge is met once** (built before this, kept): a scan that ended in a wrong result or a reset event writes that offset to the journal with the GPU id and the driver version (`{"mem_edge":2550,"rig":" / "}`, a line without an id, so older builds skip it). A later scan on the same card and driver stops four steps below the lowest such offset, and its second pass starts where the first run's did: the margin counts from the edge, not a second time from where the scan was held. With another driver or card the line does not count. **What it costs.** A card that does have a peak keeps an offset up to 200 MHz below it when the scan ends close above the peak: the margin does not know a drop from an edge. A scan that ends for a harmless reason (readings that did not settle) loses those 200 MHz as well. The second pass adds about four steps, half a minute. **Not chosen.** The long copy everywhere: see above. The short copy everywhere, as released: the offset is chance (the +100 run). Three long readings a step, or only for a step in doubt: the reset came with both. Only remembering the edge: the first meeting can cost the machine, and once per card is once too often. The stress load probing four steps ahead of the copy: it needs the proof that the load at the edge ends in a wrong result and not in a reset of its own, and getting that proof means going to the edge on purpose. A fixed top for the scan (an offset, a share of the range, a gain over +0): there is nothing to choose the number by. **Tested** on a simulated card with the reference card's numbers (wrong result at +2550, the long copy losing the device from +2450): the long copy is never taken above +2300, in the first run and in the one after it, with short readings scattering by 1.3 % on two hundred seeds, and on forty more with level changes of 1.8 % thrown in; with the edge anywhere from +300 to +2900; and the tests fail when the margin is set to nothing. **Not tested on the card.** See the row below. | -| Open: the two-pass memory search has not run on a card | The last card test (hardware check 95 d) ended with the machine restarting, and nothing has been run on the card since. Before this is merged, its owner decides whether and when to run it there. What a run would show, and what it cannot: the journal of the reference card holds the edge at +2450 now, so a run of this build stops its scan at +2250 and reads with the long copy from +2200 down, at offsets every earlier scan passed. That tests the second pass and the result, with little risk. It does not test the first meeting with the edge with the short copy: that is what the released versions do, it went well the one time it was seen, and seeing it again means removing the edge from the journal and going there. **Still not known.** Why the long copy fails where the short one did not: the amount copied in one go, or only that five runs are more than one. Whether a card with a real bandwidth peak behaves as assumed; none has been measured through a scan (the tester's RTX 3080 Ti never got a reading to settle at +0). And one observation from the run that crashed: from the first step of its scan the stress load scored 5760 it/s and the bandwidth read about 7 % higher than in every run before (486.6 GB/s at +900 against about 455), while the measurement at stock five minutes earlier was the usual 5321. It was not the monitor going to sleep, as first suspected: it never does on that machine, and the run's log shows the change in the middle of the scan, between +350 and +550 (430 to 457 GB/s within one step, 5260 to 5750 it/s), not at its start. **What it most likely is: another program using the card, here a video playing in a browser.** Measured after the restart, with the same tune applied and nothing changed (hardware check 95 g): the short copy read 480 GB/s with a scatter of 0.23 %, as steady as the long copy was all day; minutes later, with a YouTube video playing in Chrome (8.6 % of the card's video decoder, 2.8 % of its 3D engine), the same series read 434 to 473 GB/s with a scatter of 3 %, and the stress load scores about 5300 it/s in the one state and 5750 to 5830 in the other. The low state is what every measurement of that day shows until 21:31, and it fits what was going on at the machine: videos were watched during the day's tests, and none was playing at 21:31. Not proven: nobody paused and played a video on purpose while measuring. If it holds, three things follow, and none is built. The scatter of the short copy that this design is built around (0.4 to 1.3 %) was for the larger part the video, not the copy; the design stands, it only has more room than was thought. The level changes of 1.8 % are the same thing in small. And the comparison of a result with stock is worth nothing when such a program starts or stops between the two: 8 % is far more than the 1 % by which a speed profile is judged (`kNotSlowerScore`) and than anything the efficiency search decides on. What to do about that is open: say so before a run (the README names games, MSI Afterburner and RivaTuner, not a browser with a video), or read the card's use per program (Windows keeps it, per engine) and warn or wait. | +| Open: the two-pass memory search has not run on a card | The last card test (hardware check 95 d) ended with the machine restarting, and nothing has been run on the card since. Before this is merged, its owner decides whether and when to run it there. What a run would show, and what it cannot: the journal of the reference card holds the edge at +2450 now, so a run of this build stops its scan at +2250 and reads with the long copy from +2200 down, at offsets every earlier scan passed. That tests the second pass and the result, with little risk. It does not test the first meeting with the edge with the short copy: that is what the released versions do, it went well the one time it was seen, and seeing it again means removing the edge from the journal and going there. **Still not known.** Why the long copy fails where the short one did not: the amount copied in one go, or only that five runs are more than one. Whether a card with a real bandwidth peak behaves as assumed; none has been measured through a scan (the tester's RTX 3080 Ti never got a reading to settle at +0). And one observation from the run that crashed: from the first step of its scan the stress load scored 5760 it/s and the bandwidth read about 7 % higher than in every run before (486.6 GB/s at +900 against about 455), while the measurement at stock five minutes earlier was the usual 5321. It was not the monitor going to sleep, as first suspected: it never does on that machine, and the run's log shows the change in the middle of the scan, between +350 and +550 (430 to 457 GB/s within one step, 5260 to 5750 it/s), not at its start. **What it is: another program using the card.** Measured after the restart, with the same tune applied and nothing changed (hardware check 95 g): the short copy read 480 GB/s with a scatter of 0.23 %, as steady as the long copy was all day; minutes later, with a YouTube video playing in Chrome (8.6 % of the card's video decoder, 2.8 % of its 3D engine), the same series read 434 to 473 GB/s with a scatter of 3 %, and the stress load scores about 5300 it/s in the one state and 5750 to 5830 in the other. A second series, with the card's use per program sampled next to every reading, shows it to the second: 483 GB/s with no other program on the card, about 470 while Chrome decodes video, 444 to 455 while Chrome also draws, and back to 483 on the reading after Chrome stopped. So another program using the card moves the reading by 3 to 8 %, at once. **What it was on the day of the tests is not known.** The low state is what every measurement of that day shows until 21:31, but hardly any video was watched during those tests, so it was something else that used the card all day and stopped then; after the restart there is no finding out. If it holds, three things follow, and none is built. The scatter of the short copy that this design is built around (0.4 to 1.3 %) was for the larger part the video, not the copy; the design stands, it only has more room than was thought. The level changes of 1.8 % are the same thing in small. And the comparison of a result with stock is worth nothing when such a program starts or stops between the two: 8 % is far more than the 1 % by which a speed profile is judged (`kNotSlowerScore`) and than anything the efficiency search decides on. What to do about that is open: say so before a run (the README names games, MSI Afterburner and RivaTuner, not a browser with a video), or read the card's use per program (Windows keeps it, per engine) and warn or wait. | | Open: why the bandwidth measurement fails on some cards, and what then | On the tester's RTX 3080 Ti it failed at +0, and since 0.4.4 that means no memory overclock at all on such a card, though it computed right far beyond +2000 MHz. The measurement gives a number only when three runs in a row agree within 1 % (`kBwSettle`), and it throws away every run the driver's timer calls disjoint (the clock changed under it). That card was held at a power limit of 65 % and ran at 83 C when it was measured; the reference card has only ever been measured at 105 %, where the measurement itself draws 179 of 210 W. So the suspicion is the power limit, and hardware check 90 tests exactly that on the reference card. What it shows decides what is built, and nothing is built before it: **(1) runs are thrown away as disjoint, and the ones that are kept agree:** the timer is what fails, not the card. Time the copy differently (wall clock around a flush, or count a disjoint run when its neighbours agree) and keep the method. **(2) the readings themselves scatter by more than 1 %:** a peak that is found by a 1 % drop (`kBandwidthDrop`) cannot be found in that noise. Then the memory has to be measured where the card is not held back: search it before the power step, at stock power, where a probe and its measurement last five seconds; or take more readings and compare medians. **(3) it does not fail at 65 % on the reference card:** the power limit is not the cause. Then it was the heat (a card throttling at its temperature limit changes clock as well), or another program, and only the tester's `run.log` can say: since this change a measurement that never settled writes how many runs gave a reading, between which values, and how many were thrown away. In no case does the climb on stability alone come back. **Outcome of check 90: (3).** On the reference card the measurement gave a number fifteen times out of fifteen at 100, 80 and 65 %, in three or four runs, with nothing thrown away, and the bandwidth was the same at every limit. The power limit is ruled out; (1) and (2) are not built. What is left is the heat of that card, another program, or a timer query that failed, and the tester's `run.log` is the next thing that can tell them apart. **The tester's run (check 91): it is (2), but only inside a run.** Twenty readings between 763.9 and 821.2 GB/s, none thrown away, three seconds after the load began; after thirty seconds of load the same card read within 0.2 %. The suspicion now is the moment, not the card: the measurement is taken while the power limiter is still catching up. If `gao --bandwidth --after-load 3` scatters on that card and `--after-load 30` does not, that is it, and the fix is to measure when the card has settled (keep taking runs for some seconds instead of twenty runs), not to change the method. | | Every line of a run is also in `run.log` | `logged()` wraps the log hook of both front ends: a line is appended and flushed before it is shown, so the step a machine freezes in is the last line on disk. One file in the state folder, moved aside at 1 MB (`run.log.old`). Only elevated and only after `ensure_app_dir` has verified the folder: an elevated process appends to nothing a user could have put there. | | A run does not start while the NVIDIA app's Automatic Tuning is on | Switched on, a service of the driver scans the card when it sees fit: it loads it, changes its clocks, and starts again by itself (hardware check 89: a scan that was stopped started again 14 ms later, and after a restart of the service it resumed within two seconds). A run next to that measures nothing it can trust, so it is refused, like a run next to Afterburner, and with the same weight: a courtesy, not a guard. Read from where that service keeps the switch, `%ProgramData%\NVIDIA Corporation\nvtopps\nvtopps.db3`, table `ocScannerSettings`, fields `backgroundScanEnabled` and `grdOcEnabled`, joined to `TelemetryGpuCfg` so that only a card that is in the PC counts (a row of an earlier card can never be switched off again). That this is the source and not a copy of it was measured, not assumed: with the field set the service took the scan up again after every restart, with it cleared it did not, while the NVIDIA app itself showed the switch as off. Nothing documents the file. So: opened read-only and immutable (no lock, the service never waits for us), through `winsqlite3.dll` from System32 (no dependency added), and everything that is not exactly as expected reads as "unknown", which starts the run as before. The registry value the app sets next to it (`HKLM\SOFTWARE\NVIDIA Corporation\Global\NvApp\OC\`) changes at the same moment but belongs to the app, not to the service, and is not used. **The overlay is not checked:** no setting for it was found, its processes run whether it is open or not, and the one value that moved (`IsShadowPlayEnabled`) followed the service being up, not a switch. NVAPI was not tried: an undocumented call is no better than an undocumented file, and it cannot be compared with anything. | diff --git a/docs/hardware-checks.md b/docs/hardware-checks.md index 2e21dce..fc91f8e 100644 --- a/docs/hardware-checks.md +++ b/docs/hardware-checks.md @@ -122,7 +122,7 @@ root; adjust the path for a Debug build. | 92 | The undervolt profile starts with the measurement at stock every run uses, and a hot card is not refused | Elevated, the tray app exited, a copy of `gao.json` kept. (a) `gao --undervolt` to the end; then `gao --status`. (b) The same with a test build whose stock limit is lowered to a temperature the reference card passes under load (63 C; it reaches 67). Put the copy of `gao.json` back. (c) Only on a card that is above the profile's limit at stock with the fans at 100 % (the tester's RTX 3080 Ti: 82 to 83 C): the **Undervolt** profile from the window, to the end, and its `run.log` | yes | (a) `the card at stock, five minutes`, a `stock: STABLE ...` line with the driver's fan speed, then `baseline: the measurement at stock`, and no second five minutes at stock: the first candidate follows at once; no `candidates are stopped at` line; `RESULT:`; `before:` shows 300 s and the fan speed the driver chose. (b) `stock: TOO HOT ... peak=64 C`, the line that the measurement stopped there, the cool-down, then `baseline: the measurement at stock` and a search that runs to `RESULT:` with `before:` showing the shortened time; no `candidates are stopped at` line, because 64 C is under the profile's 75. (c) no `not stable at stock`; `the card reached 86 C at stock: candidates are stopped at 85 C, not at the profile's 75 C`; candidates that end `STABLE` or `WRONG RESULT`, none `TOO HOT`; a result that is cooler and draws less than stock at the same clock | First run: **not as expected**, and that found a fault (2026-10-11, reference RTX 4070, driver 617.42, a test build of d1c99e9, started elevated by a script). The first halves held: (a) `stock: STABLE ... peak=65 C fan=70 %`, then `baseline: the measurement at stock` four seconds later and no second five minutes; (b) `stock: TOO HOT ... peak=64 C`, stopped after 34 s, the run went on, no `candidates are stopped at` line. But both runs then ended with `the card held no undervolt: the first step below its own voltage already failed`: their first candidates read `1070 mV (+15 MHz): SLOW ... clock=2323 MHz` and `1050 mV (+15 MHz): SLOW ... clock=2411 MHz`, on a card that ran 2791 MHz in the measurement just before. The search's load had not run yet when that candidate began (see the decision "With its reference handed over, the undervolt search warms its load up before the first candidate"). Second run, with the warm-up: **pass** for (a) and (b) (same day, a test build of 0733695). (a) `stock: STABLE score=5426 it/s clock=2788 MHz peak=66 C fan=70 %`, four seconds later `baseline: the measurement at stock`, `warming the load up: 5 s at stock`, `target: 2790 MHz, which the card reaches at 1090 mV by itself`, and the first candidate `1085 mV (+15 MHz): STABLE ... clock=2775 MHz`; 26 candidates, none `SLOW` or `TOO HOT`, the edge at 950 mV, `RESULT: 2790 MHz at 990 mV instead of 1090 mV`; `before:` 5426 it/s, 189 W, fan 70 % (300 s), `after:` 5325 it/s, 149 W, fan 53 % (300 s). (b) `stock: TOO HOT ... peak=64 C`, stopped after 74 s, the run went on, the same warm-up and a search to the same result (990 mV, edge 950 mV); `before:` shows 75 s; no `candidates are stopped at` line. The saved tune of before the test was put back and applied. (c) not run. | | 93 | A speed profile that would end slower than stock keeps nothing | Only on a card that is warmer at stock than the profile allows (the tester's RTX 3080 Ti; the reference card ends faster than stock with every speed profile): **Best of my GPU** from the window, to the end; the result screen, the dashboard and `run.log` | yes | the run goes through all its steps and its soak; then **Not applied** with `the result is slower than stock: N it/s against M (-X %)`, the power limit it had to go down to, and the pointer to Undervolt; `The card is back at stock.`; the dashboard shows stock and the saved tune of before the run, if there was one; `run.log` ends with `stopped: the result is slower than stock ... -- card restored to stock` | Not run yet. On the reference card: nothing to see, by design. | | 94 | All in one, with the undervolt step searching for the first time | Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize all` to the end with a build that has the warm-up of the undervolt step; then `gao --status`; put the copy back | yes | five steps; in step 3 a `warming the load up` line, a first candidate that is not `SLOW`, and a descent; steps 4 and 5 run on the tune of steps 2 and 3; `RESULT:` | **Not as expected at step 4**, and that found a fault (2026-10-11, reference RTX 4070, driver 617.42, a test build of 0733695, unattended). Step 1: `stock: STABLE score=5325 it/s peak=65 C fan=70 % power=186 W`. Step 2: power 105 %, `mem: bandwidth peak +2300, confirmed +2300, applying +1600`, `core: highest stable +210, confirmed +210, applying +135`, soak `STABLE score=5568 it/s peak=70 C power=197 W`. Step 3, as wanted: `warming the load up: 5 s with the overclock`, `target: 2865 MHz, which the card reaches at 1050 mV with the overclock`, nine candidates `STABLE` from 1045 mV down, `WRONG RESULT` at 990 mV, confirmed at 1010 mV after two tries, and `applying 1050 mV (+135 MHz at that point)`: the margin left no raise above the overclock, so the curve was cut off, as in the runs before the warm-up; soak `STABLE score=5434 it/s peak=67 C power=176 W`. Step 4 did not begin: `the tune could not be applied for the fan tune: the undervolt was not applied: the point already runs more than the target -- card at stock`. The saved tune of before the test was put back and applied. Cause and fix: the decision "A flat top goes back on a card that is cooler than when it was found". Second run, with the `gao.exe` of the released v0.4.6: **pass, all five steps** (same day, started elevated by a script). Step 1: `stock: STABLE score=5309 it/s peak=66 C fan=70 % power=185 W`. Step 2: power 105 %, `mem: bandwidth peak +850, confirmed +850, applying +550`, `core: highest stable +210, confirmed +210, applying +135`, soak `STABLE score=5568 it/s`. Step 3: `target: 2850 MHz, which the card reaches at 1045 mV with the overclock`, `undervolt: lowest stable 975 mV, confirmed 985 mV, applying 1025 mV (+165 MHz at that point)`, soak `STABLE score=5363 it/s peak=66 C power=163 W`. Step 4: the tune went back on the card, `fan 60 %: 63 C`, `fan 55 %: 65 C`, `fan 52 %: 67 C`, `fan tune: 52 % holds 67 C`. Step 5: the final test ran its 300 s. `RESULT: power 105 %, core +135 MHz, mem +550 MHz, 2850 MHz at 1025 mV, fans 52 % at 67 C`; `stock: 5309 it/s, 185 W, fan 70 %, 66 C`, `now: 5353 it/s, 166 W, fan 52 %, 70 C`: `+0.8 % score and -10.3 % power against stock`; `undervolt: applied`. The saved tune of before the test was put back. **What this run did not show:** its undervolt had a raise of its own (+165 over an offset of +135), so the anchor was raised and not merely cut off, and the case of the first run (an anchor that reads a step above the target on a cooler card) did not come up. That fix has still only its tests. **The memory offset differs a lot from run to run on this card, and the two logs say why:** +1600 in the first run (bandwidth peak +2300), +550 in the second (peak +850), +650 in the run that saved the tune in use that morning (the journal shows its scan ending at +1000 too). The card was in the same state each time (68 C, power 105 %, a reading every three to four seconds). In both logged scans the bandwidth rises in a straight line, 0.36 and 0.46 % per 50 MHz step, from 420 GB/s at +0 to 509 GB/s at +2450; the first run went on to `WRONG RESULT` at +2550 without the bandwidth ever turning down. The readings scatter around that line with a standard deviation of 0.4 and 0.7 %, single ones by more than 1 %. The scan ends at a reading more than 1 % below the best so far when a second reading agrees, and in the second run that was chance: +950 read high (462.8 GB/s, where the line is at about 458), +1000 read 456.6 twice, 1.3 % below it. In the first run the one low reading (+600) was followed by a higher one, and the scan ran to the first wrong result. So on this card there is no bandwidth peak before the errors, and where the scan stops is decided by noise. | -| 95 | The memory scan no longer ends by chance | (a) Not elevated, nothing changed on the card: a measuring build takes 24 bandwidth readings in one process, with 3 s of the stress load before each as a scan has, at 64, 128 and 256 dispatches per timed run; then the same with the app's window open next to it, drawing a frame every 100 ms and every 33 ms. (b) Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize best` twice in a row with the build of the change, then the copy put back | (a) no, (b) yes | (a) how much a reading scatters, and on what that depends. (b) in both runs every `mem +N: bandwidth` line shows three readings that agree within about 1 %; the scan does not end before the first `WRONG RESULT`; the two `mem: bandwidth peak` values are within 300 MHz of each other and the applied offset is above +1000; the soak is not slower than at stock | **(a) measured** (2026-10-11, reference RTX 4070, driver 617.42, the saved tune applied: power 105 %, core +135, memory +650; standard deviation over the 24 readings of a series, and the lowest and highest against the mean). 64 dispatches, no window: 0.44 % (-0.55 to +1.07, one reading did not settle), 0.66 % (-1.58 to +1.49, one did not settle), 1.31 % (-4.83 to +1.27), 0.86 % (-1.44 to +2.05). 128 dispatches, no window: 0.43 % (-1.10 to +0.79). 256 dispatches, no window: 0.22 % (-0.43 to +0.42) and 0.60 % (-0.80 to +1.37; the last five readings of this series were 451 to 453 GB/s after nineteen of 444 to 449). With the window drawing every 100 ms: 64 dispatches 1.13 % (the first six readings 452 to 453 GB/s, the rest 439 to 451), 256 dispatches 0.24 % (-0.47 to +0.61). With the window drawing every 33 ms: 64 dispatches 0.58 %, 256 dispatches 0.25 % (-0.35 to +0.51). A reading took 6.5, 5.3 and 3.9 timed runs to settle at 64, 128 and 256 dispatches. So: the longer timed run takes the scatter from 0.4 to 1.3 % down to 0.2 %, the window's drawing adds nothing, and apart from the scatter the readings sit about 1.8 % higher for up to a minute at a time (the stretch at the end of the second 256 series ran on into the next series; the last 64 series went back and forth between 444 and 452 GB/s). What sets that level is not known. **(b) the offset is the same twice; the scan ends in a driver reset twice** (2026-10-11, same card and driver, a test build of 2764609, unattended, 40 minutes for the two runs). Run 1: `stock: STABLE score=5311 it/s peak=66 C fan=70 % power=185 W`; 49 steps from `mem +0: bandwidth 422.7 GB/s (422.7 422.6 422.7)` to `mem +2400: bandwidth 501.4 GB/s (501.3 502.3 501.4)`; at +2450 the candidate was `STABLE` and the measurement after it lost the device: `mem +2450: the bandwidth measurement failed; treated as a driver reset`, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, 20 s of rest, `health: STABLE`, `confirm core +0 / mem +2050: STABLE`, `mem: bandwidth peak +2250, confirmed +2050, applying +1400`; `core: highest stable +210, confirmed +210, applying +135`; soak `STABLE score=5536 it/s peak=69 C power=195 W`; `driver resets during this run: 1`. Run 2: stock 5312 it/s; 50 steps from 421.8 GB/s at +0 to 502.8 at +2450; the same failure at +2500; `mem: bandwidth peak +2300, confirmed +2100, applying +1450`; core +135; soak `STABLE score=5555 it/s peak=69 C power=196 W`; one driver reset. The three readings of a step differed by 0.4 % on average and by 1.2 % at most; no step was more than 0.7 % below the best before it, so neither scan came near ending early, by the old rule or the new; every step had its three readings. The scan took 5 min 25 s (about 6.5 s a step; about 3 min before the change). As expected: the two peaks are 50 MHz apart, the applied offsets are +1400 and +1450, and both results are 4 % faster than stock. **Not as expected: neither scan reached a `WRONG RESULT`.** Both ended one and two steps before +2550 with the device lost during the bandwidth copy; Windows logged it both times (System log, `nvlddmkm` events 14 and 153 at 19:51:22 and 20:11:27), and the screen goes black for a moment. In the run of check 94 that got this far, the measurements at +2450 and +2500 gave a number and the stress load reported the first wrong result at +2550, without a reset. A step now copies about seven times as much as it did (three readings of about four runs of 128 GB, against one reading of about six runs of 32 GB), and the copy checks nothing, so where the memory starts to fail it is the driver that gives out, not a verdict. Two runs against one: likely the cause, not proven. The search handles it as it should (rest, health probe, confirm four steps lower, the core is still searched), but a driver reset in every run on a card like this is not what the change was for. The saved tune of before the test was put back and applied (`applied best: power 105 %, core +135 MHz, mem +650 MHz`), the tray app restarted, Defender reported nothing. **(c) the same two runs with one reading a step, three only for a step in doubt** (2026-10-11 20:26 to 21:03, a test build of 71e7c2b, unattended): **the reset is still there.** Run 1: stock 5328 it/s; 50 steps from `mem +0: bandwidth 420.4 GB/s` to `mem +2450: bandwidth 502.4 GB/s`; at +2500 the candidate was `STABLE` and then `bandwidth measurement failed: the timing of the copy did not come back` ten seconds later (`nvlddmkm` event 14 at 20:37:07); `mem: bandwidth peak +2200, confirmed +2000, applying +1400`; core +135; soak `STABLE score=5572 it/s peak=70 C power=197 W`; one driver reset. Run 2: stock 5311 it/s; 421.9 GB/s at +0 to 508.0 at +2450; at +2500 `the device was lost during the copy failed (hr=0x887A0006)`, the device hung (event 14 at 20:55:29); `mem: bandwidth peak +2400, confirmed +2200, applying +1500`; core +135; soak `STABLE score=5571 it/s peak=70 C power=196 W`; one driver reset. No step was in doubt in either run (the lowest was 0.75 % below the best before it), so every step was read once; the scan took 3 min 33 s. The two peaks are 200 MHz apart: with one reading a step, which offsets count as within 1 % of the best moves by a few steps. Applied +1400 and +1500, both 5 % faster than stock. So the reset does not come from reading a step three times. Four runs out of four with the longer timed run ended this way, at +2450 or +2500. Not merged; the design for a scan that does not end like this is the open row in `docs/development.md`. The saved tune was put back and applied again (core +135, memory +650), the tray app restarted, Defender reported nothing. **(d) the edge is remembered** (a test build of 7c215c4; the same two runs): expected is that the first scan ends as before and logs nothing new, the journal gets one `mem_edge` line with the GPU id and the driver version, and the second run logs `mem: the memory gave out at +N in an earlier run on this card and driver; the scan stops at +(N-200)`, has no `driver resets during this run` line, applies more than +1000 and is not slower than stock. **Run 1 took the machine down; the test ended there** (2026-10-11 21:23, unattended). Stock 5321 it/s, power 105 %, then the scan: 49 steps from +0 to `mem +2400: bandwidth 539.3 GB/s`, two of them read three times (+1200 and +1250). At +2450 the candidate was `STABLE` and the copy after it failed as in the four runs before: `bandwidth measurement failed: the copy failed (hr=0x887A0007): the device was lost` at 21:35:05, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, `the driver was reset; reconnecting`. That is the last line. Windows restarted by itself within fifteen seconds: Kernel-Power event 41 at 21:35:23, `The previous system shutdown at 21:34:57 was unexpected`, no bugcheck, no dump, no `nvlddmkm` event written. So the fifth scan that met the edge with the longer timed run did not end in a driver reset the search recovers from, but in a reset of the whole machine, during the reconnect or right after it. What held: the candidate's journal entry was closed, so no ceiling of a freeze was left; the line `{"mem_edge":2450,"rig":"GPU-... / 617.42"}` was on disk before the machine went down, so no later run of this build goes there again; the run had saved nothing, and at logon the tray app applied the tune of before the test (`applied best: power 105 %, core +135 MHz, mem +650 MHz`). Not answered: the second run. Seen along the way and not explained: from the first step of the scan the stress load scored 5760 it/s and the bandwidth read about 7 % higher than in every run before (486.6 GB/s at +900 against about 455), while the measurement at stock five minutes earlier was the usual 5321. **(e) how the stress load meets the edge** (the same build, no search): `gao --reset`, then `gao --set-mem` and `gao --stress 3` at +2300, +2350, ... until the verdict is not `STABLE`, five times, at stock power and core +0; expected is one of two outcomes, `WRONG RESULT` every time (then the stress load can lead the copy to the edge) or `DEVICE LOST` (then it cannot). **Not run:** the test ended in (d). It goes to the edge on purpose, five times, and after (d) that is a risk for the machine that needs its owner's word first. **(f) the two-pass search** (the short copy in the scan, the long copy from four steps below the highest step that passed; see `docs/development.md`): tested on a simulated card only. On the reference card, with the edge at +2450 in its journal: `gao --optimize best` once; expected `mem: the memory gave out at +2450 in an earlier run on this card and driver; the scan stops at +2250`, no memory candidate above +2250, `mem: the scan passed +2250; reading from +2200 down with the long copy`, a few `mem +N: bandwidth ... GB/s, the long copy` lines, no `driver resets during this run`, an applied offset above +1000 and a soak that is not slower than stock. **Not run: waits for the owner's word.** **(g) what else uses the card** (2026-10-11 22:00, after the restart of (d), not elevated, the saved tune applied, nothing changed): `gao --stress 5` scored 5828 it/s and `gao --bandwidth --after-load 3` read 484.0 GB/s with its three runs within 0.0 %, against 5226 it/s and 444 to 452 GB/s with the same tune earlier that evening. The series of (a) again, 24 readings each: 64 dispatches 480.0 GB/s, 0.23 % (-0.66 to +0.18); 256 dispatches 475.2 GB/s, 0.68 %, sliding from 480 to 471 during the series; 64 dispatches again 454.0 GB/s, 3.02 % (-4.37 to +4.22), the readings jumping between 434 and 473. Read right after it, the card's use per program: Chrome 8.6 % of the video decoder (19 % at most) and 2.8 % of the 3D engine, with a YouTube video playing; nothing else above 1.1 %. So the short copy is as steady as the long one on a card that is left alone, and a video playing next to it costs about 8 % of the bandwidth and of the stress score and makes the readings jump. That is the likely source of the scatter and of the level changes of (a), and of the higher level in the run of (d) from 21:31 on. Not a controlled test: the video was not started and stopped on purpose. | +| 95 | The memory scan no longer ends by chance | (a) Not elevated, nothing changed on the card: a measuring build takes 24 bandwidth readings in one process, with 3 s of the stress load before each as a scan has, at 64, 128 and 256 dispatches per timed run; then the same with the app's window open next to it, drawing a frame every 100 ms and every 33 ms. (b) Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize best` twice in a row with the build of the change, then the copy put back | (a) no, (b) yes | (a) how much a reading scatters, and on what that depends. (b) in both runs every `mem +N: bandwidth` line shows three readings that agree within about 1 %; the scan does not end before the first `WRONG RESULT`; the two `mem: bandwidth peak` values are within 300 MHz of each other and the applied offset is above +1000; the soak is not slower than at stock | **(a) measured** (2026-10-11, reference RTX 4070, driver 617.42, the saved tune applied: power 105 %, core +135, memory +650; standard deviation over the 24 readings of a series, and the lowest and highest against the mean). 64 dispatches, no window: 0.44 % (-0.55 to +1.07, one reading did not settle), 0.66 % (-1.58 to +1.49, one did not settle), 1.31 % (-4.83 to +1.27), 0.86 % (-1.44 to +2.05). 128 dispatches, no window: 0.43 % (-1.10 to +0.79). 256 dispatches, no window: 0.22 % (-0.43 to +0.42) and 0.60 % (-0.80 to +1.37; the last five readings of this series were 451 to 453 GB/s after nineteen of 444 to 449). With the window drawing every 100 ms: 64 dispatches 1.13 % (the first six readings 452 to 453 GB/s, the rest 439 to 451), 256 dispatches 0.24 % (-0.47 to +0.61). With the window drawing every 33 ms: 64 dispatches 0.58 %, 256 dispatches 0.25 % (-0.35 to +0.51). A reading took 6.5, 5.3 and 3.9 timed runs to settle at 64, 128 and 256 dispatches. So: the longer timed run takes the scatter from 0.4 to 1.3 % down to 0.2 %, the window's drawing adds nothing, and apart from the scatter the readings sit about 1.8 % higher for up to a minute at a time (the stretch at the end of the second 256 series ran on into the next series; the last 64 series went back and forth between 444 and 452 GB/s). What sets that level is not known. **(b) the offset is the same twice; the scan ends in a driver reset twice** (2026-10-11, same card and driver, a test build of 2764609, unattended, 40 minutes for the two runs). Run 1: `stock: STABLE score=5311 it/s peak=66 C fan=70 % power=185 W`; 49 steps from `mem +0: bandwidth 422.7 GB/s (422.7 422.6 422.7)` to `mem +2400: bandwidth 501.4 GB/s (501.3 502.3 501.4)`; at +2450 the candidate was `STABLE` and the measurement after it lost the device: `mem +2450: the bandwidth measurement failed; treated as a driver reset`, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, 20 s of rest, `health: STABLE`, `confirm core +0 / mem +2050: STABLE`, `mem: bandwidth peak +2250, confirmed +2050, applying +1400`; `core: highest stable +210, confirmed +210, applying +135`; soak `STABLE score=5536 it/s peak=69 C power=195 W`; `driver resets during this run: 1`. Run 2: stock 5312 it/s; 50 steps from 421.8 GB/s at +0 to 502.8 at +2450; the same failure at +2500; `mem: bandwidth peak +2300, confirmed +2100, applying +1450`; core +135; soak `STABLE score=5555 it/s peak=69 C power=196 W`; one driver reset. The three readings of a step differed by 0.4 % on average and by 1.2 % at most; no step was more than 0.7 % below the best before it, so neither scan came near ending early, by the old rule or the new; every step had its three readings. The scan took 5 min 25 s (about 6.5 s a step; about 3 min before the change). As expected: the two peaks are 50 MHz apart, the applied offsets are +1400 and +1450, and both results are 4 % faster than stock. **Not as expected: neither scan reached a `WRONG RESULT`.** Both ended one and two steps before +2550 with the device lost during the bandwidth copy; Windows logged it both times (System log, `nvlddmkm` events 14 and 153 at 19:51:22 and 20:11:27), and the screen goes black for a moment. In the run of check 94 that got this far, the measurements at +2450 and +2500 gave a number and the stress load reported the first wrong result at +2550, without a reset. A step now copies about seven times as much as it did (three readings of about four runs of 128 GB, against one reading of about six runs of 32 GB), and the copy checks nothing, so where the memory starts to fail it is the driver that gives out, not a verdict. Two runs against one: likely the cause, not proven. The search handles it as it should (rest, health probe, confirm four steps lower, the core is still searched), but a driver reset in every run on a card like this is not what the change was for. The saved tune of before the test was put back and applied (`applied best: power 105 %, core +135 MHz, mem +650 MHz`), the tray app restarted, Defender reported nothing. **(c) the same two runs with one reading a step, three only for a step in doubt** (2026-10-11 20:26 to 21:03, a test build of 71e7c2b, unattended): **the reset is still there.** Run 1: stock 5328 it/s; 50 steps from `mem +0: bandwidth 420.4 GB/s` to `mem +2450: bandwidth 502.4 GB/s`; at +2500 the candidate was `STABLE` and then `bandwidth measurement failed: the timing of the copy did not come back` ten seconds later (`nvlddmkm` event 14 at 20:37:07); `mem: bandwidth peak +2200, confirmed +2000, applying +1400`; core +135; soak `STABLE score=5572 it/s peak=70 C power=197 W`; one driver reset. Run 2: stock 5311 it/s; 421.9 GB/s at +0 to 508.0 at +2450; at +2500 `the device was lost during the copy failed (hr=0x887A0006)`, the device hung (event 14 at 20:55:29); `mem: bandwidth peak +2400, confirmed +2200, applying +1500`; core +135; soak `STABLE score=5571 it/s peak=70 C power=196 W`; one driver reset. No step was in doubt in either run (the lowest was 0.75 % below the best before it), so every step was read once; the scan took 3 min 33 s. The two peaks are 200 MHz apart: with one reading a step, which offsets count as within 1 % of the best moves by a few steps. Applied +1400 and +1500, both 5 % faster than stock. So the reset does not come from reading a step three times. Four runs out of four with the longer timed run ended this way, at +2450 or +2500. Not merged; the design for a scan that does not end like this is the open row in `docs/development.md`. The saved tune was put back and applied again (core +135, memory +650), the tray app restarted, Defender reported nothing. **(d) the edge is remembered** (a test build of 7c215c4; the same two runs): expected is that the first scan ends as before and logs nothing new, the journal gets one `mem_edge` line with the GPU id and the driver version, and the second run logs `mem: the memory gave out at +N in an earlier run on this card and driver; the scan stops at +(N-200)`, has no `driver resets during this run` line, applies more than +1000 and is not slower than stock. **Run 1 took the machine down; the test ended there** (2026-10-11 21:23, unattended). Stock 5321 it/s, power 105 %, then the scan: 49 steps from +0 to `mem +2400: bandwidth 539.3 GB/s`, two of them read three times (+1200 and +1250). At +2450 the candidate was `STABLE` and the copy after it failed as in the four runs before: `bandwidth measurement failed: the copy failed (hr=0x887A0007): the device was lost` at 21:35:05, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, `the driver was reset; reconnecting`. That is the last line. Windows restarted by itself within fifteen seconds: Kernel-Power event 41 at 21:35:23, `The previous system shutdown at 21:34:57 was unexpected`, no bugcheck, no dump, no `nvlddmkm` event written. So the fifth scan that met the edge with the longer timed run did not end in a driver reset the search recovers from, but in a reset of the whole machine, during the reconnect or right after it. What held: the candidate's journal entry was closed, so no ceiling of a freeze was left; the line `{"mem_edge":2450,"rig":"GPU-... / 617.42"}` was on disk before the machine went down, so no later run of this build goes there again; the run had saved nothing, and at logon the tray app applied the tune of before the test (`applied best: power 105 %, core +135 MHz, mem +650 MHz`). Not answered: the second run. Seen along the way and not explained: from the first step of the scan the stress load scored 5760 it/s and the bandwidth read about 7 % higher than in every run before (486.6 GB/s at +900 against about 455), while the measurement at stock five minutes earlier was the usual 5321. **(e) how the stress load meets the edge** (the same build, no search): `gao --reset`, then `gao --set-mem` and `gao --stress 3` at +2300, +2350, ... until the verdict is not `STABLE`, five times, at stock power and core +0; expected is one of two outcomes, `WRONG RESULT` every time (then the stress load can lead the copy to the edge) or `DEVICE LOST` (then it cannot). **Not run:** the test ended in (d). It goes to the edge on purpose, five times, and after (d) that is a risk for the machine that needs its owner's word first. **(f) the two-pass search** (the short copy in the scan, the long copy from four steps below the highest step that passed; see `docs/development.md`): tested on a simulated card only. On the reference card, with the edge at +2450 in its journal: `gao --optimize best` once; expected `mem: the memory gave out at +2450 in an earlier run on this card and driver; the scan stops at +2250`, no memory candidate above +2250, `mem: the scan passed +2250; reading from +2200 down with the long copy`, a few `mem +N: bandwidth ... GB/s, the long copy` lines, no `driver resets during this run`, an applied offset above +1000 and a soak that is not slower than stock. **Not run: waits for the owner's word.** **(g) what else uses the card** (2026-10-11 22:00, after the restart of (d), not elevated, the saved tune applied, nothing changed): `gao --stress 5` scored 5828 it/s and `gao --bandwidth --after-load 3` read 484.0 GB/s with its three runs within 0.0 %, against 5226 it/s and 444 to 452 GB/s with the same tune earlier that evening. The series of (a) again, 24 readings each: 64 dispatches 480.0 GB/s, 0.23 % (-0.66 to +0.18); 256 dispatches 475.2 GB/s, 0.68 %, sliding from 480 to 471 during the series; 64 dispatches again 454.0 GB/s, 3.02 % (-4.37 to +4.22), the readings jumping between 434 and 473. Read right after it, the card's use per program: Chrome 8.6 % of the video decoder (19 % at most) and 2.8 % of the 3D engine, with a YouTube video playing; nothing else above 1.1 %. So the short copy is as steady as the long one on a card that is left alone, and a video playing next to it costs about 8 % of the bandwidth and of the stress score and makes the readings jump. Then 40 readings with the short copy, and next to each the other programs on the card in the seconds before it (22:13): 464 to 476 GB/s while Chrome used the video decoder, 444 to 455 while it used the 3D engine as well, and from the reading after Chrome stopped, 477 to 483.5 for the last eleven readings with no other program on the card. So another program using the card lowers the reading by 3 to 8 % from one reading to the next. That explains the scatter and the level changes of (a) in kind. Which program it was during the tests of that day is not known: hardly any video was watched then, the low level was there all day until 21:31, and the restart removed whatever it was. | ## Notes From b3cf6ec6b1ee6d8d3f0e3d87a8186cdf59dabdfc Mon Sep 17 00:00:00 2001 From: Rovey Date: Sun, 11 Oct 2026 22:20:06 +0200 Subject: [PATCH 12/15] docs: the program that used the card during the day's tests, by the second it was closed Co-Authored-By: Claude Opus 5.5 --- docs/development.md | 2 +- docs/hardware-checks.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/development.md b/docs/development.md index e129348..2e5569f 100644 --- a/docs/development.md +++ b/docs/development.md @@ -80,7 +80,7 @@ The applied offset is always at least one step below the confirmed edge. Profile | A confirm probe that is too hot at once confirms nothing | A confirm probe that ends `TOO HOT` keeps the clock edge: over 30 s the card may run a little hotter than over the power step's 20 s, and that is the soak's business, which lowers power. But the stress run stops at its first reading above the limit, so a card that is above it when the probe begins gives `TOO HOT` within a second, and that counted as holding too. It happened on the tester's RTX 3080 Ti (hardware check 91): after a driver reset the health probe loads the card at stock power, which on that card means 76 C against a limit of 75, and `confirm core +225` was confirmed by a probe that had tested nothing. Now such a probe counts only when the clock was loaded for `kConfirmMinS` (10 s, clearly longer than the 3 s probe that found the edge) before it got too hot. Otherwise the value is not confirmed, and the card rests (`io.rest`, the 20 s of a recovery, with the fans where the curve last put them) before the next try one step lower: without the rest every lower value would start just as hot, and heat alone would step the edge down to nothing, which is what keeping the edge on `TOO HOT` was for. The same value is not tried again; one step is the price. Not changed: the health probe still runs at stock power, and a soak that begins too hot still lowers the power limit by a step. | | A profile that promises speed keeps no result that is slower than stock | Best of my GPU ended 3.8 % slower than stock on the tester's RTX 3080 Ti (4749 against 4937 it/s, hardware check 91), with the fans at 80 % instead of 66 %, and showed it as **Done**, applied and kept at every logon. Every step had done what it is for: the card runs 83 C at stock with its fans at 100 %, the profile allows 75, so the power limit went to 65 %, and +150 MHz of core clock did not win that back. But a profile called "the best of my GPU" that leaves the card slower has not done what it says, and a result like that is not offered as one. After the soak, `optimize` compares its score with the measurement at stock that went before the run (the "before" the result screen shows, so the rule judges by the number the user sees): below `kNotSlowerScore` (0.99: one percent is what two five-minute runs differ by) the run ends as a failed one does, at stock, with nothing saved, and says what it found, why, and that Undervolt is the profile for a card that runs warm. `Objectives::faster_than_stock` marks the two profiles this holds for, Best of my GPU and Max performance. Not Quiet and Cool & efficient, which trade speed for noise and heat and say so. Not the overclock step of All in one either: it goes on to an undervolt and a fan tune, and its result is judged as a whole; whether that whole may be slower than stock is not decided. This is a backstop, not the cure: what such a card needs from these profiles is in `docs/planned.md` ("never hotter than at stock"). | | The memory search has two passes: a scan with the short copy finds where it ends, and the long copy reads the offset to keep, four steps below that | **What went wrong.** Four runs on one day applied +650, +1600, +550 and +100 MHz on the reference RTX 4070, a card that runs every game at +1500 (hardware check 94). That card has no bandwidth peak: bandwidth rises by about 0.4 % per 50 MHz step all the way to +2500 and the stress load computes its first wrong result at +2550, so the scan could only end early by chance, and where it did decided the offset. **What was measured** (hardware check 95 a, no search, nothing changed on the card). A reading is the median of three timed runs that agree within 1 %. With the timed run as it was, 32 GB or 75 ms, readings scatter by 0.4 to 1.3 % (standard deviation over 24 readings, six series), single ones by up to 4.8 %, and now and then one does not settle. With 128 GB per run it is 0.2 %. Apart from that the readings change level: for up to a minute every reading is about 1.8 % higher, at one and the same offset; the cause is not known. The app's own window, drawing every 100 or 33 ms next to the measurement, adds nothing, so nothing was built for it. **What was tried first, and what it did** (check 95 b to d). The long timed run at every step of the scan: the offset repeated (+1400 to +1500 in four runs, 4 to 5 % faster than stock), and every scan now ran to where the memory gives out and met it with the long copy. Five scans out of five lost the device there, at +2450 or +2500, a driver reset each time, with three readings a step and with one; and the fifth took the whole machine down within seconds (Kernel-Power 41, no bugcheck). In the one scan before all this that got to the edge, the short copy read a number at +2450 and +2500 and the stress load reported the wrong result at +2550, with no reset. One against five is thin, and it is all there is: the long copy is too heavy for memory at its edge, the short one was not. **The design.** One measurement cannot do both jobs, so there are two passes. *The scan* goes up as before and only has to find where it ends: a step that is not stable, or one that has lost more than `kBandwidthDrop` (5 %) against the best. It reads with the short copy (`kBwDispatches` 64), which is what the released versions take to the edge. Because those readings scatter, two things keep one of them from deciding: a reading that would end the scan is taken `kBandwidthReadings` (3) times and the middle one counts, and the best that readings are held against is the highest middle of three steps in a row, not the highest reading (the highest of fifty scattering readings is always one that came out high). 5 % is far more than the readings move by themselves; the scan is not where the offset is chosen any more, so it can afford to be coarse. *The second pass* chooses the offset, with the long copy (`MeasureBandwidth(true)`, `OptimizeIo::bandwidth_precise`): it starts `kPreciseMarginSteps` (4 steps, 200 MHz) below the highest step the scan passed (`precise_top`), goes down a step at a time until a reading is more than `kBandwidthTie` (1 %) below the best of them, and keeps the lowest offset within that 1 % (`bandwidth_peak_below`). On the reference card: the scan passes +2500, the long copy reads +2300, +2250, +2200 and +2150, and +2200 is kept; confirmed there, 70 % applied, +1500. The long copy lost the device at the highest step the stress load passed and one below it; four steps leave two to spare. The margin is kept whatever ended the scan, a drop and the driver's own range included: what is above was never tried. After a reset event in the scan the second pass is skipped, and confirming starts four steps below where it would have begun. **The edge is met once** (built before this, kept): a scan that ended in a wrong result or a reset event writes that offset to the journal with the GPU id and the driver version (`{"mem_edge":2550,"rig":" / "}`, a line without an id, so older builds skip it). A later scan on the same card and driver stops four steps below the lowest such offset, and its second pass starts where the first run's did: the margin counts from the edge, not a second time from where the scan was held. With another driver or card the line does not count. **What it costs.** A card that does have a peak keeps an offset up to 200 MHz below it when the scan ends close above the peak: the margin does not know a drop from an edge. A scan that ends for a harmless reason (readings that did not settle) loses those 200 MHz as well. The second pass adds about four steps, half a minute. **Not chosen.** The long copy everywhere: see above. The short copy everywhere, as released: the offset is chance (the +100 run). Three long readings a step, or only for a step in doubt: the reset came with both. Only remembering the edge: the first meeting can cost the machine, and once per card is once too often. The stress load probing four steps ahead of the copy: it needs the proof that the load at the edge ends in a wrong result and not in a reset of its own, and getting that proof means going to the edge on purpose. A fixed top for the scan (an offset, a share of the range, a gain over +0): there is nothing to choose the number by. **Tested** on a simulated card with the reference card's numbers (wrong result at +2550, the long copy losing the device from +2450): the long copy is never taken above +2300, in the first run and in the one after it, with short readings scattering by 1.3 % on two hundred seeds, and on forty more with level changes of 1.8 % thrown in; with the edge anywhere from +300 to +2900; and the tests fail when the margin is set to nothing. **Not tested on the card.** See the row below. | -| Open: the two-pass memory search has not run on a card | The last card test (hardware check 95 d) ended with the machine restarting, and nothing has been run on the card since. Before this is merged, its owner decides whether and when to run it there. What a run would show, and what it cannot: the journal of the reference card holds the edge at +2450 now, so a run of this build stops its scan at +2250 and reads with the long copy from +2200 down, at offsets every earlier scan passed. That tests the second pass and the result, with little risk. It does not test the first meeting with the edge with the short copy: that is what the released versions do, it went well the one time it was seen, and seeing it again means removing the edge from the journal and going there. **Still not known.** Why the long copy fails where the short one did not: the amount copied in one go, or only that five runs are more than one. Whether a card with a real bandwidth peak behaves as assumed; none has been measured through a scan (the tester's RTX 3080 Ti never got a reading to settle at +0). And one observation from the run that crashed: from the first step of its scan the stress load scored 5760 it/s and the bandwidth read about 7 % higher than in every run before (486.6 GB/s at +900 against about 455), while the measurement at stock five minutes earlier was the usual 5321. It was not the monitor going to sleep, as first suspected: it never does on that machine, and the run's log shows the change in the middle of the scan, between +350 and +550 (430 to 457 GB/s within one step, 5260 to 5750 it/s), not at its start. **What it is: another program using the card.** Measured after the restart, with the same tune applied and nothing changed (hardware check 95 g): the short copy read 480 GB/s with a scatter of 0.23 %, as steady as the long copy was all day; minutes later, with a YouTube video playing in Chrome (8.6 % of the card's video decoder, 2.8 % of its 3D engine), the same series read 434 to 473 GB/s with a scatter of 3 %, and the stress load scores about 5300 it/s in the one state and 5750 to 5830 in the other. A second series, with the card's use per program sampled next to every reading, shows it to the second: 483 GB/s with no other program on the card, about 470 while Chrome decodes video, 444 to 455 while Chrome also draws, and back to 483 on the reading after Chrome stopped. So another program using the card moves the reading by 3 to 8 %, at once. **What it was on the day of the tests is not known.** The low state is what every measurement of that day shows until 21:31, but hardly any video was watched during those tests, so it was something else that used the card all day and stopped then; after the restart there is no finding out. If it holds, three things follow, and none is built. The scatter of the short copy that this design is built around (0.4 to 1.3 %) was for the larger part the video, not the copy; the design stands, it only has more room than was thought. The level changes of 1.8 % are the same thing in small. And the comparison of a result with stock is worth nothing when such a program starts or stops between the two: 8 % is far more than the 1 % by which a speed profile is judged (`kNotSlowerScore`) and than anything the efficiency search decides on. What to do about that is open: say so before a run (the README names games, MSI Afterburner and RivaTuner, not a browser with a video), or read the card's use per program (Windows keeps it, per engine) and warn or wait. | +| Open: the two-pass memory search has not run on a card | The last card test (hardware check 95 d) ended with the machine restarting, and nothing has been run on the card since. Before this is merged, its owner decides whether and when to run it there. What a run would show, and what it cannot: the journal of the reference card holds the edge at +2450 now, so a run of this build stops its scan at +2250 and reads with the long copy from +2200 down, at offsets every earlier scan passed. That tests the second pass and the result, with little risk. It does not test the first meeting with the edge with the short copy: that is what the released versions do, it went well the one time it was seen, and seeing it again means removing the edge from the journal and going there. **Still not known.** Why the long copy fails where the short one did not: the amount copied in one go, or only that five runs are more than one. Whether a card with a real bandwidth peak behaves as assumed; none has been measured through a scan (the tester's RTX 3080 Ti never got a reading to settle at +0). And one observation from the run that crashed: from the first step of its scan the stress load scored 5760 it/s and the bandwidth read about 7 % higher than in every run before (486.6 GB/s at +900 against about 455), while the measurement at stock five minutes earlier was the usual 5321. It was not the monitor going to sleep, as first suspected: it never does on that machine, and the run's log shows the change in the middle of the scan, between +350 and +550 (430 to 457 GB/s within one step, 5260 to 5750 it/s), not at its start. **What it is: another program using the card.** Measured after the restart, with the same tune applied and nothing changed (hardware check 95 g): the short copy read 480 GB/s with a scatter of 0.23 %, as steady as the long copy was all day; minutes later, with a YouTube video playing in Chrome (8.6 % of the card's video decoder, 2.8 % of its 3D engine), the same series read 434 to 473 GB/s with a scatter of 3 %, and the stress load scores about 5300 it/s in the one state and 5750 to 5830 in the other. A second series, with the card's use per program sampled next to every reading, shows it to the second: 483 GB/s with no other program on the card, about 470 while Chrome decodes video, 444 to 455 while Chrome also draws, and back to 483 on the reading after Chrome stopped. So another program using the card moves the reading by 3 to 8 %, at once. **What it was on the day of the tests: most likely a code editor that stood open.** Hardly any video was watched during those tests. But two editors built on a browser engine (Trae and ZCode) had been installed that morning and stood open all day, and the last files Trae wrote are from 21:31:43 and 21:31:44, when it was closed: the second before the bandwidth went from 430 to 457 GB/s in the run's log (21:31:45). ZCode's are from 21:32:09. A match to the second, not a test: nobody has started that editor again and measured next to it. If it holds, three things follow, and none is built. The scatter of the short copy that this design is built around (0.4 to 1.3 %) was for the larger part the video, not the copy; the design stands, it only has more room than was thought. The level changes of 1.8 % are the same thing in small. And the comparison of a result with stock is worth nothing when such a program starts or stops between the two: 8 % is far more than the 1 % by which a speed profile is judged (`kNotSlowerScore`) and than anything the efficiency search decides on. What to do about that is open: say so before a run (the README names games, MSI Afterburner and RivaTuner, not a browser with a video), or read the card's use per program (Windows keeps it, per engine) and warn or wait. | | Open: why the bandwidth measurement fails on some cards, and what then | On the tester's RTX 3080 Ti it failed at +0, and since 0.4.4 that means no memory overclock at all on such a card, though it computed right far beyond +2000 MHz. The measurement gives a number only when three runs in a row agree within 1 % (`kBwSettle`), and it throws away every run the driver's timer calls disjoint (the clock changed under it). That card was held at a power limit of 65 % and ran at 83 C when it was measured; the reference card has only ever been measured at 105 %, where the measurement itself draws 179 of 210 W. So the suspicion is the power limit, and hardware check 90 tests exactly that on the reference card. What it shows decides what is built, and nothing is built before it: **(1) runs are thrown away as disjoint, and the ones that are kept agree:** the timer is what fails, not the card. Time the copy differently (wall clock around a flush, or count a disjoint run when its neighbours agree) and keep the method. **(2) the readings themselves scatter by more than 1 %:** a peak that is found by a 1 % drop (`kBandwidthDrop`) cannot be found in that noise. Then the memory has to be measured where the card is not held back: search it before the power step, at stock power, where a probe and its measurement last five seconds; or take more readings and compare medians. **(3) it does not fail at 65 % on the reference card:** the power limit is not the cause. Then it was the heat (a card throttling at its temperature limit changes clock as well), or another program, and only the tester's `run.log` can say: since this change a measurement that never settled writes how many runs gave a reading, between which values, and how many were thrown away. In no case does the climb on stability alone come back. **Outcome of check 90: (3).** On the reference card the measurement gave a number fifteen times out of fifteen at 100, 80 and 65 %, in three or four runs, with nothing thrown away, and the bandwidth was the same at every limit. The power limit is ruled out; (1) and (2) are not built. What is left is the heat of that card, another program, or a timer query that failed, and the tester's `run.log` is the next thing that can tell them apart. **The tester's run (check 91): it is (2), but only inside a run.** Twenty readings between 763.9 and 821.2 GB/s, none thrown away, three seconds after the load began; after thirty seconds of load the same card read within 0.2 %. The suspicion now is the moment, not the card: the measurement is taken while the power limiter is still catching up. If `gao --bandwidth --after-load 3` scatters on that card and `--after-load 30` does not, that is it, and the fix is to measure when the card has settled (keep taking runs for some seconds instead of twenty runs), not to change the method. | | Every line of a run is also in `run.log` | `logged()` wraps the log hook of both front ends: a line is appended and flushed before it is shown, so the step a machine freezes in is the last line on disk. One file in the state folder, moved aside at 1 MB (`run.log.old`). Only elevated and only after `ensure_app_dir` has verified the folder: an elevated process appends to nothing a user could have put there. | | A run does not start while the NVIDIA app's Automatic Tuning is on | Switched on, a service of the driver scans the card when it sees fit: it loads it, changes its clocks, and starts again by itself (hardware check 89: a scan that was stopped started again 14 ms later, and after a restart of the service it resumed within two seconds). A run next to that measures nothing it can trust, so it is refused, like a run next to Afterburner, and with the same weight: a courtesy, not a guard. Read from where that service keeps the switch, `%ProgramData%\NVIDIA Corporation\nvtopps\nvtopps.db3`, table `ocScannerSettings`, fields `backgroundScanEnabled` and `grdOcEnabled`, joined to `TelemetryGpuCfg` so that only a card that is in the PC counts (a row of an earlier card can never be switched off again). That this is the source and not a copy of it was measured, not assumed: with the field set the service took the scan up again after every restart, with it cleared it did not, while the NVIDIA app itself showed the switch as off. Nothing documents the file. So: opened read-only and immutable (no lock, the service never waits for us), through `winsqlite3.dll` from System32 (no dependency added), and everything that is not exactly as expected reads as "unknown", which starts the run as before. The registry value the app sets next to it (`HKLM\SOFTWARE\NVIDIA Corporation\Global\NvApp\OC\`) changes at the same moment but belongs to the app, not to the service, and is not used. **The overlay is not checked:** no setting for it was found, its processes run whether it is open or not, and the one value that moved (`IsShadowPlayEnabled`) followed the service being up, not a switch. NVAPI was not tried: an undocumented call is no better than an undocumented file, and it cannot be compared with anything. | diff --git a/docs/hardware-checks.md b/docs/hardware-checks.md index fc91f8e..b808862 100644 --- a/docs/hardware-checks.md +++ b/docs/hardware-checks.md @@ -122,7 +122,7 @@ root; adjust the path for a Debug build. | 92 | The undervolt profile starts with the measurement at stock every run uses, and a hot card is not refused | Elevated, the tray app exited, a copy of `gao.json` kept. (a) `gao --undervolt` to the end; then `gao --status`. (b) The same with a test build whose stock limit is lowered to a temperature the reference card passes under load (63 C; it reaches 67). Put the copy of `gao.json` back. (c) Only on a card that is above the profile's limit at stock with the fans at 100 % (the tester's RTX 3080 Ti: 82 to 83 C): the **Undervolt** profile from the window, to the end, and its `run.log` | yes | (a) `the card at stock, five minutes`, a `stock: STABLE ...` line with the driver's fan speed, then `baseline: the measurement at stock`, and no second five minutes at stock: the first candidate follows at once; no `candidates are stopped at` line; `RESULT:`; `before:` shows 300 s and the fan speed the driver chose. (b) `stock: TOO HOT ... peak=64 C`, the line that the measurement stopped there, the cool-down, then `baseline: the measurement at stock` and a search that runs to `RESULT:` with `before:` showing the shortened time; no `candidates are stopped at` line, because 64 C is under the profile's 75. (c) no `not stable at stock`; `the card reached 86 C at stock: candidates are stopped at 85 C, not at the profile's 75 C`; candidates that end `STABLE` or `WRONG RESULT`, none `TOO HOT`; a result that is cooler and draws less than stock at the same clock | First run: **not as expected**, and that found a fault (2026-10-11, reference RTX 4070, driver 617.42, a test build of d1c99e9, started elevated by a script). The first halves held: (a) `stock: STABLE ... peak=65 C fan=70 %`, then `baseline: the measurement at stock` four seconds later and no second five minutes; (b) `stock: TOO HOT ... peak=64 C`, stopped after 34 s, the run went on, no `candidates are stopped at` line. But both runs then ended with `the card held no undervolt: the first step below its own voltage already failed`: their first candidates read `1070 mV (+15 MHz): SLOW ... clock=2323 MHz` and `1050 mV (+15 MHz): SLOW ... clock=2411 MHz`, on a card that ran 2791 MHz in the measurement just before. The search's load had not run yet when that candidate began (see the decision "With its reference handed over, the undervolt search warms its load up before the first candidate"). Second run, with the warm-up: **pass** for (a) and (b) (same day, a test build of 0733695). (a) `stock: STABLE score=5426 it/s clock=2788 MHz peak=66 C fan=70 %`, four seconds later `baseline: the measurement at stock`, `warming the load up: 5 s at stock`, `target: 2790 MHz, which the card reaches at 1090 mV by itself`, and the first candidate `1085 mV (+15 MHz): STABLE ... clock=2775 MHz`; 26 candidates, none `SLOW` or `TOO HOT`, the edge at 950 mV, `RESULT: 2790 MHz at 990 mV instead of 1090 mV`; `before:` 5426 it/s, 189 W, fan 70 % (300 s), `after:` 5325 it/s, 149 W, fan 53 % (300 s). (b) `stock: TOO HOT ... peak=64 C`, stopped after 74 s, the run went on, the same warm-up and a search to the same result (990 mV, edge 950 mV); `before:` shows 75 s; no `candidates are stopped at` line. The saved tune of before the test was put back and applied. (c) not run. | | 93 | A speed profile that would end slower than stock keeps nothing | Only on a card that is warmer at stock than the profile allows (the tester's RTX 3080 Ti; the reference card ends faster than stock with every speed profile): **Best of my GPU** from the window, to the end; the result screen, the dashboard and `run.log` | yes | the run goes through all its steps and its soak; then **Not applied** with `the result is slower than stock: N it/s against M (-X %)`, the power limit it had to go down to, and the pointer to Undervolt; `The card is back at stock.`; the dashboard shows stock and the saved tune of before the run, if there was one; `run.log` ends with `stopped: the result is slower than stock ... -- card restored to stock` | Not run yet. On the reference card: nothing to see, by design. | | 94 | All in one, with the undervolt step searching for the first time | Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize all` to the end with a build that has the warm-up of the undervolt step; then `gao --status`; put the copy back | yes | five steps; in step 3 a `warming the load up` line, a first candidate that is not `SLOW`, and a descent; steps 4 and 5 run on the tune of steps 2 and 3; `RESULT:` | **Not as expected at step 4**, and that found a fault (2026-10-11, reference RTX 4070, driver 617.42, a test build of 0733695, unattended). Step 1: `stock: STABLE score=5325 it/s peak=65 C fan=70 % power=186 W`. Step 2: power 105 %, `mem: bandwidth peak +2300, confirmed +2300, applying +1600`, `core: highest stable +210, confirmed +210, applying +135`, soak `STABLE score=5568 it/s peak=70 C power=197 W`. Step 3, as wanted: `warming the load up: 5 s with the overclock`, `target: 2865 MHz, which the card reaches at 1050 mV with the overclock`, nine candidates `STABLE` from 1045 mV down, `WRONG RESULT` at 990 mV, confirmed at 1010 mV after two tries, and `applying 1050 mV (+135 MHz at that point)`: the margin left no raise above the overclock, so the curve was cut off, as in the runs before the warm-up; soak `STABLE score=5434 it/s peak=67 C power=176 W`. Step 4 did not begin: `the tune could not be applied for the fan tune: the undervolt was not applied: the point already runs more than the target -- card at stock`. The saved tune of before the test was put back and applied. Cause and fix: the decision "A flat top goes back on a card that is cooler than when it was found". Second run, with the `gao.exe` of the released v0.4.6: **pass, all five steps** (same day, started elevated by a script). Step 1: `stock: STABLE score=5309 it/s peak=66 C fan=70 % power=185 W`. Step 2: power 105 %, `mem: bandwidth peak +850, confirmed +850, applying +550`, `core: highest stable +210, confirmed +210, applying +135`, soak `STABLE score=5568 it/s`. Step 3: `target: 2850 MHz, which the card reaches at 1045 mV with the overclock`, `undervolt: lowest stable 975 mV, confirmed 985 mV, applying 1025 mV (+165 MHz at that point)`, soak `STABLE score=5363 it/s peak=66 C power=163 W`. Step 4: the tune went back on the card, `fan 60 %: 63 C`, `fan 55 %: 65 C`, `fan 52 %: 67 C`, `fan tune: 52 % holds 67 C`. Step 5: the final test ran its 300 s. `RESULT: power 105 %, core +135 MHz, mem +550 MHz, 2850 MHz at 1025 mV, fans 52 % at 67 C`; `stock: 5309 it/s, 185 W, fan 70 %, 66 C`, `now: 5353 it/s, 166 W, fan 52 %, 70 C`: `+0.8 % score and -10.3 % power against stock`; `undervolt: applied`. The saved tune of before the test was put back. **What this run did not show:** its undervolt had a raise of its own (+165 over an offset of +135), so the anchor was raised and not merely cut off, and the case of the first run (an anchor that reads a step above the target on a cooler card) did not come up. That fix has still only its tests. **The memory offset differs a lot from run to run on this card, and the two logs say why:** +1600 in the first run (bandwidth peak +2300), +550 in the second (peak +850), +650 in the run that saved the tune in use that morning (the journal shows its scan ending at +1000 too). The card was in the same state each time (68 C, power 105 %, a reading every three to four seconds). In both logged scans the bandwidth rises in a straight line, 0.36 and 0.46 % per 50 MHz step, from 420 GB/s at +0 to 509 GB/s at +2450; the first run went on to `WRONG RESULT` at +2550 without the bandwidth ever turning down. The readings scatter around that line with a standard deviation of 0.4 and 0.7 %, single ones by more than 1 %. The scan ends at a reading more than 1 % below the best so far when a second reading agrees, and in the second run that was chance: +950 read high (462.8 GB/s, where the line is at about 458), +1000 read 456.6 twice, 1.3 % below it. In the first run the one low reading (+600) was followed by a higher one, and the scan ran to the first wrong result. So on this card there is no bandwidth peak before the errors, and where the scan stops is decided by noise. | -| 95 | The memory scan no longer ends by chance | (a) Not elevated, nothing changed on the card: a measuring build takes 24 bandwidth readings in one process, with 3 s of the stress load before each as a scan has, at 64, 128 and 256 dispatches per timed run; then the same with the app's window open next to it, drawing a frame every 100 ms and every 33 ms. (b) Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize best` twice in a row with the build of the change, then the copy put back | (a) no, (b) yes | (a) how much a reading scatters, and on what that depends. (b) in both runs every `mem +N: bandwidth` line shows three readings that agree within about 1 %; the scan does not end before the first `WRONG RESULT`; the two `mem: bandwidth peak` values are within 300 MHz of each other and the applied offset is above +1000; the soak is not slower than at stock | **(a) measured** (2026-10-11, reference RTX 4070, driver 617.42, the saved tune applied: power 105 %, core +135, memory +650; standard deviation over the 24 readings of a series, and the lowest and highest against the mean). 64 dispatches, no window: 0.44 % (-0.55 to +1.07, one reading did not settle), 0.66 % (-1.58 to +1.49, one did not settle), 1.31 % (-4.83 to +1.27), 0.86 % (-1.44 to +2.05). 128 dispatches, no window: 0.43 % (-1.10 to +0.79). 256 dispatches, no window: 0.22 % (-0.43 to +0.42) and 0.60 % (-0.80 to +1.37; the last five readings of this series were 451 to 453 GB/s after nineteen of 444 to 449). With the window drawing every 100 ms: 64 dispatches 1.13 % (the first six readings 452 to 453 GB/s, the rest 439 to 451), 256 dispatches 0.24 % (-0.47 to +0.61). With the window drawing every 33 ms: 64 dispatches 0.58 %, 256 dispatches 0.25 % (-0.35 to +0.51). A reading took 6.5, 5.3 and 3.9 timed runs to settle at 64, 128 and 256 dispatches. So: the longer timed run takes the scatter from 0.4 to 1.3 % down to 0.2 %, the window's drawing adds nothing, and apart from the scatter the readings sit about 1.8 % higher for up to a minute at a time (the stretch at the end of the second 256 series ran on into the next series; the last 64 series went back and forth between 444 and 452 GB/s). What sets that level is not known. **(b) the offset is the same twice; the scan ends in a driver reset twice** (2026-10-11, same card and driver, a test build of 2764609, unattended, 40 minutes for the two runs). Run 1: `stock: STABLE score=5311 it/s peak=66 C fan=70 % power=185 W`; 49 steps from `mem +0: bandwidth 422.7 GB/s (422.7 422.6 422.7)` to `mem +2400: bandwidth 501.4 GB/s (501.3 502.3 501.4)`; at +2450 the candidate was `STABLE` and the measurement after it lost the device: `mem +2450: the bandwidth measurement failed; treated as a driver reset`, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, 20 s of rest, `health: STABLE`, `confirm core +0 / mem +2050: STABLE`, `mem: bandwidth peak +2250, confirmed +2050, applying +1400`; `core: highest stable +210, confirmed +210, applying +135`; soak `STABLE score=5536 it/s peak=69 C power=195 W`; `driver resets during this run: 1`. Run 2: stock 5312 it/s; 50 steps from 421.8 GB/s at +0 to 502.8 at +2450; the same failure at +2500; `mem: bandwidth peak +2300, confirmed +2100, applying +1450`; core +135; soak `STABLE score=5555 it/s peak=69 C power=196 W`; one driver reset. The three readings of a step differed by 0.4 % on average and by 1.2 % at most; no step was more than 0.7 % below the best before it, so neither scan came near ending early, by the old rule or the new; every step had its three readings. The scan took 5 min 25 s (about 6.5 s a step; about 3 min before the change). As expected: the two peaks are 50 MHz apart, the applied offsets are +1400 and +1450, and both results are 4 % faster than stock. **Not as expected: neither scan reached a `WRONG RESULT`.** Both ended one and two steps before +2550 with the device lost during the bandwidth copy; Windows logged it both times (System log, `nvlddmkm` events 14 and 153 at 19:51:22 and 20:11:27), and the screen goes black for a moment. In the run of check 94 that got this far, the measurements at +2450 and +2500 gave a number and the stress load reported the first wrong result at +2550, without a reset. A step now copies about seven times as much as it did (three readings of about four runs of 128 GB, against one reading of about six runs of 32 GB), and the copy checks nothing, so where the memory starts to fail it is the driver that gives out, not a verdict. Two runs against one: likely the cause, not proven. The search handles it as it should (rest, health probe, confirm four steps lower, the core is still searched), but a driver reset in every run on a card like this is not what the change was for. The saved tune of before the test was put back and applied (`applied best: power 105 %, core +135 MHz, mem +650 MHz`), the tray app restarted, Defender reported nothing. **(c) the same two runs with one reading a step, three only for a step in doubt** (2026-10-11 20:26 to 21:03, a test build of 71e7c2b, unattended): **the reset is still there.** Run 1: stock 5328 it/s; 50 steps from `mem +0: bandwidth 420.4 GB/s` to `mem +2450: bandwidth 502.4 GB/s`; at +2500 the candidate was `STABLE` and then `bandwidth measurement failed: the timing of the copy did not come back` ten seconds later (`nvlddmkm` event 14 at 20:37:07); `mem: bandwidth peak +2200, confirmed +2000, applying +1400`; core +135; soak `STABLE score=5572 it/s peak=70 C power=197 W`; one driver reset. Run 2: stock 5311 it/s; 421.9 GB/s at +0 to 508.0 at +2450; at +2500 `the device was lost during the copy failed (hr=0x887A0006)`, the device hung (event 14 at 20:55:29); `mem: bandwidth peak +2400, confirmed +2200, applying +1500`; core +135; soak `STABLE score=5571 it/s peak=70 C power=196 W`; one driver reset. No step was in doubt in either run (the lowest was 0.75 % below the best before it), so every step was read once; the scan took 3 min 33 s. The two peaks are 200 MHz apart: with one reading a step, which offsets count as within 1 % of the best moves by a few steps. Applied +1400 and +1500, both 5 % faster than stock. So the reset does not come from reading a step three times. Four runs out of four with the longer timed run ended this way, at +2450 or +2500. Not merged; the design for a scan that does not end like this is the open row in `docs/development.md`. The saved tune was put back and applied again (core +135, memory +650), the tray app restarted, Defender reported nothing. **(d) the edge is remembered** (a test build of 7c215c4; the same two runs): expected is that the first scan ends as before and logs nothing new, the journal gets one `mem_edge` line with the GPU id and the driver version, and the second run logs `mem: the memory gave out at +N in an earlier run on this card and driver; the scan stops at +(N-200)`, has no `driver resets during this run` line, applies more than +1000 and is not slower than stock. **Run 1 took the machine down; the test ended there** (2026-10-11 21:23, unattended). Stock 5321 it/s, power 105 %, then the scan: 49 steps from +0 to `mem +2400: bandwidth 539.3 GB/s`, two of them read three times (+1200 and +1250). At +2450 the candidate was `STABLE` and the copy after it failed as in the four runs before: `bandwidth measurement failed: the copy failed (hr=0x887A0007): the device was lost` at 21:35:05, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, `the driver was reset; reconnecting`. That is the last line. Windows restarted by itself within fifteen seconds: Kernel-Power event 41 at 21:35:23, `The previous system shutdown at 21:34:57 was unexpected`, no bugcheck, no dump, no `nvlddmkm` event written. So the fifth scan that met the edge with the longer timed run did not end in a driver reset the search recovers from, but in a reset of the whole machine, during the reconnect or right after it. What held: the candidate's journal entry was closed, so no ceiling of a freeze was left; the line `{"mem_edge":2450,"rig":"GPU-... / 617.42"}` was on disk before the machine went down, so no later run of this build goes there again; the run had saved nothing, and at logon the tray app applied the tune of before the test (`applied best: power 105 %, core +135 MHz, mem +650 MHz`). Not answered: the second run. Seen along the way and not explained: from the first step of the scan the stress load scored 5760 it/s and the bandwidth read about 7 % higher than in every run before (486.6 GB/s at +900 against about 455), while the measurement at stock five minutes earlier was the usual 5321. **(e) how the stress load meets the edge** (the same build, no search): `gao --reset`, then `gao --set-mem` and `gao --stress 3` at +2300, +2350, ... until the verdict is not `STABLE`, five times, at stock power and core +0; expected is one of two outcomes, `WRONG RESULT` every time (then the stress load can lead the copy to the edge) or `DEVICE LOST` (then it cannot). **Not run:** the test ended in (d). It goes to the edge on purpose, five times, and after (d) that is a risk for the machine that needs its owner's word first. **(f) the two-pass search** (the short copy in the scan, the long copy from four steps below the highest step that passed; see `docs/development.md`): tested on a simulated card only. On the reference card, with the edge at +2450 in its journal: `gao --optimize best` once; expected `mem: the memory gave out at +2450 in an earlier run on this card and driver; the scan stops at +2250`, no memory candidate above +2250, `mem: the scan passed +2250; reading from +2200 down with the long copy`, a few `mem +N: bandwidth ... GB/s, the long copy` lines, no `driver resets during this run`, an applied offset above +1000 and a soak that is not slower than stock. **Not run: waits for the owner's word.** **(g) what else uses the card** (2026-10-11 22:00, after the restart of (d), not elevated, the saved tune applied, nothing changed): `gao --stress 5` scored 5828 it/s and `gao --bandwidth --after-load 3` read 484.0 GB/s with its three runs within 0.0 %, against 5226 it/s and 444 to 452 GB/s with the same tune earlier that evening. The series of (a) again, 24 readings each: 64 dispatches 480.0 GB/s, 0.23 % (-0.66 to +0.18); 256 dispatches 475.2 GB/s, 0.68 %, sliding from 480 to 471 during the series; 64 dispatches again 454.0 GB/s, 3.02 % (-4.37 to +4.22), the readings jumping between 434 and 473. Read right after it, the card's use per program: Chrome 8.6 % of the video decoder (19 % at most) and 2.8 % of the 3D engine, with a YouTube video playing; nothing else above 1.1 %. So the short copy is as steady as the long one on a card that is left alone, and a video playing next to it costs about 8 % of the bandwidth and of the stress score and makes the readings jump. Then 40 readings with the short copy, and next to each the other programs on the card in the seconds before it (22:13): 464 to 476 GB/s while Chrome used the video decoder, 444 to 455 while it used the 3D engine as well, and from the reading after Chrome stopped, 477 to 483.5 for the last eleven readings with no other program on the card. So another program using the card lowers the reading by 3 to 8 % from one reading to the next. That explains the scatter and the level changes of (a) in kind. Which program it was during the tests of that day is not known: hardly any video was watched then, the low level was there all day until 21:31, and the restart removed whatever it was. | +| 95 | The memory scan no longer ends by chance | (a) Not elevated, nothing changed on the card: a measuring build takes 24 bandwidth readings in one process, with 3 s of the stress load before each as a scan has, at 64, 128 and 256 dispatches per timed run; then the same with the app's window open next to it, drawing a frame every 100 ms and every 33 ms. (b) Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize best` twice in a row with the build of the change, then the copy put back | (a) no, (b) yes | (a) how much a reading scatters, and on what that depends. (b) in both runs every `mem +N: bandwidth` line shows three readings that agree within about 1 %; the scan does not end before the first `WRONG RESULT`; the two `mem: bandwidth peak` values are within 300 MHz of each other and the applied offset is above +1000; the soak is not slower than at stock | **(a) measured** (2026-10-11, reference RTX 4070, driver 617.42, the saved tune applied: power 105 %, core +135, memory +650; standard deviation over the 24 readings of a series, and the lowest and highest against the mean). 64 dispatches, no window: 0.44 % (-0.55 to +1.07, one reading did not settle), 0.66 % (-1.58 to +1.49, one did not settle), 1.31 % (-4.83 to +1.27), 0.86 % (-1.44 to +2.05). 128 dispatches, no window: 0.43 % (-1.10 to +0.79). 256 dispatches, no window: 0.22 % (-0.43 to +0.42) and 0.60 % (-0.80 to +1.37; the last five readings of this series were 451 to 453 GB/s after nineteen of 444 to 449). With the window drawing every 100 ms: 64 dispatches 1.13 % (the first six readings 452 to 453 GB/s, the rest 439 to 451), 256 dispatches 0.24 % (-0.47 to +0.61). With the window drawing every 33 ms: 64 dispatches 0.58 %, 256 dispatches 0.25 % (-0.35 to +0.51). A reading took 6.5, 5.3 and 3.9 timed runs to settle at 64, 128 and 256 dispatches. So: the longer timed run takes the scatter from 0.4 to 1.3 % down to 0.2 %, the window's drawing adds nothing, and apart from the scatter the readings sit about 1.8 % higher for up to a minute at a time (the stretch at the end of the second 256 series ran on into the next series; the last 64 series went back and forth between 444 and 452 GB/s). What sets that level is not known. **(b) the offset is the same twice; the scan ends in a driver reset twice** (2026-10-11, same card and driver, a test build of 2764609, unattended, 40 minutes for the two runs). Run 1: `stock: STABLE score=5311 it/s peak=66 C fan=70 % power=185 W`; 49 steps from `mem +0: bandwidth 422.7 GB/s (422.7 422.6 422.7)` to `mem +2400: bandwidth 501.4 GB/s (501.3 502.3 501.4)`; at +2450 the candidate was `STABLE` and the measurement after it lost the device: `mem +2450: the bandwidth measurement failed; treated as a driver reset`, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, 20 s of rest, `health: STABLE`, `confirm core +0 / mem +2050: STABLE`, `mem: bandwidth peak +2250, confirmed +2050, applying +1400`; `core: highest stable +210, confirmed +210, applying +135`; soak `STABLE score=5536 it/s peak=69 C power=195 W`; `driver resets during this run: 1`. Run 2: stock 5312 it/s; 50 steps from 421.8 GB/s at +0 to 502.8 at +2450; the same failure at +2500; `mem: bandwidth peak +2300, confirmed +2100, applying +1450`; core +135; soak `STABLE score=5555 it/s peak=69 C power=196 W`; one driver reset. The three readings of a step differed by 0.4 % on average and by 1.2 % at most; no step was more than 0.7 % below the best before it, so neither scan came near ending early, by the old rule or the new; every step had its three readings. The scan took 5 min 25 s (about 6.5 s a step; about 3 min before the change). As expected: the two peaks are 50 MHz apart, the applied offsets are +1400 and +1450, and both results are 4 % faster than stock. **Not as expected: neither scan reached a `WRONG RESULT`.** Both ended one and two steps before +2550 with the device lost during the bandwidth copy; Windows logged it both times (System log, `nvlddmkm` events 14 and 153 at 19:51:22 and 20:11:27), and the screen goes black for a moment. In the run of check 94 that got this far, the measurements at +2450 and +2500 gave a number and the stress load reported the first wrong result at +2550, without a reset. A step now copies about seven times as much as it did (three readings of about four runs of 128 GB, against one reading of about six runs of 32 GB), and the copy checks nothing, so where the memory starts to fail it is the driver that gives out, not a verdict. Two runs against one: likely the cause, not proven. The search handles it as it should (rest, health probe, confirm four steps lower, the core is still searched), but a driver reset in every run on a card like this is not what the change was for. The saved tune of before the test was put back and applied (`applied best: power 105 %, core +135 MHz, mem +650 MHz`), the tray app restarted, Defender reported nothing. **(c) the same two runs with one reading a step, three only for a step in doubt** (2026-10-11 20:26 to 21:03, a test build of 71e7c2b, unattended): **the reset is still there.** Run 1: stock 5328 it/s; 50 steps from `mem +0: bandwidth 420.4 GB/s` to `mem +2450: bandwidth 502.4 GB/s`; at +2500 the candidate was `STABLE` and then `bandwidth measurement failed: the timing of the copy did not come back` ten seconds later (`nvlddmkm` event 14 at 20:37:07); `mem: bandwidth peak +2200, confirmed +2000, applying +1400`; core +135; soak `STABLE score=5572 it/s peak=70 C power=197 W`; one driver reset. Run 2: stock 5311 it/s; 421.9 GB/s at +0 to 508.0 at +2450; at +2500 `the device was lost during the copy failed (hr=0x887A0006)`, the device hung (event 14 at 20:55:29); `mem: bandwidth peak +2400, confirmed +2200, applying +1500`; core +135; soak `STABLE score=5571 it/s peak=70 C power=196 W`; one driver reset. No step was in doubt in either run (the lowest was 0.75 % below the best before it), so every step was read once; the scan took 3 min 33 s. The two peaks are 200 MHz apart: with one reading a step, which offsets count as within 1 % of the best moves by a few steps. Applied +1400 and +1500, both 5 % faster than stock. So the reset does not come from reading a step three times. Four runs out of four with the longer timed run ended this way, at +2450 or +2500. Not merged; the design for a scan that does not end like this is the open row in `docs/development.md`. The saved tune was put back and applied again (core +135, memory +650), the tray app restarted, Defender reported nothing. **(d) the edge is remembered** (a test build of 7c215c4; the same two runs): expected is that the first scan ends as before and logs nothing new, the journal gets one `mem_edge` line with the GPU id and the driver version, and the second run logs `mem: the memory gave out at +N in an earlier run on this card and driver; the scan stops at +(N-200)`, has no `driver resets during this run` line, applies more than +1000 and is not slower than stock. **Run 1 took the machine down; the test ended there** (2026-10-11 21:23, unattended). Stock 5321 it/s, power 105 %, then the scan: 49 steps from +0 to `mem +2400: bandwidth 539.3 GB/s`, two of them read three times (+1200 and +1250). At +2450 the candidate was `STABLE` and the copy after it failed as in the four runs before: `bandwidth measurement failed: the copy failed (hr=0x887A0007): the device was lost` at 21:35:05, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, `the driver was reset; reconnecting`. That is the last line. Windows restarted by itself within fifteen seconds: Kernel-Power event 41 at 21:35:23, `The previous system shutdown at 21:34:57 was unexpected`, no bugcheck, no dump, no `nvlddmkm` event written. So the fifth scan that met the edge with the longer timed run did not end in a driver reset the search recovers from, but in a reset of the whole machine, during the reconnect or right after it. What held: the candidate's journal entry was closed, so no ceiling of a freeze was left; the line `{"mem_edge":2450,"rig":"GPU-... / 617.42"}` was on disk before the machine went down, so no later run of this build goes there again; the run had saved nothing, and at logon the tray app applied the tune of before the test (`applied best: power 105 %, core +135 MHz, mem +650 MHz`). Not answered: the second run. Seen along the way and not explained: from the first step of the scan the stress load scored 5760 it/s and the bandwidth read about 7 % higher than in every run before (486.6 GB/s at +900 against about 455), while the measurement at stock five minutes earlier was the usual 5321. **(e) how the stress load meets the edge** (the same build, no search): `gao --reset`, then `gao --set-mem` and `gao --stress 3` at +2300, +2350, ... until the verdict is not `STABLE`, five times, at stock power and core +0; expected is one of two outcomes, `WRONG RESULT` every time (then the stress load can lead the copy to the edge) or `DEVICE LOST` (then it cannot). **Not run:** the test ended in (d). It goes to the edge on purpose, five times, and after (d) that is a risk for the machine that needs its owner's word first. **(f) the two-pass search** (the short copy in the scan, the long copy from four steps below the highest step that passed; see `docs/development.md`): tested on a simulated card only. On the reference card, with the edge at +2450 in its journal: `gao --optimize best` once; expected `mem: the memory gave out at +2450 in an earlier run on this card and driver; the scan stops at +2250`, no memory candidate above +2250, `mem: the scan passed +2250; reading from +2200 down with the long copy`, a few `mem +N: bandwidth ... GB/s, the long copy` lines, no `driver resets during this run`, an applied offset above +1000 and a soak that is not slower than stock. **Not run: waits for the owner's word.** **(g) what else uses the card** (2026-10-11 22:00, after the restart of (d), not elevated, the saved tune applied, nothing changed): `gao --stress 5` scored 5828 it/s and `gao --bandwidth --after-load 3` read 484.0 GB/s with its three runs within 0.0 %, against 5226 it/s and 444 to 452 GB/s with the same tune earlier that evening. The series of (a) again, 24 readings each: 64 dispatches 480.0 GB/s, 0.23 % (-0.66 to +0.18); 256 dispatches 475.2 GB/s, 0.68 %, sliding from 480 to 471 during the series; 64 dispatches again 454.0 GB/s, 3.02 % (-4.37 to +4.22), the readings jumping between 434 and 473. Read right after it, the card's use per program: Chrome 8.6 % of the video decoder (19 % at most) and 2.8 % of the 3D engine, with a YouTube video playing; nothing else above 1.1 %. So the short copy is as steady as the long one on a card that is left alone, and a video playing next to it costs about 8 % of the bandwidth and of the stress score and makes the readings jump. Then 40 readings with the short copy, and next to each the other programs on the card in the seconds before it (22:13): 464 to 476 GB/s while Chrome used the video decoder, 444 to 455 while it used the 3D engine as well, and from the reading after Chrome stopped, 477 to 483.5 for the last eleven readings with no other program on the card. So another program using the card lowers the reading by 3 to 8 % from one reading to the next. That explains the scatter and the level changes of (a) in kind. During the tests of that day it was most likely not a video (hardly any was watched) but a code editor built on a browser engine that had stood open since the morning: its last files were written at 21:31:43 and 21:31:44, when it was closed, and the run of (d) logs the bandwidth going from 430.4 to 457.1 GB/s at 21:31:45. So every card measurement of checks 94 and 95 up to then was taken with that program on the card: about 7 % lower, and more scattered, than the card alone. A match to the second; not measured with the editor started again. | ## Notes From 2fbed885969247dd6669686f12505c401a45bb3d Mon Sep 17 00:00:00 2001 From: Rovey Date: Sun, 11 Oct 2026 23:31:41 +0200 Subject: [PATCH 13/15] refactor: the memory search reads with one copy; the long timed run is gone The long bandwidth copy was added because the short one scattered. That scatter came from another program on the card (hardware check 96): on a quiet card the short copy is as steady, and main now looks for such programs before a run. The long copy was also what lost the device at the memory's edge in five scans out of five, once with the machine. The two passes stay: the scan finds where it ends, and the offset is read again from four steps below the highest step it passed. What makes the result repeat is that structure, not a steadier reading. The remembered edge and the coarser, single-reading-proof end of the scan stay as well. Co-Authored-By: Claude Opus 5.5 --- README.md | 2 +- docs/development.md | 4 +- src/app/common.cpp | 6 +- src/core/search.cpp | 19 ++--- src/core/search.hpp | 41 +++++----- src/hw/stress.cpp | 22 +++--- src/hw/stress.hpp | 7 +- tests/test_search.cpp | 174 ++++++++++++++++++------------------------ 8 files changed, 116 insertions(+), 159 deletions(-) diff --git a/README.md b/README.md index 5c1811d..457e724 100644 --- a/README.md +++ b/README.md @@ -176,7 +176,7 @@ flowchart LR - **Every run starts with the card at stock for five minutes,** as long as the result is tested at the end, with the fans left to the NVIDIA driver. That is the "before" of the result: a card is not heated through after 30 s, so a shorter run would make every result look warmer and louder than stock. Some cards get hot there: the driver's own fan control lets them (a 3080 Ti reached 85 °C with its fans at 65 %). The measurement stops at 85 °C, what it measured until then is the "before", and the run goes on. From there every step keeps the fans on a curve that reaches 100 % at the profile's temperature limit. - **Stress test.** A DirectX 11 compute load in which the GPU checks every value it computes against a known answer. Each probe ends in a verdict: `STABLE`, `WRONG RESULT`, `DEVICE LOST`, `STALLED` (the values were right but the card computed almost nothing), `TOO HOT` or `NO TELEMETRY`. -- **Memory comes first and stops at the bandwidth peak,** not at the first error. GDDR6 and GDDR6X retry failed transfers, so an overclocked memory bus loses speed long before it returns a wrong result. The search raises the offset 50 MHz at a time, measures bandwidth at each step until a step fails or has clearly lost bandwidth, then measures again more precisely from 200 MHz below that point down, and keeps the lowest offset within 1 % of the best. The precise measurement is heavy, and memory at its very edge does not survive it; that is why it stays below. Where bandwidth cannot be measured, the memory is left at stock: without that measurement nothing says where to stop, and the first wrong result comes far too late. +- **Memory comes first and stops at the bandwidth peak,** not at the first error. GDDR6 and GDDR6X retry failed transfers, so an overclocked memory bus loses speed long before it returns a wrong result. The search raises the offset 50 MHz at a time, measures bandwidth at each step until a step fails or has clearly lost bandwidth, then measures again from 200 MHz below that point down and keeps the lowest offset within 1 % of the best. Where the memory gave out is remembered for that card and driver, so a later run stays below it without going there again. Where bandwidth cannot be measured, the memory is left at stock: without that measurement nothing says where to stop, and the first wrong result comes far too late. - **The core search climbs from stock.** With the chosen memory offset applied, it raises the core offset 15 MHz at a time until a probe fails, and keeps the last value that passed. How far it may go comes from the range the driver reports for your card, so a card that can hold more is not stopped at a fixed limit. - **Each edge is confirmed.** The value a search ends on gets a 30 s probe, and a lower value is tried if it does not hold. The safety margin is applied to the confirmed value, and the result as a whole must pass the 300 s soak. - **A driver reset ends that clock's search.** On a card the search can push past its edge, a search resets the driver: the screen goes black for a moment. The run then reconnects to the driver, sets the card to stock, leaves it without load for 20 seconds and checks with a short probe at stock that it computes normally again. After that the search tries no new values for that clock: it keeps the last value that passed and confirms four steps below it. Each clock has its own search, so a reset in the memory search does not keep the core from being searched; a run can therefore reset the driver twice, once per clock. One reset more than that ends the run at stock with nothing saved, and so does a card that does not come back. Results on real cards are in [docs/hardware-checks.md](docs/hardware-checks.md), checks 51 and 52. diff --git a/docs/development.md b/docs/development.md index bcf2edc..063f82c 100644 --- a/docs/development.md +++ b/docs/development.md @@ -79,8 +79,8 @@ The applied offset is always at least one step below the confirmed edge. Profile | Without the bandwidth measurement the memory is left at stock | The search used to fall back to a climb on stability alone when the measurement failed at +0. On a tester's 3080 Ti (0.4.3) that climb went to +2300 MHz with the score falling from +2150 on, got its first wrong result there, and the machine froze with a corrupted screen in the 30 s confirm probes just below it. The first wrong result is no edge to stand next to: the memory was correcting errors for hundreds of MHz before it. Now a measurement that does not work means no memory overclock (`left_at_stock`), the run says so, and the core is still searched. `optimize` keeps the stability climb only for a caller that gives no measurement at all; the app always gives one. Why the measurement failed on that card is not known: the log was only in the window and went with the reboot. | | A confirm probe that is too hot at once confirms nothing | A confirm probe that ends `TOO HOT` keeps the clock edge: over 30 s the card may run a little hotter than over the power step's 20 s, and that is the soak's business, which lowers power. But the stress run stops at its first reading above the limit, so a card that is above it when the probe begins gives `TOO HOT` within a second, and that counted as holding too. It happened on the tester's RTX 3080 Ti (hardware check 91): after a driver reset the health probe loads the card at stock power, which on that card means 76 C against a limit of 75, and `confirm core +225` was confirmed by a probe that had tested nothing. Now such a probe counts only when the clock was loaded for `kConfirmMinS` (10 s, clearly longer than the 3 s probe that found the edge) before it got too hot. Otherwise the value is not confirmed, and the card rests (`io.rest`, the 20 s of a recovery, with the fans where the curve last put them) before the next try one step lower: without the rest every lower value would start just as hot, and heat alone would step the edge down to nothing, which is what keeping the edge on `TOO HOT` was for. The same value is not tried again; one step is the price. Not changed: the health probe still runs at stock power, and a soak that begins too hot still lowers the power limit by a step. | | A profile that promises speed keeps no result that is slower than stock | Best of my GPU ended 3.8 % slower than stock on the tester's RTX 3080 Ti (4749 against 4937 it/s, hardware check 91), with the fans at 80 % instead of 66 %, and showed it as **Done**, applied and kept at every logon. Every step had done what it is for: the card runs 83 C at stock with its fans at 100 %, the profile allows 75, so the power limit went to 65 %, and +150 MHz of core clock did not win that back. But a profile called "the best of my GPU" that leaves the card slower has not done what it says, and a result like that is not offered as one. After the soak, `optimize` compares its score with the measurement at stock that went before the run (the "before" the result screen shows, so the rule judges by the number the user sees): below `kNotSlowerScore` (0.99: one percent is what two five-minute runs differ by) the run ends as a failed one does, at stock, with nothing saved, and says what it found, why, and that Undervolt is the profile for a card that runs warm. `Objectives::faster_than_stock` marks the two profiles this holds for, Best of my GPU and Max performance. Not Quiet and Cool & efficient, which trade speed for noise and heat and say so. Not the overclock step of All in one either: it goes on to an undervolt and a fan tune, and its result is judged as a whole; whether that whole may be slower than stock is not decided. This is a backstop, not the cure: what such a card needs from these profiles is in `docs/planned.md` ("never hotter than at stock"). | -| The memory search has two passes: a scan with the short copy finds where it ends, and the long copy reads the offset to keep, four steps below that | **What went wrong.** Four runs on one day applied +650, +1600, +550 and +100 MHz on the reference RTX 4070, a card that runs every game at +1500 (hardware check 94). That card has no bandwidth peak: bandwidth rises by about 0.4 % per 50 MHz step all the way to +2500 and the stress load computes its first wrong result at +2550, so the scan could only end early by chance, and where it did decided the offset. **What was measured** (hardware check 95 a, no search, nothing changed on the card). A reading is the median of three timed runs that agree within 1 %. With the timed run as it was, 32 GB or 75 ms, readings scatter by 0.4 to 1.3 % (standard deviation over 24 readings, six series), single ones by up to 4.8 %, and now and then one does not settle. With 128 GB per run it is 0.2 %. Apart from that the readings change level: for up to a minute every reading is about 1.8 % higher, at one and the same offset; the cause is not known. The app's own window, drawing every 100 or 33 ms next to the measurement, adds nothing, so nothing was built for it. **What was tried first, and what it did** (check 95 b to d). The long timed run at every step of the scan: the offset repeated (+1400 to +1500 in four runs, 4 to 5 % faster than stock), and every scan now ran to where the memory gives out and met it with the long copy. Five scans out of five lost the device there, at +2450 or +2500, a driver reset each time, with three readings a step and with one; and the fifth took the whole machine down within seconds (Kernel-Power 41, no bugcheck). In the one scan before all this that got to the edge, the short copy read a number at +2450 and +2500 and the stress load reported the wrong result at +2550, with no reset. One against five is thin, and it is all there is: the long copy is too heavy for memory at its edge, the short one was not. **The design.** One measurement cannot do both jobs, so there are two passes. *The scan* goes up as before and only has to find where it ends: a step that is not stable, or one that has lost more than `kBandwidthDrop` (5 %) against the best. It reads with the short copy (`kBwDispatches` 64), which is what the released versions take to the edge. Because those readings scatter, two things keep one of them from deciding: a reading that would end the scan is taken `kBandwidthReadings` (3) times and the middle one counts, and the best that readings are held against is the highest middle of three steps in a row, not the highest reading (the highest of fifty scattering readings is always one that came out high). 5 % is far more than the readings move by themselves; the scan is not where the offset is chosen any more, so it can afford to be coarse. *The second pass* chooses the offset, with the long copy (`MeasureBandwidth(true)`, `OptimizeIo::bandwidth_precise`): it starts `kPreciseMarginSteps` (4 steps, 200 MHz) below the highest step the scan passed (`precise_top`), goes down a step at a time until a reading is more than `kBandwidthTie` (1 %) below the best of them, and keeps the lowest offset within that 1 % (`bandwidth_peak_below`). On the reference card: the scan passes +2500, the long copy reads +2300, +2250, +2200 and +2150, and +2200 is kept; confirmed there, 70 % applied, +1500. The long copy lost the device at the highest step the stress load passed and one below it; four steps leave two to spare. The margin is kept whatever ended the scan, a drop and the driver's own range included: what is above was never tried. After a reset event in the scan the second pass is skipped, and confirming starts four steps below where it would have begun. **The edge is met once** (built before this, kept): a scan that ended in a wrong result or a reset event writes that offset to the journal with the GPU id and the driver version (`{"mem_edge":2550,"rig":" / "}`, a line without an id, so older builds skip it). A later scan on the same card and driver stops four steps below the lowest such offset, and its second pass starts where the first run's did: the margin counts from the edge, not a second time from where the scan was held. With another driver or card the line does not count. **What it costs.** A card that does have a peak keeps an offset up to 200 MHz below it when the scan ends close above the peak: the margin does not know a drop from an edge. A scan that ends for a harmless reason (readings that did not settle) loses those 200 MHz as well. The second pass adds about four steps, half a minute. **Not chosen.** The long copy everywhere: see above. The short copy everywhere, as released: the offset is chance (the +100 run). Three long readings a step, or only for a step in doubt: the reset came with both. Only remembering the edge: the first meeting can cost the machine, and once per card is once too often. The stress load probing four steps ahead of the copy: it needs the proof that the load at the edge ends in a wrong result and not in a reset of its own, and getting that proof means going to the edge on purpose. A fixed top for the scan (an offset, a share of the range, a gain over +0): there is nothing to choose the number by. **Tested** on a simulated card with the reference card's numbers (wrong result at +2550, the long copy losing the device from +2450): the long copy is never taken above +2300, in the first run and in the one after it, with short readings scattering by 1.3 % on two hundred seeds, and on forty more with level changes of 1.8 % thrown in; with the edge anywhere from +300 to +2900; and the tests fail when the margin is set to nothing. **Not tested on the card.** See the row below. | -| Open: the two-pass memory search has not run on a card | The last card test (hardware check 95 d) ended with the machine restarting, and nothing has been run on the card since. Before this is merged, its owner decides whether and when to run it there. What a run would show, and what it cannot: the journal of the reference card holds the edge at +2450 now, so a run of this build stops its scan at +2250 and reads with the long copy from +2200 down, at offsets every earlier scan passed. That tests the second pass and the result, with little risk. It does not test the first meeting with the edge with the short copy: that is what the released versions do, it went well the one time it was seen, and seeing it again means removing the edge from the journal and going there. **Still not known.** Why the long copy fails where the short one did not: the amount copied in one go, or only that five runs are more than one. Whether a card with a real bandwidth peak behaves as assumed; none has been measured through a scan (the tester's RTX 3080 Ti never got a reading to settle at +0). And one observation from the run that crashed: from the first step of its scan the stress load scored 5760 it/s and the bandwidth read about 7 % higher than in every run before (486.6 GB/s at +900 against about 455), while the measurement at stock five minutes earlier was the usual 5321. It was not the monitor going to sleep, as first suspected: it never does on that machine, and the run's log shows the change in the middle of the scan, between +350 and +550 (430 to 457 GB/s within one step, 5260 to 5750 it/s), not at its start. **What it is: another program using the card.** Measured after the restart, with the same tune applied and nothing changed (hardware check 95 g): the short copy read 480 GB/s with a scatter of 0.23 %, as steady as the long copy was all day; minutes later, with a YouTube video playing in Chrome (8.6 % of the card's video decoder, 2.8 % of its 3D engine), the same series read 434 to 473 GB/s with a scatter of 3 %, and the stress load scores about 5300 it/s in the one state and 5750 to 5830 in the other. A second series, with the card's use per program sampled next to every reading, shows it to the second: 483 GB/s with no other program on the card, about 470 while Chrome decodes video, 444 to 455 while Chrome also draws, and back to 483 on the reading after Chrome stopped. So another program using the card moves the reading by 3 to 8 %, at once. **What it was on the day of the tests: most likely a code editor that stood open.** Hardly any video was watched during those tests. But two editors built on a browser engine (Trae and ZCode) had been installed that morning and stood open all day, and the last files Trae wrote are from 21:31:43 and 21:31:44, when it was closed: the second before the bandwidth went from 430 to 457 GB/s in the run's log (21:31:45). ZCode's are from 21:32:09. A match to the second, not a test: nobody has started that editor again and measured next to it. If it holds, three things follow, and none is built. The scatter of the short copy that this design is built around (0.4 to 1.3 %) was for the larger part the video, not the copy; the design stands, it only has more room than was thought. The level changes of 1.8 % are the same thing in small. And the comparison of a result with stock is worth nothing when such a program starts or stops between the two: 8 % is far more than the 1 % by which a speed profile is judged (`kNotSlowerScore`) and than anything the efficiency search decides on. What to do about that is open: say so before a run (the README names games, MSI Afterburner and RivaTuner, not a browser with a video), or read the card's use per program (Windows keeps it, per engine) and warn or wait. | +| The memory search has two passes, and remembers where a card's memory gave out | **What went wrong.** Four runs on one day applied +650, +1600, +550 and +100 MHz on the reference RTX 4070, a card that runs every game at +1500 (hardware check 94). That card has no bandwidth peak: bandwidth rises by about 0.4 % per 50 MHz step all the way to +2500 and the stress load computes its first wrong result at +2550. So the scan could only end early by chance, and where it did decided the offset. **The cause, found late.** The chance came from another program on the card: an editor stood open all that day and took 2.2 % of the 3D engine, and with it the bandwidth readings scattered by 0.4 to 1.3 % and changed level by 1.8 %. On a card that is left alone they scatter by 0.2 % (hardware check 96). A run now looks for such programs before it begins and asks (the row on other programs on the card). What is left for the search to do is to give a sound result also when the user goes on next to them, and not to go to the memory's edge more often than it has to. **A detour, taken back.** The first answer to the scatter was a timed run four times as long (128 GB copied in one go). It was steadier next to the editor, and no steadier than the short one on a quiet card. With it every scan ran to where the memory gives out and met it with that copy: five scans out of five lost the device there, at +2450 or +2500, and the fifth took the whole machine down within seconds (check 95 b to d). In the one scan before that got to the edge, the short copy read a number at +2450 and +2500 and the stress load reported the wrong result at +2550, with no reset. The long run is gone; `kBwDispatches` is 64 as it was. **The design.** *The scan* goes up and only has to find where it ends: a step that is not stable, or one more than `kBandwidthDrop` (5 %) below the best. Two things keep a single reading from deciding: a reading that would end the scan is taken `kBandwidthReadings` (3) times and the middle one counts, and the best that readings are held against is the highest middle of three steps in a row, not the highest reading (the highest of fifty scattering readings is always one that came out high). 5 % is far more than readings move by themselves, also next to another program; the scan is not where the offset is chosen any more, so it can afford to be coarse. *The second pass* chooses the offset: it starts `kPeakMarginSteps` (4 steps, 200 MHz) below the highest step the scan passed (`second_pass_top`), reads again a step at a time going down until a reading is more than `kBandwidthTie` (1 %) below the best of them, and keeps the lowest offset within that 1 % (`bandwidth_peak_below`). On the reference card: the scan passes +2500, the second pass reads +2300, +2250, +2200 and +2150, and +2200 is kept; confirmed there, 70 % applied, +1500. Whatever the readings do, the offset comes out of a few steps below a point the card itself sets, so it repeats: +2050 to +2300 at the scatter of a quiet card. The margin is kept whatever ended the scan, a drop and the driver's own range included: what is above was never tried. After a reset event in the scan the second pass is skipped, and confirming starts four steps lower still. **The edge is met once.** A scan that ended in a wrong result or a reset event writes that offset to the journal with the GPU id and the driver version (`{"mem_edge":2550,"rig":" / "}`, a line without an id, so older builds skip it). A later scan on the same card and driver stops four steps below the lowest such offset, and says so; its second pass starts where the first run's did, because the margin counts from the edge and not a second time from where the scan was held. With another driver or another card the line does not count. A scan that ended for another reason (a bandwidth peak, readings that did not settle, the driver's range) writes nothing. **What it costs.** A card that does have a peak keeps an offset up to 200 MHz below it when the scan ends close above the peak: the margin does not know a drop from an edge. A scan that ends for a harmless reason (readings that did not settle) loses those 200 MHz as well. The second pass adds about four steps, a quarter of a minute. **Not chosen.** The long copy, anywhere: see above. Three readings at every step: two minutes a scan, and it decided nothing a single reading did not. Only remembering the edge: the result would still be chance for whoever goes on next to another program. The stress load probing four steps ahead of the copy: there is no heavy copy left to protect the edge from. A fixed top for the scan (an offset, a share of the range, a gain over +0): there is nothing to choose the number by. **Tested** on a simulated card with the reference card's numbers (wrong result at +2550): the second pass never reads above +2300, in the first run and in the one after it; with readings scattering by 1.3 % in both passes and level changes thrown in; with the edge anywhere from +300 to +2900; and the tests fail when the margin is set to nothing. **Not tested on a card.** See the row below. | +| Open: the two-pass memory search has not run on a card | The last card test of this work (hardware check 95 d) ended with the machine restarting, with the long copy that has since been taken out. Nothing of the search as it is now has run on a card. What a run on the reference card would show, and what it cannot: its journal holds the edge at +2450, so a run stops its scan at +2250 and reads again from +2200 down, at offsets every earlier scan passed. That tests the second pass and the result, with little risk (hardware check 95 f). It does not test the first meeting with the edge: that is what the released versions do with the same copy, it ended in a wrong result the one time it was seen, and seeing it again means removing the edge from the journal and going there. **Still not known.** Whether a card with a real bandwidth peak behaves as assumed; none has been measured through a scan (the tester's RTX 3080 Ti never got a reading to settle at +0). And why the long copy failed at the edge where the short one did not: the amount copied in one go, or only that five runs are more than one. It no longer matters for the search, and is noted because it was never explained. | | Open: why the bandwidth measurement fails on some cards, and what then | On the tester's RTX 3080 Ti it failed at +0, and since 0.4.4 that means no memory overclock at all on such a card, though it computed right far beyond +2000 MHz. The measurement gives a number only when three runs in a row agree within 1 % (`kBwSettle`), and it throws away every run the driver's timer calls disjoint (the clock changed under it). That card was held at a power limit of 65 % and ran at 83 C when it was measured; the reference card has only ever been measured at 105 %, where the measurement itself draws 179 of 210 W. So the suspicion is the power limit, and hardware check 90 tests exactly that on the reference card. What it shows decides what is built, and nothing is built before it: **(1) runs are thrown away as disjoint, and the ones that are kept agree:** the timer is what fails, not the card. Time the copy differently (wall clock around a flush, or count a disjoint run when its neighbours agree) and keep the method. **(2) the readings themselves scatter by more than 1 %:** a peak that is found by a 1 % drop (`kBandwidthDrop`) cannot be found in that noise. Then the memory has to be measured where the card is not held back: search it before the power step, at stock power, where a probe and its measurement last five seconds; or take more readings and compare medians. **(3) it does not fail at 65 % on the reference card:** the power limit is not the cause. Then it was the heat (a card throttling at its temperature limit changes clock as well), or another program, and only the tester's `run.log` can say: since this change a measurement that never settled writes how many runs gave a reading, between which values, and how many were thrown away. In no case does the climb on stability alone come back. **Outcome of check 90: (3).** On the reference card the measurement gave a number fifteen times out of fifteen at 100, 80 and 65 %, in three or four runs, with nothing thrown away, and the bandwidth was the same at every limit. The power limit is ruled out; (1) and (2) are not built. What is left is the heat of that card, another program, or a timer query that failed, and the tester's `run.log` is the next thing that can tell them apart. **The tester's run (check 91): it is (2), but only inside a run.** Twenty readings between 763.9 and 821.2 GB/s, none thrown away, three seconds after the load began; after thirty seconds of load the same card read within 0.2 %. The suspicion now is the moment, not the card: the measurement is taken while the power limiter is still catching up. If `gao --bandwidth --after-load 3` scatters on that card and `--after-load 30` does not, that is it, and the fix is to measure when the card has settled (keep taking runs for some seconds instead of twenty runs), not to change the method. | | Every line of a run is also in `run.log` | `logged()` wraps the log hook of both front ends: a line is appended and flushed before it is shown, so the step a machine freezes in is the last line on disk. One file in the state folder, moved aside at 1 MB (`run.log.old`). Only elevated and only after `ensure_app_dir` has verified the folder: an elevated process appends to nothing a user could have put there. | | A run does not start while the NVIDIA app's Automatic Tuning is on | Switched on, a service of the driver scans the card when it sees fit: it loads it, changes its clocks, and starts again by itself (hardware check 89: a scan that was stopped started again 14 ms later, and after a restart of the service it resumed within two seconds). A run next to that measures nothing it can trust, so it is refused, like a run next to Afterburner, and with the same weight: a courtesy, not a guard. Read from where that service keeps the switch, `%ProgramData%\NVIDIA Corporation\nvtopps\nvtopps.db3`, table `ocScannerSettings`, fields `backgroundScanEnabled` and `grdOcEnabled`, joined to `TelemetryGpuCfg` so that only a card that is in the PC counts (a row of an earlier card can never be switched off again). That this is the source and not a copy of it was measured, not assumed: with the field set the service took the scan up again after every restart, with it cleared it did not, while the NVIDIA app itself showed the switch as off. Nothing documents the file. So: opened read-only and immutable (no lock, the service never waits for us), through `winsqlite3.dll` from System32 (no dependency added), and everything that is not exactly as expected reads as "unknown", which starts the run as before. The registry value the app sets next to it (`HKLM\SOFTWARE\NVIDIA Corporation\Global\NvApp\OC\`) changes at the same moment but belongs to the app, not to the service, and is not used. **The overlay is not checked:** no setting for it was found, its processes run whether it is open or not, and the one value that moved (`IsShadowPlayEnabled`) followed the service being up, not a switch. NVAPI was not tried: an undocumented call is no better than an undocumented file, and it cannot be compared with anything. | diff --git a/src/app/common.cpp b/src/app/common.cpp index 4e60ece..81c022a 100644 --- a/src/app/common.cpp +++ b/src/app/common.cpp @@ -381,13 +381,11 @@ OptimizeOutcome run_optimize(Preset preset, const OptimizeHooks& hooks, const st }; io.aborted = hooks.aborted; io.log = hooks.log; - const auto measure = [&](bool precise) { - const auto gbps = load->MeasureBandwidth(precise); + io.bandwidth = [&] { + const auto gbps = load->MeasureBandwidth(); if (!gbps) log("bandwidth measurement failed: " + load->Error()); return gbps; }; - io.bandwidth = [=] { return measure(false); }; - io.bandwidth_precise = [=] { return measure(true); }; io.bandwidth_unsettled = [&] { return load->BandwidthUnsettled(); }; // The wait between a driver reset and the next load. The card is left // alone: no load, no driver call, no fan tick, and no log line (the search diff --git a/src/core/search.cpp b/src/core/search.cpp index 67306ba..12b6c04 100644 --- a/src/core/search.cpp +++ b/src/core/search.cpp @@ -93,11 +93,11 @@ MemScanEnd bandwidth_scan(int lo, int hi, int step, int ceiling, const std::func return end; } -int precise_top(const MemScanEnd& end, int lo, int step, int known_edge) { +int second_pass_top(const MemScanEnd& end, int lo, int step, int known_edge) { // The lowest offset that is known, or has to be assumed, not to hold. int bound = end.top + step; if (end.why == MemScanEnd::Why::Limit && known_edge != INT_MAX && known_edge > bound) bound = known_edge; - return std::max(lo, bound - step - kPreciseMarginSteps * step); + return std::max(lo, bound - step - kPeakMarginSteps * step); } int bandwidth_peak_below(int top, int lo, int step, const std::function& sample) { @@ -632,9 +632,8 @@ OptimizeResult optimize(const GpuControl& gpu, const Objectives& obj, Journal& j if (by_bandwidth) { bool unavailable = false; // One step of either pass: the candidate, then the reading while - // it is still applied. `precise`: with the long copy. - const auto sample = [&](int v, bool precise) { - const auto& read = precise && io.bandwidth_precise ? io.bandwidth_precise : io.bandwidth; + // it is still applied. `again`: in the second pass. + const auto sample = [&](int v, bool again) { MemSample m; // One reading. One that only did not settle is taken again, // kBandwidthReadings times at most; any other failure ends @@ -648,7 +647,7 @@ OptimizeResult optimize(const GpuControl& gpu, const Objectives& obj, Journal& j m.stable = explore(v, reset_at, [&] { const auto verdict = clock_candidate(std::nullopt, mem_j, 0, v, kClockProbeS, [&] { for (int i = 0; i < kBandwidthReadings && !gbps && !failed; ++i) { - gbps = read(); + gbps = io.bandwidth(); failed = !gbps && !(io.bandwidth_unsettled && io.bandwidth_unsettled()); } }); @@ -678,21 +677,19 @@ OptimizeResult optimize(const GpuControl& gpu, const Objectives& obj, Journal& j if (m.stable) { m.gbps = *gbps; char buf[64]; - std::snprintf(buf, sizeof(buf), "mem +%d: bandwidth %.1f GB/s%s", v, m.gbps, - precise && io.bandwidth_precise ? ", the long copy" : ""); + std::snprintf(buf, sizeof(buf), "mem +%d: bandwidth %.1f GB/s%s", v, m.gbps, again ? ", read again" : ""); log(buf); } return m; }; const MemScanEnd end = bandwidth_scan(0, bounds.mem_max_mhz, kMemStep, ceiling, [&](int v) { return sample(v, false); }); - int peak = precise_top(end, 0, kMemStep, journal.mem_edge()); + int peak = second_pass_top(end, 0, kMemStep, journal.mem_edge()); // The second pass, when there is something to read and the scan // left the card in a state to read it: not after a reset event // (nothing new is explored then, and confirming starts // kResetBackoffSteps below `peak`), not on a stopped run. if (!unavailable && stopped.empty() && !explored_enough && peak > 0) { - log("mem: the scan passed " + offset_text(end.top) + "; reading from " + offset_text(peak) + " down" + - (io.bandwidth_precise ? " with the long copy" : " once more")); + log("mem: the scan passed " + offset_text(end.top) + "; reading again from " + offset_text(peak) + " down"); peak = bandwidth_peak_below(peak, 0, kMemStep, [&](int v) { return sample(v, true); }); } // An unavailable measurement ended the scan at +0, and nothing diff --git a/src/core/search.hpp b/src/core/search.hpp index 613c832..63e5e98 100644 --- a/src/core/search.hpp +++ b/src/core/search.hpp @@ -68,20 +68,18 @@ int apply_margin(int max, int step, float perf_push); // errors: GDDR6X/GDDR6 retry bad transfers. The memory search therefore looks // for the bandwidth peak, not the stability ceiling. // -// It does so in two passes, because one measurement cannot do both jobs. -// The scan goes up and only has to find where it ends: a step that fails, or -// one that has clearly lost bandwidth. It reads with the short copy, which -// scatters (0.4 to 1.3 % on the reference RTX 4070, single readings by more) -// and is light: the one scan that took it to the memory's edge got a wrong -// result from the stress load there. Then the offset to keep is read with the -// long copy, which scatters by 0.2 % and is heavy: in five scans out of five -// it lost the device at the edge, and once the machine with it (hardware -// check 95). So the long copy is only taken kPreciseMarginSteps below the -// highest step the scan passed, going down from there. +// It does so in two passes. The scan goes up and only has to find where it +// ends: a step that fails, or one that has clearly lost bandwidth. Then the +// offset to keep is read again, starting kPeakMarginSteps below the highest +// step the scan passed and going down from there. Where the scan ended no +// longer decides the offset by itself: on the reference RTX 4070, whose +// bandwidth rises until its memory fails, scans that ended by chance saved +// +100, +650 and +1600 MHz on one day (hardware check 94), and the cause was +// another program on the card that made the readings scatter (check 96). // // The scan ends once a step is this far below the best. Far more than the -// short readings move by themselves, level changes of 1.8 % included: where -// the scan ends by chance, the offset is chance again. +// readings move by themselves, also next to another program (up to 3 %): +// where the scan ends by chance, the offset is chance again. inline constexpr double kBandwidthDrop = 0.05; // Within 1 % of the best counts as the best, in the second pass: one 50 MHz // step is worth ~0.5 %, so a narrower band would let noise pick the offset. @@ -94,11 +92,12 @@ inline constexpr double kBandwidthTie = 0.01; // scans at +250, at +1000 and not at all on one day, and the runs saved +100, // +650 and +1600 MHz (hardware check 94). inline constexpr int kBandwidthReadings = 3; -// How far below the highest step the scan passed the long copy stays, in -// steps. On the reference card the stress load passed +2500 and the long copy -// lost the device at +2450 and +2500: up to one step below the highest passed -// one. Four steps is where confirming starts after a reset, too. -inline constexpr int kPreciseMarginSteps = 4; +// How far below the highest step the scan passed the second pass begins, in +// steps. The highest steps are where the memory is about to give out: the +// stress load passes them for three seconds, and a longer look at them is +// what a confirm probe is for, further down. Four steps is where confirming +// starts after a reset, too. +inline constexpr int kPeakMarginSteps = 4; // A confirm probe that ends too hot keeps the clock edge (heat is the power // step's and the soak's business), but only when the clock was loaded this @@ -155,11 +154,11 @@ struct MemScanEnd { // times first, and its middle reading counts. MemScanEnd bandwidth_scan(int lo, int hi, int step, int ceiling, const std::function& sample); -// The highest offset the second pass may read: kPreciseMarginSteps below the +// The highest offset the second pass reads: kPeakMarginSteps below the // highest offset that can have passed. `known_edge` is where the memory gave // out in an earlier run (INT_MAX: nowhere): a scan that was held below it // counts its margin from that edge, not a second time from where it was held. -int precise_top(const MemScanEnd& end, int lo, int step, int known_edge); +int second_pass_top(const MemScanEnd& end, int lo, int step, int known_edge); // The second pass: samples top, top-step, ... (>= lo) until a sample is more // than kBandwidthTie below the best of them, or not stable. Returns the @@ -203,10 +202,6 @@ struct OptimizeIo { // getting faster. The app always gives one; when it fails at stock, the // memory is left at stock. std::function()> bandwidth; - // The same with the long copy: steadier, and too heavy for memory at its - // edge. Only called kPreciseMarginSteps below the highest step a scan - // passed. Empty: `bandwidth` is used there as well. - std::function()> bandwidth_precise; // Asked after `bandwidth` returned nothing: true when the load measured // but its readings did not agree (other programs using the card, say). // That is no sign of a driver reset. Empty: every failure may be one. diff --git a/src/hw/stress.cpp b/src/hw/stress.cpp index 85ac3ee..0365426 100644 --- a/src/hw/stress.cpp +++ b/src/hw/stress.cpp @@ -28,16 +28,15 @@ constexpr double kTargetBatchMs = 250.0; constexpr int kMaxDispatches = 2048; constexpr UINT kBwBytes = 256u * 1024 * 1024; // per buffer; far above the 4070's 36 MB L2 constexpr UINT kBwGroups = 1024; -// Dispatches per timed run. The short run copies 32 GB, about 75 ms on the -// reference RTX 4070: short enough for whatever else the machine does to land -// in one run and not in the next, so readings scatter by 0.4 to 1.3 % -// (standard deviation) and now and then do not settle. The long run copies -// 128 GB in 0.3 s and scatters by 0.2 % (hardware check 95); what it costs is -// in stress.hpp. +// Dispatches per timed run: 32 GB copied, about 75 ms on the reference RTX +// 4070. Readings of it scatter by 0.2 % on a card that is left alone, and by +// 0.4 to 3 % when another program uses the card (hardware checks 95 and 96). +// A run four times as long was tried for steadier readings and taken out +// again: it was no steadier on a quiet card, and with it memory at its edge +// lost the device in five scans out of five, once the machine with it. constexpr int kBwDispatches = 64; -constexpr int kBwDispatchesPrecise = 256; constexpr size_t kBwRuns = 3; // consecutive runs that must agree -constexpr int kBwMaxRuns = 20; // ~1.5 s at full speed (~6 s for the long run); more if clocks are low +constexpr int kBwMaxRuns = 20; // ~1.5 s at full speed; more if clocks are low constexpr double kBwSettle = 0.01; // agree = within 1 % // Grid-stride copy of 16-byte elements between two raw buffers. @@ -291,9 +290,8 @@ StressBatch Stress::Batch() { return out; } -std::optional Stress::MeasureBandwidth(bool precise) { +std::optional Stress::MeasureBandwidth() { Impl& d = *impl_; - const int dispatches = precise ? kBwDispatchesPrecise : kBwDispatches; bw_unsettled_ = false; bw_runs_.clear(); if (!d.device && !CreateDevice()) return std::nullopt; @@ -365,7 +363,7 @@ std::optional Stress::MeasureBandwidth(bool precise) { d.ctx->CSSetUnorderedAccessViews(0, 2, uavs, nullptr); d.ctx->Begin(d.q_disjoint.Get()); d.ctx->End(d.q_begin.Get()); - for (int i = 0; i < dispatches; ++i) d.ctx->Dispatch(kBwGroups, 1, 1); + for (int i = 0; i < kBwDispatches; ++i) d.ctx->Dispatch(kBwGroups, 1, 1); d.ctx->End(d.q_end.Get()); d.ctx->End(d.q_disjoint.Get()); d.ctx->CSSetUnorderedAccessViews(0, 2, none, nullptr); @@ -384,7 +382,7 @@ std::optional Stress::MeasureBandwidth(bool precise) { continue; } const double seconds = double(t1 - t0) / double(dj.Frequency); - runs.push_back(2.0 * kBwBytes * dispatches / seconds / 1e9); + runs.push_back(2.0 * kBwBytes * kBwDispatches / seconds / 1e9); bw_runs_.push_back(runs.back()); if (runs.size() < kBwRuns) continue; std::vector last(runs.end() - kBwRuns, runs.end()); diff --git a/src/hw/stress.hpp b/src/hw/stress.hpp index 21429f2..f9a144d 100644 --- a/src/hw/stress.hpp +++ b/src/hw/stress.hpp @@ -41,12 +41,7 @@ class Stress { // Device-memory bandwidth in GB/s (read + write) from a 256 MB buffer copy, // timed with GPU timestamps; median of 3 runs. nullopt when the device was // lost or the timing was unusable. - // precise: with the long timed run. Steadier readings (0.2 % against 0.4 - // to 1.3 % on the reference RTX 4070), and four times the copying in one - // go: memory at its edge lost the device under it in five scans out of - // five, and once the machine (hardware check 95). The search only asks - // for it well below that edge. - std::optional MeasureBandwidth(bool precise = false); + std::optional MeasureBandwidth(); // Why the last MeasureBandwidth() returned nothing: true when it measured // but the runs never agreed (other programs using the card, say); false // when the device was lost or a timing query failed. diff --git a/tests/test_search.cpp b/tests/test_search.cpp index b137cbd..aec58cc 100644 --- a/tests/test_search.cpp +++ b/tests/test_search.cpp @@ -131,14 +131,14 @@ struct FakeCard { bool lost_on_first_core_confirm = false; // the first 30 s probe with a non-zero core offset ends DEVICE LOST bool lost_on_first_mem_confirm = false; // the first 30 s probe with a non-zero memory offset ends DEVICE LOST int bw_measurements = 0; // bandwidth measurements taken with the short copy - // The long copy. Quiet unless bw_precise_noise says otherwise, and from - // this memory offset on it takes the device down, as it did on the - // reference card at its edge (hardware check 95). - int bw_precise_loses_device_from = INT_MAX; - std::function bw_precise_noise; - int bw_precise_measurements = 0; - int max_precise_mem = -1; // the highest offset it was taken at - std::map bw_precise_at; + // The readings of the second pass: those taken below an offset the card + // has already been at. bw_again_noise: what they are off by, when that is + // to differ from bw_noise. From bw_again_loses_device_from on, such a + // reading takes the device down. + int bw_again_loses_device_from = INT_MAX; + std::function bw_again_noise; + int bw_again_measurements = 0; + int max_again_mem = -1; // the highest offset one was taken at bool bw_fails_at_zero = false; // the measurement at memory +0 fails; nothing is wrong with the card bool lost_on_first_soak = false; // the first 300 s probe ends DEVICE LOST bool fail_set_on_first_soak = false; // the first write of the soak state fails once @@ -304,7 +304,6 @@ struct Run { bool no_rest = false; // the application gives the search no way to wait std::optional stock; // the measurement at stock that went before the run, when there was one std::string rig = "card-a / 617.42"; // the card and driver the journal's memory edges are bound to - bool no_precise = false; // the application gives the search one way of measuring bandwidth only std::vector> steps; // what the search said it was doing, and for how long OptimizeResult go(Preset preset, GpuControl gpu) { Journal j(card.journal, [this](const std::string& l) { @@ -336,8 +335,16 @@ struct Run { }; if (card.bw_curve) io.bandwidth = [this]() -> std::optional { - ++card.bw_measurements; const int nth = card.bw_readings_at[card.mem]++; + if (card.mem < card.max_mem_seen) { // the second pass + ++card.bw_again_measurements; + card.max_again_mem = std::max(card.max_again_mem, card.mem); + card.bw_unsettled = false; + if (card.mem >= card.bw_again_loses_device_from) { card.driver_reset(); return std::nullopt; } + const auto& noise = card.bw_again_noise ? card.bw_again_noise : card.bw_noise; + return card.bw_curve(card.mem) * (noise ? noise(card.mem, nth) : 1.0); + } + ++card.bw_measurements; card.bw_unsettled = card.mem == card.bw_unsettled_at_mem || (card.mem == card.bw_unsettled_once_at_mem && nth == 0); if (card.bw_unsettled) return std::nullopt; if (card.bw_loses_device) { card.driver_reset(); return std::nullopt; } @@ -350,15 +357,6 @@ struct Run { if (card.bw_fails) return std::nullopt; return card.bw_curve(card.mem) * (card.bw_noise ? card.bw_noise(card.mem, nth) : 1.0); }; - if (card.bw_curve && !no_precise) - io.bandwidth_precise = [this]() -> std::optional { - ++card.bw_precise_measurements; - const int nth = card.bw_precise_at[card.mem]++; - card.max_precise_mem = std::max(card.max_precise_mem, card.mem); - card.bw_unsettled = false; - if (card.mem >= card.bw_precise_loses_device_from) { card.driver_reset(); return std::nullopt; } - return card.bw_curve(card.mem) * (card.bw_precise_noise ? card.bw_precise_noise(card.mem, nth) : 1.0); - }; if (card.bw_curve) io.bandwidth_unsettled = [this] { return card.bw_unsettled; }; io.measuring = [this](const std::string& what, double seconds) { steps.emplace_back(what, seconds); }; return optimize(gpu, objectives_for(preset), j, io, stock); @@ -783,31 +781,31 @@ TEST_CASE("the bandwidth scan samples lo and respects the ceiling") { CHECK(none.why == MemScanEnd::Why::Failed); } -TEST_CASE("the long copy stays four steps below the highest step the scan passed") { +TEST_CASE("the second pass begins four steps below the highest step the scan passed") { using Why = MemScanEnd::Why; // The step above +2500 failed: +2500 passed, so +2300. - CHECK(precise_top({2500, Why::Failed}, 0, 50, INT_MAX) == 2300); + CHECK(second_pass_top({2500, Why::Failed}, 0, 50, INT_MAX) == 2300); // +1000 passed and had lost bandwidth: +800. - CHECK(precise_top({1000, Why::Dropped}, 0, 50, INT_MAX) == 800); + CHECK(second_pass_top({1000, Why::Dropped}, 0, 50, INT_MAX) == 800); // The driver's range, or the ceiling of a freeze: nothing failed, and the // margin is kept all the same. What is above was never tried. - CHECK(precise_top({3000, Why::Limit}, 0, 50, INT_MAX) == 2800); + CHECK(second_pass_top({3000, Why::Limit}, 0, 50, INT_MAX) == 2800); // Never below lo, and a scan that passed nothing reads nothing above it. - CHECK(precise_top({150, Why::Failed}, 0, 50, INT_MAX) == 0); - CHECK(precise_top({0, Why::Failed}, 0, 50, INT_MAX) == 0); + CHECK(second_pass_top({150, Why::Failed}, 0, 50, INT_MAX) == 0); + CHECK(second_pass_top({0, Why::Failed}, 0, 50, INT_MAX) == 0); } TEST_CASE("a scan held below a known edge counts the margin from that edge, once") { using Why = MemScanEnd::Why; // The run that found the edge at +2550 read from +2300. The run after it is // held at +2350 and reads from +2300 as well, not from +2150. - CHECK(precise_top({2500, Why::Failed}, 0, 50, INT_MAX) == 2300); - CHECK(precise_top({2350, Why::Limit}, 0, 50, 2550) == 2300); + CHECK(second_pass_top({2500, Why::Failed}, 0, 50, INT_MAX) == 2300); + CHECK(second_pass_top({2350, Why::Limit}, 0, 50, 2550) == 2300); // A step that failed below the known edge counts, not the edge. - CHECK(precise_top({2000, Why::Failed}, 0, 50, 2550) == 1800); - CHECK(precise_top({2000, Why::Dropped}, 0, 50, 2550) == 1800); + CHECK(second_pass_top({2000, Why::Failed}, 0, 50, 2550) == 1800); + CHECK(second_pass_top({2000, Why::Dropped}, 0, 50, 2550) == 1800); // An edge that is no higher than where the scan got changes nothing. - CHECK(precise_top({2350, Why::Limit}, 0, 50, 2400) == 2150); + CHECK(second_pass_top({2350, Why::Limit}, 0, 50, 2400) == 2150); } TEST_CASE("the second pass goes down from the top and keeps the lowest offset within the tie") { @@ -832,8 +830,8 @@ TEST_CASE("the second pass goes down from the top and keeps the lowest offset wi // A flat curve keeps the lowest offset there is. CHECK(bandwidth_peak_below(400, 0, 50, [](int v) { return MemSample{true, 500.0 + ((v / 50) % 2)}; }) == 0); - // Readings that scatter by twice what the long copy does on the 4070 leave - // a flat curve at the bottom. + // Readings that scatter by twice what they do on a quiet 4070 leave a flat + // curve at the bottom. CHECK(bandwidth_peak_below(400, 0, 50, [](int v) { return MemSample{true, 500.0 * (1 + (((v / 50) % 3) - 1) * 0.004)}; }) == 0); } @@ -870,12 +868,12 @@ TEST_CASE("memory search stops at the bandwidth peak, not the stability edge") { const auto r = run.go(Preset::BestOfMyGpu); REQUIRE(r.ok); // The scan ends at +1000, the first step that has lost 5 % (7.8 % below - // the peak at +900), far below the first wrong result at +1450. The long - // copy reads from four steps below that, +800, and keeps +800: the peak + // the peak at +900), far below the first wrong result at +1450. The second + // pass reads from four steps below that, +800, and keeps +800: the peak // itself is inside the margin. That is the price of the margin on a card // that has a peak. CHECK(run.card.max_mem_seen == 1000); - CHECK(run.card.max_precise_mem == 800); + CHECK(run.card.max_again_mem == 800); CHECK(r.mem_max_stable == 800); CHECK(r.mem_confirmed == 800); CHECK(r.mem_mhz == 550); @@ -2308,7 +2306,7 @@ TEST_CASE("a failed bandwidth measurement ends the memory scan; the core is stil REQUIRE(r.ok); CHECK(r.driver_resets == 1); // The scan samples +0 .. +300. +250 was the highest step that passed, so - // the long copy would read from +50; after a reset nothing is read, and + // the second pass would begin at +50; after a reset nothing is read, and // confirming starts four steps below that, which is stock. The +0 sample // names no clock in the journal. CHECK(mem_begins(run.card) == climb(50, 300, 50)); @@ -2319,7 +2317,7 @@ TEST_CASE("a failed bandwidth measurement ends the memory scan; the core is stil CHECK(run.logged("mem +0: bandwidth 500.0 GB/s")); CHECK(run.card.max_mem_seen == 300); CHECK(run.card.bw_measurements == 7); - CHECK(run.card.bw_precise_measurements == 0); + CHECK(run.card.bw_again_measurements == 0); CHECK(r.mem_max_stable == 50); CHECK(r.mem_confirmed == 0); CHECK(r.mem_mhz == 0); @@ -2344,10 +2342,10 @@ TEST_CASE("bandwidth readings that do not settle end the memory scan and are no CHECK_FALSE(run.logged("treated as a driver reset")); CHECK(run.logged("mem +300: the bandwidth readings did not settle; the memory scan ends here")); // The scan ends at +300; +250 was the highest step that passed, so the - // long copy reads from +50, and the confirm probe runs at what it kept, + // second pass begins at +50, and the confirm probe runs at what it kept, // not four steps below that. CHECK(run.card.max_mem_seen == 300); - CHECK(run.card.max_precise_mem == 50); + CHECK(run.card.max_again_mem == 50); CHECK(r.mem_max_stable == 50); CHECK(r.mem_confirmed == 50); // The core is searched as in a run without any event. @@ -2585,7 +2583,7 @@ TEST_CASE("the +0 sample of the bandwidth scan journals no clock") { run.card.bw_curve = [](int m) { return 500 + m * 0.3; }; const auto r = run.go(Preset::BestOfMyGpu); REQUIRE(r.ok); - CHECK(r.mem_max_stable == 600); // +800 passed; the long copy read from +600 + CHECK(r.mem_max_stable == 600); // +800 passed; the second pass began at +600 // The first entry of the run is the +0 sample: it ran and was measured, // and its `begin` line names neither clock. CHECK(run.logged("core +0 / mem +0: STABLE")); @@ -3059,7 +3057,7 @@ TEST_CASE("one bandwidth reading that is far too high does not end the memory sc CHECK(run.card.max_mem_seen == 2550); // the scan went to the first wrong result CHECK(run.logged("mem +0: bandwidth 434.0 GB/s")); CHECK(run.card.bw_measurements == 51); // +0 .. +2500, each once: no step was in doubt - // +2500 passed; the long copy reads from +2300 down, and +2200 is the + // +2500 passed; the second pass reads from +2300 down, and +2200 is the // lowest offset within 1 % of what it read there. CHECK(r.mem_max_stable == 2200); CHECK(r.mem_mhz == 1500); @@ -3083,29 +3081,27 @@ TEST_CASE("one low bandwidth reading does not end the memory scan; three low one const auto r = run.go(Preset::BestOfMyGpu); REQUIRE(r.ok); CHECK(run.card.bw_readings_at[1000] == kBandwidthReadings); - CHECK(run.logged("mem: the scan passed +1000; reading from +800 down with the long copy")); + CHECK(run.logged("mem: the scan passed +1000; reading again from +800 down")); CHECK(run.card.max_mem_seen == 1000); - CHECK(run.card.max_precise_mem == 800); + CHECK(run.card.max_again_mem == 800); CHECK(r.mem_max_stable == 700); // the lowest offset within 1 % of the reading at +800 } } TEST_CASE("readings that scatter as they do on the reference card leave the memory scan where the card puts it") { - // Measured there: the short copy scatters by 0.4 to 1.3 % (standard - // deviation), the long one by 0.2 %. The worst of the first and a little - // more than the second here, on two hundred different days. Every scan - // gets to the first wrong result, and the offset that is kept moves by - // four steps. + // Measured there: readings scatter by 0.2 % on a card that is left alone + // (hardware check 96). A little more than that here, on two hundred + // different days. Every scan gets to the first wrong result, and the + // offset that is kept moves by five steps: +2050 to +2300. for (unsigned seed = 1; seed <= 200; ++seed) { Run run = rising_to_the_edge(); - run.card.bw_noise = scatter(seed, 0.013); - run.card.bw_precise_noise = scatter(seed + 1000, 0.0025); + run.card.bw_noise = scatter(seed, 0.0025); const auto r = run.go(Preset::BestOfMyGpu); CAPTURE(seed); REQUIRE(r.ok); CHECK(run.card.max_mem_seen == 2550); - CHECK(run.card.max_precise_mem == 2300); - CHECK(r.mem_max_stable >= 2100); + CHECK(run.card.max_again_mem == 2300); + CHECK(r.mem_max_stable >= 2050); CHECK(r.mem_max_stable <= 2300); } } @@ -3118,7 +3114,6 @@ TEST_CASE("a real peak is still found where it is when the readings scatter") { run.card.mem_edge = 1400; run.card.bw_curve = [](int m) { return m <= 900 ? 500 + m * 0.3 : 770 - (m - 900) * 0.6; }; run.card.bw_noise = scatter(seed, 0.013); - run.card.bw_precise_noise = scatter(seed + 1000, 0.0045); const auto r = run.go(Preset::BestOfMyGpu); CAPTURE(seed); REQUIRE(r.ok); @@ -3126,7 +3121,7 @@ TEST_CASE("a real peak is still found where it is when the readings scatter") { // scan ends at one of the two, now and then one step later. CHECK(run.card.max_mem_seen >= 950); CHECK(run.card.max_mem_seen <= 1050); - CHECK(run.card.max_precise_mem <= run.card.max_mem_seen - kPreciseMarginSteps * 50); + CHECK(run.card.max_again_mem <= run.card.max_mem_seen - kPeakMarginSteps * 50); CHECK(r.mem_max_stable >= 700); CHECK(r.mem_max_stable <= 900); } @@ -3173,9 +3168,8 @@ OptimizeResult again(Run& run) { run.card.max_mem_seen = 0; run.card.bw_measurements = 0; run.card.bw_readings_at.clear(); - run.card.bw_precise_measurements = 0; - run.card.bw_precise_at.clear(); - run.card.max_precise_mem = -1; + run.card.bw_again_measurements = 0; + run.card.max_again_mem = -1; return run.go(Preset::BestOfMyGpu); } @@ -3204,7 +3198,7 @@ TEST_CASE("the next run stays four steps below the memory edge of the run before CHECK(second.driver_resets == 0); // The same offset as the run that found the edge: the margin counts from // the edge, not a second time from where the scan was held. - CHECK(run.card.max_precise_mem == 2300); + CHECK(run.card.max_again_mem == 2300); CHECK(second.mem_max_stable == first.mem_max_stable); CHECK(second.mem_max_stable == 2200); CHECK(second.mem_mhz == 1500); @@ -3272,34 +3266,32 @@ TEST_CASE("a scan that ended for another reason leaves no memory edge") { namespace { -// The reference RTX 4070 as hardware check 95 found it: bandwidth rises by -// 0.4 % a step until the memory gives out, the stress load computes right up -// to +2500 and wrong at +2550, and the long copy takes the device down from -// +2450 on (in five scans out of five; once the machine went with it). The -// short copy got through there in the one scan that took it that far. +// The reference RTX 4070: bandwidth rises by 0.4 % a step until the memory +// gives out, and the stress load computes right up to +2500 and wrong at +// +2550. Nothing announces that edge (hardware checks 94 and 95). Run reference_card() { Run run = recovering(); run.card.mem_edge = 2500; run.card.ranges = ClockOffsetRanges{{-500, 1000}, {-1000, 3000}}; run.card.bw_curve = [](int m) { return 434.0 * (1 + 0.004 * m / 50); }; - run.card.bw_precise_loses_device_from = 2450; return run; } } -TEST_CASE("on a card whose long copy loses the device at the edge, the first run never takes it there") { +TEST_CASE("on a card whose bandwidth rises to the edge, the offset is read again well below it") { Run run = reference_card(); const auto r = run.go(Preset::BestOfMyGpu); REQUIRE(r.ok); // The scan finds the end with the short copy and the stress load. CHECK(run.card.max_mem_seen == 2550); CHECK(run.logged("core +0 / mem +2550: WRONG RESULT")); - CHECK(run.logged("mem: the scan passed +2500; reading from +2300 down with the long copy")); - // The long copy: four readings, none above +2300, and the device stayed. - CHECK(run.card.max_precise_mem == 2300); - CHECK(run.card.bw_precise_measurements == 4); - CHECK(run.logged("mem +2300: bandwidth 513.9 GB/s, the long copy")); + CHECK(run.logged("mem: the scan passed +2500; reading again from +2300 down")); + // The second pass: four readings, none above +2300. + CHECK(run.card.max_again_mem == 2300); + CHECK(run.card.bw_readings_at[2300] == 2); // once going up, once coming down + CHECK(run.card.bw_again_measurements == 4); + CHECK(run.logged("mem +2300: bandwidth 513.9 GB/s, read again")); CHECK(r.driver_resets == 0); CHECK(run.card.recover_calls == 0); CHECK_FALSE(run.logged("treated as a driver reset")); @@ -3310,15 +3302,15 @@ TEST_CASE("on a card whose long copy loses the device at the edge, the first run const auto second = again(run); REQUIRE(second.ok); CHECK(run.card.max_mem_seen == 2350); - CHECK(run.card.max_precise_mem == 2300); + CHECK(run.card.max_again_mem == 2300); CHECK(second.driver_resets == 0); CHECK(second.mem_mhz == 1500); check_journal_rules(run); } -TEST_CASE("the long copy stays below the margin on that card whatever the readings do") { - // Short readings that scatter as the worst series measured, with a high - // and a low stretch thrown in; the long copy at twice its measured scatter. +TEST_CASE("the second pass stays below the margin on that card whatever the readings do") { + // Readings that scatter as the worst series measured next to another + // program, in both passes, with a high and a low stretch thrown in. for (unsigned seed = 1; seed <= 40; ++seed) { Run run = reference_card(); const auto noise = scatter(seed, 0.013); @@ -3327,45 +3319,40 @@ TEST_CASE("the long copy stays below the margin on that card whatever the readin const double level = n > 10 && n <= 20 ? 1.018 : n > 30 + static_cast(seed % 10) && n <= 45 ? 0.982 : 1.0; return noise(mem, nth) * level; }; - run.card.bw_precise_noise = scatter(seed + 1000, 0.0045); const auto r = run.go(Preset::BestOfMyGpu); CAPTURE(seed); REQUIRE(r.ok); - CHECK(run.card.max_precise_mem <= run.card.max_mem_seen - 50 - kPreciseMarginSteps * 50); - CHECK(run.card.max_precise_mem < 2450); + CHECK(run.card.max_again_mem <= run.card.max_mem_seen - 50 - kPeakMarginSteps * 50); CHECK(r.driver_resets == 0); CHECK(run.card.max_mem_seen == 2550); - CHECK(r.mem_mhz >= 1400); // +2050 or higher kept, at twice the long copy's scatter + CHECK(r.mem_mhz >= 1400); // +2050 or higher kept } } TEST_CASE("the margin holds for an edge anywhere") { - // The same kind of card with its edge elsewhere: the long copy dies one - // step below the highest offset the stress load passes, as on the - // reference card, and is never taken there. + // The same kind of card with its edge elsewhere. for (int edge = 300; edge <= 2900; edge += 150) { Run run = reference_card(); run.card.mem_edge = edge; - run.card.bw_precise_loses_device_from = edge - 50; const auto r = run.go(Preset::BestOfMyGpu); CAPTURE(edge); REQUIRE(r.ok); CHECK(run.card.max_mem_seen == edge + 50); - CHECK(run.card.max_precise_mem == edge - 200); + CHECK(run.card.max_again_mem == edge - 200); CHECK(r.driver_resets == 0); CHECK(r.mem_max_stable <= edge - 200); } } -TEST_CASE("a long copy that loses the device below the margin after all is a reset the search recovers from") { +TEST_CASE("a reading of the second pass that loses the device is a reset the search recovers from") { // A card that is worse than the margin allows for. The run goes on as // after any reset in the memory search, and the offset is remembered. Run run = reference_card(); - run.card.bw_precise_loses_device_from = 2300; + run.card.bw_again_loses_device_from = 2300; const auto r = run.go(Preset::BestOfMyGpu); REQUIRE(r.ok); CHECK(r.driver_resets == 1); - CHECK(run.card.bw_precise_measurements == 1); + CHECK(run.card.bw_again_measurements == 1); CHECK(run.logged("mem +2300: the bandwidth measurement failed; treated as a driver reset")); CHECK(r.mem_max_stable == 2300); CHECK(r.mem_confirmed == 2300 - kResetBackoffSteps * 50); @@ -3376,20 +3363,7 @@ TEST_CASE("a long copy that loses the device below the margin after all is a res REQUIRE(second.ok); CHECK(second.driver_resets == 0); CHECK(run.card.max_mem_seen == 2100); - CHECK(run.card.max_precise_mem == 2050); + CHECK(run.card.max_again_mem == 2050); check_journal_rules(run); } -TEST_CASE("without a long copy the second pass reads with the short one") { - // A caller that gives only one way of measuring: the search is the same. - Run run = rising_to_the_edge(); - run.no_precise = true; - const auto r = run.go(Preset::BestOfMyGpu); - REQUIRE(r.ok); - CHECK(run.card.bw_precise_measurements == 0); - CHECK(run.card.bw_readings_at[2300] == 2); // once going up, once coming down - CHECK_FALSE(run.logged("the long copy")); - CHECK(run.logged("mem: the scan passed +2500; reading from +2300 down once more")); - CHECK(r.mem_max_stable == 2200); -} - From e5433eeecac3f7761e4ebc386e648318eccbd908 Mon Sep 17 00:00:00 2001 From: Rovey Date: Sun, 11 Oct 2026 23:31:51 +0200 Subject: [PATCH 14/15] docs: hardware check 95 (f) for the search as it is now Co-Authored-By: Claude Opus 5.5 --- docs/hardware-checks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hardware-checks.md b/docs/hardware-checks.md index 912f51e..3ab602f 100644 --- a/docs/hardware-checks.md +++ b/docs/hardware-checks.md @@ -122,7 +122,7 @@ root; adjust the path for a Debug build. | 92 | The undervolt profile starts with the measurement at stock every run uses, and a hot card is not refused | Elevated, the tray app exited, a copy of `gao.json` kept. (a) `gao --undervolt` to the end; then `gao --status`. (b) The same with a test build whose stock limit is lowered to a temperature the reference card passes under load (63 C; it reaches 67). Put the copy of `gao.json` back. (c) Only on a card that is above the profile's limit at stock with the fans at 100 % (the tester's RTX 3080 Ti: 82 to 83 C): the **Undervolt** profile from the window, to the end, and its `run.log` | yes | (a) `the card at stock, five minutes`, a `stock: STABLE ...` line with the driver's fan speed, then `baseline: the measurement at stock`, and no second five minutes at stock: the first candidate follows at once; no `candidates are stopped at` line; `RESULT:`; `before:` shows 300 s and the fan speed the driver chose. (b) `stock: TOO HOT ... peak=64 C`, the line that the measurement stopped there, the cool-down, then `baseline: the measurement at stock` and a search that runs to `RESULT:` with `before:` showing the shortened time; no `candidates are stopped at` line, because 64 C is under the profile's 75. (c) no `not stable at stock`; `the card reached 86 C at stock: candidates are stopped at 85 C, not at the profile's 75 C`; candidates that end `STABLE` or `WRONG RESULT`, none `TOO HOT`; a result that is cooler and draws less than stock at the same clock | First run: **not as expected**, and that found a fault (2026-10-11, reference RTX 4070, driver 617.42, a test build of d1c99e9, started elevated by a script). The first halves held: (a) `stock: STABLE ... peak=65 C fan=70 %`, then `baseline: the measurement at stock` four seconds later and no second five minutes; (b) `stock: TOO HOT ... peak=64 C`, stopped after 34 s, the run went on, no `candidates are stopped at` line. But both runs then ended with `the card held no undervolt: the first step below its own voltage already failed`: their first candidates read `1070 mV (+15 MHz): SLOW ... clock=2323 MHz` and `1050 mV (+15 MHz): SLOW ... clock=2411 MHz`, on a card that ran 2791 MHz in the measurement just before. The search's load had not run yet when that candidate began (see the decision "With its reference handed over, the undervolt search warms its load up before the first candidate"). Second run, with the warm-up: **pass** for (a) and (b) (same day, a test build of 0733695). (a) `stock: STABLE score=5426 it/s clock=2788 MHz peak=66 C fan=70 %`, four seconds later `baseline: the measurement at stock`, `warming the load up: 5 s at stock`, `target: 2790 MHz, which the card reaches at 1090 mV by itself`, and the first candidate `1085 mV (+15 MHz): STABLE ... clock=2775 MHz`; 26 candidates, none `SLOW` or `TOO HOT`, the edge at 950 mV, `RESULT: 2790 MHz at 990 mV instead of 1090 mV`; `before:` 5426 it/s, 189 W, fan 70 % (300 s), `after:` 5325 it/s, 149 W, fan 53 % (300 s). (b) `stock: TOO HOT ... peak=64 C`, stopped after 74 s, the run went on, the same warm-up and a search to the same result (990 mV, edge 950 mV); `before:` shows 75 s; no `candidates are stopped at` line. The saved tune of before the test was put back and applied. (c) not run. | | 93 | A speed profile that would end slower than stock keeps nothing | Only on a card that is warmer at stock than the profile allows (the tester's RTX 3080 Ti; the reference card ends faster than stock with every speed profile): **Best of my GPU** from the window, to the end; the result screen, the dashboard and `run.log` | yes | the run goes through all its steps and its soak; then **Not applied** with `the result is slower than stock: N it/s against M (-X %)`, the power limit it had to go down to, and the pointer to Undervolt; `The card is back at stock.`; the dashboard shows stock and the saved tune of before the run, if there was one; `run.log` ends with `stopped: the result is slower than stock ... -- card restored to stock` | Not run yet. On the reference card: nothing to see, by design. | | 94 | All in one, with the undervolt step searching for the first time | Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize all` to the end with a build that has the warm-up of the undervolt step; then `gao --status`; put the copy back | yes | five steps; in step 3 a `warming the load up` line, a first candidate that is not `SLOW`, and a descent; steps 4 and 5 run on the tune of steps 2 and 3; `RESULT:` | **Not as expected at step 4**, and that found a fault (2026-10-11, reference RTX 4070, driver 617.42, a test build of 0733695, unattended). Step 1: `stock: STABLE score=5325 it/s peak=65 C fan=70 % power=186 W`. Step 2: power 105 %, `mem: bandwidth peak +2300, confirmed +2300, applying +1600`, `core: highest stable +210, confirmed +210, applying +135`, soak `STABLE score=5568 it/s peak=70 C power=197 W`. Step 3, as wanted: `warming the load up: 5 s with the overclock`, `target: 2865 MHz, which the card reaches at 1050 mV with the overclock`, nine candidates `STABLE` from 1045 mV down, `WRONG RESULT` at 990 mV, confirmed at 1010 mV after two tries, and `applying 1050 mV (+135 MHz at that point)`: the margin left no raise above the overclock, so the curve was cut off, as in the runs before the warm-up; soak `STABLE score=5434 it/s peak=67 C power=176 W`. Step 4 did not begin: `the tune could not be applied for the fan tune: the undervolt was not applied: the point already runs more than the target -- card at stock`. The saved tune of before the test was put back and applied. Cause and fix: the decision "A flat top goes back on a card that is cooler than when it was found". Second run, with the `gao.exe` of the released v0.4.6: **pass, all five steps** (same day, started elevated by a script). Step 1: `stock: STABLE score=5309 it/s peak=66 C fan=70 % power=185 W`. Step 2: power 105 %, `mem: bandwidth peak +850, confirmed +850, applying +550`, `core: highest stable +210, confirmed +210, applying +135`, soak `STABLE score=5568 it/s`. Step 3: `target: 2850 MHz, which the card reaches at 1045 mV with the overclock`, `undervolt: lowest stable 975 mV, confirmed 985 mV, applying 1025 mV (+165 MHz at that point)`, soak `STABLE score=5363 it/s peak=66 C power=163 W`. Step 4: the tune went back on the card, `fan 60 %: 63 C`, `fan 55 %: 65 C`, `fan 52 %: 67 C`, `fan tune: 52 % holds 67 C`. Step 5: the final test ran its 300 s. `RESULT: power 105 %, core +135 MHz, mem +550 MHz, 2850 MHz at 1025 mV, fans 52 % at 67 C`; `stock: 5309 it/s, 185 W, fan 70 %, 66 C`, `now: 5353 it/s, 166 W, fan 52 %, 70 C`: `+0.8 % score and -10.3 % power against stock`; `undervolt: applied`. The saved tune of before the test was put back. **What this run did not show:** its undervolt had a raise of its own (+165 over an offset of +135), so the anchor was raised and not merely cut off, and the case of the first run (an anchor that reads a step above the target on a cooler card) did not come up. That fix has still only its tests. **The memory offset differs a lot from run to run on this card, and the two logs say why:** +1600 in the first run (bandwidth peak +2300), +550 in the second (peak +850), +650 in the run that saved the tune in use that morning (the journal shows its scan ending at +1000 too). The card was in the same state each time (68 C, power 105 %, a reading every three to four seconds). In both logged scans the bandwidth rises in a straight line, 0.36 and 0.46 % per 50 MHz step, from 420 GB/s at +0 to 509 GB/s at +2450; the first run went on to `WRONG RESULT` at +2550 without the bandwidth ever turning down. The readings scatter around that line with a standard deviation of 0.4 and 0.7 %, single ones by more than 1 %. The scan ends at a reading more than 1 % below the best so far when a second reading agrees, and in the second run that was chance: +950 read high (462.8 GB/s, where the line is at about 458), +1000 read 456.6 twice, 1.3 % below it. In the first run the one low reading (+600) was followed by a higher one, and the scan ran to the first wrong result. So on this card there is no bandwidth peak before the errors, and where the scan stops is decided by noise. | -| 95 | The memory scan no longer ends by chance | (a) Not elevated, nothing changed on the card: a measuring build takes 24 bandwidth readings in one process, with 3 s of the stress load before each as a scan has, at 64, 128 and 256 dispatches per timed run; then the same with the app's window open next to it, drawing a frame every 100 ms and every 33 ms. (b) Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize best` twice in a row with the build of the change, then the copy put back | (a) no, (b) yes | (a) how much a reading scatters, and on what that depends. (b) in both runs every `mem +N: bandwidth` line shows three readings that agree within about 1 %; the scan does not end before the first `WRONG RESULT`; the two `mem: bandwidth peak` values are within 300 MHz of each other and the applied offset is above +1000; the soak is not slower than at stock | **(a) measured** (2026-10-11, reference RTX 4070, driver 617.42, the saved tune applied: power 105 %, core +135, memory +650; standard deviation over the 24 readings of a series, and the lowest and highest against the mean). 64 dispatches, no window: 0.44 % (-0.55 to +1.07, one reading did not settle), 0.66 % (-1.58 to +1.49, one did not settle), 1.31 % (-4.83 to +1.27), 0.86 % (-1.44 to +2.05). 128 dispatches, no window: 0.43 % (-1.10 to +0.79). 256 dispatches, no window: 0.22 % (-0.43 to +0.42) and 0.60 % (-0.80 to +1.37; the last five readings of this series were 451 to 453 GB/s after nineteen of 444 to 449). With the window drawing every 100 ms: 64 dispatches 1.13 % (the first six readings 452 to 453 GB/s, the rest 439 to 451), 256 dispatches 0.24 % (-0.47 to +0.61). With the window drawing every 33 ms: 64 dispatches 0.58 %, 256 dispatches 0.25 % (-0.35 to +0.51). A reading took 6.5, 5.3 and 3.9 timed runs to settle at 64, 128 and 256 dispatches. So: the longer timed run takes the scatter from 0.4 to 1.3 % down to 0.2 %, the window's drawing adds nothing, and apart from the scatter the readings sit about 1.8 % higher for up to a minute at a time (the stretch at the end of the second 256 series ran on into the next series; the last 64 series went back and forth between 444 and 452 GB/s). What sets that level is not known. **(b) the offset is the same twice; the scan ends in a driver reset twice** (2026-10-11, same card and driver, a test build of 2764609, unattended, 40 minutes for the two runs). Run 1: `stock: STABLE score=5311 it/s peak=66 C fan=70 % power=185 W`; 49 steps from `mem +0: bandwidth 422.7 GB/s (422.7 422.6 422.7)` to `mem +2400: bandwidth 501.4 GB/s (501.3 502.3 501.4)`; at +2450 the candidate was `STABLE` and the measurement after it lost the device: `mem +2450: the bandwidth measurement failed; treated as a driver reset`, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, 20 s of rest, `health: STABLE`, `confirm core +0 / mem +2050: STABLE`, `mem: bandwidth peak +2250, confirmed +2050, applying +1400`; `core: highest stable +210, confirmed +210, applying +135`; soak `STABLE score=5536 it/s peak=69 C power=195 W`; `driver resets during this run: 1`. Run 2: stock 5312 it/s; 50 steps from 421.8 GB/s at +0 to 502.8 at +2450; the same failure at +2500; `mem: bandwidth peak +2300, confirmed +2100, applying +1450`; core +135; soak `STABLE score=5555 it/s peak=69 C power=196 W`; one driver reset. The three readings of a step differed by 0.4 % on average and by 1.2 % at most; no step was more than 0.7 % below the best before it, so neither scan came near ending early, by the old rule or the new; every step had its three readings. The scan took 5 min 25 s (about 6.5 s a step; about 3 min before the change). As expected: the two peaks are 50 MHz apart, the applied offsets are +1400 and +1450, and both results are 4 % faster than stock. **Not as expected: neither scan reached a `WRONG RESULT`.** Both ended one and two steps before +2550 with the device lost during the bandwidth copy; Windows logged it both times (System log, `nvlddmkm` events 14 and 153 at 19:51:22 and 20:11:27), and the screen goes black for a moment. In the run of check 94 that got this far, the measurements at +2450 and +2500 gave a number and the stress load reported the first wrong result at +2550, without a reset. A step now copies about seven times as much as it did (three readings of about four runs of 128 GB, against one reading of about six runs of 32 GB), and the copy checks nothing, so where the memory starts to fail it is the driver that gives out, not a verdict. Two runs against one: likely the cause, not proven. The search handles it as it should (rest, health probe, confirm four steps lower, the core is still searched), but a driver reset in every run on a card like this is not what the change was for. The saved tune of before the test was put back and applied (`applied best: power 105 %, core +135 MHz, mem +650 MHz`), the tray app restarted, Defender reported nothing. **(c) the same two runs with one reading a step, three only for a step in doubt** (2026-10-11 20:26 to 21:03, a test build of 71e7c2b, unattended): **the reset is still there.** Run 1: stock 5328 it/s; 50 steps from `mem +0: bandwidth 420.4 GB/s` to `mem +2450: bandwidth 502.4 GB/s`; at +2500 the candidate was `STABLE` and then `bandwidth measurement failed: the timing of the copy did not come back` ten seconds later (`nvlddmkm` event 14 at 20:37:07); `mem: bandwidth peak +2200, confirmed +2000, applying +1400`; core +135; soak `STABLE score=5572 it/s peak=70 C power=197 W`; one driver reset. Run 2: stock 5311 it/s; 421.9 GB/s at +0 to 508.0 at +2450; at +2500 `the device was lost during the copy failed (hr=0x887A0006)`, the device hung (event 14 at 20:55:29); `mem: bandwidth peak +2400, confirmed +2200, applying +1500`; core +135; soak `STABLE score=5571 it/s peak=70 C power=196 W`; one driver reset. No step was in doubt in either run (the lowest was 0.75 % below the best before it), so every step was read once; the scan took 3 min 33 s. The two peaks are 200 MHz apart: with one reading a step, which offsets count as within 1 % of the best moves by a few steps. Applied +1400 and +1500, both 5 % faster than stock. So the reset does not come from reading a step three times. Four runs out of four with the longer timed run ended this way, at +2450 or +2500. Not merged; the design for a scan that does not end like this is the open row in `docs/development.md`. The saved tune was put back and applied again (core +135, memory +650), the tray app restarted, Defender reported nothing. **(d) the edge is remembered** (a test build of 7c215c4; the same two runs): expected is that the first scan ends as before and logs nothing new, the journal gets one `mem_edge` line with the GPU id and the driver version, and the second run logs `mem: the memory gave out at +N in an earlier run on this card and driver; the scan stops at +(N-200)`, has no `driver resets during this run` line, applies more than +1000 and is not slower than stock. **Run 1 took the machine down; the test ended there** (2026-10-11 21:23, unattended). Stock 5321 it/s, power 105 %, then the scan: 49 steps from +0 to `mem +2400: bandwidth 539.3 GB/s`, two of them read three times (+1200 and +1250). At +2450 the candidate was `STABLE` and the copy after it failed as in the four runs before: `bandwidth measurement failed: the copy failed (hr=0x887A0007): the device was lost` at 21:35:05, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, `the driver was reset; reconnecting`. That is the last line. Windows restarted by itself within fifteen seconds: Kernel-Power event 41 at 21:35:23, `The previous system shutdown at 21:34:57 was unexpected`, no bugcheck, no dump, no `nvlddmkm` event written. So the fifth scan that met the edge with the longer timed run did not end in a driver reset the search recovers from, but in a reset of the whole machine, during the reconnect or right after it. What held: the candidate's journal entry was closed, so no ceiling of a freeze was left; the line `{"mem_edge":2450,"rig":"GPU-... / 617.42"}` was on disk before the machine went down, so no later run of this build goes there again; the run had saved nothing, and at logon the tray app applied the tune of before the test (`applied best: power 105 %, core +135 MHz, mem +650 MHz`). Not answered: the second run. Seen along the way and not explained: from the first step of the scan the stress load scored 5760 it/s and the bandwidth read about 7 % higher than in every run before (486.6 GB/s at +900 against about 455), while the measurement at stock five minutes earlier was the usual 5321. **(e) how the stress load meets the edge** (the same build, no search): `gao --reset`, then `gao --set-mem` and `gao --stress 3` at +2300, +2350, ... until the verdict is not `STABLE`, five times, at stock power and core +0; expected is one of two outcomes, `WRONG RESULT` every time (then the stress load can lead the copy to the edge) or `DEVICE LOST` (then it cannot). **Not run:** the test ended in (d). It goes to the edge on purpose, five times, and after (d) that is a risk for the machine that needs its owner's word first. **(f) the two-pass search** (the short copy in the scan, the long copy from four steps below the highest step that passed; see `docs/development.md`): tested on a simulated card only. On the reference card, with the edge at +2450 in its journal: `gao --optimize best` once; expected `mem: the memory gave out at +2450 in an earlier run on this card and driver; the scan stops at +2250`, no memory candidate above +2250, `mem: the scan passed +2250; reading from +2200 down with the long copy`, a few `mem +N: bandwidth ... GB/s, the long copy` lines, no `driver resets during this run`, an applied offset above +1000 and a soak that is not slower than stock. **Not run: waits for the owner's word.** **(g) what else uses the card** (2026-10-11 22:00, after the restart of (d), not elevated, the saved tune applied, nothing changed): `gao --stress 5` scored 5828 it/s and `gao --bandwidth --after-load 3` read 484.0 GB/s with its three runs within 0.0 %, against 5226 it/s and 444 to 452 GB/s with the same tune earlier that evening. The series of (a) again, 24 readings each: 64 dispatches 480.0 GB/s, 0.23 % (-0.66 to +0.18); 256 dispatches 475.2 GB/s, 0.68 %, sliding from 480 to 471 during the series; 64 dispatches again 454.0 GB/s, 3.02 % (-4.37 to +4.22), the readings jumping between 434 and 473. Read right after it, the card's use per program: Chrome 8.6 % of the video decoder (19 % at most) and 2.8 % of the 3D engine, with a YouTube video playing; nothing else above 1.1 %. So the short copy is as steady as the long one on a card that is left alone, and a video playing next to it costs about 8 % of the bandwidth and of the stress score and makes the readings jump. Then 40 readings with the short copy, and next to each the other programs on the card in the seconds before it (22:13): 464 to 476 GB/s while Chrome used the video decoder, 444 to 455 while it used the 3D engine as well, and from the reading after Chrome stopped, 477 to 483.5 for the last eleven readings with no other program on the card. So another program using the card lowers the reading by 3 to 8 % from one reading to the next. That explains the scatter and the level changes of (a) in kind. During the tests of that day it was most likely not a video (hardly any was watched) but a code editor built on a browser engine that had stood open since the morning: its last files were written at 21:31:43 and 21:31:44, when it was closed, and the run of (d) logs the bandwidth going from 430.4 to 457.1 GB/s at 21:31:45. So every card measurement of checks 94 and 95 up to then was taken with that program on the card: about 7 % lower, and more scattered, than the card alone. A match to the second; not measured with the editor started again. | +| 95 | The memory scan no longer ends by chance | (a) Not elevated, nothing changed on the card: a measuring build takes 24 bandwidth readings in one process, with 3 s of the stress load before each as a scan has, at 64, 128 and 256 dispatches per timed run; then the same with the app's window open next to it, drawing a frame every 100 ms and every 33 ms. (b) Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize best` twice in a row with the build of the change, then the copy put back | (a) no, (b) yes | (a) how much a reading scatters, and on what that depends. (b) in both runs every `mem +N: bandwidth` line shows three readings that agree within about 1 %; the scan does not end before the first `WRONG RESULT`; the two `mem: bandwidth peak` values are within 300 MHz of each other and the applied offset is above +1000; the soak is not slower than at stock | **(a) measured** (2026-10-11, reference RTX 4070, driver 617.42, the saved tune applied: power 105 %, core +135, memory +650; standard deviation over the 24 readings of a series, and the lowest and highest against the mean). 64 dispatches, no window: 0.44 % (-0.55 to +1.07, one reading did not settle), 0.66 % (-1.58 to +1.49, one did not settle), 1.31 % (-4.83 to +1.27), 0.86 % (-1.44 to +2.05). 128 dispatches, no window: 0.43 % (-1.10 to +0.79). 256 dispatches, no window: 0.22 % (-0.43 to +0.42) and 0.60 % (-0.80 to +1.37; the last five readings of this series were 451 to 453 GB/s after nineteen of 444 to 449). With the window drawing every 100 ms: 64 dispatches 1.13 % (the first six readings 452 to 453 GB/s, the rest 439 to 451), 256 dispatches 0.24 % (-0.47 to +0.61). With the window drawing every 33 ms: 64 dispatches 0.58 %, 256 dispatches 0.25 % (-0.35 to +0.51). A reading took 6.5, 5.3 and 3.9 timed runs to settle at 64, 128 and 256 dispatches. So: the longer timed run takes the scatter from 0.4 to 1.3 % down to 0.2 %, the window's drawing adds nothing, and apart from the scatter the readings sit about 1.8 % higher for up to a minute at a time (the stretch at the end of the second 256 series ran on into the next series; the last 64 series went back and forth between 444 and 452 GB/s). What sets that level is not known. **(b) the offset is the same twice; the scan ends in a driver reset twice** (2026-10-11, same card and driver, a test build of 2764609, unattended, 40 minutes for the two runs). Run 1: `stock: STABLE score=5311 it/s peak=66 C fan=70 % power=185 W`; 49 steps from `mem +0: bandwidth 422.7 GB/s (422.7 422.6 422.7)` to `mem +2400: bandwidth 501.4 GB/s (501.3 502.3 501.4)`; at +2450 the candidate was `STABLE` and the measurement after it lost the device: `mem +2450: the bandwidth measurement failed; treated as a driver reset`, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, 20 s of rest, `health: STABLE`, `confirm core +0 / mem +2050: STABLE`, `mem: bandwidth peak +2250, confirmed +2050, applying +1400`; `core: highest stable +210, confirmed +210, applying +135`; soak `STABLE score=5536 it/s peak=69 C power=195 W`; `driver resets during this run: 1`. Run 2: stock 5312 it/s; 50 steps from 421.8 GB/s at +0 to 502.8 at +2450; the same failure at +2500; `mem: bandwidth peak +2300, confirmed +2100, applying +1450`; core +135; soak `STABLE score=5555 it/s peak=69 C power=196 W`; one driver reset. The three readings of a step differed by 0.4 % on average and by 1.2 % at most; no step was more than 0.7 % below the best before it, so neither scan came near ending early, by the old rule or the new; every step had its three readings. The scan took 5 min 25 s (about 6.5 s a step; about 3 min before the change). As expected: the two peaks are 50 MHz apart, the applied offsets are +1400 and +1450, and both results are 4 % faster than stock. **Not as expected: neither scan reached a `WRONG RESULT`.** Both ended one and two steps before +2550 with the device lost during the bandwidth copy; Windows logged it both times (System log, `nvlddmkm` events 14 and 153 at 19:51:22 and 20:11:27), and the screen goes black for a moment. In the run of check 94 that got this far, the measurements at +2450 and +2500 gave a number and the stress load reported the first wrong result at +2550, without a reset. A step now copies about seven times as much as it did (three readings of about four runs of 128 GB, against one reading of about six runs of 32 GB), and the copy checks nothing, so where the memory starts to fail it is the driver that gives out, not a verdict. Two runs against one: likely the cause, not proven. The search handles it as it should (rest, health probe, confirm four steps lower, the core is still searched), but a driver reset in every run on a card like this is not what the change was for. The saved tune of before the test was put back and applied (`applied best: power 105 %, core +135 MHz, mem +650 MHz`), the tray app restarted, Defender reported nothing. **(c) the same two runs with one reading a step, three only for a step in doubt** (2026-10-11 20:26 to 21:03, a test build of 71e7c2b, unattended): **the reset is still there.** Run 1: stock 5328 it/s; 50 steps from `mem +0: bandwidth 420.4 GB/s` to `mem +2450: bandwidth 502.4 GB/s`; at +2500 the candidate was `STABLE` and then `bandwidth measurement failed: the timing of the copy did not come back` ten seconds later (`nvlddmkm` event 14 at 20:37:07); `mem: bandwidth peak +2200, confirmed +2000, applying +1400`; core +135; soak `STABLE score=5572 it/s peak=70 C power=197 W`; one driver reset. Run 2: stock 5311 it/s; 421.9 GB/s at +0 to 508.0 at +2450; at +2500 `the device was lost during the copy failed (hr=0x887A0006)`, the device hung (event 14 at 20:55:29); `mem: bandwidth peak +2400, confirmed +2200, applying +1500`; core +135; soak `STABLE score=5571 it/s peak=70 C power=196 W`; one driver reset. No step was in doubt in either run (the lowest was 0.75 % below the best before it), so every step was read once; the scan took 3 min 33 s. The two peaks are 200 MHz apart: with one reading a step, which offsets count as within 1 % of the best moves by a few steps. Applied +1400 and +1500, both 5 % faster than stock. So the reset does not come from reading a step three times. Four runs out of four with the longer timed run ended this way, at +2450 or +2500. Not merged; the design for a scan that does not end like this is the open row in `docs/development.md`. The saved tune was put back and applied again (core +135, memory +650), the tray app restarted, Defender reported nothing. **(d) the edge is remembered** (a test build of 7c215c4; the same two runs): expected is that the first scan ends as before and logs nothing new, the journal gets one `mem_edge` line with the GPU id and the driver version, and the second run logs `mem: the memory gave out at +N in an earlier run on this card and driver; the scan stops at +(N-200)`, has no `driver resets during this run` line, applies more than +1000 and is not slower than stock. **Run 1 took the machine down; the test ended there** (2026-10-11 21:23, unattended). Stock 5321 it/s, power 105 %, then the scan: 49 steps from +0 to `mem +2400: bandwidth 539.3 GB/s`, two of them read three times (+1200 and +1250). At +2450 the candidate was `STABLE` and the copy after it failed as in the four runs before: `bandwidth measurement failed: the copy failed (hr=0x887A0007): the device was lost` at 21:35:05, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, `the driver was reset; reconnecting`. That is the last line. Windows restarted by itself within fifteen seconds: Kernel-Power event 41 at 21:35:23, `The previous system shutdown at 21:34:57 was unexpected`, no bugcheck, no dump, no `nvlddmkm` event written. So the fifth scan that met the edge with the longer timed run did not end in a driver reset the search recovers from, but in a reset of the whole machine, during the reconnect or right after it. What held: the candidate's journal entry was closed, so no ceiling of a freeze was left; the line `{"mem_edge":2450,"rig":"GPU-... / 617.42"}` was on disk before the machine went down, so no later run of this build goes there again; the run had saved nothing, and at logon the tray app applied the tune of before the test (`applied best: power 105 %, core +135 MHz, mem +650 MHz`). Not answered: the second run. Seen along the way and not explained: from the first step of the scan the stress load scored 5760 it/s and the bandwidth read about 7 % higher than in every run before (486.6 GB/s at +900 against about 455), while the measurement at stock five minutes earlier was the usual 5321. **(e) how the stress load meets the edge** (the same build, no search): `gao --reset`, then `gao --set-mem` and `gao --stress 3` at +2300, +2350, ... until the verdict is not `STABLE`, five times, at stock power and core +0; expected is one of two outcomes, `WRONG RESULT` every time (then the stress load can lead the copy to the edge) or `DEVICE LOST` (then it cannot). **Not run:** the test ended in (d). It goes to the edge on purpose, five times, and after (d) that is a risk for the machine that needs its owner's word first. **(f) the two-pass search as it is now** (one copy; the second pass reads again from four steps below the highest step that passed; see `docs/development.md`): tested on a simulated card only. On the reference card, with the edge at +2450 in its journal and no other program on the card: `gao --optimize best` once; expected `other programs on the card: none`, `mem: the memory gave out at +2450 in an earlier run on this card and driver; the scan stops at +2250`, no memory candidate above +2250, `mem: the scan passed +2250; reading again from +2200 down`, a few `mem +N: bandwidth ... GB/s, read again` lines, no `driver resets during this run`, an applied offset above +1000 and a soak that is not slower than stock. **Not run: waits for the owner's word.** **(g) what else uses the card** (2026-10-11 22:00, after the restart of (d), not elevated, the saved tune applied, nothing changed): `gao --stress 5` scored 5828 it/s and `gao --bandwidth --after-load 3` read 484.0 GB/s with its three runs within 0.0 %, against 5226 it/s and 444 to 452 GB/s with the same tune earlier that evening. The series of (a) again, 24 readings each: 64 dispatches 480.0 GB/s, 0.23 % (-0.66 to +0.18); 256 dispatches 475.2 GB/s, 0.68 %, sliding from 480 to 471 during the series; 64 dispatches again 454.0 GB/s, 3.02 % (-4.37 to +4.22), the readings jumping between 434 and 473. Read right after it, the card's use per program: Chrome 8.6 % of the video decoder (19 % at most) and 2.8 % of the 3D engine, with a YouTube video playing; nothing else above 1.1 %. So the short copy is as steady as the long one on a card that is left alone, and a video playing next to it costs about 8 % of the bandwidth and of the stress score and makes the readings jump. Then 40 readings with the short copy, and next to each the other programs on the card in the seconds before it (22:13): 464 to 476 GB/s while Chrome used the video decoder, 444 to 455 while it used the 3D engine as well, and from the reading after Chrome stopped, 477 to 483.5 for the last eleven readings with no other program on the card. So another program using the card lowers the reading by 3 to 8 % from one reading to the next. That explains the scatter and the level changes of (a) in kind. During the tests of that day it was most likely not a video (hardly any was watched) but a code editor built on a browser engine that had stood open since the morning: its last files were written at 21:31:43 and 21:31:44, when it was closed, and the run of (d) logs the bandwidth going from 430.4 to 457.1 GB/s at 21:31:45. So every card measurement of checks 94 and 95 up to then was taken with that program on the card: about 7 % lower, and more scattered, than the card alone. A match to the second; not measured with the editor started again. | | 96 | Other programs on the card are found and named before a run | (a) Not elevated, nothing changed on the card, the saved tune applied: `gao --stress 5` and bandwidth readings with nothing else on the card, with an editor (Trae) standing open, and with a video playing in Chrome; next to every reading the card's use per program. (b) Not elevated: `gao --busy` with the editor open and with it closed. (c) The window's question, from a look build that shows it without a run. (d) Elevated, with the editor open: `gao --retest`, then `gao --retest --busy-ok`; and **Optimize GPU** in the window, each of the three answers | (a) to (c) no, (d) yes | (a) how much another program costs. (b) `other programs on the card: Trae (2 %)`, and `none`. (c) the list, the explanation and the three buttons, nothing cut off. (d) the first prints the list and the advice and exits 1 with nothing changed on the card; the second logs the list and `going on with them` and runs; in the window **Check again** with the editor closed starts the run, **Go on anyway** starts it and logs the list, **Cancel** ends in **Not started** with the list | **(a) measured** (2026-10-11, reference RTX 4070, driver 617.42, saved tune: power 105 %, core +135, memory +650). Nothing else on the card: 5828 it/s, bandwidth 480 GB/s over 24 readings scattering by 0.23 %, and 477 to 483.5 in a later series. Trae open and idle, 2.2 % of the 3D engine: 5329 it/s, 452 to 457 GB/s over 28 readings, then two at 441 and 444. A YouTube video in Chrome, 8.6 % of the video decoder and 2.8 % of the 3D engine while the card idles: 464 to 476 GB/s while it decodes, 444 to 455 while it draws as well, 24 readings scattering by 3.0 %; back to 483 on the first reading after it stopped. Under the run's own load the other program's share reads 1 % or less, so it can only be seen while the card idles. One sample of the counters read 183 455 208 183 776 %. That day's card tests had all been taken with the editor open, from the morning until 21:31:44, when its last files were written; the log of the run under way shows the bandwidth going from 430.4 to 457.1 GB/s at 21:31:45. **(b) as expected with the editor open**: `other programs on the card: Trae (2 %)`, twice. Not run with it closed. **(c) as expected**: the heading, `chrome (12 %, video), Trae (2 %).`, the explanation in two lines, **Check again**, **Go on anyway** and **Cancel**, at 1180 x 900. **(d) the command line's refusal: as expected** (2026-10-11 22:46, elevated, a test build of ff80e3d, the editor and the chat application this was written in open). `gao --busy`: `other programs on the card: Trae (1 %)`. `gao --retest`: `other programs on the card: Trae (2 %), msedgewebview2 (1 %)`, the four lines of advice, `RESULT: not applied -- other programs are using the graphics card: Trae (2 %), msedgewebview2 (1 %)`, exit 1, four seconds after it began; the profile and what is applied read the same before and after, and `run.log` has the line. Two things seen: the editor read 1 % in one look and 2 % in the next, so the threshold went from 1 % to 0.5 % after this run; and `msedgewebview2` is the browser engine inside another application, a name that tells its user nothing. **Not run:** `--busy-ok`, and the three answers in the window, on a real run. | ## Notes From 5e3583c775197a5c2359a813125367eec190e4dc Mon Sep 17 00:00:00 2001 From: Rovey Date: Sun, 11 Oct 2026 23:51:54 +0200 Subject: [PATCH 15/15] docs: hardware check 95 (f), the two-pass memory search on the reference card One Best of my GPU run on a quiet card with the edge in its journal: the scan stopped at +2250, the second pass kept +2100, +1450 was applied, no reset, 3.1 % faster than stock. Co-Authored-By: Claude Opus 5.5 --- docs/development.md | 4 ++-- docs/hardware-checks.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/development.md b/docs/development.md index 063f82c..2042e69 100644 --- a/docs/development.md +++ b/docs/development.md @@ -79,8 +79,8 @@ The applied offset is always at least one step below the confirmed edge. Profile | Without the bandwidth measurement the memory is left at stock | The search used to fall back to a climb on stability alone when the measurement failed at +0. On a tester's 3080 Ti (0.4.3) that climb went to +2300 MHz with the score falling from +2150 on, got its first wrong result there, and the machine froze with a corrupted screen in the 30 s confirm probes just below it. The first wrong result is no edge to stand next to: the memory was correcting errors for hundreds of MHz before it. Now a measurement that does not work means no memory overclock (`left_at_stock`), the run says so, and the core is still searched. `optimize` keeps the stability climb only for a caller that gives no measurement at all; the app always gives one. Why the measurement failed on that card is not known: the log was only in the window and went with the reboot. | | A confirm probe that is too hot at once confirms nothing | A confirm probe that ends `TOO HOT` keeps the clock edge: over 30 s the card may run a little hotter than over the power step's 20 s, and that is the soak's business, which lowers power. But the stress run stops at its first reading above the limit, so a card that is above it when the probe begins gives `TOO HOT` within a second, and that counted as holding too. It happened on the tester's RTX 3080 Ti (hardware check 91): after a driver reset the health probe loads the card at stock power, which on that card means 76 C against a limit of 75, and `confirm core +225` was confirmed by a probe that had tested nothing. Now such a probe counts only when the clock was loaded for `kConfirmMinS` (10 s, clearly longer than the 3 s probe that found the edge) before it got too hot. Otherwise the value is not confirmed, and the card rests (`io.rest`, the 20 s of a recovery, with the fans where the curve last put them) before the next try one step lower: without the rest every lower value would start just as hot, and heat alone would step the edge down to nothing, which is what keeping the edge on `TOO HOT` was for. The same value is not tried again; one step is the price. Not changed: the health probe still runs at stock power, and a soak that begins too hot still lowers the power limit by a step. | | A profile that promises speed keeps no result that is slower than stock | Best of my GPU ended 3.8 % slower than stock on the tester's RTX 3080 Ti (4749 against 4937 it/s, hardware check 91), with the fans at 80 % instead of 66 %, and showed it as **Done**, applied and kept at every logon. Every step had done what it is for: the card runs 83 C at stock with its fans at 100 %, the profile allows 75, so the power limit went to 65 %, and +150 MHz of core clock did not win that back. But a profile called "the best of my GPU" that leaves the card slower has not done what it says, and a result like that is not offered as one. After the soak, `optimize` compares its score with the measurement at stock that went before the run (the "before" the result screen shows, so the rule judges by the number the user sees): below `kNotSlowerScore` (0.99: one percent is what two five-minute runs differ by) the run ends as a failed one does, at stock, with nothing saved, and says what it found, why, and that Undervolt is the profile for a card that runs warm. `Objectives::faster_than_stock` marks the two profiles this holds for, Best of my GPU and Max performance. Not Quiet and Cool & efficient, which trade speed for noise and heat and say so. Not the overclock step of All in one either: it goes on to an undervolt and a fan tune, and its result is judged as a whole; whether that whole may be slower than stock is not decided. This is a backstop, not the cure: what such a card needs from these profiles is in `docs/planned.md` ("never hotter than at stock"). | -| The memory search has two passes, and remembers where a card's memory gave out | **What went wrong.** Four runs on one day applied +650, +1600, +550 and +100 MHz on the reference RTX 4070, a card that runs every game at +1500 (hardware check 94). That card has no bandwidth peak: bandwidth rises by about 0.4 % per 50 MHz step all the way to +2500 and the stress load computes its first wrong result at +2550. So the scan could only end early by chance, and where it did decided the offset. **The cause, found late.** The chance came from another program on the card: an editor stood open all that day and took 2.2 % of the 3D engine, and with it the bandwidth readings scattered by 0.4 to 1.3 % and changed level by 1.8 %. On a card that is left alone they scatter by 0.2 % (hardware check 96). A run now looks for such programs before it begins and asks (the row on other programs on the card). What is left for the search to do is to give a sound result also when the user goes on next to them, and not to go to the memory's edge more often than it has to. **A detour, taken back.** The first answer to the scatter was a timed run four times as long (128 GB copied in one go). It was steadier next to the editor, and no steadier than the short one on a quiet card. With it every scan ran to where the memory gives out and met it with that copy: five scans out of five lost the device there, at +2450 or +2500, and the fifth took the whole machine down within seconds (check 95 b to d). In the one scan before that got to the edge, the short copy read a number at +2450 and +2500 and the stress load reported the wrong result at +2550, with no reset. The long run is gone; `kBwDispatches` is 64 as it was. **The design.** *The scan* goes up and only has to find where it ends: a step that is not stable, or one more than `kBandwidthDrop` (5 %) below the best. Two things keep a single reading from deciding: a reading that would end the scan is taken `kBandwidthReadings` (3) times and the middle one counts, and the best that readings are held against is the highest middle of three steps in a row, not the highest reading (the highest of fifty scattering readings is always one that came out high). 5 % is far more than readings move by themselves, also next to another program; the scan is not where the offset is chosen any more, so it can afford to be coarse. *The second pass* chooses the offset: it starts `kPeakMarginSteps` (4 steps, 200 MHz) below the highest step the scan passed (`second_pass_top`), reads again a step at a time going down until a reading is more than `kBandwidthTie` (1 %) below the best of them, and keeps the lowest offset within that 1 % (`bandwidth_peak_below`). On the reference card: the scan passes +2500, the second pass reads +2300, +2250, +2200 and +2150, and +2200 is kept; confirmed there, 70 % applied, +1500. Whatever the readings do, the offset comes out of a few steps below a point the card itself sets, so it repeats: +2050 to +2300 at the scatter of a quiet card. The margin is kept whatever ended the scan, a drop and the driver's own range included: what is above was never tried. After a reset event in the scan the second pass is skipped, and confirming starts four steps lower still. **The edge is met once.** A scan that ended in a wrong result or a reset event writes that offset to the journal with the GPU id and the driver version (`{"mem_edge":2550,"rig":" / "}`, a line without an id, so older builds skip it). A later scan on the same card and driver stops four steps below the lowest such offset, and says so; its second pass starts where the first run's did, because the margin counts from the edge and not a second time from where the scan was held. With another driver or another card the line does not count. A scan that ended for another reason (a bandwidth peak, readings that did not settle, the driver's range) writes nothing. **What it costs.** A card that does have a peak keeps an offset up to 200 MHz below it when the scan ends close above the peak: the margin does not know a drop from an edge. A scan that ends for a harmless reason (readings that did not settle) loses those 200 MHz as well. The second pass adds about four steps, a quarter of a minute. **Not chosen.** The long copy, anywhere: see above. Three readings at every step: two minutes a scan, and it decided nothing a single reading did not. Only remembering the edge: the result would still be chance for whoever goes on next to another program. The stress load probing four steps ahead of the copy: there is no heavy copy left to protect the edge from. A fixed top for the scan (an offset, a share of the range, a gain over +0): there is nothing to choose the number by. **Tested** on a simulated card with the reference card's numbers (wrong result at +2550): the second pass never reads above +2300, in the first run and in the one after it; with readings scattering by 1.3 % in both passes and level changes thrown in; with the edge anywhere from +300 to +2900; and the tests fail when the margin is set to nothing. **Not tested on a card.** See the row below. | -| Open: the two-pass memory search has not run on a card | The last card test of this work (hardware check 95 d) ended with the machine restarting, with the long copy that has since been taken out. Nothing of the search as it is now has run on a card. What a run on the reference card would show, and what it cannot: its journal holds the edge at +2450, so a run stops its scan at +2250 and reads again from +2200 down, at offsets every earlier scan passed. That tests the second pass and the result, with little risk (hardware check 95 f). It does not test the first meeting with the edge: that is what the released versions do with the same copy, it ended in a wrong result the one time it was seen, and seeing it again means removing the edge from the journal and going there. **Still not known.** Whether a card with a real bandwidth peak behaves as assumed; none has been measured through a scan (the tester's RTX 3080 Ti never got a reading to settle at +0). And why the long copy failed at the edge where the short one did not: the amount copied in one go, or only that five runs are more than one. It no longer matters for the search, and is noted because it was never explained. | +| The memory search has two passes, and remembers where a card's memory gave out | **What went wrong.** Four runs on one day applied +650, +1600, +550 and +100 MHz on the reference RTX 4070, a card that runs every game at +1500 (hardware check 94). That card has no bandwidth peak: bandwidth rises by about 0.4 % per 50 MHz step all the way to +2500 and the stress load computes its first wrong result at +2550. So the scan could only end early by chance, and where it did decided the offset. **The cause, found late.** The chance came from another program on the card: an editor stood open all that day and took 2.2 % of the 3D engine, and with it the bandwidth readings scattered by 0.4 to 1.3 % and changed level by 1.8 %. On a card that is left alone they scatter by 0.2 % (hardware check 96). A run now looks for such programs before it begins and asks (the row on other programs on the card). What is left for the search to do is to give a sound result also when the user goes on next to them, and not to go to the memory's edge more often than it has to. **A detour, taken back.** The first answer to the scatter was a timed run four times as long (128 GB copied in one go). It was steadier next to the editor, and no steadier than the short one on a quiet card. With it every scan ran to where the memory gives out and met it with that copy: five scans out of five lost the device there, at +2450 or +2500, and the fifth took the whole machine down within seconds (check 95 b to d). In the one scan before that got to the edge, the short copy read a number at +2450 and +2500 and the stress load reported the wrong result at +2550, with no reset. The long run is gone; `kBwDispatches` is 64 as it was. **The design.** *The scan* goes up and only has to find where it ends: a step that is not stable, or one more than `kBandwidthDrop` (5 %) below the best. Two things keep a single reading from deciding: a reading that would end the scan is taken `kBandwidthReadings` (3) times and the middle one counts, and the best that readings are held against is the highest middle of three steps in a row, not the highest reading (the highest of fifty scattering readings is always one that came out high). 5 % is far more than readings move by themselves, also next to another program; the scan is not where the offset is chosen any more, so it can afford to be coarse. *The second pass* chooses the offset: it starts `kPeakMarginSteps` (4 steps, 200 MHz) below the highest step the scan passed (`second_pass_top`), reads again a step at a time going down until a reading is more than `kBandwidthTie` (1 %) below the best of them, and keeps the lowest offset within that 1 % (`bandwidth_peak_below`). On the reference card: the scan passes +2500, the second pass reads +2300, +2250, +2200 and +2150, and +2200 is kept; confirmed there, 70 % applied, +1500. Whatever the readings do, the offset comes out of a few steps below a point the card itself sets, so it repeats: +2050 to +2300 at the scatter of a quiet card. The margin is kept whatever ended the scan, a drop and the driver's own range included: what is above was never tried. After a reset event in the scan the second pass is skipped, and confirming starts four steps lower still. **The edge is met once.** A scan that ended in a wrong result or a reset event writes that offset to the journal with the GPU id and the driver version (`{"mem_edge":2550,"rig":" / "}`, a line without an id, so older builds skip it). A later scan on the same card and driver stops four steps below the lowest such offset, and says so; its second pass starts where the first run's did, because the margin counts from the edge and not a second time from where the scan was held. With another driver or another card the line does not count. A scan that ended for another reason (a bandwidth peak, readings that did not settle, the driver's range) writes nothing. **What it costs.** A card that does have a peak keeps an offset up to 200 MHz below it when the scan ends close above the peak: the margin does not know a drop from an edge. A scan that ends for a harmless reason (readings that did not settle) loses those 200 MHz as well. The second pass adds about four steps, a quarter of a minute. **Not chosen.** The long copy, anywhere: see above. Three readings at every step: two minutes a scan, and it decided nothing a single reading did not. Only remembering the edge: the result would still be chance for whoever goes on next to another program. The stress load probing four steps ahead of the copy: there is no heavy copy left to protect the edge from. A fixed top for the scan (an offset, a share of the range, a gain over +0): there is nothing to choose the number by. **Tested** on a simulated card with the reference card's numbers (wrong result at +2550): the second pass never reads above +2300, in the first run and in the one after it; with readings scattering by 1.3 % in both passes and level changes thrown in; with the edge anywhere from +300 to +2900; and the tests fail when the margin is set to nothing. **On the card:** one run, with the edge already in the journal (hardware check 95 f): as designed. See the row below for what that leaves. | +| Open: the first meeting with a memory edge has not been seen with the two-pass search | The search as it is now ran once on the reference card (hardware check 95 f), on a quiet card and with the edge at +2450 in its journal: the scan stopped at +2250, the second pass read +2200 down to +2050 and kept +2100, +1450 was applied, there was no reset, and the result was 3.1 % faster than stock. That is the run after the one that finds the edge. The run that finds it has not been seen: on a card whose journal has no edge the scan goes up until the stress load computes wrong or something gives, with the same copy the released versions use; that ended in a wrong result the one time it was seen before (check 94), and seeing it again means removing the edge from a journal and going there. **Still not known.** Whether a card with a real bandwidth peak behaves as assumed; none has been measured through a scan (the tester's RTX 3080 Ti never got a reading to settle at +0). And why the long copy failed at the edge where the short one did not: the amount copied in one go, or only that five runs are more than one. It no longer matters for the search, and is noted because it was never explained. | | Open: why the bandwidth measurement fails on some cards, and what then | On the tester's RTX 3080 Ti it failed at +0, and since 0.4.4 that means no memory overclock at all on such a card, though it computed right far beyond +2000 MHz. The measurement gives a number only when three runs in a row agree within 1 % (`kBwSettle`), and it throws away every run the driver's timer calls disjoint (the clock changed under it). That card was held at a power limit of 65 % and ran at 83 C when it was measured; the reference card has only ever been measured at 105 %, where the measurement itself draws 179 of 210 W. So the suspicion is the power limit, and hardware check 90 tests exactly that on the reference card. What it shows decides what is built, and nothing is built before it: **(1) runs are thrown away as disjoint, and the ones that are kept agree:** the timer is what fails, not the card. Time the copy differently (wall clock around a flush, or count a disjoint run when its neighbours agree) and keep the method. **(2) the readings themselves scatter by more than 1 %:** a peak that is found by a 1 % drop (`kBandwidthDrop`) cannot be found in that noise. Then the memory has to be measured where the card is not held back: search it before the power step, at stock power, where a probe and its measurement last five seconds; or take more readings and compare medians. **(3) it does not fail at 65 % on the reference card:** the power limit is not the cause. Then it was the heat (a card throttling at its temperature limit changes clock as well), or another program, and only the tester's `run.log` can say: since this change a measurement that never settled writes how many runs gave a reading, between which values, and how many were thrown away. In no case does the climb on stability alone come back. **Outcome of check 90: (3).** On the reference card the measurement gave a number fifteen times out of fifteen at 100, 80 and 65 %, in three or four runs, with nothing thrown away, and the bandwidth was the same at every limit. The power limit is ruled out; (1) and (2) are not built. What is left is the heat of that card, another program, or a timer query that failed, and the tester's `run.log` is the next thing that can tell them apart. **The tester's run (check 91): it is (2), but only inside a run.** Twenty readings between 763.9 and 821.2 GB/s, none thrown away, three seconds after the load began; after thirty seconds of load the same card read within 0.2 %. The suspicion now is the moment, not the card: the measurement is taken while the power limiter is still catching up. If `gao --bandwidth --after-load 3` scatters on that card and `--after-load 30` does not, that is it, and the fix is to measure when the card has settled (keep taking runs for some seconds instead of twenty runs), not to change the method. | | Every line of a run is also in `run.log` | `logged()` wraps the log hook of both front ends: a line is appended and flushed before it is shown, so the step a machine freezes in is the last line on disk. One file in the state folder, moved aside at 1 MB (`run.log.old`). Only elevated and only after `ensure_app_dir` has verified the folder: an elevated process appends to nothing a user could have put there. | | A run does not start while the NVIDIA app's Automatic Tuning is on | Switched on, a service of the driver scans the card when it sees fit: it loads it, changes its clocks, and starts again by itself (hardware check 89: a scan that was stopped started again 14 ms later, and after a restart of the service it resumed within two seconds). A run next to that measures nothing it can trust, so it is refused, like a run next to Afterburner, and with the same weight: a courtesy, not a guard. Read from where that service keeps the switch, `%ProgramData%\NVIDIA Corporation\nvtopps\nvtopps.db3`, table `ocScannerSettings`, fields `backgroundScanEnabled` and `grdOcEnabled`, joined to `TelemetryGpuCfg` so that only a card that is in the PC counts (a row of an earlier card can never be switched off again). That this is the source and not a copy of it was measured, not assumed: with the field set the service took the scan up again after every restart, with it cleared it did not, while the NVIDIA app itself showed the switch as off. Nothing documents the file. So: opened read-only and immutable (no lock, the service never waits for us), through `winsqlite3.dll` from System32 (no dependency added), and everything that is not exactly as expected reads as "unknown", which starts the run as before. The registry value the app sets next to it (`HKLM\SOFTWARE\NVIDIA Corporation\Global\NvApp\OC\`) changes at the same moment but belongs to the app, not to the service, and is not used. **The overlay is not checked:** no setting for it was found, its processes run whether it is open or not, and the one value that moved (`IsShadowPlayEnabled`) followed the service being up, not a switch. NVAPI was not tried: an undocumented call is no better than an undocumented file, and it cannot be compared with anything. | diff --git a/docs/hardware-checks.md b/docs/hardware-checks.md index 3ab602f..e0c814b 100644 --- a/docs/hardware-checks.md +++ b/docs/hardware-checks.md @@ -122,7 +122,7 @@ root; adjust the path for a Debug build. | 92 | The undervolt profile starts with the measurement at stock every run uses, and a hot card is not refused | Elevated, the tray app exited, a copy of `gao.json` kept. (a) `gao --undervolt` to the end; then `gao --status`. (b) The same with a test build whose stock limit is lowered to a temperature the reference card passes under load (63 C; it reaches 67). Put the copy of `gao.json` back. (c) Only on a card that is above the profile's limit at stock with the fans at 100 % (the tester's RTX 3080 Ti: 82 to 83 C): the **Undervolt** profile from the window, to the end, and its `run.log` | yes | (a) `the card at stock, five minutes`, a `stock: STABLE ...` line with the driver's fan speed, then `baseline: the measurement at stock`, and no second five minutes at stock: the first candidate follows at once; no `candidates are stopped at` line; `RESULT:`; `before:` shows 300 s and the fan speed the driver chose. (b) `stock: TOO HOT ... peak=64 C`, the line that the measurement stopped there, the cool-down, then `baseline: the measurement at stock` and a search that runs to `RESULT:` with `before:` showing the shortened time; no `candidates are stopped at` line, because 64 C is under the profile's 75. (c) no `not stable at stock`; `the card reached 86 C at stock: candidates are stopped at 85 C, not at the profile's 75 C`; candidates that end `STABLE` or `WRONG RESULT`, none `TOO HOT`; a result that is cooler and draws less than stock at the same clock | First run: **not as expected**, and that found a fault (2026-10-11, reference RTX 4070, driver 617.42, a test build of d1c99e9, started elevated by a script). The first halves held: (a) `stock: STABLE ... peak=65 C fan=70 %`, then `baseline: the measurement at stock` four seconds later and no second five minutes; (b) `stock: TOO HOT ... peak=64 C`, stopped after 34 s, the run went on, no `candidates are stopped at` line. But both runs then ended with `the card held no undervolt: the first step below its own voltage already failed`: their first candidates read `1070 mV (+15 MHz): SLOW ... clock=2323 MHz` and `1050 mV (+15 MHz): SLOW ... clock=2411 MHz`, on a card that ran 2791 MHz in the measurement just before. The search's load had not run yet when that candidate began (see the decision "With its reference handed over, the undervolt search warms its load up before the first candidate"). Second run, with the warm-up: **pass** for (a) and (b) (same day, a test build of 0733695). (a) `stock: STABLE score=5426 it/s clock=2788 MHz peak=66 C fan=70 %`, four seconds later `baseline: the measurement at stock`, `warming the load up: 5 s at stock`, `target: 2790 MHz, which the card reaches at 1090 mV by itself`, and the first candidate `1085 mV (+15 MHz): STABLE ... clock=2775 MHz`; 26 candidates, none `SLOW` or `TOO HOT`, the edge at 950 mV, `RESULT: 2790 MHz at 990 mV instead of 1090 mV`; `before:` 5426 it/s, 189 W, fan 70 % (300 s), `after:` 5325 it/s, 149 W, fan 53 % (300 s). (b) `stock: TOO HOT ... peak=64 C`, stopped after 74 s, the run went on, the same warm-up and a search to the same result (990 mV, edge 950 mV); `before:` shows 75 s; no `candidates are stopped at` line. The saved tune of before the test was put back and applied. (c) not run. | | 93 | A speed profile that would end slower than stock keeps nothing | Only on a card that is warmer at stock than the profile allows (the tester's RTX 3080 Ti; the reference card ends faster than stock with every speed profile): **Best of my GPU** from the window, to the end; the result screen, the dashboard and `run.log` | yes | the run goes through all its steps and its soak; then **Not applied** with `the result is slower than stock: N it/s against M (-X %)`, the power limit it had to go down to, and the pointer to Undervolt; `The card is back at stock.`; the dashboard shows stock and the saved tune of before the run, if there was one; `run.log` ends with `stopped: the result is slower than stock ... -- card restored to stock` | Not run yet. On the reference card: nothing to see, by design. | | 94 | All in one, with the undervolt step searching for the first time | Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize all` to the end with a build that has the warm-up of the undervolt step; then `gao --status`; put the copy back | yes | five steps; in step 3 a `warming the load up` line, a first candidate that is not `SLOW`, and a descent; steps 4 and 5 run on the tune of steps 2 and 3; `RESULT:` | **Not as expected at step 4**, and that found a fault (2026-10-11, reference RTX 4070, driver 617.42, a test build of 0733695, unattended). Step 1: `stock: STABLE score=5325 it/s peak=65 C fan=70 % power=186 W`. Step 2: power 105 %, `mem: bandwidth peak +2300, confirmed +2300, applying +1600`, `core: highest stable +210, confirmed +210, applying +135`, soak `STABLE score=5568 it/s peak=70 C power=197 W`. Step 3, as wanted: `warming the load up: 5 s with the overclock`, `target: 2865 MHz, which the card reaches at 1050 mV with the overclock`, nine candidates `STABLE` from 1045 mV down, `WRONG RESULT` at 990 mV, confirmed at 1010 mV after two tries, and `applying 1050 mV (+135 MHz at that point)`: the margin left no raise above the overclock, so the curve was cut off, as in the runs before the warm-up; soak `STABLE score=5434 it/s peak=67 C power=176 W`. Step 4 did not begin: `the tune could not be applied for the fan tune: the undervolt was not applied: the point already runs more than the target -- card at stock`. The saved tune of before the test was put back and applied. Cause and fix: the decision "A flat top goes back on a card that is cooler than when it was found". Second run, with the `gao.exe` of the released v0.4.6: **pass, all five steps** (same day, started elevated by a script). Step 1: `stock: STABLE score=5309 it/s peak=66 C fan=70 % power=185 W`. Step 2: power 105 %, `mem: bandwidth peak +850, confirmed +850, applying +550`, `core: highest stable +210, confirmed +210, applying +135`, soak `STABLE score=5568 it/s`. Step 3: `target: 2850 MHz, which the card reaches at 1045 mV with the overclock`, `undervolt: lowest stable 975 mV, confirmed 985 mV, applying 1025 mV (+165 MHz at that point)`, soak `STABLE score=5363 it/s peak=66 C power=163 W`. Step 4: the tune went back on the card, `fan 60 %: 63 C`, `fan 55 %: 65 C`, `fan 52 %: 67 C`, `fan tune: 52 % holds 67 C`. Step 5: the final test ran its 300 s. `RESULT: power 105 %, core +135 MHz, mem +550 MHz, 2850 MHz at 1025 mV, fans 52 % at 67 C`; `stock: 5309 it/s, 185 W, fan 70 %, 66 C`, `now: 5353 it/s, 166 W, fan 52 %, 70 C`: `+0.8 % score and -10.3 % power against stock`; `undervolt: applied`. The saved tune of before the test was put back. **What this run did not show:** its undervolt had a raise of its own (+165 over an offset of +135), so the anchor was raised and not merely cut off, and the case of the first run (an anchor that reads a step above the target on a cooler card) did not come up. That fix has still only its tests. **The memory offset differs a lot from run to run on this card, and the two logs say why:** +1600 in the first run (bandwidth peak +2300), +550 in the second (peak +850), +650 in the run that saved the tune in use that morning (the journal shows its scan ending at +1000 too). The card was in the same state each time (68 C, power 105 %, a reading every three to four seconds). In both logged scans the bandwidth rises in a straight line, 0.36 and 0.46 % per 50 MHz step, from 420 GB/s at +0 to 509 GB/s at +2450; the first run went on to `WRONG RESULT` at +2550 without the bandwidth ever turning down. The readings scatter around that line with a standard deviation of 0.4 and 0.7 %, single ones by more than 1 %. The scan ends at a reading more than 1 % below the best so far when a second reading agrees, and in the second run that was chance: +950 read high (462.8 GB/s, where the line is at about 458), +1000 read 456.6 twice, 1.3 % below it. In the first run the one low reading (+600) was followed by a higher one, and the scan ran to the first wrong result. So on this card there is no bandwidth peak before the errors, and where the scan stops is decided by noise. | -| 95 | The memory scan no longer ends by chance | (a) Not elevated, nothing changed on the card: a measuring build takes 24 bandwidth readings in one process, with 3 s of the stress load before each as a scan has, at 64, 128 and 256 dispatches per timed run; then the same with the app's window open next to it, drawing a frame every 100 ms and every 33 ms. (b) Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize best` twice in a row with the build of the change, then the copy put back | (a) no, (b) yes | (a) how much a reading scatters, and on what that depends. (b) in both runs every `mem +N: bandwidth` line shows three readings that agree within about 1 %; the scan does not end before the first `WRONG RESULT`; the two `mem: bandwidth peak` values are within 300 MHz of each other and the applied offset is above +1000; the soak is not slower than at stock | **(a) measured** (2026-10-11, reference RTX 4070, driver 617.42, the saved tune applied: power 105 %, core +135, memory +650; standard deviation over the 24 readings of a series, and the lowest and highest against the mean). 64 dispatches, no window: 0.44 % (-0.55 to +1.07, one reading did not settle), 0.66 % (-1.58 to +1.49, one did not settle), 1.31 % (-4.83 to +1.27), 0.86 % (-1.44 to +2.05). 128 dispatches, no window: 0.43 % (-1.10 to +0.79). 256 dispatches, no window: 0.22 % (-0.43 to +0.42) and 0.60 % (-0.80 to +1.37; the last five readings of this series were 451 to 453 GB/s after nineteen of 444 to 449). With the window drawing every 100 ms: 64 dispatches 1.13 % (the first six readings 452 to 453 GB/s, the rest 439 to 451), 256 dispatches 0.24 % (-0.47 to +0.61). With the window drawing every 33 ms: 64 dispatches 0.58 %, 256 dispatches 0.25 % (-0.35 to +0.51). A reading took 6.5, 5.3 and 3.9 timed runs to settle at 64, 128 and 256 dispatches. So: the longer timed run takes the scatter from 0.4 to 1.3 % down to 0.2 %, the window's drawing adds nothing, and apart from the scatter the readings sit about 1.8 % higher for up to a minute at a time (the stretch at the end of the second 256 series ran on into the next series; the last 64 series went back and forth between 444 and 452 GB/s). What sets that level is not known. **(b) the offset is the same twice; the scan ends in a driver reset twice** (2026-10-11, same card and driver, a test build of 2764609, unattended, 40 minutes for the two runs). Run 1: `stock: STABLE score=5311 it/s peak=66 C fan=70 % power=185 W`; 49 steps from `mem +0: bandwidth 422.7 GB/s (422.7 422.6 422.7)` to `mem +2400: bandwidth 501.4 GB/s (501.3 502.3 501.4)`; at +2450 the candidate was `STABLE` and the measurement after it lost the device: `mem +2450: the bandwidth measurement failed; treated as a driver reset`, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, 20 s of rest, `health: STABLE`, `confirm core +0 / mem +2050: STABLE`, `mem: bandwidth peak +2250, confirmed +2050, applying +1400`; `core: highest stable +210, confirmed +210, applying +135`; soak `STABLE score=5536 it/s peak=69 C power=195 W`; `driver resets during this run: 1`. Run 2: stock 5312 it/s; 50 steps from 421.8 GB/s at +0 to 502.8 at +2450; the same failure at +2500; `mem: bandwidth peak +2300, confirmed +2100, applying +1450`; core +135; soak `STABLE score=5555 it/s peak=69 C power=196 W`; one driver reset. The three readings of a step differed by 0.4 % on average and by 1.2 % at most; no step was more than 0.7 % below the best before it, so neither scan came near ending early, by the old rule or the new; every step had its three readings. The scan took 5 min 25 s (about 6.5 s a step; about 3 min before the change). As expected: the two peaks are 50 MHz apart, the applied offsets are +1400 and +1450, and both results are 4 % faster than stock. **Not as expected: neither scan reached a `WRONG RESULT`.** Both ended one and two steps before +2550 with the device lost during the bandwidth copy; Windows logged it both times (System log, `nvlddmkm` events 14 and 153 at 19:51:22 and 20:11:27), and the screen goes black for a moment. In the run of check 94 that got this far, the measurements at +2450 and +2500 gave a number and the stress load reported the first wrong result at +2550, without a reset. A step now copies about seven times as much as it did (three readings of about four runs of 128 GB, against one reading of about six runs of 32 GB), and the copy checks nothing, so where the memory starts to fail it is the driver that gives out, not a verdict. Two runs against one: likely the cause, not proven. The search handles it as it should (rest, health probe, confirm four steps lower, the core is still searched), but a driver reset in every run on a card like this is not what the change was for. The saved tune of before the test was put back and applied (`applied best: power 105 %, core +135 MHz, mem +650 MHz`), the tray app restarted, Defender reported nothing. **(c) the same two runs with one reading a step, three only for a step in doubt** (2026-10-11 20:26 to 21:03, a test build of 71e7c2b, unattended): **the reset is still there.** Run 1: stock 5328 it/s; 50 steps from `mem +0: bandwidth 420.4 GB/s` to `mem +2450: bandwidth 502.4 GB/s`; at +2500 the candidate was `STABLE` and then `bandwidth measurement failed: the timing of the copy did not come back` ten seconds later (`nvlddmkm` event 14 at 20:37:07); `mem: bandwidth peak +2200, confirmed +2000, applying +1400`; core +135; soak `STABLE score=5572 it/s peak=70 C power=197 W`; one driver reset. Run 2: stock 5311 it/s; 421.9 GB/s at +0 to 508.0 at +2450; at +2500 `the device was lost during the copy failed (hr=0x887A0006)`, the device hung (event 14 at 20:55:29); `mem: bandwidth peak +2400, confirmed +2200, applying +1500`; core +135; soak `STABLE score=5571 it/s peak=70 C power=196 W`; one driver reset. No step was in doubt in either run (the lowest was 0.75 % below the best before it), so every step was read once; the scan took 3 min 33 s. The two peaks are 200 MHz apart: with one reading a step, which offsets count as within 1 % of the best moves by a few steps. Applied +1400 and +1500, both 5 % faster than stock. So the reset does not come from reading a step three times. Four runs out of four with the longer timed run ended this way, at +2450 or +2500. Not merged; the design for a scan that does not end like this is the open row in `docs/development.md`. The saved tune was put back and applied again (core +135, memory +650), the tray app restarted, Defender reported nothing. **(d) the edge is remembered** (a test build of 7c215c4; the same two runs): expected is that the first scan ends as before and logs nothing new, the journal gets one `mem_edge` line with the GPU id and the driver version, and the second run logs `mem: the memory gave out at +N in an earlier run on this card and driver; the scan stops at +(N-200)`, has no `driver resets during this run` line, applies more than +1000 and is not slower than stock. **Run 1 took the machine down; the test ended there** (2026-10-11 21:23, unattended). Stock 5321 it/s, power 105 %, then the scan: 49 steps from +0 to `mem +2400: bandwidth 539.3 GB/s`, two of them read three times (+1200 and +1250). At +2450 the candidate was `STABLE` and the copy after it failed as in the four runs before: `bandwidth measurement failed: the copy failed (hr=0x887A0007): the device was lost` at 21:35:05, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, `the driver was reset; reconnecting`. That is the last line. Windows restarted by itself within fifteen seconds: Kernel-Power event 41 at 21:35:23, `The previous system shutdown at 21:34:57 was unexpected`, no bugcheck, no dump, no `nvlddmkm` event written. So the fifth scan that met the edge with the longer timed run did not end in a driver reset the search recovers from, but in a reset of the whole machine, during the reconnect or right after it. What held: the candidate's journal entry was closed, so no ceiling of a freeze was left; the line `{"mem_edge":2450,"rig":"GPU-... / 617.42"}` was on disk before the machine went down, so no later run of this build goes there again; the run had saved nothing, and at logon the tray app applied the tune of before the test (`applied best: power 105 %, core +135 MHz, mem +650 MHz`). Not answered: the second run. Seen along the way and not explained: from the first step of the scan the stress load scored 5760 it/s and the bandwidth read about 7 % higher than in every run before (486.6 GB/s at +900 against about 455), while the measurement at stock five minutes earlier was the usual 5321. **(e) how the stress load meets the edge** (the same build, no search): `gao --reset`, then `gao --set-mem` and `gao --stress 3` at +2300, +2350, ... until the verdict is not `STABLE`, five times, at stock power and core +0; expected is one of two outcomes, `WRONG RESULT` every time (then the stress load can lead the copy to the edge) or `DEVICE LOST` (then it cannot). **Not run:** the test ended in (d). It goes to the edge on purpose, five times, and after (d) that is a risk for the machine that needs its owner's word first. **(f) the two-pass search as it is now** (one copy; the second pass reads again from four steps below the highest step that passed; see `docs/development.md`): tested on a simulated card only. On the reference card, with the edge at +2450 in its journal and no other program on the card: `gao --optimize best` once; expected `other programs on the card: none`, `mem: the memory gave out at +2450 in an earlier run on this card and driver; the scan stops at +2250`, no memory candidate above +2250, `mem: the scan passed +2250; reading again from +2200 down`, a few `mem +N: bandwidth ... GB/s, read again` lines, no `driver resets during this run`, an applied offset above +1000 and a soak that is not slower than stock. **Not run: waits for the owner's word.** **(g) what else uses the card** (2026-10-11 22:00, after the restart of (d), not elevated, the saved tune applied, nothing changed): `gao --stress 5` scored 5828 it/s and `gao --bandwidth --after-load 3` read 484.0 GB/s with its three runs within 0.0 %, against 5226 it/s and 444 to 452 GB/s with the same tune earlier that evening. The series of (a) again, 24 readings each: 64 dispatches 480.0 GB/s, 0.23 % (-0.66 to +0.18); 256 dispatches 475.2 GB/s, 0.68 %, sliding from 480 to 471 during the series; 64 dispatches again 454.0 GB/s, 3.02 % (-4.37 to +4.22), the readings jumping between 434 and 473. Read right after it, the card's use per program: Chrome 8.6 % of the video decoder (19 % at most) and 2.8 % of the 3D engine, with a YouTube video playing; nothing else above 1.1 %. So the short copy is as steady as the long one on a card that is left alone, and a video playing next to it costs about 8 % of the bandwidth and of the stress score and makes the readings jump. Then 40 readings with the short copy, and next to each the other programs on the card in the seconds before it (22:13): 464 to 476 GB/s while Chrome used the video decoder, 444 to 455 while it used the 3D engine as well, and from the reading after Chrome stopped, 477 to 483.5 for the last eleven readings with no other program on the card. So another program using the card lowers the reading by 3 to 8 % from one reading to the next. That explains the scatter and the level changes of (a) in kind. During the tests of that day it was most likely not a video (hardly any was watched) but a code editor built on a browser engine that had stood open since the morning: its last files were written at 21:31:43 and 21:31:44, when it was closed, and the run of (d) logs the bandwidth going from 430.4 to 457.1 GB/s at 21:31:45. So every card measurement of checks 94 and 95 up to then was taken with that program on the card: about 7 % lower, and more scattered, than the card alone. A match to the second; not measured with the editor started again. | +| 95 | The memory scan no longer ends by chance | (a) Not elevated, nothing changed on the card: a measuring build takes 24 bandwidth readings in one process, with 3 s of the stress load before each as a scan has, at 64, 128 and 256 dispatches per timed run; then the same with the app's window open next to it, drawing a frame every 100 ms and every 33 ms. (b) Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize best` twice in a row with the build of the change, then the copy put back | (a) no, (b) yes | (a) how much a reading scatters, and on what that depends. (b) in both runs every `mem +N: bandwidth` line shows three readings that agree within about 1 %; the scan does not end before the first `WRONG RESULT`; the two `mem: bandwidth peak` values are within 300 MHz of each other and the applied offset is above +1000; the soak is not slower than at stock | **(a) measured** (2026-10-11, reference RTX 4070, driver 617.42, the saved tune applied: power 105 %, core +135, memory +650; standard deviation over the 24 readings of a series, and the lowest and highest against the mean). 64 dispatches, no window: 0.44 % (-0.55 to +1.07, one reading did not settle), 0.66 % (-1.58 to +1.49, one did not settle), 1.31 % (-4.83 to +1.27), 0.86 % (-1.44 to +2.05). 128 dispatches, no window: 0.43 % (-1.10 to +0.79). 256 dispatches, no window: 0.22 % (-0.43 to +0.42) and 0.60 % (-0.80 to +1.37; the last five readings of this series were 451 to 453 GB/s after nineteen of 444 to 449). With the window drawing every 100 ms: 64 dispatches 1.13 % (the first six readings 452 to 453 GB/s, the rest 439 to 451), 256 dispatches 0.24 % (-0.47 to +0.61). With the window drawing every 33 ms: 64 dispatches 0.58 %, 256 dispatches 0.25 % (-0.35 to +0.51). A reading took 6.5, 5.3 and 3.9 timed runs to settle at 64, 128 and 256 dispatches. So: the longer timed run takes the scatter from 0.4 to 1.3 % down to 0.2 %, the window's drawing adds nothing, and apart from the scatter the readings sit about 1.8 % higher for up to a minute at a time (the stretch at the end of the second 256 series ran on into the next series; the last 64 series went back and forth between 444 and 452 GB/s). What sets that level is not known. **(b) the offset is the same twice; the scan ends in a driver reset twice** (2026-10-11, same card and driver, a test build of 2764609, unattended, 40 minutes for the two runs). Run 1: `stock: STABLE score=5311 it/s peak=66 C fan=70 % power=185 W`; 49 steps from `mem +0: bandwidth 422.7 GB/s (422.7 422.6 422.7)` to `mem +2400: bandwidth 501.4 GB/s (501.3 502.3 501.4)`; at +2450 the candidate was `STABLE` and the measurement after it lost the device: `mem +2450: the bandwidth measurement failed; treated as a driver reset`, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, 20 s of rest, `health: STABLE`, `confirm core +0 / mem +2050: STABLE`, `mem: bandwidth peak +2250, confirmed +2050, applying +1400`; `core: highest stable +210, confirmed +210, applying +135`; soak `STABLE score=5536 it/s peak=69 C power=195 W`; `driver resets during this run: 1`. Run 2: stock 5312 it/s; 50 steps from 421.8 GB/s at +0 to 502.8 at +2450; the same failure at +2500; `mem: bandwidth peak +2300, confirmed +2100, applying +1450`; core +135; soak `STABLE score=5555 it/s peak=69 C power=196 W`; one driver reset. The three readings of a step differed by 0.4 % on average and by 1.2 % at most; no step was more than 0.7 % below the best before it, so neither scan came near ending early, by the old rule or the new; every step had its three readings. The scan took 5 min 25 s (about 6.5 s a step; about 3 min before the change). As expected: the two peaks are 50 MHz apart, the applied offsets are +1400 and +1450, and both results are 4 % faster than stock. **Not as expected: neither scan reached a `WRONG RESULT`.** Both ended one and two steps before +2550 with the device lost during the bandwidth copy; Windows logged it both times (System log, `nvlddmkm` events 14 and 153 at 19:51:22 and 20:11:27), and the screen goes black for a moment. In the run of check 94 that got this far, the measurements at +2450 and +2500 gave a number and the stress load reported the first wrong result at +2550, without a reset. A step now copies about seven times as much as it did (three readings of about four runs of 128 GB, against one reading of about six runs of 32 GB), and the copy checks nothing, so where the memory starts to fail it is the driver that gives out, not a verdict. Two runs against one: likely the cause, not proven. The search handles it as it should (rest, health probe, confirm four steps lower, the core is still searched), but a driver reset in every run on a card like this is not what the change was for. The saved tune of before the test was put back and applied (`applied best: power 105 %, core +135 MHz, mem +650 MHz`), the tray app restarted, Defender reported nothing. **(c) the same two runs with one reading a step, three only for a step in doubt** (2026-10-11 20:26 to 21:03, a test build of 71e7c2b, unattended): **the reset is still there.** Run 1: stock 5328 it/s; 50 steps from `mem +0: bandwidth 420.4 GB/s` to `mem +2450: bandwidth 502.4 GB/s`; at +2500 the candidate was `STABLE` and then `bandwidth measurement failed: the timing of the copy did not come back` ten seconds later (`nvlddmkm` event 14 at 20:37:07); `mem: bandwidth peak +2200, confirmed +2000, applying +1400`; core +135; soak `STABLE score=5572 it/s peak=70 C power=197 W`; one driver reset. Run 2: stock 5311 it/s; 421.9 GB/s at +0 to 508.0 at +2450; at +2500 `the device was lost during the copy failed (hr=0x887A0006)`, the device hung (event 14 at 20:55:29); `mem: bandwidth peak +2400, confirmed +2200, applying +1500`; core +135; soak `STABLE score=5571 it/s peak=70 C power=196 W`; one driver reset. No step was in doubt in either run (the lowest was 0.75 % below the best before it), so every step was read once; the scan took 3 min 33 s. The two peaks are 200 MHz apart: with one reading a step, which offsets count as within 1 % of the best moves by a few steps. Applied +1400 and +1500, both 5 % faster than stock. So the reset does not come from reading a step three times. Four runs out of four with the longer timed run ended this way, at +2450 or +2500. Not merged; the design for a scan that does not end like this is the open row in `docs/development.md`. The saved tune was put back and applied again (core +135, memory +650), the tray app restarted, Defender reported nothing. **(d) the edge is remembered** (a test build of 7c215c4; the same two runs): expected is that the first scan ends as before and logs nothing new, the journal gets one `mem_edge` line with the GPU id and the driver version, and the second run logs `mem: the memory gave out at +N in an earlier run on this card and driver; the scan stops at +(N-200)`, has no `driver resets during this run` line, applies more than +1000 and is not slower than stock. **Run 1 took the machine down; the test ended there** (2026-10-11 21:23, unattended). Stock 5321 it/s, power 105 %, then the scan: 49 steps from +0 to `mem +2400: bandwidth 539.3 GB/s`, two of them read three times (+1200 and +1250). At +2450 the candidate was `STABLE` and the copy after it failed as in the four runs before: `bandwidth measurement failed: the copy failed (hr=0x887A0007): the device was lost` at 21:35:05, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, `the driver was reset; reconnecting`. That is the last line. Windows restarted by itself within fifteen seconds: Kernel-Power event 41 at 21:35:23, `The previous system shutdown at 21:34:57 was unexpected`, no bugcheck, no dump, no `nvlddmkm` event written. So the fifth scan that met the edge with the longer timed run did not end in a driver reset the search recovers from, but in a reset of the whole machine, during the reconnect or right after it. What held: the candidate's journal entry was closed, so no ceiling of a freeze was left; the line `{"mem_edge":2450,"rig":"GPU-... / 617.42"}` was on disk before the machine went down, so no later run of this build goes there again; the run had saved nothing, and at logon the tray app applied the tune of before the test (`applied best: power 105 %, core +135 MHz, mem +650 MHz`). Not answered: the second run. Seen along the way and not explained: from the first step of the scan the stress load scored 5760 it/s and the bandwidth read about 7 % higher than in every run before (486.6 GB/s at +900 against about 455), while the measurement at stock five minutes earlier was the usual 5321. **(e) how the stress load meets the edge** (the same build, no search): `gao --reset`, then `gao --set-mem` and `gao --stress 3` at +2300, +2350, ... until the verdict is not `STABLE`, five times, at stock power and core +0; expected is one of two outcomes, `WRONG RESULT` every time (then the stress load can lead the copy to the edge) or `DEVICE LOST` (then it cannot). **Not run:** the test ended in (d). It goes to the edge on purpose, five times, and after (d) that is a risk for the machine that needs its owner's word first. **(f) the two-pass search as it is now** (one copy; the second pass reads again from four steps below the highest step that passed; see `docs/development.md`): tested on a simulated card only. On the reference card, with the edge at +2450 in its journal and no other program on the card: `gao --optimize best` once; expected `other programs on the card: none`, `mem: the memory gave out at +2450 in an earlier run on this card and driver; the scan stops at +2250`, no memory candidate above +2250, `mem: the scan passed +2250; reading again from +2200 down`, a few `mem +N: bandwidth ... GB/s, read again` lines, no `driver resets during this run`, an applied offset above +1000 and a soak that is not slower than stock. **As expected** (2026-10-11 23:34 to 23:51, elevated, a test build of e5433ee, the editor and the videos closed). `other programs on the card: none`; `stock: STABLE score=5741 it/s clock=2790 MHz peak=67 C fan=72 % power=196 W`, 8 % above every stock measurement of that day, which were all taken next to the editor; power 105 %. `mem: the memory gave out at +2450 in an earlier run on this card and driver; the scan stops at +2250`: 46 steps in 2 min 41 s, from `mem +0: bandwidth 452.9 GB/s` to `mem +2250: bandwidth 533.3 GB/s`, each read once. `mem: the scan passed +2250; reading again from +2200 down`: 533.8, 528.4, 530.7 and 527.8 GB/s at +2200 to +2050; `confirm core +0 / mem +2100: STABLE`; `mem: bandwidth peak +2100, confirmed +2100, applying +1450`. `core: highest stable +210, confirmed +210, applying +135` after `WRONG RESULT` at +225, as in every run. Soak `STABLE score=5918 it/s peak=70 C power=200 W`: 3.1 % faster than stock. No reset event, no `nvlddmkm` event in the System log, nothing new in the journal, sixteen and a half minutes in all. The second look after the run found the card as before it. The saved tune of before the test was put back and applied (core +135, memory +650), the tray app restarted, Defender reported nothing. Not tested by this: a first scan on a card whose edge is not in the journal yet. **(g) what else uses the card** (2026-10-11 22:00, after the restart of (d), not elevated, the saved tune applied, nothing changed): `gao --stress 5` scored 5828 it/s and `gao --bandwidth --after-load 3` read 484.0 GB/s with its three runs within 0.0 %, against 5226 it/s and 444 to 452 GB/s with the same tune earlier that evening. The series of (a) again, 24 readings each: 64 dispatches 480.0 GB/s, 0.23 % (-0.66 to +0.18); 256 dispatches 475.2 GB/s, 0.68 %, sliding from 480 to 471 during the series; 64 dispatches again 454.0 GB/s, 3.02 % (-4.37 to +4.22), the readings jumping between 434 and 473. Read right after it, the card's use per program: Chrome 8.6 % of the video decoder (19 % at most) and 2.8 % of the 3D engine, with a YouTube video playing; nothing else above 1.1 %. So the short copy is as steady as the long one on a card that is left alone, and a video playing next to it costs about 8 % of the bandwidth and of the stress score and makes the readings jump. Then 40 readings with the short copy, and next to each the other programs on the card in the seconds before it (22:13): 464 to 476 GB/s while Chrome used the video decoder, 444 to 455 while it used the 3D engine as well, and from the reading after Chrome stopped, 477 to 483.5 for the last eleven readings with no other program on the card. So another program using the card lowers the reading by 3 to 8 % from one reading to the next. That explains the scatter and the level changes of (a) in kind. During the tests of that day it was most likely not a video (hardly any was watched) but a code editor built on a browser engine that had stood open since the morning: its last files were written at 21:31:43 and 21:31:44, when it was closed, and the run of (d) logs the bandwidth going from 430.4 to 457.1 GB/s at 21:31:45. So every card measurement of checks 94 and 95 up to then was taken with that program on the card: about 7 % lower, and more scattered, than the card alone. A match to the second; not measured with the editor started again. | | 96 | Other programs on the card are found and named before a run | (a) Not elevated, nothing changed on the card, the saved tune applied: `gao --stress 5` and bandwidth readings with nothing else on the card, with an editor (Trae) standing open, and with a video playing in Chrome; next to every reading the card's use per program. (b) Not elevated: `gao --busy` with the editor open and with it closed. (c) The window's question, from a look build that shows it without a run. (d) Elevated, with the editor open: `gao --retest`, then `gao --retest --busy-ok`; and **Optimize GPU** in the window, each of the three answers | (a) to (c) no, (d) yes | (a) how much another program costs. (b) `other programs on the card: Trae (2 %)`, and `none`. (c) the list, the explanation and the three buttons, nothing cut off. (d) the first prints the list and the advice and exits 1 with nothing changed on the card; the second logs the list and `going on with them` and runs; in the window **Check again** with the editor closed starts the run, **Go on anyway** starts it and logs the list, **Cancel** ends in **Not started** with the list | **(a) measured** (2026-10-11, reference RTX 4070, driver 617.42, saved tune: power 105 %, core +135, memory +650). Nothing else on the card: 5828 it/s, bandwidth 480 GB/s over 24 readings scattering by 0.23 %, and 477 to 483.5 in a later series. Trae open and idle, 2.2 % of the 3D engine: 5329 it/s, 452 to 457 GB/s over 28 readings, then two at 441 and 444. A YouTube video in Chrome, 8.6 % of the video decoder and 2.8 % of the 3D engine while the card idles: 464 to 476 GB/s while it decodes, 444 to 455 while it draws as well, 24 readings scattering by 3.0 %; back to 483 on the first reading after it stopped. Under the run's own load the other program's share reads 1 % or less, so it can only be seen while the card idles. One sample of the counters read 183 455 208 183 776 %. That day's card tests had all been taken with the editor open, from the morning until 21:31:44, when its last files were written; the log of the run under way shows the bandwidth going from 430.4 to 457.1 GB/s at 21:31:45. **(b) as expected with the editor open**: `other programs on the card: Trae (2 %)`, twice. Not run with it closed. **(c) as expected**: the heading, `chrome (12 %, video), Trae (2 %).`, the explanation in two lines, **Check again**, **Go on anyway** and **Cancel**, at 1180 x 900. **(d) the command line's refusal: as expected** (2026-10-11 22:46, elevated, a test build of ff80e3d, the editor and the chat application this was written in open). `gao --busy`: `other programs on the card: Trae (1 %)`. `gao --retest`: `other programs on the card: Trae (2 %), msedgewebview2 (1 %)`, the four lines of advice, `RESULT: not applied -- other programs are using the graphics card: Trae (2 %), msedgewebview2 (1 %)`, exit 1, four seconds after it began; the profile and what is applied read the same before and after, and `run.log` has the line. Two things seen: the editor read 1 % in one look and 2 % in the next, so the threshold went from 1 % to 0.5 % after this run; and `msedgewebview2` is the browser engine inside another application, a name that tells its user nothing. **Not run:** `--busy-ok`, and the three answers in the window, on a real run. | ## Notes