diff --git a/README.md b/README.md index 9ca4284..457e724 100644 --- a/README.md +++ b/README.md @@ -176,7 +176,7 @@ flowchart LR - **Every run starts with the card at stock for five minutes,** as long as the result is tested at the end, with the fans left to the NVIDIA driver. That is the "before" of the result: a card is not heated through after 30 s, so a shorter run would make every result look warmer and louder than stock. Some cards get hot there: the driver's own fan control lets them (a 3080 Ti reached 85 °C with its fans at 65 %). The measurement stops at 85 °C, what it measured until then is the "before", and the run goes on. From there every step keeps the fans on a curve that reaches 100 % at the profile's temperature limit. - **Stress test.** A DirectX 11 compute load in which the GPU checks every value it computes against a known answer. Each probe ends in a verdict: `STABLE`, `WRONG RESULT`, `DEVICE LOST`, `STALLED` (the values were right but the card computed almost nothing), `TOO HOT` or `NO TELEMETRY`. -- **Memory comes first and stops at the bandwidth peak,** not at the first error. GDDR6 and GDDR6X retry failed transfers, so an overclocked memory bus loses speed long before it returns a wrong result. The search raises the offset 50 MHz at a time, measures bandwidth at each step and keeps the lowest offset within 1 % of the best. Where bandwidth cannot be measured, the memory is left at stock: without that measurement nothing says where to stop, and the first wrong result comes far too late. +- **Memory comes first and stops at the bandwidth peak,** not at the first error. GDDR6 and GDDR6X retry failed transfers, so an overclocked memory bus loses speed long before it returns a wrong result. The search raises the offset 50 MHz at a time, measures bandwidth at each step until a step fails or has clearly lost bandwidth, then measures again from 200 MHz below that point down and keeps the lowest offset within 1 % of the best. Where the memory gave out is remembered for that card and driver, so a later run stays below it without going there again. Where bandwidth cannot be measured, the memory is left at stock: without that measurement nothing says where to stop, and the first wrong result comes far too late. - **The core search climbs from stock.** With the chosen memory offset applied, it raises the core offset 15 MHz at a time until a probe fails, and keeps the last value that passed. How far it may go comes from the range the driver reports for your card, so a card that can hold more is not stopped at a fixed limit. - **Each edge is confirmed.** The value a search ends on gets a 30 s probe, and a lower value is tried if it does not hold. The safety margin is applied to the confirmed value, and the result as a whole must pass the 300 s soak. - **A driver reset ends that clock's search.** On a card the search can push past its edge, a search resets the driver: the screen goes black for a moment. The run then reconnects to the driver, sets the card to stock, leaves it without load for 20 seconds and checks with a short probe at stock that it computes normally again. After that the search tries no new values for that clock: it keeps the last value that passed and confirms four steps below it. Each clock has its own search, so a reset in the memory search does not keep the core from being searched; a run can therefore reset the driver twice, once per clock. One reset more than that ends the run at stock with nothing saved, and so does a card that does not come back. Results on real cards are in [docs/hardware-checks.md](docs/hardware-checks.md), checks 51 and 52. diff --git a/docs/development.md b/docs/development.md index beccb8e..2042e69 100644 --- a/docs/development.md +++ b/docs/development.md @@ -79,7 +79,8 @@ The applied offset is always at least one step below the confirmed edge. Profile | Without the bandwidth measurement the memory is left at stock | The search used to fall back to a climb on stability alone when the measurement failed at +0. On a tester's 3080 Ti (0.4.3) that climb went to +2300 MHz with the score falling from +2150 on, got its first wrong result there, and the machine froze with a corrupted screen in the 30 s confirm probes just below it. The first wrong result is no edge to stand next to: the memory was correcting errors for hundreds of MHz before it. Now a measurement that does not work means no memory overclock (`left_at_stock`), the run says so, and the core is still searched. `optimize` keeps the stability climb only for a caller that gives no measurement at all; the app always gives one. Why the measurement failed on that card is not known: the log was only in the window and went with the reboot. | | A confirm probe that is too hot at once confirms nothing | A confirm probe that ends `TOO HOT` keeps the clock edge: over 30 s the card may run a little hotter than over the power step's 20 s, and that is the soak's business, which lowers power. But the stress run stops at its first reading above the limit, so a card that is above it when the probe begins gives `TOO HOT` within a second, and that counted as holding too. It happened on the tester's RTX 3080 Ti (hardware check 91): after a driver reset the health probe loads the card at stock power, which on that card means 76 C against a limit of 75, and `confirm core +225` was confirmed by a probe that had tested nothing. Now such a probe counts only when the clock was loaded for `kConfirmMinS` (10 s, clearly longer than the 3 s probe that found the edge) before it got too hot. Otherwise the value is not confirmed, and the card rests (`io.rest`, the 20 s of a recovery, with the fans where the curve last put them) before the next try one step lower: without the rest every lower value would start just as hot, and heat alone would step the edge down to nothing, which is what keeping the edge on `TOO HOT` was for. The same value is not tried again; one step is the price. Not changed: the health probe still runs at stock power, and a soak that begins too hot still lowers the power limit by a step. | | A profile that promises speed keeps no result that is slower than stock | Best of my GPU ended 3.8 % slower than stock on the tester's RTX 3080 Ti (4749 against 4937 it/s, hardware check 91), with the fans at 80 % instead of 66 %, and showed it as **Done**, applied and kept at every logon. Every step had done what it is for: the card runs 83 C at stock with its fans at 100 %, the profile allows 75, so the power limit went to 65 %, and +150 MHz of core clock did not win that back. But a profile called "the best of my GPU" that leaves the card slower has not done what it says, and a result like that is not offered as one. After the soak, `optimize` compares its score with the measurement at stock that went before the run (the "before" the result screen shows, so the rule judges by the number the user sees): below `kNotSlowerScore` (0.99: one percent is what two five-minute runs differ by) the run ends as a failed one does, at stock, with nothing saved, and says what it found, why, and that Undervolt is the profile for a card that runs warm. `Objectives::faster_than_stock` marks the two profiles this holds for, Best of my GPU and Max performance. Not Quiet and Cool & efficient, which trade speed for noise and heat and say so. Not the overclock step of All in one either: it goes on to an undervolt and a fan tune, and its result is judged as a whole; whether that whole may be slower than stock is not decided. This is a backstop, not the cure: what such a card needs from these profiles is in `docs/planned.md` ("never hotter than at stock"). | -| Open: on the reference card the memory offset is decided by noise | Three runs on one day applied +650, +1600 and +550 (hardware check 94). The memory search assumes that bandwidth peaks before the memory makes errors (`kBandwidthDrop`: stop at a reading 1 % below the best, re-measured once). On the reference RTX 4070 it does not: bandwidth rises by about 0.4 % per 50 MHz step all the way to +2500, and the first wrong result comes at +2550. The readings scatter by 0.4 to 0.7 % (standard deviation), so two readings in a row 1 % under a high one happen by chance, and where that happens is where the scan ends: at +1000 twice, not at all once. When it does not happen the search stands next to the first wrong result after all, which is what the bandwidth method exists to avoid; the tie rule and the margin still put the applied offset 950 MHz below it. Nothing is changed yet. What a fix has to decide: whether a card like this should get a memory overclock by another rule (a fixed share of the range the scan covered, or of the offset of the first error), and how a real peak is told from scatter on the cards that have one (several readings per step and their median, or a drop that holds over more than one step). | +| The memory search has two passes, and remembers where a card's memory gave out | **What went wrong.** Four runs on one day applied +650, +1600, +550 and +100 MHz on the reference RTX 4070, a card that runs every game at +1500 (hardware check 94). That card has no bandwidth peak: bandwidth rises by about 0.4 % per 50 MHz step all the way to +2500 and the stress load computes its first wrong result at +2550. So the scan could only end early by chance, and where it did decided the offset. **The cause, found late.** The chance came from another program on the card: an editor stood open all that day and took 2.2 % of the 3D engine, and with it the bandwidth readings scattered by 0.4 to 1.3 % and changed level by 1.8 %. On a card that is left alone they scatter by 0.2 % (hardware check 96). A run now looks for such programs before it begins and asks (the row on other programs on the card). What is left for the search to do is to give a sound result also when the user goes on next to them, and not to go to the memory's edge more often than it has to. **A detour, taken back.** The first answer to the scatter was a timed run four times as long (128 GB copied in one go). It was steadier next to the editor, and no steadier than the short one on a quiet card. With it every scan ran to where the memory gives out and met it with that copy: five scans out of five lost the device there, at +2450 or +2500, and the fifth took the whole machine down within seconds (check 95 b to d). In the one scan before that got to the edge, the short copy read a number at +2450 and +2500 and the stress load reported the wrong result at +2550, with no reset. The long run is gone; `kBwDispatches` is 64 as it was. **The design.** *The scan* goes up and only has to find where it ends: a step that is not stable, or one more than `kBandwidthDrop` (5 %) below the best. Two things keep a single reading from deciding: a reading that would end the scan is taken `kBandwidthReadings` (3) times and the middle one counts, and the best that readings are held against is the highest middle of three steps in a row, not the highest reading (the highest of fifty scattering readings is always one that came out high). 5 % is far more than readings move by themselves, also next to another program; the scan is not where the offset is chosen any more, so it can afford to be coarse. *The second pass* chooses the offset: it starts `kPeakMarginSteps` (4 steps, 200 MHz) below the highest step the scan passed (`second_pass_top`), reads again a step at a time going down until a reading is more than `kBandwidthTie` (1 %) below the best of them, and keeps the lowest offset within that 1 % (`bandwidth_peak_below`). On the reference card: the scan passes +2500, the second pass reads +2300, +2250, +2200 and +2150, and +2200 is kept; confirmed there, 70 % applied, +1500. Whatever the readings do, the offset comes out of a few steps below a point the card itself sets, so it repeats: +2050 to +2300 at the scatter of a quiet card. The margin is kept whatever ended the scan, a drop and the driver's own range included: what is above was never tried. After a reset event in the scan the second pass is skipped, and confirming starts four steps lower still. **The edge is met once.** A scan that ended in a wrong result or a reset event writes that offset to the journal with the GPU id and the driver version (`{"mem_edge":2550,"rig":" / "}`, a line without an id, so older builds skip it). A later scan on the same card and driver stops four steps below the lowest such offset, and says so; its second pass starts where the first run's did, because the margin counts from the edge and not a second time from where the scan was held. With another driver or another card the line does not count. A scan that ended for another reason (a bandwidth peak, readings that did not settle, the driver's range) writes nothing. **What it costs.** A card that does have a peak keeps an offset up to 200 MHz below it when the scan ends close above the peak: the margin does not know a drop from an edge. A scan that ends for a harmless reason (readings that did not settle) loses those 200 MHz as well. The second pass adds about four steps, a quarter of a minute. **Not chosen.** The long copy, anywhere: see above. Three readings at every step: two minutes a scan, and it decided nothing a single reading did not. Only remembering the edge: the result would still be chance for whoever goes on next to another program. The stress load probing four steps ahead of the copy: there is no heavy copy left to protect the edge from. A fixed top for the scan (an offset, a share of the range, a gain over +0): there is nothing to choose the number by. **Tested** on a simulated card with the reference card's numbers (wrong result at +2550): the second pass never reads above +2300, in the first run and in the one after it; with readings scattering by 1.3 % in both passes and level changes thrown in; with the edge anywhere from +300 to +2900; and the tests fail when the margin is set to nothing. **On the card:** one run, with the edge already in the journal (hardware check 95 f): as designed. See the row below for what that leaves. | +| Open: the first meeting with a memory edge has not been seen with the two-pass search | The search as it is now ran once on the reference card (hardware check 95 f), on a quiet card and with the edge at +2450 in its journal: the scan stopped at +2250, the second pass read +2200 down to +2050 and kept +2100, +1450 was applied, there was no reset, and the result was 3.1 % faster than stock. That is the run after the one that finds the edge. The run that finds it has not been seen: on a card whose journal has no edge the scan goes up until the stress load computes wrong or something gives, with the same copy the released versions use; that ended in a wrong result the one time it was seen before (check 94), and seeing it again means removing the edge from a journal and going there. **Still not known.** Whether a card with a real bandwidth peak behaves as assumed; none has been measured through a scan (the tester's RTX 3080 Ti never got a reading to settle at +0). And why the long copy failed at the edge where the short one did not: the amount copied in one go, or only that five runs are more than one. It no longer matters for the search, and is noted because it was never explained. | | Open: why the bandwidth measurement fails on some cards, and what then | On the tester's RTX 3080 Ti it failed at +0, and since 0.4.4 that means no memory overclock at all on such a card, though it computed right far beyond +2000 MHz. The measurement gives a number only when three runs in a row agree within 1 % (`kBwSettle`), and it throws away every run the driver's timer calls disjoint (the clock changed under it). That card was held at a power limit of 65 % and ran at 83 C when it was measured; the reference card has only ever been measured at 105 %, where the measurement itself draws 179 of 210 W. So the suspicion is the power limit, and hardware check 90 tests exactly that on the reference card. What it shows decides what is built, and nothing is built before it: **(1) runs are thrown away as disjoint, and the ones that are kept agree:** the timer is what fails, not the card. Time the copy differently (wall clock around a flush, or count a disjoint run when its neighbours agree) and keep the method. **(2) the readings themselves scatter by more than 1 %:** a peak that is found by a 1 % drop (`kBandwidthDrop`) cannot be found in that noise. Then the memory has to be measured where the card is not held back: search it before the power step, at stock power, where a probe and its measurement last five seconds; or take more readings and compare medians. **(3) it does not fail at 65 % on the reference card:** the power limit is not the cause. Then it was the heat (a card throttling at its temperature limit changes clock as well), or another program, and only the tester's `run.log` can say: since this change a measurement that never settled writes how many runs gave a reading, between which values, and how many were thrown away. In no case does the climb on stability alone come back. **Outcome of check 90: (3).** On the reference card the measurement gave a number fifteen times out of fifteen at 100, 80 and 65 %, in three or four runs, with nothing thrown away, and the bandwidth was the same at every limit. The power limit is ruled out; (1) and (2) are not built. What is left is the heat of that card, another program, or a timer query that failed, and the tester's `run.log` is the next thing that can tell them apart. **The tester's run (check 91): it is (2), but only inside a run.** Twenty readings between 763.9 and 821.2 GB/s, none thrown away, three seconds after the load began; after thirty seconds of load the same card read within 0.2 %. The suspicion now is the moment, not the card: the measurement is taken while the power limiter is still catching up. If `gao --bandwidth --after-load 3` scatters on that card and `--after-load 30` does not, that is it, and the fix is to measure when the card has settled (keep taking runs for some seconds instead of twenty runs), not to change the method. | | Every line of a run is also in `run.log` | `logged()` wraps the log hook of both front ends: a line is appended and flushed before it is shown, so the step a machine freezes in is the last line on disk. One file in the state folder, moved aside at 1 MB (`run.log.old`). Only elevated and only after `ensure_app_dir` has verified the folder: an elevated process appends to nothing a user could have put there. | | A run does not start while the NVIDIA app's Automatic Tuning is on | Switched on, a service of the driver scans the card when it sees fit: it loads it, changes its clocks, and starts again by itself (hardware check 89: a scan that was stopped started again 14 ms later, and after a restart of the service it resumed within two seconds). A run next to that measures nothing it can trust, so it is refused, like a run next to Afterburner, and with the same weight: a courtesy, not a guard. Read from where that service keeps the switch, `%ProgramData%\NVIDIA Corporation\nvtopps\nvtopps.db3`, table `ocScannerSettings`, fields `backgroundScanEnabled` and `grdOcEnabled`, joined to `TelemetryGpuCfg` so that only a card that is in the PC counts (a row of an earlier card can never be switched off again). That this is the source and not a copy of it was measured, not assumed: with the field set the service took the scan up again after every restart, with it cleared it did not, while the NVIDIA app itself showed the switch as off. Nothing documents the file. So: opened read-only and immutable (no lock, the service never waits for us), through `winsqlite3.dll` from System32 (no dependency added), and everything that is not exactly as expected reads as "unknown", which starts the run as before. The registry value the app sets next to it (`HKLM\SOFTWARE\NVIDIA Corporation\Global\NvApp\OC\`) changes at the same moment but belongs to the app, not to the service, and is not used. **The overlay is not checked:** no setting for it was found, its processes run whether it is open or not, and the one value that moved (`IsShadowPlayEnabled`) followed the service being up, not a switch. NVAPI was not tried: an undocumented call is no better than an undocumented file, and it cannot be compared with anything. | diff --git a/docs/hardware-checks.md b/docs/hardware-checks.md index 2c3ffe1..e0c814b 100644 --- a/docs/hardware-checks.md +++ b/docs/hardware-checks.md @@ -122,6 +122,7 @@ root; adjust the path for a Debug build. | 92 | The undervolt profile starts with the measurement at stock every run uses, and a hot card is not refused | Elevated, the tray app exited, a copy of `gao.json` kept. (a) `gao --undervolt` to the end; then `gao --status`. (b) The same with a test build whose stock limit is lowered to a temperature the reference card passes under load (63 C; it reaches 67). Put the copy of `gao.json` back. (c) Only on a card that is above the profile's limit at stock with the fans at 100 % (the tester's RTX 3080 Ti: 82 to 83 C): the **Undervolt** profile from the window, to the end, and its `run.log` | yes | (a) `the card at stock, five minutes`, a `stock: STABLE ...` line with the driver's fan speed, then `baseline: the measurement at stock`, and no second five minutes at stock: the first candidate follows at once; no `candidates are stopped at` line; `RESULT:`; `before:` shows 300 s and the fan speed the driver chose. (b) `stock: TOO HOT ... peak=64 C`, the line that the measurement stopped there, the cool-down, then `baseline: the measurement at stock` and a search that runs to `RESULT:` with `before:` showing the shortened time; no `candidates are stopped at` line, because 64 C is under the profile's 75. (c) no `not stable at stock`; `the card reached 86 C at stock: candidates are stopped at 85 C, not at the profile's 75 C`; candidates that end `STABLE` or `WRONG RESULT`, none `TOO HOT`; a result that is cooler and draws less than stock at the same clock | First run: **not as expected**, and that found a fault (2026-10-11, reference RTX 4070, driver 617.42, a test build of d1c99e9, started elevated by a script). The first halves held: (a) `stock: STABLE ... peak=65 C fan=70 %`, then `baseline: the measurement at stock` four seconds later and no second five minutes; (b) `stock: TOO HOT ... peak=64 C`, stopped after 34 s, the run went on, no `candidates are stopped at` line. But both runs then ended with `the card held no undervolt: the first step below its own voltage already failed`: their first candidates read `1070 mV (+15 MHz): SLOW ... clock=2323 MHz` and `1050 mV (+15 MHz): SLOW ... clock=2411 MHz`, on a card that ran 2791 MHz in the measurement just before. The search's load had not run yet when that candidate began (see the decision "With its reference handed over, the undervolt search warms its load up before the first candidate"). Second run, with the warm-up: **pass** for (a) and (b) (same day, a test build of 0733695). (a) `stock: STABLE score=5426 it/s clock=2788 MHz peak=66 C fan=70 %`, four seconds later `baseline: the measurement at stock`, `warming the load up: 5 s at stock`, `target: 2790 MHz, which the card reaches at 1090 mV by itself`, and the first candidate `1085 mV (+15 MHz): STABLE ... clock=2775 MHz`; 26 candidates, none `SLOW` or `TOO HOT`, the edge at 950 mV, `RESULT: 2790 MHz at 990 mV instead of 1090 mV`; `before:` 5426 it/s, 189 W, fan 70 % (300 s), `after:` 5325 it/s, 149 W, fan 53 % (300 s). (b) `stock: TOO HOT ... peak=64 C`, stopped after 74 s, the run went on, the same warm-up and a search to the same result (990 mV, edge 950 mV); `before:` shows 75 s; no `candidates are stopped at` line. The saved tune of before the test was put back and applied. (c) not run. | | 93 | A speed profile that would end slower than stock keeps nothing | Only on a card that is warmer at stock than the profile allows (the tester's RTX 3080 Ti; the reference card ends faster than stock with every speed profile): **Best of my GPU** from the window, to the end; the result screen, the dashboard and `run.log` | yes | the run goes through all its steps and its soak; then **Not applied** with `the result is slower than stock: N it/s against M (-X %)`, the power limit it had to go down to, and the pointer to Undervolt; `The card is back at stock.`; the dashboard shows stock and the saved tune of before the run, if there was one; `run.log` ends with `stopped: the result is slower than stock ... -- card restored to stock` | Not run yet. On the reference card: nothing to see, by design. | | 94 | All in one, with the undervolt step searching for the first time | Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize all` to the end with a build that has the warm-up of the undervolt step; then `gao --status`; put the copy back | yes | five steps; in step 3 a `warming the load up` line, a first candidate that is not `SLOW`, and a descent; steps 4 and 5 run on the tune of steps 2 and 3; `RESULT:` | **Not as expected at step 4**, and that found a fault (2026-10-11, reference RTX 4070, driver 617.42, a test build of 0733695, unattended). Step 1: `stock: STABLE score=5325 it/s peak=65 C fan=70 % power=186 W`. Step 2: power 105 %, `mem: bandwidth peak +2300, confirmed +2300, applying +1600`, `core: highest stable +210, confirmed +210, applying +135`, soak `STABLE score=5568 it/s peak=70 C power=197 W`. Step 3, as wanted: `warming the load up: 5 s with the overclock`, `target: 2865 MHz, which the card reaches at 1050 mV with the overclock`, nine candidates `STABLE` from 1045 mV down, `WRONG RESULT` at 990 mV, confirmed at 1010 mV after two tries, and `applying 1050 mV (+135 MHz at that point)`: the margin left no raise above the overclock, so the curve was cut off, as in the runs before the warm-up; soak `STABLE score=5434 it/s peak=67 C power=176 W`. Step 4 did not begin: `the tune could not be applied for the fan tune: the undervolt was not applied: the point already runs more than the target -- card at stock`. The saved tune of before the test was put back and applied. Cause and fix: the decision "A flat top goes back on a card that is cooler than when it was found". Second run, with the `gao.exe` of the released v0.4.6: **pass, all five steps** (same day, started elevated by a script). Step 1: `stock: STABLE score=5309 it/s peak=66 C fan=70 % power=185 W`. Step 2: power 105 %, `mem: bandwidth peak +850, confirmed +850, applying +550`, `core: highest stable +210, confirmed +210, applying +135`, soak `STABLE score=5568 it/s`. Step 3: `target: 2850 MHz, which the card reaches at 1045 mV with the overclock`, `undervolt: lowest stable 975 mV, confirmed 985 mV, applying 1025 mV (+165 MHz at that point)`, soak `STABLE score=5363 it/s peak=66 C power=163 W`. Step 4: the tune went back on the card, `fan 60 %: 63 C`, `fan 55 %: 65 C`, `fan 52 %: 67 C`, `fan tune: 52 % holds 67 C`. Step 5: the final test ran its 300 s. `RESULT: power 105 %, core +135 MHz, mem +550 MHz, 2850 MHz at 1025 mV, fans 52 % at 67 C`; `stock: 5309 it/s, 185 W, fan 70 %, 66 C`, `now: 5353 it/s, 166 W, fan 52 %, 70 C`: `+0.8 % score and -10.3 % power against stock`; `undervolt: applied`. The saved tune of before the test was put back. **What this run did not show:** its undervolt had a raise of its own (+165 over an offset of +135), so the anchor was raised and not merely cut off, and the case of the first run (an anchor that reads a step above the target on a cooler card) did not come up. That fix has still only its tests. **The memory offset differs a lot from run to run on this card, and the two logs say why:** +1600 in the first run (bandwidth peak +2300), +550 in the second (peak +850), +650 in the run that saved the tune in use that morning (the journal shows its scan ending at +1000 too). The card was in the same state each time (68 C, power 105 %, a reading every three to four seconds). In both logged scans the bandwidth rises in a straight line, 0.36 and 0.46 % per 50 MHz step, from 420 GB/s at +0 to 509 GB/s at +2450; the first run went on to `WRONG RESULT` at +2550 without the bandwidth ever turning down. The readings scatter around that line with a standard deviation of 0.4 and 0.7 %, single ones by more than 1 %. The scan ends at a reading more than 1 % below the best so far when a second reading agrees, and in the second run that was chance: +950 read high (462.8 GB/s, where the line is at about 458), +1000 read 456.6 twice, 1.3 % below it. In the first run the one low reading (+600) was followed by a higher one, and the scan ran to the first wrong result. So on this card there is no bandwidth peak before the errors, and where the scan stops is decided by noise. | +| 95 | The memory scan no longer ends by chance | (a) Not elevated, nothing changed on the card: a measuring build takes 24 bandwidth readings in one process, with 3 s of the stress load before each as a scan has, at 64, 128 and 256 dispatches per timed run; then the same with the app's window open next to it, drawing a frame every 100 ms and every 33 ms. (b) Elevated, the tray app exited, a copy of `gao.json` kept: `gao --optimize best` twice in a row with the build of the change, then the copy put back | (a) no, (b) yes | (a) how much a reading scatters, and on what that depends. (b) in both runs every `mem +N: bandwidth` line shows three readings that agree within about 1 %; the scan does not end before the first `WRONG RESULT`; the two `mem: bandwidth peak` values are within 300 MHz of each other and the applied offset is above +1000; the soak is not slower than at stock | **(a) measured** (2026-10-11, reference RTX 4070, driver 617.42, the saved tune applied: power 105 %, core +135, memory +650; standard deviation over the 24 readings of a series, and the lowest and highest against the mean). 64 dispatches, no window: 0.44 % (-0.55 to +1.07, one reading did not settle), 0.66 % (-1.58 to +1.49, one did not settle), 1.31 % (-4.83 to +1.27), 0.86 % (-1.44 to +2.05). 128 dispatches, no window: 0.43 % (-1.10 to +0.79). 256 dispatches, no window: 0.22 % (-0.43 to +0.42) and 0.60 % (-0.80 to +1.37; the last five readings of this series were 451 to 453 GB/s after nineteen of 444 to 449). With the window drawing every 100 ms: 64 dispatches 1.13 % (the first six readings 452 to 453 GB/s, the rest 439 to 451), 256 dispatches 0.24 % (-0.47 to +0.61). With the window drawing every 33 ms: 64 dispatches 0.58 %, 256 dispatches 0.25 % (-0.35 to +0.51). A reading took 6.5, 5.3 and 3.9 timed runs to settle at 64, 128 and 256 dispatches. So: the longer timed run takes the scatter from 0.4 to 1.3 % down to 0.2 %, the window's drawing adds nothing, and apart from the scatter the readings sit about 1.8 % higher for up to a minute at a time (the stretch at the end of the second 256 series ran on into the next series; the last 64 series went back and forth between 444 and 452 GB/s). What sets that level is not known. **(b) the offset is the same twice; the scan ends in a driver reset twice** (2026-10-11, same card and driver, a test build of 2764609, unattended, 40 minutes for the two runs). Run 1: `stock: STABLE score=5311 it/s peak=66 C fan=70 % power=185 W`; 49 steps from `mem +0: bandwidth 422.7 GB/s (422.7 422.6 422.7)` to `mem +2400: bandwidth 501.4 GB/s (501.3 502.3 501.4)`; at +2450 the candidate was `STABLE` and the measurement after it lost the device: `mem +2450: the bandwidth measurement failed; treated as a driver reset`, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, 20 s of rest, `health: STABLE`, `confirm core +0 / mem +2050: STABLE`, `mem: bandwidth peak +2250, confirmed +2050, applying +1400`; `core: highest stable +210, confirmed +210, applying +135`; soak `STABLE score=5536 it/s peak=69 C power=195 W`; `driver resets during this run: 1`. Run 2: stock 5312 it/s; 50 steps from 421.8 GB/s at +0 to 502.8 at +2450; the same failure at +2500; `mem: bandwidth peak +2300, confirmed +2100, applying +1450`; core +135; soak `STABLE score=5555 it/s peak=69 C power=196 W`; one driver reset. The three readings of a step differed by 0.4 % on average and by 1.2 % at most; no step was more than 0.7 % below the best before it, so neither scan came near ending early, by the old rule or the new; every step had its three readings. The scan took 5 min 25 s (about 6.5 s a step; about 3 min before the change). As expected: the two peaks are 50 MHz apart, the applied offsets are +1400 and +1450, and both results are 4 % faster than stock. **Not as expected: neither scan reached a `WRONG RESULT`.** Both ended one and two steps before +2550 with the device lost during the bandwidth copy; Windows logged it both times (System log, `nvlddmkm` events 14 and 153 at 19:51:22 and 20:11:27), and the screen goes black for a moment. In the run of check 94 that got this far, the measurements at +2450 and +2500 gave a number and the stress load reported the first wrong result at +2550, without a reset. A step now copies about seven times as much as it did (three readings of about four runs of 128 GB, against one reading of about six runs of 32 GB), and the copy checks nothing, so where the memory starts to fail it is the driver that gives out, not a verdict. Two runs against one: likely the cause, not proven. The search handles it as it should (rest, health probe, confirm four steps lower, the core is still searched), but a driver reset in every run on a card like this is not what the change was for. The saved tune of before the test was put back and applied (`applied best: power 105 %, core +135 MHz, mem +650 MHz`), the tray app restarted, Defender reported nothing. **(c) the same two runs with one reading a step, three only for a step in doubt** (2026-10-11 20:26 to 21:03, a test build of 71e7c2b, unattended): **the reset is still there.** Run 1: stock 5328 it/s; 50 steps from `mem +0: bandwidth 420.4 GB/s` to `mem +2450: bandwidth 502.4 GB/s`; at +2500 the candidate was `STABLE` and then `bandwidth measurement failed: the timing of the copy did not come back` ten seconds later (`nvlddmkm` event 14 at 20:37:07); `mem: bandwidth peak +2200, confirmed +2000, applying +1400`; core +135; soak `STABLE score=5572 it/s peak=70 C power=197 W`; one driver reset. Run 2: stock 5311 it/s; 421.9 GB/s at +0 to 508.0 at +2450; at +2500 `the device was lost during the copy failed (hr=0x887A0006)`, the device hung (event 14 at 20:55:29); `mem: bandwidth peak +2400, confirmed +2200, applying +1500`; core +135; soak `STABLE score=5571 it/s peak=70 C power=196 W`; one driver reset. No step was in doubt in either run (the lowest was 0.75 % below the best before it), so every step was read once; the scan took 3 min 33 s. The two peaks are 200 MHz apart: with one reading a step, which offsets count as within 1 % of the best moves by a few steps. Applied +1400 and +1500, both 5 % faster than stock. So the reset does not come from reading a step three times. Four runs out of four with the longer timed run ended this way, at +2450 or +2500. Not merged; the design for a scan that does not end like this is the open row in `docs/development.md`. The saved tune was put back and applied again (core +135, memory +650), the tray app restarted, Defender reported nothing. **(d) the edge is remembered** (a test build of 7c215c4; the same two runs): expected is that the first scan ends as before and logs nothing new, the journal gets one `mem_edge` line with the GPU id and the driver version, and the second run logs `mem: the memory gave out at +N in an earlier run on this card and driver; the scan stops at +(N-200)`, has no `driver resets during this run` line, applies more than +1000 and is not slower than stock. **Run 1 took the machine down; the test ended there** (2026-10-11 21:23, unattended). Stock 5321 it/s, power 105 %, then the scan: 49 steps from +0 to `mem +2400: bandwidth 539.3 GB/s`, two of them read three times (+1200 and +1250). At +2450 the candidate was `STABLE` and the copy after it failed as in the four runs before: `bandwidth measurement failed: the copy failed (hr=0x887A0007): the device was lost` at 21:35:05, `mem: the driver reset at +2450; using +2250, the best of the values that passed`, `the driver was reset; reconnecting`. That is the last line. Windows restarted by itself within fifteen seconds: Kernel-Power event 41 at 21:35:23, `The previous system shutdown at 21:34:57 was unexpected`, no bugcheck, no dump, no `nvlddmkm` event written. So the fifth scan that met the edge with the longer timed run did not end in a driver reset the search recovers from, but in a reset of the whole machine, during the reconnect or right after it. What held: the candidate's journal entry was closed, so no ceiling of a freeze was left; the line `{"mem_edge":2450,"rig":"GPU-... / 617.42"}` was on disk before the machine went down, so no later run of this build goes there again; the run had saved nothing, and at logon the tray app applied the tune of before the test (`applied best: power 105 %, core +135 MHz, mem +650 MHz`). Not answered: the second run. Seen along the way and not explained: from the first step of the scan the stress load scored 5760 it/s and the bandwidth read about 7 % higher than in every run before (486.6 GB/s at +900 against about 455), while the measurement at stock five minutes earlier was the usual 5321. **(e) how the stress load meets the edge** (the same build, no search): `gao --reset`, then `gao --set-mem` and `gao --stress 3` at +2300, +2350, ... until the verdict is not `STABLE`, five times, at stock power and core +0; expected is one of two outcomes, `WRONG RESULT` every time (then the stress load can lead the copy to the edge) or `DEVICE LOST` (then it cannot). **Not run:** the test ended in (d). It goes to the edge on purpose, five times, and after (d) that is a risk for the machine that needs its owner's word first. **(f) the two-pass search as it is now** (one copy; the second pass reads again from four steps below the highest step that passed; see `docs/development.md`): tested on a simulated card only. On the reference card, with the edge at +2450 in its journal and no other program on the card: `gao --optimize best` once; expected `other programs on the card: none`, `mem: the memory gave out at +2450 in an earlier run on this card and driver; the scan stops at +2250`, no memory candidate above +2250, `mem: the scan passed +2250; reading again from +2200 down`, a few `mem +N: bandwidth ... GB/s, read again` lines, no `driver resets during this run`, an applied offset above +1000 and a soak that is not slower than stock. **As expected** (2026-10-11 23:34 to 23:51, elevated, a test build of e5433ee, the editor and the videos closed). `other programs on the card: none`; `stock: STABLE score=5741 it/s clock=2790 MHz peak=67 C fan=72 % power=196 W`, 8 % above every stock measurement of that day, which were all taken next to the editor; power 105 %. `mem: the memory gave out at +2450 in an earlier run on this card and driver; the scan stops at +2250`: 46 steps in 2 min 41 s, from `mem +0: bandwidth 452.9 GB/s` to `mem +2250: bandwidth 533.3 GB/s`, each read once. `mem: the scan passed +2250; reading again from +2200 down`: 533.8, 528.4, 530.7 and 527.8 GB/s at +2200 to +2050; `confirm core +0 / mem +2100: STABLE`; `mem: bandwidth peak +2100, confirmed +2100, applying +1450`. `core: highest stable +210, confirmed +210, applying +135` after `WRONG RESULT` at +225, as in every run. Soak `STABLE score=5918 it/s peak=70 C power=200 W`: 3.1 % faster than stock. No reset event, no `nvlddmkm` event in the System log, nothing new in the journal, sixteen and a half minutes in all. The second look after the run found the card as before it. The saved tune of before the test was put back and applied (core +135, memory +650), the tray app restarted, Defender reported nothing. Not tested by this: a first scan on a card whose edge is not in the journal yet. **(g) what else uses the card** (2026-10-11 22:00, after the restart of (d), not elevated, the saved tune applied, nothing changed): `gao --stress 5` scored 5828 it/s and `gao --bandwidth --after-load 3` read 484.0 GB/s with its three runs within 0.0 %, against 5226 it/s and 444 to 452 GB/s with the same tune earlier that evening. The series of (a) again, 24 readings each: 64 dispatches 480.0 GB/s, 0.23 % (-0.66 to +0.18); 256 dispatches 475.2 GB/s, 0.68 %, sliding from 480 to 471 during the series; 64 dispatches again 454.0 GB/s, 3.02 % (-4.37 to +4.22), the readings jumping between 434 and 473. Read right after it, the card's use per program: Chrome 8.6 % of the video decoder (19 % at most) and 2.8 % of the 3D engine, with a YouTube video playing; nothing else above 1.1 %. So the short copy is as steady as the long one on a card that is left alone, and a video playing next to it costs about 8 % of the bandwidth and of the stress score and makes the readings jump. Then 40 readings with the short copy, and next to each the other programs on the card in the seconds before it (22:13): 464 to 476 GB/s while Chrome used the video decoder, 444 to 455 while it used the 3D engine as well, and from the reading after Chrome stopped, 477 to 483.5 for the last eleven readings with no other program on the card. So another program using the card lowers the reading by 3 to 8 % from one reading to the next. That explains the scatter and the level changes of (a) in kind. During the tests of that day it was most likely not a video (hardly any was watched) but a code editor built on a browser engine that had stood open since the morning: its last files were written at 21:31:43 and 21:31:44, when it was closed, and the run of (d) logs the bandwidth going from 430.4 to 457.1 GB/s at 21:31:45. So every card measurement of checks 94 and 95 up to then was taken with that program on the card: about 7 % lower, and more scattered, than the card alone. A match to the second; not measured with the editor started again. | | 96 | Other programs on the card are found and named before a run | (a) Not elevated, nothing changed on the card, the saved tune applied: `gao --stress 5` and bandwidth readings with nothing else on the card, with an editor (Trae) standing open, and with a video playing in Chrome; next to every reading the card's use per program. (b) Not elevated: `gao --busy` with the editor open and with it closed. (c) The window's question, from a look build that shows it without a run. (d) Elevated, with the editor open: `gao --retest`, then `gao --retest --busy-ok`; and **Optimize GPU** in the window, each of the three answers | (a) to (c) no, (d) yes | (a) how much another program costs. (b) `other programs on the card: Trae (2 %)`, and `none`. (c) the list, the explanation and the three buttons, nothing cut off. (d) the first prints the list and the advice and exits 1 with nothing changed on the card; the second logs the list and `going on with them` and runs; in the window **Check again** with the editor closed starts the run, **Go on anyway** starts it and logs the list, **Cancel** ends in **Not started** with the list | **(a) measured** (2026-10-11, reference RTX 4070, driver 617.42, saved tune: power 105 %, core +135, memory +650). Nothing else on the card: 5828 it/s, bandwidth 480 GB/s over 24 readings scattering by 0.23 %, and 477 to 483.5 in a later series. Trae open and idle, 2.2 % of the 3D engine: 5329 it/s, 452 to 457 GB/s over 28 readings, then two at 441 and 444. A YouTube video in Chrome, 8.6 % of the video decoder and 2.8 % of the 3D engine while the card idles: 464 to 476 GB/s while it decodes, 444 to 455 while it draws as well, 24 readings scattering by 3.0 %; back to 483 on the first reading after it stopped. Under the run's own load the other program's share reads 1 % or less, so it can only be seen while the card idles. One sample of the counters read 183 455 208 183 776 %. That day's card tests had all been taken with the editor open, from the morning until 21:31:44, when its last files were written; the log of the run under way shows the bandwidth going from 430.4 to 457.1 GB/s at 21:31:45. **(b) as expected with the editor open**: `other programs on the card: Trae (2 %)`, twice. Not run with it closed. **(c) as expected**: the heading, `chrome (12 %, video), Trae (2 %).`, the explanation in two lines, **Check again**, **Go on anyway** and **Cancel**, at 1180 x 900. **(d) the command line's refusal: as expected** (2026-10-11 22:46, elevated, a test build of ff80e3d, the editor and the chat application this was written in open). `gao --busy`: `other programs on the card: Trae (1 %)`. `gao --retest`: `other programs on the card: Trae (2 %), msedgewebview2 (1 %)`, the four lines of advice, `RESULT: not applied -- other programs are using the graphics card: Trae (2 %), msedgewebview2 (1 %)`, exit 1, four seconds after it began; the profile and what is applied read the same before and after, and `run.log` has the line. Two things seen: the editor read 1 % in one look and 2 % in the next, so the threshold went from 1 % to 0.5 % after this run; and `msedgewebview2` is the browser engine inside another application, a name that tells its user nothing. **Not run:** `--busy-ok`, and the three answers in the window, on a real run. | ## Notes diff --git a/src/app/common.cpp b/src/app/common.cpp index 179f2c1..81c022a 100644 --- a/src/app/common.cpp +++ b/src/app/common.cpp @@ -355,7 +355,10 @@ OptimizeOutcome run_optimize(Preset preset, const OptimizeHooks& hooks, const st if (path.empty()) return fail("the ProgramData folder could not be resolved; cannot keep the crash journal"); const auto lines = read_lines(path); if (!lines) return fail("the crash journal " + path.string() + " exists but cannot be read; not tuning without it"); - Journal journal(*lines, [&path](const std::string& l) { return append_line_durable(path, l); }); + // What a memory edge in the journal is bound to: this card with this driver. + const std::string rig_card = hw.GpuUuid(), rig_driver = hw.DriverVersion(); + Journal journal(*lines, [&path](const std::string& l) { return append_line_durable(path, l); }, + rig_card.empty() || rig_driver.empty() ? std::string() : rig_card + " / " + rig_driver); // Prove the journal is writable before any clock is touched; the parser // ignores lines without an id, so this one never becomes a ceiling. if (!append_line_durable(path, "{\"session\":\"" + now_text() + "\"}")) diff --git a/src/core/journal.cpp b/src/core/journal.cpp index 22a9ecc..26722d4 100644 --- a/src/core/journal.cpp +++ b/src/core/journal.cpp @@ -43,8 +43,8 @@ std::optional int_field(const nlohmann::json& j, const char* key) { } } -Journal::Journal(const std::vector& existing_lines, Append append) - : append_(std::move(append)) { +Journal::Journal(const std::vector& existing_lines, Append append, std::string rig) + : append_(std::move(append)), rig_(std::move(rig)) { std::map pending; // ordered by id = file order int max_id = 0; for (std::string line : existing_lines) { @@ -53,6 +53,14 @@ Journal::Journal(const std::vector& existing_lines, Append append) const bool parsed = !j.is_discarded() && j.is_object(); torn_tail_ = !line.empty() && !parsed; // only the last line's value survives the loop if (!parsed) continue; + if (const auto edge = j.find("mem_edge"); edge != j.end()) { + // A line of its own, without an id: builds that do not know it skip it. + const auto mhz = int_field(j, "mem_edge"); + const auto rig_of = j.find("rig"); + if (mhz && *mhz > 0 && *mhz <= kMaxClockMhz && !rig_.empty() && rig_of != j.end() && rig_of->is_string() && *rig_of == rig_) + mem_edge_ = std::min(mem_edge_, *mhz); + continue; + } const auto id = int_field(j, "id"); const auto state = j.find("state"); if (!id || *id <= 0 || *id >= kMaxJournalId || state == j.end() || !state->is_string()) continue; @@ -92,6 +100,14 @@ bool Journal::complete(int id, const std::string& verdict) { return true; } +bool Journal::note_mem_edge(int mhz) { + if (mhz <= 0 || rig_.empty()) return true; // nothing to bind it to: nothing is kept + const nlohmann::json j = {{"mem_edge", mhz}, {"rig", rig_}}; + if (!write(j.dump())) return false; + mem_edge_ = std::min(mem_edge_, mhz); + return true; +} + bool Journal::write(const std::string& line) { if (!append_(torn_tail_ ? "\n" + line : line)) return false; torn_tail_ = false; diff --git a/src/core/journal.hpp b/src/core/journal.hpp index 942fa9e..0364d92 100644 --- a/src/core/journal.hpp +++ b/src/core/journal.hpp @@ -30,11 +30,18 @@ struct Freeze { // that froze the machine, and it becomes a ceiling for every later run. An // offset of 0 is never a ceiling: that would end the search of that clock. // Pure: the file lives in hw/app_files, reached through `append`. +// +// The journal also remembers where the memory of a card gave out in a scan +// that finished (a wrong result, a driver reset, a lost device): a card whose +// bandwidth rises until its memory fails has nothing else to stop a scan, and +// every run would meet that edge again. Such an edge belongs to a card with a +// driver, `rig`; with another of either it does not count. An empty rig keeps +// and uses none. class Journal { public: using Append = std::function; - Journal(const std::vector& existing_lines, Append append); + Journal(const std::vector& existing_lines, Append append, std::string rig = {}); const Ceilings& ceilings() const { return ceilings_; } // Human-readable description of each unfinished candidate, in file order: @@ -43,6 +50,12 @@ class Journal { // The same candidates, each with whether it lowered a ceiling. const std::vector& freeze_entries() const { return freeze_entries_; } int next_id() const { return next_id_; } + // The lowest memory offset at which a scan on this rig gave out; INT_MAX + // when none is known. Not a ceiling: the search decides how far below it + // to stay. + int mem_edge() const { return mem_edge_; } + // Writes one down. False when the line could not be written. + bool note_mem_edge(int mhz); // The entry begin() opened and complete() has not closed; -1 when none. int open_id() const { return open_id_; } @@ -60,6 +73,8 @@ class Journal { // by a power cut, without its newline. The next write starts with '\n' // so it is not glued onto that fragment. bool torn_tail_ = false; + std::string rig_; + int mem_edge_ = INT_MAX; Ceilings ceilings_; std::vector freezes_; std::vector freeze_entries_; diff --git a/src/core/search.cpp b/src/core/search.cpp index 3bd426d..12b6c04 100644 --- a/src/core/search.cpp +++ b/src/core/search.cpp @@ -61,26 +61,59 @@ int apply_margin(int max, int step, float perf_push) { return std::max(0, std::min(steps * step, max - step)); } -int best_bandwidth_offset(int lo, int hi, int step, int ceiling, const std::function& sample) { - const int top = std::min(hi, ceiling - 1); - std::vector> seen; +MemScanEnd bandwidth_scan(int lo, int hi, int step, int ceiling, const std::function& sample) { + MemScanEnd end{lo, MemScanEnd::Why::Limit}; + auto middle = [](std::vector of) { // of an even number: the lower one + std::sort(of.begin(), of.end()); + return of[(of.size() - 1) / 2]; + }; + std::vector steps; // the reading of every step that passed double best = 0; - for (int v = lo; v <= top; v += step) { + for (int v = lo; v <= std::min(hi, ceiling - 1); v += step) { MemSample s = sample(v); - if (!s.stable) break; - // A drop is measured twice before it ends the scan: a single reading - // can land while the memory clock is still in a lower P-state. - if (s.gbps < best * (1 - kBandwidthDrop)) { - s = sample(v); - if (!s.stable) break; + // A reading that would end the scan is in doubt: the step is read + // kBandwidthReadings times and the middle one counts. Only such a + // step: three readings at every step cost two minutes a scan and + // decided nothing a single reading did not (hardware check 95). + if (s.stable && s.gbps < best * (1 - kBandwidthDrop)) { + std::vector readings{s.gbps}; + while (static_cast(readings.size()) < kBandwidthReadings && (s = sample(v)).stable) readings.push_back(s.gbps); + s.gbps = middle(readings); } + if (!s.stable) { end.why = MemScanEnd::Why::Failed; break; } + end.top = v; + if (s.gbps < best * (1 - kBandwidthDrop)) { end.why = MemScanEnd::Why::Dropped; break; } + // The best is the highest middle of three steps in a row, not the + // highest reading: one reading that came out high would be a best + // every later step is held against, and the highest of fifty readings + // that scatter is always such a one. + steps.push_back(s.gbps); + best = std::max(best, middle({steps.end() - static_cast(std::min(steps.size(), 3)), steps.end()})); + } + return end; +} + +int second_pass_top(const MemScanEnd& end, int lo, int step, int known_edge) { + // The lowest offset that is known, or has to be assumed, not to hold. + int bound = end.top + step; + if (end.why == MemScanEnd::Why::Limit && known_edge != INT_MAX && known_edge > bound) bound = known_edge; + return std::max(lo, bound - step - kPeakMarginSteps * step); +} + +int bandwidth_peak_below(int top, int lo, int step, const std::function& sample) { + std::vector> seen; + double best = 0; + for (int v = top; v >= lo; v -= step) { + const MemSample s = sample(v); + if (!s.stable) break; seen.emplace_back(v, s.gbps); best = std::max(best, s.gbps); - if (s.gbps < best * (1 - kBandwidthDrop)) break; + if (s.gbps < best * (1 - kBandwidthTie)) break; } + int peak = top; for (const auto& [v, gbps] : seen) - if (gbps >= best * (1 - kBandwidthTie)) return v; - return lo; + if (gbps >= best * (1 - kBandwidthTie)) peak = std::min(peak, v); + return peak; } int confirm_edge(int edge, int lo, int step, int tries, const std::function& holds, @@ -580,25 +613,49 @@ OptimizeResult optimize(const GpuControl& gpu, const Objectives& obj, Journal& j // Memory comes first, at core 0: the core climb is where a card is most // likely to give out, and it must not keep the memory from being tuned. if (obj.mem_oc) { - const int ceiling = journal.ceilings().mem_mhz; + int ceiling = journal.ceilings().mem_mhz; + // Where the memory gave out in an earlier scan on this card and + // driver, the scan does not go again: it stops as far below it as + // confirming starts below a reset. A card whose bandwidth rises until + // its memory fails would otherwise end every run in a wrong result or + // a driver reset (hardware check 95). + if (const int edge = journal.mem_edge(); edge != INT_MAX) { + const int top = std::max(0, edge - kResetBackoffSteps * kMemStep); + log("mem: the memory gave out at " + offset_text(edge) + " in an earlier run on this card and driver; the scan stops at " + + offset_text(top)); + ceiling = std::min(ceiling, top + 1); + } int reset_at = -1; // the memory offset at which a reset event ended the scan + int gave_out_at = 0; // the offset whose candidate computed a wrong result const bool by_bandwidth = static_cast(io.bandwidth); bool left_at_stock = false; // the measurement does not work: nothing was searched if (by_bandwidth) { bool unavailable = false; - const int peak = best_bandwidth_offset(0, bounds.mem_max_mhz, kMemStep, ceiling, [&](int v) { + // One step of either pass: the candidate, then the reading while + // it is still applied. `again`: in the second pass. + const auto sample = [&](int v, bool again) { MemSample m; + // One reading. One that only did not settle is taken again, + // kBandwidthReadings times at most; any other failure ends + // the step at once. std::optional gbps; + bool failed = false; // The +0 sample is journaled without a clock, like a soak at // stock: if the machine froze in it, an open entry that said // memory 0 would read as "never search memory again". const std::optional mem_j = v > 0 ? std::optional(v) : std::nullopt; m.stable = explore(v, reset_at, [&] { - if (!is_stable(clock_candidate(std::nullopt, mem_j, 0, v, kClockProbeS, [&] { gbps = io.bandwidth(); }))) - return false; + const auto verdict = clock_candidate(std::nullopt, mem_j, 0, v, kClockProbeS, [&] { + for (int i = 0; i < kBandwidthReadings && !gbps && !failed; ++i) { + gbps = io.bandwidth(); + failed = !gbps && !(io.bandwidth_unsettled && io.bandwidth_unsettled()); + } + }); + if (verdict == Verdict::WrongResult) gave_out_at = v; + if (!is_stable(verdict)) return false; if (gbps) return true; // The candidate passed, the measurement failed: the step is unusable. - if (v > 0 && io.bandwidth_unsettled && io.bandwidth_unsettled()) { + if (v > 0 && !failed) { // The load computed and the readings disagreed: noise, // not the card giving out. The scan ends with the best // of what was measured; the core is still searched. @@ -620,11 +677,21 @@ OptimizeResult optimize(const GpuControl& gpu, const Objectives& obj, Journal& j if (m.stable) { m.gbps = *gbps; char buf[64]; - std::snprintf(buf, sizeof(buf), "mem +%d: bandwidth %.1f GB/s", v, m.gbps); + std::snprintf(buf, sizeof(buf), "mem +%d: bandwidth %.1f GB/s%s", v, m.gbps, again ? ", read again" : ""); log(buf); } return m; - }); + }; + const MemScanEnd end = bandwidth_scan(0, bounds.mem_max_mhz, kMemStep, ceiling, [&](int v) { return sample(v, false); }); + int peak = second_pass_top(end, 0, kMemStep, journal.mem_edge()); + // The second pass, when there is something to read and the scan + // left the card in a state to read it: not after a reset event + // (nothing new is explored then, and confirming starts + // kResetBackoffSteps below `peak`), not on a stopped run. + if (!unavailable && stopped.empty() && !explored_enough && peak > 0) { + log("mem: the scan passed " + offset_text(end.top) + "; reading again from " + offset_text(peak) + " down"); + peak = bandwidth_peak_below(peak, 0, kMemStep, [&](int v) { return sample(v, true); }); + } // An unavailable measurement ended the scan at +0, and nothing // is searched without it. Memory that corrects its own errors // computes right far beyond the point where it stops getting @@ -641,6 +708,12 @@ OptimizeResult optimize(const GpuControl& gpu, const Objectives& obj, Journal& j reset_event(why); } left_at_stock = unavailable; + // The edge of this card's memory, for the runs after this one. A + // reset event counts as much as a wrong result: the scan ended + // there because the memory did. + if (reset_at > 0) gave_out_at = reset_at; + if (gave_out_at > 0 && !journal.note_mem_edge(gave_out_at)) + log("warning: the memory's edge could not be written to the journal; the next run will find it again"); } if (!by_bandwidth) { r.mem_max_stable = climb_to_edge(0, bounds.mem_max_mhz, kMemStep, kMemStride, ceiling, [&](int v) { @@ -650,7 +723,7 @@ OptimizeResult optimize(const GpuControl& gpu, const Objectives& obj, Journal& j if (!stopped.empty()) return finish_fail(stopped); if (reset_at >= 0) log("mem: the driver reset at " + offset_text(reset_at) + "; using " + offset_text(r.mem_max_stable) + - (by_bandwidth ? ", the best of the values that passed" : ", the last value that passed")); + (by_bandwidth ? ", well below the values that passed" : ", the last value that passed")); r.mem_confirmed = confirm_edge(confirm_from(r.mem_max_stable, kMemStep, reset_at), 0, kMemStep, kConfirmTries, [&](int v) { return holds(clock_candidate(std::nullopt, v, 0, v, kConfirmProbeS)); }, confirm_step_down); diff --git a/src/core/search.hpp b/src/core/search.hpp index e971d8d..63e5e98 100644 --- a/src/core/search.hpp +++ b/src/core/search.hpp @@ -67,11 +67,37 @@ int apply_margin(int max, int step, float perf_push); // Memory overclocks fail by losing bandwidth before they fail by producing // errors: GDDR6X/GDDR6 retry bad transfers. The memory search therefore looks // for the bandwidth peak, not the stability ceiling. -inline constexpr double kBandwidthDrop = 0.01; // stop once 1 % below the best -// Within 1 % of the best counts as the best: measured noise on the 4070 is -// ~0.4 % and one 50 MHz step is worth ~0.5 %, so a narrower band would let -// noise pick the offset. +// +// It does so in two passes. The scan goes up and only has to find where it +// ends: a step that fails, or one that has clearly lost bandwidth. Then the +// offset to keep is read again, starting kPeakMarginSteps below the highest +// step the scan passed and going down from there. Where the scan ended no +// longer decides the offset by itself: on the reference RTX 4070, whose +// bandwidth rises until its memory fails, scans that ended by chance saved +// +100, +650 and +1600 MHz on one day (hardware check 94), and the cause was +// another program on the card that made the readings scatter (check 96). +// +// The scan ends once a step is this far below the best. Far more than the +// readings move by themselves, also next to another program (up to 3 %): +// where the scan ends by chance, the offset is chance again. +inline constexpr double kBandwidthDrop = 0.05; +// Within 1 % of the best counts as the best, in the second pass: one 50 MHz +// step is worth ~0.5 %, so a narrower band would let noise pick the offset. inline constexpr double kBandwidthTie = 0.01; +// A step of the scan whose reading would end it is the middle one of this +// many readings, so one low reading decides nothing; and the best that +// readings are held against is the highest middle of three steps in a row, so +// one high reading does not either. A reading that did not settle is taken +// again up to this many times. On the reference card single readings ended +// scans at +250, at +1000 and not at all on one day, and the runs saved +100, +// +650 and +1600 MHz (hardware check 94). +inline constexpr int kBandwidthReadings = 3; +// How far below the highest step the scan passed the second pass begins, in +// steps. The highest steps are where the memory is about to give out: the +// stress load passes them for three seconds, and a longer look at them is +// what a confirm probe is for, further down. Four steps is where confirming +// starts after a reset, too. +inline constexpr int kPeakMarginSteps = 4; // A confirm probe that ends too hot keeps the clock edge (heat is the power // step's and the soak's business), but only when the clock was loaded this @@ -111,12 +137,34 @@ struct MemSample { double gbps = 0; }; -// Scans lo, lo+step, ... (<= hi, < ceiling), sampling lo too. Stops at the -// first unstable sample or once gbps falls more than kBandwidthDrop below the -// best so far (a drop is re-measured once before it counts). Returns the -// lowest stable offset within kBandwidthTie of the best, so noise on a flat -// curve never drifts the result upward; lo when no sample was stable. -int best_bandwidth_offset(int lo, int hi, int step, int ceiling, const std::function& sample); +// Where a bandwidth scan ended, and why. +struct MemScanEnd { + int top = 0; // the highest offset that passed; lo when none did + enum class Why { + Failed, // the step above `top` was not stable + Dropped, // `top` itself was stable but more than kBandwidthDrop below the best + Limit, // hi or the ceiling + } why = Why::Limit; +}; + +// The first pass: scans lo, lo+step, ... (<= hi, < ceiling), sampling lo too, +// until a sample is not stable or its gbps is more than kBandwidthDrop below +// the best so far, where the best is the highest middle value of three steps +// in a row. A step that would end the scan is sampled kBandwidthReadings +// times first, and its middle reading counts. +MemScanEnd bandwidth_scan(int lo, int hi, int step, int ceiling, const std::function& sample); + +// The highest offset the second pass reads: kPeakMarginSteps below the +// highest offset that can have passed. `known_edge` is where the memory gave +// out in an earlier run (INT_MAX: nowhere): a scan that was held below it +// counts its margin from that edge, not a second time from where it was held. +int second_pass_top(const MemScanEnd& end, int lo, int step, int known_edge); + +// The second pass: samples top, top-step, ... (>= lo) until a sample is more +// than kBandwidthTie below the best of them, or not stable. Returns the +// lowest offset within kBandwidthTie of the best, so noise on a flat curve +// never drifts the result upward; `top` when not one sample was stable. +int bandwidth_peak_below(int top, int lo, int step, const std::function& sample); // "Search fast, confirm long": re-checks edge with a long probe; on failure // steps down and retries, at most `tries` probes, never probing lo (stock) or diff --git a/src/hw/stress.cpp b/src/hw/stress.cpp index ba46542..0365426 100644 --- a/src/hw/stress.cpp +++ b/src/hw/stress.cpp @@ -28,6 +28,12 @@ constexpr double kTargetBatchMs = 250.0; constexpr int kMaxDispatches = 2048; constexpr UINT kBwBytes = 256u * 1024 * 1024; // per buffer; far above the 4070's 36 MB L2 constexpr UINT kBwGroups = 1024; +// Dispatches per timed run: 32 GB copied, about 75 ms on the reference RTX +// 4070. Readings of it scatter by 0.2 % on a card that is left alone, and by +// 0.4 to 3 % when another program uses the card (hardware checks 95 and 96). +// A run four times as long was tried for steadier readings and taken out +// again: it was no steadier on a quiet card, and with it memory at its edge +// lost the device in five scans out of five, once the machine with it. constexpr int kBwDispatches = 64; constexpr size_t kBwRuns = 3; // consecutive runs that must agree constexpr int kBwMaxRuns = 20; // ~1.5 s at full speed; more if clocks are low @@ -365,7 +371,10 @@ std::optional Stress::MeasureBandwidth() { UINT64 t0 = 0, t1 = 0; if (!wait(d.q_disjoint.Get(), &dj, sizeof(dj)) || !wait(d.q_begin.Get(), &t0, sizeof(t0)) || !wait(d.q_end.Get(), &t1, sizeof(t1))) { - if (d.device->GetDeviceRemovedReason() != S_OK) d.device.Reset(); + // Say which: the log line used to end in nothing here. + const HRESULT removed = d.device->GetDeviceRemovedReason(); + error_ = removed != S_OK ? Hr("the copy", removed) + ": the device was lost" : "the timing of the copy did not come back"; + if (removed != S_OK) d.device.Reset(); return std::nullopt; } if (dj.Disjoint || dj.Frequency == 0 || t1 <= t0) { // clock changed mid-run: discard diff --git a/tests/test_journal.cpp b/tests/test_journal.cpp index 47269f3..ea2c6d2 100644 --- a/tests/test_journal.cpp +++ b/tests/test_journal.cpp @@ -227,3 +227,70 @@ TEST_CASE("an empty journal has no freeze entries") { Journal j({}, s.append()); CHECK(j.freeze_entries().empty()); } + +TEST_CASE("a journal without a memory edge has none") { + std::vector out; + Journal j({"{\"id\":1,\"mem\":2500,\"state\":\"begin\"}", "{\"id\":1,\"state\":\"complete\",\"verdict\":\"WRONG RESULT\"}"}, + [&](const std::string& l) { out.push_back(l); return true; }, "card-a / 617.42"); + CHECK(j.mem_edge() == INT_MAX); + CHECK(j.ceilings().mem_mhz == INT_MAX); +} + +TEST_CASE("a memory edge is written with the card and driver it was found on, and read back") { + std::vector out; + Journal j({}, [&](const std::string& l) { out.push_back(l); return true; }, "card-a / 617.42"); + CHECK(j.note_mem_edge(2500)); + REQUIRE(out.size() == 1); + CHECK(out[0].find("\"mem_edge\":2500") != std::string::npos); + CHECK(out[0].find("card-a / 617.42") != std::string::npos); + CHECK(j.mem_edge() == 2500); + + Journal again(out, [](const std::string&) { return true; }, "card-a / 617.42"); + CHECK(again.mem_edge() == 2500); + CHECK(again.ceilings().mem_mhz == INT_MAX); // not a freeze + CHECK(again.freezes().empty()); + CHECK(again.next_id() == 1); // not a candidate either +} + +TEST_CASE("the lowest memory edge of this card and driver counts") { + const std::vector lines = { + "{\"mem_edge\":2500,\"rig\":\"card-a / 617.42\"}", + "{\"mem_edge\":2450,\"rig\":\"card-a / 617.42\"}", + "{\"mem_edge\":1200,\"rig\":\"card-b / 617.42\"}", + "{\"mem_edge\":900,\"rig\":\"card-a / 620.10\"}", + }; + auto ok = [](const std::string&) { return true; }; + CHECK(Journal(lines, ok, "card-a / 617.42").mem_edge() == 2450); + CHECK(Journal(lines, ok, "card-b / 617.42").mem_edge() == 1200); +} + +TEST_CASE("a memory edge lapses with another driver or another card") { + const std::vector lines = {"{\"mem_edge\":2500,\"rig\":\"card-a / 617.42\"}"}; + auto ok = [](const std::string&) { return true; }; + CHECK(Journal(lines, ok, "card-a / 620.10").mem_edge() == INT_MAX); + CHECK(Journal(lines, ok, "card-b / 617.42").mem_edge() == INT_MAX); + // Back on the driver it was found with, it counts again. + CHECK(Journal(lines, ok, "card-a / 617.42").mem_edge() == 2500); +} + +TEST_CASE("without a card and driver to bind it to, no memory edge is kept or used") { + std::vector out; + Journal j({"{\"mem_edge\":2500,\"rig\":\"\"}", "{\"mem_edge\":2400}"}, [&](const std::string& l) { out.push_back(l); return true; }); + CHECK(j.mem_edge() == INT_MAX); + CHECK(j.note_mem_edge(2300)); + CHECK(out.empty()); + CHECK(j.mem_edge() == INT_MAX); +} + +TEST_CASE("memory edges that make no sense are ignored, and a failed write is reported") { + const std::vector lines = { + "{\"mem_edge\":0,\"rig\":\"r\"}", "{\"mem_edge\":-50,\"rig\":\"r\"}", "{\"mem_edge\":4294967305,\"rig\":\"r\"}", + "{\"mem_edge\":\"2500\",\"rig\":\"r\"}", "{\"mem_edge\":2500,\"rig\":7}", + }; + Journal j(lines, [](const std::string&) { return false; }, "r"); + CHECK(j.mem_edge() == INT_MAX); + CHECK_FALSE(j.note_mem_edge(2500)); + CHECK(j.mem_edge() == INT_MAX); + CHECK(j.note_mem_edge(0)); // nothing to note +} + diff --git a/tests/test_search.cpp b/tests/test_search.cpp index 92f1333..aec58cc 100644 --- a/tests/test_search.cpp +++ b/tests/test_search.cpp @@ -130,12 +130,25 @@ struct FakeCard { int flaky_power_sets = 0; // this many set_power_limit calls fail although nothing is wrong bool lost_on_first_core_confirm = false; // the first 30 s probe with a non-zero core offset ends DEVICE LOST bool lost_on_first_mem_confirm = false; // the first 30 s probe with a non-zero memory offset ends DEVICE LOST - int bw_measurements = 0; // bandwidth measurements taken + int bw_measurements = 0; // bandwidth measurements taken with the short copy + // The readings of the second pass: those taken below an offset the card + // has already been at. bw_again_noise: what they are off by, when that is + // to differ from bw_noise. From bw_again_loses_device_from on, such a + // reading takes the device down. + int bw_again_loses_device_from = INT_MAX; + std::function bw_again_noise; + int bw_again_measurements = 0; + int max_again_mem = -1; // the highest offset one was taken at bool bw_fails_at_zero = false; // the measurement at memory +0 fails; nothing is wrong with the card bool lost_on_first_soak = false; // the first 300 s probe ends DEVICE LOST bool fail_set_on_first_soak = false; // the first write of the soak state fails once int bw_fails_once_at_mem = -1; // the bandwidth measurement at this offset fails once and the driver resets int bw_unsettled_at_mem = -1; // the bandwidth readings at this offset do not agree; the driver is fine + int bw_unsettled_once_at_mem = -1; // the first reading at this offset does not settle, the next ones do + // What a reading is off by: a factor on the curve, for the n-th reading + // (from 0) taken at that offset. Empty: none. + std::function bw_noise; + std::map bw_readings_at; // readings taken so far, by offset bool bw_unsettled = false; // what the last failed measurement was int lost_mem_from = -1; // at or above this memory offset a 3 s probe ends DEVICE LOST int recover_fails_from_call = -1; // recover() fails from this call on (1 = the first) @@ -290,6 +303,7 @@ struct Run { int abort_after_probes = -1; bool no_rest = false; // the application gives the search no way to wait std::optional stock; // the measurement at stock that went before the run, when there was one + std::string rig = "card-a / 617.42"; // the card and driver the journal's memory edges are bound to std::vector> steps; // what the search said it was doing, and for how long OptimizeResult go(Preset preset, GpuControl gpu) { Journal j(card.journal, [this](const std::string& l) { @@ -298,7 +312,7 @@ struct Run { if (l.find("\"begin\"") != std::string::npos) card.events.push_back("begin"); card.journal.push_back(l); return true; - }); + }, rig); OptimizeIo io; io.probe = card.probe(); io.aborted = [this] { return card.aborted_now || (abort_after_probes >= 0 && card.probes >= abort_after_probes); }; @@ -321,8 +335,17 @@ struct Run { }; if (card.bw_curve) io.bandwidth = [this]() -> std::optional { + const int nth = card.bw_readings_at[card.mem]++; + if (card.mem < card.max_mem_seen) { // the second pass + ++card.bw_again_measurements; + card.max_again_mem = std::max(card.max_again_mem, card.mem); + card.bw_unsettled = false; + if (card.mem >= card.bw_again_loses_device_from) { card.driver_reset(); return std::nullopt; } + const auto& noise = card.bw_again_noise ? card.bw_again_noise : card.bw_noise; + return card.bw_curve(card.mem) * (noise ? noise(card.mem, nth) : 1.0); + } ++card.bw_measurements; - card.bw_unsettled = card.mem == card.bw_unsettled_at_mem; + card.bw_unsettled = card.mem == card.bw_unsettled_at_mem || (card.mem == card.bw_unsettled_once_at_mem && nth == 0); if (card.bw_unsettled) return std::nullopt; if (card.bw_loses_device) { card.driver_reset(); return std::nullopt; } if (card.bw_fails_at_zero && card.mem == 0) return std::nullopt; @@ -332,7 +355,7 @@ struct Run { return std::nullopt; } if (card.bw_fails) return std::nullopt; - return card.bw_curve(card.mem); + return card.bw_curve(card.mem) * (card.bw_noise ? card.bw_noise(card.mem, nth) : 1.0); }; if (card.bw_curve) io.bandwidth_unsettled = [this] { return card.bw_unsettled; }; io.measuring = [this](const std::string& what, double seconds) { steps.emplace_back(what, seconds); }; @@ -656,59 +679,172 @@ TEST_CASE("a soak that runs too hot lowers power, not clocks") { CHECK(r.mem_mhz == 550); } -TEST_CASE("bandwidth scan stops at the peak of a rising-then-falling curve") { +TEST_CASE("the bandwidth scan ends at the first step that has clearly lost bandwidth") { int max_sampled = -1; - const int r = best_bandwidth_offset(0, 1500, 50, INT_MAX, [&](int v) { + const MemScanEnd end = bandwidth_scan(0, 1500, 50, INT_MAX, [&](int v) { max_sampled = std::max(max_sampled, v); return MemSample{true, v <= 600 ? 500 + v * 0.3 : 680 - (v - 600) * 0.9}; }); - CHECK(r == 600); - CHECK(max_sampled == 650); // stopped at the first step more than 1 % below the peak -} - -TEST_CASE("a single low bandwidth reading is re-measured before it ends the scan") { - // Measured on the 4070: an occasional reading lands at half speed when the - // memory clock is still in a lower P-state. One such dip must not end the - // scan; a drop that repeats does. + CHECK(end.why == MemScanEnd::Why::Dropped); + // The best is the middle of three steps, 665 GB/s around the peak at + // +600: +650 is 4.5 % below it, +700 is 11 %. It was stable, so it passed. + CHECK(end.top == 700); + CHECK(max_sampled == 700); +} + +TEST_CASE("a single low bandwidth reading does not end the scan; three do") { + // A step that is further from the best than readings scatter is read + // kBandwidthReadings times and its middle reading counts. One dip (measured + // on the 4070: a reading at half speed, the memory clock still in a lower + // P-state) ends nothing; a drop that is there every time does. std::map calls; - const int r = best_bandwidth_offset(0, 1500, 50, INT_MAX, [&](int v) { + MemScanEnd end = bandwidth_scan(0, 1500, 50, INT_MAX, [&](int v) { const bool dip = v == 300 && calls[v] == 0; ++calls[v]; - return MemSample{true, dip ? 250.0 : (v <= 600 ? 500 + v * 0.3 : 680 - (v - 600) * 0.9)}; + return MemSample{true, dip ? 250.0 : 500 + v * 0.1}; + }); + CHECK(end.why == MemScanEnd::Why::Limit); + CHECK(end.top == 1500); + CHECK(calls[250] == 1); // nothing in doubt: one reading + CHECK(calls[300] == 3); + CHECK(calls[350] == 1); + + // Two low readings of three are the middle one. + calls.clear(); + end = bandwidth_scan(0, 1500, 50, INT_MAX, [&](int v) { + const int nth = calls[v]++; + return MemSample{true, v == 300 && nth < 2 ? 250.0 : 500 + v * 0.1}; + }); + CHECK(end.why == MemScanEnd::Why::Dropped); + CHECK(end.top == 300); + CHECK(calls[300] == 3); + CHECK(calls.count(350) == 0); + + // A step that stops being stable while it is read again ends the scan below it. + calls.clear(); + end = bandwidth_scan(0, 1500, 50, INT_MAX, [&](int v) { + const int nth = calls[v]++; + return MemSample{!(v == 300 && nth == 1), v == 300 ? 250.0 : 500 + v * 0.1}; }); - CHECK(r == 600); + CHECK(end.why == MemScanEnd::Why::Failed); + CHECK(end.top == 250); CHECK(calls[300] == 2); - CHECK(calls[650] == 2); // the real drop is confirmed once too } -TEST_CASE("bandwidth scan stops at the first unstable step") { - int max_sampled = -1; - const int r = best_bandwidth_offset(0, 1500, 50, INT_MAX, [&](int v) { - max_sampled = std::max(max_sampled, v); - return MemSample{v <= 400, 500 + v * 0.3}; +TEST_CASE("a single high bandwidth reading does not become the best the scan is held against") { + // The run that saved +100 MHz on a card that runs every game at +1500: one + // reading came out far above its neighbours, and the steps after it were + // "a drop". + std::map calls; + const MemScanEnd end = bandwidth_scan(0, 1500, 50, INT_MAX, [&](int v) { + const bool spike = v == 200 && calls[v] == 0; + ++calls[v]; + return MemSample{true, (500 + v * 0.1) * (spike ? 1.08 : 1.0)}; }); - CHECK(r == 400); - CHECK(max_sampled == 450); + CHECK(end.why == MemScanEnd::Why::Limit); + CHECK(end.top == 1500); + for (const auto& [v, n] : calls) CHECK(n == 1); // and no step after it was in doubt + + // Two high steps in a row are the middle of three, and so a best: what + // comes after them is that far below it, read three times, and ends the scan. + const MemScanEnd two = bandwidth_scan(0, 1500, 50, INT_MAX, [&](int v) { + return MemSample{true, (500 + v * 0.1) * (v == 200 || v == 250 ? 1.08 : 1.0)}; + }); + CHECK(two.why == MemScanEnd::Why::Dropped); + CHECK(two.top == 300); } -TEST_CASE("a flat curve keeps the lowest offset") { - const int r = best_bandwidth_offset(0, 1500, 50, INT_MAX, [](int v) { - return MemSample{true, 500.0 + ((v / 50) % 2)}; // +-1 GB/s noise +TEST_CASE("the bandwidth scan ends below the first unstable step") { + int max_sampled = -1; + const MemScanEnd end = bandwidth_scan(0, 1500, 50, INT_MAX, [&](int v) { + max_sampled = std::max(max_sampled, v); + return MemSample{v <= 400, 500 + v * 0.1}; }); - CHECK(r == 0); + CHECK(end.why == MemScanEnd::Why::Failed); + CHECK(end.top == 400); + CHECK(max_sampled == 450); } -TEST_CASE("bandwidth scan samples lo and respects the ceiling") { +TEST_CASE("the bandwidth scan samples lo and respects the ceiling") { std::vector sampled; - const int r = best_bandwidth_offset(0, 1500, 50, 300, [&](int v) { + const MemScanEnd end = bandwidth_scan(0, 1500, 50, 300, [&](int v) { sampled.push_back(v); - return MemSample{true, 500 + v * 0.3}; + return MemSample{true, 500 + v * 0.1}; }); REQUIRE_FALSE(sampled.empty()); CHECK(sampled.front() == 0); CHECK(sampled.back() == 250); - CHECK(r == 250); - CHECK(best_bandwidth_offset(0, 1500, 50, 300, [](int) { return MemSample{false, 0}; }) == 0); + CHECK(end.top == 250); + CHECK(end.why == MemScanEnd::Why::Limit); + const MemScanEnd none = bandwidth_scan(0, 1500, 50, 300, [](int) { return MemSample{false, 0}; }); + CHECK(none.top == 0); + CHECK(none.why == MemScanEnd::Why::Failed); +} + +TEST_CASE("the second pass begins four steps below the highest step the scan passed") { + using Why = MemScanEnd::Why; + // The step above +2500 failed: +2500 passed, so +2300. + CHECK(second_pass_top({2500, Why::Failed}, 0, 50, INT_MAX) == 2300); + // +1000 passed and had lost bandwidth: +800. + CHECK(second_pass_top({1000, Why::Dropped}, 0, 50, INT_MAX) == 800); + // The driver's range, or the ceiling of a freeze: nothing failed, and the + // margin is kept all the same. What is above was never tried. + CHECK(second_pass_top({3000, Why::Limit}, 0, 50, INT_MAX) == 2800); + // Never below lo, and a scan that passed nothing reads nothing above it. + CHECK(second_pass_top({150, Why::Failed}, 0, 50, INT_MAX) == 0); + CHECK(second_pass_top({0, Why::Failed}, 0, 50, INT_MAX) == 0); +} + +TEST_CASE("a scan held below a known edge counts the margin from that edge, once") { + using Why = MemScanEnd::Why; + // The run that found the edge at +2550 read from +2300. The run after it is + // held at +2350 and reads from +2300 as well, not from +2150. + CHECK(second_pass_top({2500, Why::Failed}, 0, 50, INT_MAX) == 2300); + CHECK(second_pass_top({2350, Why::Limit}, 0, 50, 2550) == 2300); + // A step that failed below the known edge counts, not the edge. + CHECK(second_pass_top({2000, Why::Failed}, 0, 50, 2550) == 1800); + CHECK(second_pass_top({2000, Why::Dropped}, 0, 50, 2550) == 1800); + // An edge that is no higher than where the scan got changes nothing. + CHECK(second_pass_top({2350, Why::Limit}, 0, 50, 2400) == 2150); +} + +TEST_CASE("the second pass goes down from the top and keeps the lowest offset within the tie") { + // A curve that still rises at the top: 0.4 % a step. + std::vector sampled; + int r = bandwidth_peak_below(2300, 0, 50, [&](int v) { + sampled.push_back(v); + return MemSample{true, 434.0 * (1 + 0.004 * v / 50)}; + }); + CHECK(r == 2200); + CHECK(sampled == std::vector{2300, 2250, 2200, 2150}); + + // A peak below the top: the readings rise on the way down, then fall. + sampled.clear(); + r = bandwidth_peak_below(1200, 0, 50, [&](int v) { + sampled.push_back(v); + return MemSample{true, v <= 900 ? 500 + v * 0.3 : 770 - (v - 900) * 0.6}; + }); + CHECK(r == 900); + CHECK(sampled.front() == 1200); + CHECK(sampled.back() == 850); // 1.9 % below the peak: the pass ends + + // A flat curve keeps the lowest offset there is. + CHECK(bandwidth_peak_below(400, 0, 50, [](int v) { return MemSample{true, 500.0 + ((v / 50) % 2)}; }) == 0); + // Readings that scatter by twice what they do on a quiet 4070 leave a flat + // curve at the bottom. + CHECK(bandwidth_peak_below(400, 0, 50, [](int v) { return MemSample{true, 500.0 * (1 + (((v / 50) % 3) - 1) * 0.004)}; }) == 0); +} + +TEST_CASE("the second pass keeps what it has when a step is not stable") { + // Not one sample: the top it was given. + CHECK(bandwidth_peak_below(2300, 0, 50, [](int) { return MemSample{false, 0}; }) == 2300); + // One sample, then none: that one. + int n = 0; + CHECK(bandwidth_peak_below(2300, 0, 50, [&](int v) { return MemSample{n++ == 0, 500.0 + v * 0.1}; }) == 2300); + // Never below lo. + std::vector sampled; + CHECK(bandwidth_peak_below(100, 0, 50, [&](int v) { sampled.push_back(v); return MemSample{true, 500.0}; }) == 0); + CHECK(sampled == std::vector{100, 50, 0}); } TEST_CASE("confirm_edge keeps a holding edge and steps down otherwise") { @@ -731,10 +867,16 @@ TEST_CASE("memory search stops at the bandwidth peak, not the stability edge") { run.card.bw_curve = [](int m) { return m <= 900 ? 500 + m * 0.3 : 770 - (m - 900) * 0.6; }; const auto r = run.go(Preset::BestOfMyGpu); REQUIRE(r.ok); - CHECK(r.mem_max_stable == 900); - CHECK(r.mem_confirmed == 900); - CHECK(r.mem_mhz == 600); - CHECK(run.card.max_mem_seen <= 950); + // The scan ends at +1000, the first step that has lost 5 % (7.8 % below + // the peak at +900), far below the first wrong result at +1450. The second + // pass reads from four steps below that, +800, and keeps +800: the peak + // itself is inside the margin. That is the price of the margin on a card + // that has a peak. + CHECK(run.card.max_mem_seen == 1000); + CHECK(run.card.max_again_mem == 800); + CHECK(r.mem_max_stable == 800); + CHECK(r.mem_confirmed == 800); + CHECK(r.mem_mhz == 550); Journal reread(run.card.journal, [](const std::string&) { return true; }); CHECK(reread.freezes().empty()); } @@ -777,16 +919,6 @@ TEST_CASE("a confirm probe that only runs too hot keeps the clock edge") { CHECK(r.mem_confirmed == 800); } -TEST_CASE("bandwidth noise wider than one step does not pick a higher offset") { - // Measured spread on the 4070 is ~0.4 %; one 50 MHz step is worth ~0.5 %. - // Noise of +-0.45 % (0.9 % spread, twice what was measured) on a flat - // curve must leave the result at the bottom. - const int r = best_bandwidth_offset(0, 1500, 50, INT_MAX, [](int v) { - return MemSample{true, 500.0 * (1 + (((v / 50) % 3) - 1) * 0.0045)}; - }); - CHECK(r == 0); -} - TEST_CASE("a journal entry that cannot be closed stops the run at stock") { // An open entry would blacklist the candidate forever as if it had frozen // the machine; carrying on would hide that the journal is failing. @@ -2173,19 +2305,21 @@ TEST_CASE("a failed bandwidth measurement ends the memory scan; the core is stil const auto r = run.go(Preset::BestOfMyGpu); REQUIRE(r.ok); CHECK(r.driver_resets == 1); - // The scan samples +0 .. +300; the confirm probe runs four steps below - // +250, the best of the values that were measured. The +0 sample names no - // clock in the journal. - CHECK(mem_begins(run.card) == climb(50, 300, 50, {50})); + // The scan samples +0 .. +300. +250 was the highest step that passed, so + // the second pass would begin at +50; after a reset nothing is read, and + // confirming starts four steps below that, which is stock. The +0 sample + // names no clock in the journal. + CHECK(mem_begins(run.card) == climb(50, 300, 50)); CHECK(clockless_begins(run.card) == 1); - // +0, +50 .. +300, the memory confirm probe, core +15 .. +165, the core confirm probe, the soak - CHECK(run.card.count("\"begin\"") == 21); + // +0, +50 .. +300, core +15 .. +165, the core confirm probe, the soak + CHECK(run.card.count("\"begin\"") == 20); CHECK(run.logged("core +0 / mem +0: STABLE")); CHECK(run.logged("mem +0: bandwidth 500.0 GB/s")); CHECK(run.card.max_mem_seen == 300); CHECK(run.card.bw_measurements == 7); - CHECK(r.mem_max_stable == 250); - CHECK(r.mem_confirmed == 50); + CHECK(run.card.bw_again_measurements == 0); + CHECK(r.mem_max_stable == 50); + CHECK(r.mem_confirmed == 0); CHECK(r.mem_mhz == 0); CHECK(run.card.max_core_seen == 165); CHECK(r.core_mhz == 105); @@ -2207,11 +2341,13 @@ TEST_CASE("bandwidth readings that do not settle end the memory scan and are no CHECK(run.card.recover_calls == 0); CHECK_FALSE(run.logged("treated as a driver reset")); CHECK(run.logged("mem +300: the bandwidth readings did not settle; the memory scan ends here")); - // The scan ends at +300 with the best of the measured values, +250, and - // the confirm probe runs there, not four steps below. + // The scan ends at +300; +250 was the highest step that passed, so the + // second pass begins at +50, and the confirm probe runs at what it kept, + // not four steps below that. CHECK(run.card.max_mem_seen == 300); - CHECK(r.mem_max_stable == 250); - CHECK(r.mem_confirmed == 250); + CHECK(run.card.max_again_mem == 50); + CHECK(r.mem_max_stable == 50); + CHECK(r.mem_confirmed == 50); // The core is searched as in a run without any event. CHECK(run.card.max_core_seen > 0); CHECK(r.core_mhz > 0); @@ -2447,7 +2583,7 @@ TEST_CASE("the +0 sample of the bandwidth scan journals no clock") { run.card.bw_curve = [](int m) { return 500 + m * 0.3; }; const auto r = run.go(Preset::BestOfMyGpu); REQUIRE(r.ok); - CHECK(r.mem_max_stable == 800); + CHECK(r.mem_max_stable == 600); // +800 passed; the second pass began at +600 // The first entry of the run is the +0 sample: it ran and was measured, // and its `begin` line names neither clock. CHECK(run.logged("core +0 / mem +0: STABLE")); @@ -2478,7 +2614,7 @@ TEST_CASE("the +0 sample of the bandwidth scan journals no clock") { next.card.journal = frozen; const auto b = next.go(Preset::BestOfMyGpu); REQUIRE(b.ok); - CHECK(b.mem_max_stable == 800); + CHECK(b.mem_max_stable == 600); CHECK(b.core_max_stable == 150); } @@ -2879,3 +3015,355 @@ TEST_CASE("without a measurement at stock there is nothing to be slower than") { Run unscored = warm_card(0); // a measurement that scored nothing says nothing CHECK(unscored.go(Preset::BestOfMyGpu).ok); } + +namespace { + +// A card like the reference RTX 4070 (hardware check 94): the bandwidth rises +// by 0.4 % per 50 MHz step all the way to the first wrong result. There is no +// peak to find; only scatter can end the scan before the edge. +Run rising_to_the_edge(int edge = 2500) { + Run run; + run.card.mem_edge = edge; + run.card.ranges = ClockOffsetRanges{{-500, 1000}, {-1000, 3000}}; + run.card.bw_curve = [](int m) { return 434.0 * (1 + 0.004 * m / 50); }; + return run; +} + +// Scatter with a standard deviation of about `sd`, the same for the same +// (seed, offset, reading): three uniform numbers summed, which is bell-shaped +// enough and needs no library whose numbers differ between compilers. +std::function scatter(unsigned seed, double sd) { + return [seed, sd](int mem, int nth) { + unsigned x = seed * 2654435761u + static_cast(mem) * 40503u + static_cast(nth) * 9176u + 12345u; + double sum = 0; + for (int i = 0; i < 3; ++i) { + x ^= x << 13; x ^= x >> 17; x ^= x << 5; + sum += (x % 100000) / 100000.0 - 0.5; // uniform in -0.5 .. 0.5, variance 1/12 + } + return 1.0 + sum * 2.0 * sd; // three of them: variance 1/4, so a deviation of sd + }; +} + +} + +TEST_CASE("one bandwidth reading that is far too high does not end the memory scan") { + // The run that saved +100 MHz on a card that runs every game at +1500: + // one reading came out far above its neighbours, the two after it were + // under it, and that was "the peak". + Run run = rising_to_the_edge(); + run.card.bw_noise = [](int mem, int nth) { return mem == 200 && nth == 0 ? 1.08 : 1.0; }; + const auto r = run.go(Preset::BestOfMyGpu); + REQUIRE(r.ok); + CHECK(run.card.max_mem_seen == 2550); // the scan went to the first wrong result + CHECK(run.logged("mem +0: bandwidth 434.0 GB/s")); + CHECK(run.card.bw_measurements == 51); // +0 .. +2500, each once: no step was in doubt + // +2500 passed; the second pass reads from +2300 down, and +2200 is the + // lowest offset within 1 % of what it read there. + CHECK(r.mem_max_stable == 2200); + CHECK(r.mem_mhz == 1500); +} + +TEST_CASE("one low bandwidth reading does not end the memory scan; three low ones do") { + { + Run run = rising_to_the_edge(); + run.card.bw_noise = [](int mem, int nth) { return mem == 1000 && nth == 0 ? 0.9 : 1.0; }; + const auto r = run.go(Preset::BestOfMyGpu); + REQUIRE(r.ok); + CHECK(run.card.bw_readings_at[950] == 1); + CHECK(run.card.bw_readings_at[1000] == kBandwidthReadings); + CHECK(run.card.max_mem_seen == 2550); + CHECK(r.mem_max_stable == 2200); + } + { + // From +1000 on the memory has lost 8 %: that is a peak, whichever copy reads it. + Run run = rising_to_the_edge(); + run.card.bw_curve = [](int m) { return 434.0 * (1 + 0.004 * m / 50) * (m >= 1000 ? 0.92 : 1.0); }; + const auto r = run.go(Preset::BestOfMyGpu); + REQUIRE(r.ok); + CHECK(run.card.bw_readings_at[1000] == kBandwidthReadings); + CHECK(run.logged("mem: the scan passed +1000; reading again from +800 down")); + CHECK(run.card.max_mem_seen == 1000); + CHECK(run.card.max_again_mem == 800); + CHECK(r.mem_max_stable == 700); // the lowest offset within 1 % of the reading at +800 + } +} + +TEST_CASE("readings that scatter as they do on the reference card leave the memory scan where the card puts it") { + // Measured there: readings scatter by 0.2 % on a card that is left alone + // (hardware check 96). A little more than that here, on two hundred + // different days. Every scan gets to the first wrong result, and the + // offset that is kept moves by five steps: +2050 to +2300. + for (unsigned seed = 1; seed <= 200; ++seed) { + Run run = rising_to_the_edge(); + run.card.bw_noise = scatter(seed, 0.0025); + const auto r = run.go(Preset::BestOfMyGpu); + CAPTURE(seed); + REQUIRE(r.ok); + CHECK(run.card.max_mem_seen == 2550); + CHECK(run.card.max_again_mem == 2300); + CHECK(r.mem_max_stable >= 2050); + CHECK(r.mem_max_stable <= 2300); + } +} + +TEST_CASE("a real peak is still found where it is when the readings scatter") { + // The card the method was made for: bandwidth turns down long before the + // first wrong result. Scatter must not carry the scan past the peak. + for (unsigned seed = 1; seed <= 20; ++seed) { + Run run; + run.card.mem_edge = 1400; + run.card.bw_curve = [](int m) { return m <= 900 ? 500 + m * 0.3 : 770 - (m - 900) * 0.6; }; + run.card.bw_noise = scatter(seed, 0.013); + const auto r = run.go(Preset::BestOfMyGpu); + CAPTURE(seed); + REQUIRE(r.ok); + // +950 is 3.9 % below the peak and +1000 7.8 %: with this scatter the + // scan ends at one of the two, now and then one step later. + CHECK(run.card.max_mem_seen >= 950); + CHECK(run.card.max_mem_seen <= 1050); + CHECK(run.card.max_again_mem <= run.card.max_mem_seen - kPeakMarginSteps * 50); + CHECK(r.mem_max_stable >= 700); + CHECK(r.mem_max_stable <= 900); + } +} + +TEST_CASE("a reading that does not settle is taken again") { + // On the reference card one reading in twenty-four did not settle, with + // nothing wrong. One such reading used to end the scan. + Run run = recovering(); + run.card.bw_curve = [](int m) { return 500 + m * 0.3; }; + run.card.bw_unsettled_once_at_mem = 300; + const auto r = run.go(Preset::BestOfMyGpu); + REQUIRE(r.ok); + CHECK_FALSE(run.logged("did not settle")); + CHECK(run.logged("mem +300: bandwidth 590.0 GB/s")); + CHECK(run.card.bw_readings_at[300] == 2); + CHECK(run.card.max_mem_seen > 300); + CHECK(r.driver_resets == 0); +} + +TEST_CASE("a stretch in which every reading is 2 % off does not end the memory scan") { + // Measured on the reference card (hardware check 95): for tens of seconds + // at a time every reading is about 1.8 % higher, whatever the offset, and + // then it is back. The readings of one step are all inside such a + // stretch, so reading it three times changes nothing. + for (const double off : {0.98, 1.02}) { + Run run = rising_to_the_edge(); + run.card.bw_noise = [&card = run.card, off](int, int) { return card.bw_measurements > 20 && card.bw_measurements <= 45 ? off : 1.0; }; + const auto r = run.go(Preset::BestOfMyGpu); + CAPTURE(off); + REQUIRE(r.ok); + CHECK(run.card.max_mem_seen == 2550); + // The stretch can leave a few steps that read as high as the top of + // the scan; the tie rule then keeps the lowest of them. + CHECK(r.mem_max_stable >= 2000); + } +} + +namespace { + +// The run after: the same card and journal, the counters of the first run gone. +OptimizeResult again(Run& run) { + run.log.clear(); + run.card.max_mem_seen = 0; + run.card.bw_measurements = 0; + run.card.bw_readings_at.clear(); + run.card.bw_again_measurements = 0; + run.card.max_again_mem = -1; + return run.go(Preset::BestOfMyGpu); +} + +} + +TEST_CASE("a memory scan that ran into a wrong result leaves the edge in the journal") { + Run run = rising_to_the_edge(); + REQUIRE(run.go(Preset::BestOfMyGpu).ok); + CHECK(run.card.max_mem_seen == 2550); + CHECK(run.card.count("\"mem_edge\":2550") == 1); + CHECK(run.card.count("card-a / 617.42") == 1); + check_journal_rules(run); +} + +TEST_CASE("the next run stays four steps below the memory edge of the run before") { + // On the reference card every scan ended in a driver reset at the edge + // (hardware check 95). Once is enough. + Run run = rising_to_the_edge(); + const auto first = run.go(Preset::BestOfMyGpu); + REQUIRE(first.ok); + const auto second = again(run); + REQUIRE(second.ok); + CHECK(run.card.max_mem_seen == 2550 - kResetBackoffSteps * 50); + CHECK(run.logged("mem: the memory gave out at +2550 in an earlier run on this card and driver; the scan stops at +2350")); + CHECK_FALSE(run.logged("mem +2400")); // no candidate above it, so none that could fail + CHECK(second.driver_resets == 0); + // The same offset as the run that found the edge: the margin counts from + // the edge, not a second time from where the scan was held. + CHECK(run.card.max_again_mem == 2300); + CHECK(second.mem_max_stable == first.mem_max_stable); + CHECK(second.mem_max_stable == 2200); + CHECK(second.mem_mhz == 1500); + CHECK(run.card.count("\"mem_edge\"") == 1); // nothing new was found + // And the run after that is the same. + const auto third = again(run); + REQUIRE(third.ok); + CHECK(third.mem_mhz == second.mem_mhz); +} + +TEST_CASE("a bandwidth measurement that lost the device leaves the edge too, and the next run has no reset") { + Run run = recovering(); + run.card.bw_curve = [](int m) { return 500 + m * 0.3; }; + run.card.bw_fails_once_at_mem = 500; + const auto first = run.go(Preset::BestOfMyGpu); + REQUIRE(first.ok); + CHECK(first.driver_resets == 1); + CHECK(run.card.count("\"mem_edge\":500") == 1); + const auto second = again(run); + REQUIRE(second.ok); + CHECK(second.driver_resets == 0); + CHECK(run.card.max_mem_seen == 300); + CHECK(second.mem_max_stable == 250); // the margin counts from the edge at +500, once +} + +TEST_CASE("the memory edge lapses with another driver") { + Run run = rising_to_the_edge(); + REQUIRE(run.go(Preset::BestOfMyGpu).ok); + run.rig = "card-a / 620.10"; + REQUIRE(again(run).ok); + CHECK(run.card.max_mem_seen == 2550); // searched afresh + CHECK_FALSE(run.logged("in an earlier run")); + CHECK(run.card.count("\"mem_edge\":2550") == 2); // and found again, for the new driver +} + +TEST_CASE("a scan that ended for another reason leaves no memory edge") { + { // a bandwidth peak + Run run; + run.card.mem_edge = 1400; + run.card.bw_curve = [](int m) { return m <= 900 ? 500 + m * 0.3 : 770 - (m - 900) * 0.6; }; + REQUIRE(run.go(Preset::BestOfMyGpu).ok); + CHECK(run.card.count("mem_edge") == 0); + } + { // readings that did not settle + Run run = recovering(); + run.card.bw_curve = [](int m) { return 500 + m * 0.3; }; + run.card.bw_unsettled_at_mem = 300; + REQUIRE(run.go(Preset::BestOfMyGpu).ok); + CHECK(run.card.count("mem_edge") == 0); + } + { // a measurement that does not work on this machine + Run run = recovering(); + run.card.bw_curve = [](int m) { return 500 + m * 0.3; }; + run.card.bw_fails_at_zero = true; + REQUIRE(run.go(Preset::BestOfMyGpu).ok); + CHECK(run.card.count("mem_edge") == 0); + } + { // the driver's own range + Run run = rising_to_the_edge(5000); + REQUIRE(run.go(Preset::BestOfMyGpu).ok); + CHECK(run.card.max_mem_seen == 3000); + CHECK(run.card.count("mem_edge") == 0); + } +} + +namespace { + +// The reference RTX 4070: bandwidth rises by 0.4 % a step until the memory +// gives out, and the stress load computes right up to +2500 and wrong at +// +2550. Nothing announces that edge (hardware checks 94 and 95). +Run reference_card() { + Run run = recovering(); + run.card.mem_edge = 2500; + run.card.ranges = ClockOffsetRanges{{-500, 1000}, {-1000, 3000}}; + run.card.bw_curve = [](int m) { return 434.0 * (1 + 0.004 * m / 50); }; + return run; +} + +} + +TEST_CASE("on a card whose bandwidth rises to the edge, the offset is read again well below it") { + Run run = reference_card(); + const auto r = run.go(Preset::BestOfMyGpu); + REQUIRE(r.ok); + // The scan finds the end with the short copy and the stress load. + CHECK(run.card.max_mem_seen == 2550); + CHECK(run.logged("core +0 / mem +2550: WRONG RESULT")); + CHECK(run.logged("mem: the scan passed +2500; reading again from +2300 down")); + // The second pass: four readings, none above +2300. + CHECK(run.card.max_again_mem == 2300); + CHECK(run.card.bw_readings_at[2300] == 2); // once going up, once coming down + CHECK(run.card.bw_again_measurements == 4); + CHECK(run.logged("mem +2300: bandwidth 513.9 GB/s, read again")); + CHECK(r.driver_resets == 0); + CHECK(run.card.recover_calls == 0); + CHECK_FALSE(run.logged("treated as a driver reset")); + CHECK(run.logged("mem: bandwidth peak +2200, confirmed +2200, applying +1500")); + CHECK(r.mem_mhz == 1500); + // The edge is written down, and the next run does not even meet the wrong result. + CHECK(run.card.count("\"mem_edge\":2550") == 1); + const auto second = again(run); + REQUIRE(second.ok); + CHECK(run.card.max_mem_seen == 2350); + CHECK(run.card.max_again_mem == 2300); + CHECK(second.driver_resets == 0); + CHECK(second.mem_mhz == 1500); + check_journal_rules(run); +} + +TEST_CASE("the second pass stays below the margin on that card whatever the readings do") { + // Readings that scatter as the worst series measured next to another + // program, in both passes, with a high and a low stretch thrown in. + for (unsigned seed = 1; seed <= 40; ++seed) { + Run run = reference_card(); + const auto noise = scatter(seed, 0.013); + run.card.bw_noise = [&card = run.card, noise, seed](int mem, int nth) { + const int n = card.bw_measurements; + const double level = n > 10 && n <= 20 ? 1.018 : n > 30 + static_cast(seed % 10) && n <= 45 ? 0.982 : 1.0; + return noise(mem, nth) * level; + }; + const auto r = run.go(Preset::BestOfMyGpu); + CAPTURE(seed); + REQUIRE(r.ok); + CHECK(run.card.max_again_mem <= run.card.max_mem_seen - 50 - kPeakMarginSteps * 50); + CHECK(r.driver_resets == 0); + CHECK(run.card.max_mem_seen == 2550); + CHECK(r.mem_mhz >= 1400); // +2050 or higher kept + } +} + +TEST_CASE("the margin holds for an edge anywhere") { + // The same kind of card with its edge elsewhere. + for (int edge = 300; edge <= 2900; edge += 150) { + Run run = reference_card(); + run.card.mem_edge = edge; + const auto r = run.go(Preset::BestOfMyGpu); + CAPTURE(edge); + REQUIRE(r.ok); + CHECK(run.card.max_mem_seen == edge + 50); + CHECK(run.card.max_again_mem == edge - 200); + CHECK(r.driver_resets == 0); + CHECK(r.mem_max_stable <= edge - 200); + } +} + +TEST_CASE("a reading of the second pass that loses the device is a reset the search recovers from") { + // A card that is worse than the margin allows for. The run goes on as + // after any reset in the memory search, and the offset is remembered. + Run run = reference_card(); + run.card.bw_again_loses_device_from = 2300; + const auto r = run.go(Preset::BestOfMyGpu); + REQUIRE(r.ok); + CHECK(r.driver_resets == 1); + CHECK(run.card.bw_again_measurements == 1); + CHECK(run.logged("mem +2300: the bandwidth measurement failed; treated as a driver reset")); + CHECK(r.mem_max_stable == 2300); + CHECK(r.mem_confirmed == 2300 - kResetBackoffSteps * 50); + CHECK(run.card.max_core_seen > 0); // the core is still searched + CHECK(run.card.count("\"mem_edge\":2300") == 1); + // The next run stays below that as well, with both copies. + const auto second = again(run); + REQUIRE(second.ok); + CHECK(second.driver_resets == 0); + CHECK(run.card.max_mem_seen == 2100); + CHECK(run.card.max_again_mem == 2050); + check_journal_rules(run); +} +