Repository navigation
perf: add benchmark scenarios for Set reads, Map iteration, moved rows and unchanged collections - #198
Merged
Conversation
…nd unchanged collections
…ow and unchanged collection scenarios
|
Coverage after merging perf/micro-benchmark-scenarios into main will be
Coverage Report
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #168.
#183 and #184 improved four paths that only micro-benchmarks measured, so neither the benchmark suite nor the CI budgets would notice if one of them regressed. This PR adds a scenario for each, budgets the new scenarios in CI, and adds their measurements to the published results, so that the rerun for the release covers them.
set-readlooks up the middle number and a missing number in a Set of--array-sizenumbers and reads its size, without changing the Set. It covers the lazy item map of Set drafts from perf: frozen returns, Set reads, frozen array copies and moved-element edits, plus a v1 to v2 migration guide #183: Mutative 1.3.0 mapped every item as soon as the recipe read the Set, and copied the Set when a lookup missed.map-forEachsums a Map of numbers withforEach(). It covers the direct entry reads of Map drafts from fix: patches of moved drafts, Map and Set freezing and iterators, and other review findings #184: before them, every entry calledget()through the draft's proxy.shift-and-updateremoves the first row withshift()and updates the middle row, which drafts a moved row. It covers the search of the original array for the first moved elements from perf: frozen returns, Set reads, frozen array copies and moved-element edits, plus a v1 to v2 migration guide #183: before it, the first lookup built a map of all original indices.map-set-besideinserts a row into a Map of rows and a number into a Set of numbers, which copies both, and then updates a value beside them in nine more producers. With auto-freeze it covers the freezing of Map and Set instances from fix: patches of moved drafts, Map and Set freezing and iterators, and other review findings #184: before it, the instances were never frozen, so every later producer walked both copies again. The input is pre-frozen like any other, so the copies are the only collections that the library freezes itself.shift-and-updatejoinscore. The Map and Set scenarios move fromcollectionsintomapsandsets, two jobs instead of one: with the three new scenarios, thecollectionsjob would have taken an estimated 10–11 minutes instead of 8. The five jobs take 4–8 minutes each.shift-and-updatealso with Immer's array-method plugin. Their cells join the published results, layered over the batch of October 6 and 7 as the Set and patch application cells were for perf: change the copy of a Set draft directly, and keep the changes of a Set draft that left its key #195 and perf: read the type of a draft from its state when applying patches #196. The README, the website andperf-testing/reports/SUMMARY.mdnow cover 97 workloads, and the summary has a section for the new scenarios.The published figures change
Cells faster / within 5% / slower, and the geometric mean of comparator time over Mutative time. The headline figures become 3.6x (was 3.7x) and 6.7x (was 6.8x): in
set-read, Mutative and Immer both read the original Set (0.6–0.7 µs at every size, 4 of 12 cells within 5%), andmap-forEachis 1.1–1.3 times as fast as Immer. None of the new cells is slower than Immer.The new scenarios
Microseconds per scenario at 1,000 rows, patches off, medians of three isolated runs on an Apple M1 Max with Node.js 24.16.0:
At 10,000 rows,
set-readtook 0.62 µs against 558 µs for Mutative 1.3.0, andshift-and-update15.9 µs against 8,768 µs for Immer. With Immer's array-method plugin, Mutative was faster in everyshift-and-updatecell, 1.1–24 times.Each scenario catches the regression it covers
Mutative alone, 1,000 rows, three rounds in alternating order, the commit that made each improvement against the commit before it:
A budget fails a cell at 1.30 times.
Calibration on GitHub runners
Before budgeting the new scenarios,
--self-controlcompared identical builds on 30 GitHub runners, 10 for each ofcore,mapsandsets, as for #194. All 30 runs pass, and all 1,160 decisions:mutation-density-100pct), 1.084 for allocation (array-reverse-nestedwith auto-freeze, 4.3 KiB, below the 8 KiB floor) and 1.003 for retained heap.map-forEachwith patches, whose processes in one run differed by up to 1.29 times. The fastest-process statistic of perf: budget the scenarios added in #177 and #180, and compare the fastest processes of each build #194 absorbs this. The medians of paired ratios that the gate used before reached 1.303 onshift-and-update.core, 6.6–7.8 formapsand 4.3–5.2 forsets.Found while measuring
With auto-freeze,
shift-and-updatetook 301 µs at 10,000 rows, against 15.9 µs without, and 189 µs forarray-shift-nested, which only removes the row. The moved row's original index is searched withlastIndexOfon the original array, which is frozen then. V8 runslastIndexOfon a frozen array about 15 times as slowly: 116 µs against 7.9 µs to find the middle element of 10,000.indexOftakes 1.1 µs on both. Searching forward withindexOf, then again from after each hit until there is none, would find the same last index. This PR does not change the runtime; the summary lists the cost under its limits.Checks
node perf-testing/run-benchmarks.mjs --listlists 97 scenarios.--patches bothand bounded fixtures, and 18 forapply-*. Each measured process also validated its scenario at 100, 1,000 and 10,000 rows.pnpm test:benchmarks: 35 tests pass.pnpm lint,pnpm format --check,oxfmt --checkofperf-testing.src/changes, so the production artifacts are unchanged (bd8ff3a6…, the same as perf: read the type of a draft from its state when applying patches #196's).Commits
perf: add a scenario that reads a Set without changing itperf: add a scenario that iterates a Map of numbers with forEach()perf: add a scenario that updates a row that shift() movedperf: add a scenario of producers beside a Map and a Set that they leave unchangedci: run the Map and the Set budgets in jobs of their ownperf: budget the scenarios for Set reads, Map iteration, moved rows and unchanged collectionsperf: record the measurements of the Set read, Map iteration, moved row and unchanged collection scenarios