Skip to content

perf: build no SpellXSpellVisual hotfix records a client is never told about - #595

Merged
Xian55 merged 2 commits into
Xian55:masterfrom
hanming5:perf/skip-unadvertised-hotfix-records
Nov 19, 2026
Merged

Xian55 merged 2 commits into
Xian55:masterfrom
hanming5:perf/skip-unadvertised-hotfix-records

Conversation

@hanming5

Copy link
Copy Markdown

Part of #187.

What

From 3.4.3 on, SMSG_AVAILABLE_HOTFIXES advertises only AreaTrigger, ItemExtendedCost, CurrencyTypes and GlobalStrings. LoadSpellXSpellVisualHotfixes still turned every CSV row into a HotfixRecord that nothing could send. Its real output is the two lookups, SpellVisuals and SpellXSpellVisualToSpellVisual.

  • The advertised set moves out of WorldSocket.SendAvailableHotfixes into AdvertisedHotfixTables, so the packet and the loaders read the same decision.
  • LoadSpellXSpellVisualHotfixes still fills both lookups on every build, but only builds records when its table is advertised. When it doesn't, the record columns are not parsed either.
  • 1.14 and 2.5, which advertise everything they load, are unchanged.

Measured

GameData.LoadEverything(), Debug build, records left in GameData.Hotfixes and bytes allocated during the load:

client records allocated
3.4.3 / 3.3.5a 58,257 → 24,791 135 MB → 119 MB
4.4.2 / 4.3.4 63,765 → 16,826 166 MB → 143 MB
1.14.2 / 1.12.1 2,785 → 2,785 unchanged

A smaller store also makes FindHotfixesByRecordIdAndTable cheaper, since it walks every record and the item-query path calls it twice per hotfix.

Why ItemEffect is not in this PR

The issue lists ItemEffect as unreachable too, but on current master it isn't. LoadItemEffectHotfixes numbers its records from HotfixItemEffectBegin (2,500,000), which is inside the range UpdateHotfix searches from HotfixItemBegin (2,100,000). So a runtime item record with the same id overwrites the CSV record in place, and GenerateHotFixMessage sends ItemEffect records in SMSG_HOTFIX_MESSAGE without consulting the filter. Dropping them could change which records reach the client for item tooltips, so I left them alone and noted this in the AdvertisedHotfixTables remarks. The rest of #187 is still open because of this.

Tests

AdvertisedHotfixTablesTests (15 cases): the advertised set per build, and ParseSpellXSpellVisual against the shipped 1.14, 3.4.3 and 4.4.2 CSVs. With records off it yields identical lookups. With records on it yields one record per row with the same hotfix ids as before. Loading goes through a pure ParseSpellXSpellVisual, following ParseAreaTriggerProximity, so tests don't touch the global stores.

docs/opcode-coverage.md is regenerated: its one version-check site moved from WorldSocket.cs to AdvertisedHotfixTables.cs.

dotnet build: 0 warnings. dotnet test: 4,763 passed, 37 skipped, 0 failed.

Not play-tested in a client. The only behaviour change is to records no client is told about.

hanming5 and others added 2 commits November 19, 2026 23:12
…d about

From 3.4.3 on, SMSG_AVAILABLE_HOTFIXES advertises four tables, and
SpellXSpellVisual is not one of them. Its loader still turned every CSV
row into a HotfixRecord that nothing could send: 33,466 on 3.4.3 and
46,939 on 4.4.2. The two lookups it fills, SpellVisuals and
SpellXSpellVisualToSpellVisual, are what the proxy actually reads.

The advertised set moves out of WorldSocket into AdvertisedHotfixTables,
so the packet and the loaders read the same decision. The loader keeps
filling both lookups on every build and only builds records when its
table is advertised; 1.14 and 2.5 builds, which advertise everything
they load, are unchanged.

ItemEffect is not advertised either but stays loaded: its CSV records
get ids at HotfixItemEffectBegin, inside the range UpdateHotfix searches,
and GenerateHotFixMessage sends them without consulting the filter.

GameData.LoadEverything, records built / bytes allocated:
  3.4.3   58,257 -> 24,791   135 MB -> 119 MB
  4.4.2   63,765 -> 16,826   166 MB -> 143 MB

Part of Xian55#187.
GameData.SpellVisuals was assigned by two loaders running in parallel:
LoadSpellVisuals in LoadEverything (the SpellVisuals CSV) and the
SpellXSpellVisual hotfix loader inside LoadHotfixes, which copies the
field, lays its rows over the copy and assigns it back. Whichever
finished last won.

On 3.4.3 and 4.4.2 the two files hold the same map, so it never showed.
On 1.14 they differ: the hotfix file names three spells the base file
does not have. A Release build lost those three rows on every start
(30 of 30). A Debug test host ended with only the three rows, and
none of the 14,232 base rows, in 15 of 36 starts.

One job now reads both files in order and is the only writer of the
field. It stays among the hotfix jobs because it writes the hotfix
records. ReadSpellVisuals is the pure part, for tests.

After: 1.14 loads 14,235 rows in 30 of 30 Release starts and 36 of 36
Debug starts. Records, lookups and load time on 2.5, 3.4.3 and 4.4.2
are unchanged.

Co-Authored-By: Claude <noreply@anthropic.com>
@Xian55

Xian55 commented Nov 19, 2026

Copy link
Copy Markdown
Owner

Thanks for this. I reviewed and measured it locally and pushed one commit on top, cdc9b98. Here is why.

Your commit

It does what the description says. Against master, with the real GameData.LoadEverything():

client records allocated heap kept after load
3.4.3 / 3.3.5a 58,257 → 24,791 137.7 → 122.8 MB 102.7 → 89.3 MB
4.4.2 / 4.3.4 63,765 → 16,826 166.0 → 146.0 MB 125.5 → 107.0 MB
1.14.2 / 1.12.1 2,785 → 2,785 unchanged unchanged
2.5.3 / 2.4.3 85 → 85 unchanged unchanged

Every record outside SpellXSpellVisual is byte-identical, both lookups are identical, and the set of advertised hotfix ids is the same. In live sessions on a 3.4.3 and a 4.4.2 client, SMSG_AVAILABLE_HOTFIXES had the same length and ids as on master, and every spell cast in both sessions carried the same SpellXSpellVisualID.

What the extra commit fixes

While comparing the lookups I found a race that is already on master, in the function this PR touches. GameData.SpellVisuals had two writers running in parallel:

  • LoadSpellVisuals, a job in LoadEverything, reads SpellVisuals{N}.csv and assigns the field.
  • LoadSpellXSpellVisualHotfixes, a job in LoadHotfixes, copies the field, lays the hotfix rows over the copy and assigns it back.

Whichever finished last won. On 3.4.3 and 4.4.2 both files hold the same map, so it never shows. On 1.14 they differ: the hotfix file names three spells the base file does not have.

1.14 SpellVisuals after load master this PR before with cdc9b98
Release build, 30 starts each 14,232 rows, the 3 hotfix rows lost same 14,235, merged
Debug test host, 36 starts each merged 1, base only 20, hotfix rows only 15 merged 0, base only 6, hotfix rows only 30 merged 36

"Hotfix rows only" means a map of 3 entries, with none of the 14,232 base rows. I only saw that in the Debug test host, not in a Release build. Your change did not cause any of this, but making the hotfix loader faster moved the odds in that host, which is how it came up.

The fix: one job reads both files in order and is the only writer of the field. LoadSpellVisuals left the LoadEverything list and took the place of LoadSpellXSpellVisualHotfixes among the hotfix jobs, because it also writes the hotfix records. The ordered read is a pure ReadSpellVisuals, next to your ParseSpellXSpellVisual, with tests in SpellVisualsLoadTests. Those test the merge. They cannot fail on the old code for the race itself, since that needs the global stores, so the table above is the evidence for it.

Records, lookups and load time on 2.5.3, 3.4.3 and 4.4.2 are the same as with your commit alone. The suite is at 4,773 passed, 35 skipped, 0 failed.

One correction to the description

"1.14 and 2.5 are unchanged" now holds for the hotfix records only. On 1.14 the three rows of SpellXSpellVisual1.csv (spells 32950, 33389, 33392) take effect, where a Release build used to drop them. Live sessions on 1.14, 2.5 and 3.4.3 clients came through clean with both commits. The 4.4.2 session has one failing check that master fails the same way, so it is not from this PR.

@Xian55 Xian55 added performance Performance improvement ai Artificial Intelligence contribution labels Nov 19, 2026
@Xian55
Xian55 merged commit a6e8e87 into Xian55:master Nov 19, 2026
@Xian55

Xian55 commented Nov 19, 2026

Copy link
Copy Markdown
Owner

No release for this merge yet: GitHub has no macOS runner for us

The merge is on master (a6e8e87), but the release that normally follows it has not been published. The latest release is still v4.7.7. Nothing is wrong with the change itself.

What is failing. Both workflows the merge started stop at the same job, the macOS build:

workflow macOS build everything else
Build Proxy cancelled test, Windows, Ubuntu: success
Release cancelled test, version, Windows, Ubuntu x64, Ubuntu arm64: success

The macOS job never starts. It waits about 15 minutes for a hosted runner and GitHub then cancels it with:

The job was not acquired by Runner of type hosted even after multiple attempts

Due to capacity constraints, jobs targeting macOS arm64 runners may experience longer queue times.

In the Release workflow the verify job reports that missing build as a failure, and release and docker are skipped because of it. That is why there is no new version and no new container image.

What has been tried. The failed jobs have been rerun 10 times between 15:33 and 18:08 UTC on 2026-11-19, each with the same result. The run on the contributor's fork was cancelled the same way, so it is not tied to this repository. The workflows still ask for macos-latest, as they did when v4.7.7 was built on 2026-11-10. GitHub's status page lists no incident for Actions.

What happens next. The reruns continue. As soon as a macOS runner picks the job up, the Release workflow goes on by itself and publishes the version that contains this PR. No action is needed from anyone here.

@hanming5

Copy link
Copy Markdown
Author

Update: the release went out. On the 13th attempt of the Release run, a macOS runner picked up the job at 18:44 UTC on 2026-11-19, and the workflow then finished by itself. v4.7.8 is built from a6e8e87 and contains this PR, with all four platform builds and the container image. Nothing more is needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai Artificial Intelligence contribution performance Performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants