Skip to content

Improve performance with large CodeSystems - #1637

Open
vadi2 wants to merge 4 commits into
FHIR:masterfrom
vadi2:large-codesystem-performance
Open

Improve performance with large CodeSystems#1637
vadi2 wants to merge 4 commits into
FHIR:masterfrom
vadi2:large-codesystem-performance

Conversation

@vadi2

@vadi2 vadi2 commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Description:

SUSHI slows down sharply as a CodeSystem grows. Profiling a 9,347-concept CodeSystem (about 56,000 code caret rules for designations and properties) showed two costs that grow with the number of rules, so this PR fixes both:

  1. Rebuilding the StructureDefinition for every caret rule. setPropertyOnDefinitionInstance called instance.getOwnStructureDefinition(fisher) (a package lookup plus StructureDefinition.fromJSON) on each call. For a CodeSystem that is once per code caret rule, and it was 83% of the build. The function now takes the StructureDefinition as an optional final parameter (default unchanged), and CodeSystemExporter and ValueSetExporter pass in the one they already built. Sharing one StructureDefinition across an entity's rules has precedent: the validation pass in these exporters already did it, and InstanceExporter reuses cached StructureDefinitions across all of an instance's rules. In the same code path, findConceptPath and setConcepts index concepts by code instead of scanning the concept list for every rule, and setPropertyOnInstance fills a large array from its current end instead of from index 0 (the old loop re-read the array length each iteration, and each iteration pushed one element, so starting at the current length adds exactly the same elements).

  2. Scanning every rule to find a definition's id, url, name, or version. findAssignmentByPath did a findLast over all rules, and FSHTank.fish does that for every entity it checks on every fish. In an IG with several large CodeSystems, every lookup of a Reference or code system walked tens of thousands of rules, which was 63% of what remained after fix 1. The found rule is now cached per rules array. Rules arrays are only appended to or replaced wholesale (as applyInsertRules does), so the cache is invalidated when the array is replaced, when its length changes, or when its last rule changes. Because the rule rather than its value is cached, changes to a rule's value are still seen, and a cached rule is re-checked against the lookup on each hit because CodeSystemExporter rewrites a code caret rule's path in place once it resolves the concept.

Measurements (Node 22, sushi on the same machine):

Build master fix 1 fix 1 + 2
One CodeSystem with 9,347 concepts, FSHOnly 147 s, 750 MB 15 s, 594 MB 8 s, 598 MB
IG with 32 large CodeSystems written in FSH not measured (many minutes) 142 s 52 s

For comparison, the same IG with those 32 CodeSystems pre-rendered to JSON in input/ (the workaround the project uses today) builds in 31 s. Most of the remaining gap is ANTLR parsing of the large FSH files.

Output is unchanged: the generated JSON for the 9,347-concept CodeSystem is byte-identical to master, and every other resource in the IG is identical to the pre-rendered build.

Developed with AI assistance (Claude Code). The diff, tests, and measurements were reviewed before submitting.

Testing Instructions:

Run npm run check. New tests in test/fshtypes/FshCodeSystem.test.ts, test/fshtypes/Instance.test.ts and test/import/FSHTank.test.ts cover the cache being invalidated when rules are added, removed, replaced, their values change, or a code caret rule's path is resolved. New tests in test/export/CodeSystemExporter.test.ts and test/export/ValueSetExporter.test.ts pin the output when several caret rules target the same extension slice and a numerically indexed extension, since the shared StructureDefinition now carries the slices added by earlier rules.

To see the performance difference, run sushi on a project containing a CodeSystem with a few thousand concepts that each have a designation or property.

Related Issue:

None.

https://claude.ai/code/session_01AKNbxASp6vABedSLJMGW97

setPropertyOnDefinitionInstance rebuilt the instance's StructureDefinition
for every rule. A CodeSystem with thousands of concepts has tens of
thousands of code caret rules, so that dominated the build. The exporters
now pass in the StructureDefinition they already built, concepts are
indexed by code instead of scanned, and filling a large array no longer
restarts from index 0.

Claude-Session: https://claude.ai/code/session_01AKNbxASp6vABedSLJMGW97
findAssignmentByPath scanned every rule on a definition, and the tank does
that for every entity on every fish. Cache the result per rules array,
invalidating when rules are added or removed or the array is replaced.

Claude-Session: https://claude.ai/code/session_01AKNbxASp6vABedSLJMGW97
CodeSystemExporter rewrites a code caret rule's path in place, so a cached
rule may stop matching its lookup. Re-check it on a hit and rescan if it no
longer matches. Also clarify comments and add tests for the Instance branch
and for reusing the StructureDefinition across extension caret rules.

Claude-Session: https://claude.ai/code/session_01AKNbxASp6vABedSLJMGW97
Rename the cache for what it holds, return early on a cache hit, keep the
original conceptIndices local in findConceptPath, and flatten the array
fill branches. No behavior change.

Claude-Session: https://claude.ai/code/session_01AKNbxASp6vABedSLJMGW97
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant