perf(scope,export): reuse the model type resolver, class loader and genmodel lookups within a generation pass by joaodinissf · Pull Request #1547 · dsldevkit/dsl-devkit

joaodinissf · 2026-09-25T09:20:20Z

Part of #1545

Why the change

Generating the Java for a Scope or Export model repeated the same type resolution, class-loader setup and genmodel lookups for every expression; doing each once per generated resource cuts that work without changing the output.

Special things to note

The generated output is byte-identical (92 generated .java and ._trace files). On the affected Windows machine a full build allocates 7% less (20.1 → 18.7 GB); the wall-time gain is not claimed, because full builds varied between 103 and 141 s.
Memoized values live only while one resource is generated: a thread-local scope opened by GeneratorSupport.executeWithProjectResourceLoader and dropped in its finally block. Failed lookups (null) are not cached.
Format and Check don't use these helpers. Check has one similar uncached genmodel lookup, left for a separate change.

Change outline

What happens for each generated resource:

 JvmModelGenerator.doGenerate(resource)
+  GeneratorSupport.executeWithProjectResourceLoader(project)   # one loader and memo scope per resource
     for each method body and initializer
-      new URLClassLoader(full project classpath)
-      new Scope/ExportModelTypeResolver(model)                   # resolves every visible EPackage
-      GenModelUtilX.genPackage(ePackage)                         # global-scope query every time
+      reuse the resource's loader
+      Scope/ExportModelTypeResolver.forModel(model)              # memoized per model
+      GenModelUtilX.genPackage(ePackage)                         # memoized per (resource, EPackage)

Counted on 10 corpus models in one full build: resolvers built 103 → 6, EPackages resolved 7,210 → 420, class loaders 62 → 6, classpath entries resolved 18,296 → 1,769.

 com.avaloq.tools.ddk.xtext.expression/generator/
+  GeneratorSupport.java           # memo scope around executeWithProjectResourceLoader
   GenModelUtilX.java              # genPackage memoized
 com.avaloq.tools.ddk.xtext.scope/generator/
+  ScopeJvmModelGenerator.java     # opens the scope per generated resource (bound in ScopeRuntimeModule)
   ScopeModelTypeResolver.java     # forModel
 com.avaloq.tools.ddk.xtext.export/generator/
+  ExportJvmModelGenerator.java
   ExportModelTypeResolver.java

Tests: GeneratorSupportTest (scope lifetime, reuse, exceptions) and JvmModelGeneratorBindingTest.

🤖 Generated with Claude Code

… pass ScopeExpressionTranslator and ExportExpressionTranslator built a new Scope/ExportModelTypeResolver for every compilation context, i.e. for nearly every expression rendered into a generated method body. Each construction queries the IMPORT__PACKAGE scope and EcoreUtil.resolves every EPackage visible to the model (the EMF registry plus every EPackage in the index), so the cost was O(visible EPackages) per expression. The pre-dsldevkit#1405 generator built one CompilationContext per generated file. GeneratorSupport gains memoize(key, supplier): values memoized by the innermost executeWithProjectResourceLoader call on the current thread, discarded when that call returns; outside of such a call nothing is memoized. The translators memoize the resolver under a key holding the model. Equivalence: a resolver is immutable once constructed and is fully determined by its model and by the EPackages visible to that model, i.e. by the index and the resource set. Neither changes while one body is rendered, and EcoreUtil.resolve returns the same (already loaded) instances when it runs again, so a resolver built by the first compilation context is equal to the one each later context would have built. Lifetime: the memoized values live on a ThreadLocal of the executeWithProjectResourceLoader call and are dropped in its finally block. Today every method body is rendered inside its own such call, so a resolver is shared by the compilation contexts of one body and never outlives it - never across resources or builds, where the index and the classpath may change. Compilation contexts created outside such a call (tests, inference) still build a fresh resolver each time. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…esource Since the Scope and Export inferrers render their method bodies lazily during emission (4424094 and the export equivalent), every body and field initializer goes through its own GeneratorSupport.executeWithProjectResourceLoader call, which resolves the project's full classpath and builds a new URLClassLoader - once per body instead of once per file as before. executeWithProjectResourceLoader is now re-entrant: when the current thread already executes within a call for the same project and that call's resource loader is still the installed one, the operation runs directly within it, sharing its resource loader and memoized values. The new ScopeJvmModelGenerator (bound in ScopeRuntimeModule) and ExportJvmModelGenerator wrap JvmModelGenerator.doGenerate(Resource, ..) in one such call for the resource's project, so the bodies rendered while the resource is emitted reuse that loader. A resource without any inferred type (e.g. a header-less export model whose grammar cannot be loaded) renders no body, so it is generated without the wrapper and still builds no class loader at all. Equivalence: the nested call would have installed a loader built from the same project's resolved classpath (GeneratorSupport.projectOf is the inferrers' projectOf applied to the same resource URI), and a generation pass does not change the classpath. No DDK code reads the thread's resource loader outside of these calls, so installing it for the whole of doGenerate rather than for each body does not change any other emitted text. Inference is untouched: bodies are still rendered only during emission, preserving the intent of 4424094. Callers that emit without doGenerate (e.g. JvmModelGenerator.generateType in tests) keep the previous behaviour of one loader per body. Lifetime: the loader and the memoized values (see the previous commit) live for one doGenerate call of one resource and are closed/dropped in its finally block - never across resources or builds. Memoized model type resolvers are therefore now shared by all bodies of the resource. GeneratorSupportTest covers memoize inside and outside of a resource loader call, that the values are dropped when the call returns, and that a nested call for the same project shares them. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…tion pass GenModelUtilX.genPackage - behind qualifiedPackageInterfaceName, literalIdentifier, instanceClassName, genClass and genDataType - queried the global GenPackage scope and EcoreUtil.resolved the result on every call, falling back to an index query and to a scan over all indexed GenPackages. The methods were marked "/*cached*/" in the Xtend source, a leftover of Xpand's "cached" keyword; neither the Xtend nor the Java version ever cached anything, so a generated file repeated the lookup for every literal and instance class name it contains. genPackage now memoizes the lookup through GeneratorSupport.memoize under a key of the context resource and the EPackage, whenever a context resource is set; without one (the lookup then depends on the element's own resource set) it is not memoized. Equivalence: with a context resource, the lookup reads only that resource, its resource set and the index, and uses the element only to find its EPackage. Within one generation pass the index and the resource set's content do not change; the only side effect, EcoreUtil.resolve loading a .genmodel into the context's resource set, makes a repeated lookup return the very instance the first one returned. The context is part of the key, so switching the context during the pass cannot serve a GenPackage looked up for another resource. A failing lookup is not memoized and fails again, as before. Lifetime: GeneratorSupport.memoize only memoizes within an executeWithProjectResourceLoader call, i.e. within one doGenerate of one resource (previous commit), and drops the values when it returns - never across resources or builds. Lookups outside of such a call or without a context resource are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

rubenporras · 2026-09-25T09:37:45Z

      runnable.run();
    } finally {
+      if (enclosingScope == null) {
+        CURRENT_SCOPE.remove();


I wonder, since we already depend on Guava, does Guava not offer us a good fit for this use case without rolling our own cache?

Good question, I looked into it. Most of this code is the scope handling (a thread-local scope for one generation pass, reused when calls nest, dropped in finally), which Guava doesn't cover; the memo table itself is ~6 lines. Guava's Cache turns out to fit that part poorly: it rejects null (a missing GenPackage is a valid, memoizable result here), and it wraps loader exceptions in (Unchecked)ExecutionException, which would change the errors generation reports. computeIfAbsent has the same null problem, and it fails on re-entrant calls. Happy to switch to Cache with Optional values if you prefer the library type, though.

joaodinissf · 2026-09-25T18:30:29Z

@rubenporras: Windows results for this change on its own (master vs master + this PR, 4 GB heap, 3 rounds):

Full build: allocation 20.1 → 18.7 GB (−7%), GC 1.49 → 1.32 s; wall time 122 → 105 s median, but full builds varied 103–141 s between rounds, so I'm not claiming the wall-time gain.
Branch switches barely change (allocation −2%), as expected: a Java-only switch rarely regenerates .scope/.export.
Generated output byte-identical; 370 tests green.

Marking this ready for review; I'll answer the Guava question in its thread.

🤖 Generated with Claude Code

- Do not memoize null values: a GenPackage lookup that finds nothing may succeed later in the same pass, once more GenModels are loaded. - Make GeneratorSupport.memoize static and drop the injected instances in GenModelUtilX and the expression translators. - Memoize the model type resolvers in their own factories (forModel), so ExportExpressionTranslator keeps using ExportModelTypeResolver.forElement. - Correct the lifetime Javadoc of memoize and CURRENT_SCOPE, and the GenModelUtilX context comment (the utility is not a singleton). - Test the Scope/Export generator bindings, a nested call with a replaced resource loader, and that null values are not memoized. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LgVRfQCTxSDwGrN8J5s6fd

joaodinissf · 2026-09-26T00:16:26Z

Pushed 34a4f40 with fixes from a self-review:

memoize no longer caches null. A GenPackage lookup that finds nothing can succeed later in the same pass (its last fallback only searches GenModels already loaded), so a cached miss could have diverged from master.
memoize is static; GenModelUtilX and the translators no longer inject GeneratorSupport.
The resolvers memoize in their own factories (forModel), so ExportExpressionTranslator is back to master's forElement call.
Javadoc fixes: the memo lifetime, and GenModelUtilX isn't a singleton.
New tests: the Scope/Export generator bindings, a nested call with a replaced resource loader, and null not being memoized.

Full reactor verify is green (374 tests, 0 Checkstyle violations).

🤖 Generated with Claude Code

rubenporras · 2026-09-28T06:51:34Z

Do Format and Check not have the same problem that could be fixed in this PR as well?

joaodinissf and others added 3 commits September 24, 2026 21:02

joaodinissf mentioned this pull request Sep 25, 2026

Builds of large DDK-based workspaces are slower and retain more memory since #1405 #1545

Open

rubenporras reviewed Sep 25, 2026

View reviewed changes

joaodinissf marked this pull request as ready for review September 25, 2026 18:30

joaodinissf mentioned this pull request Sep 26, 2026

perf(export): ignore Java-only changes when deciding whether an export model is affected #1548

Closed

Navigation Menu

perf(scope,export): reuse the model type resolver, class loader and genmodel lookups within a generation pass - #1547

perf(scope,export): reuse the model type resolver, class loader and genmodel lookups within a generation pass#1547
joaodinissf wants to merge 4 commits into
dsldevkit:masterfrom
joaodinissf:perf/scope-export-generation-reuse

joaodinissf commented Sep 25, 2026 •

edited

Loading

Uh oh!

rubenporras Sep 25, 2026

Uh oh!

joaodinissf Sep 25, 2026

Uh oh!

joaodinissf commented Sep 25, 2026

Uh oh!

joaodinissf commented Sep 26, 2026

Uh oh!

rubenporras commented Sep 28, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

joaodinissf commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why the change

Special things to note

Change outline

Uh oh!

rubenporras Sep 25, 2026

Choose a reason for hiding this comment

Uh oh!

joaodinissf Sep 25, 2026

Choose a reason for hiding this comment

Uh oh!

joaodinissf commented Sep 25, 2026

Uh oh!

joaodinissf commented Sep 26, 2026

Uh oh!

rubenporras commented Sep 28, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

joaodinissf commented Sep 25, 2026 •

edited

Loading