3.6.0 regression: forked JVM silently vanishes on Windows for tests using FFM upcalls from background threads

[dfa1]

Affected version

3.6.0 (regression). Bisected to commit
90529e9
("Use StackWalker in StackTraceProvider when available (Java 9+)", #3374).
Last known good: 3.6.0-M1 and everything before that commit.

Environment

• OS: windows-latest GitHub Actions runner (Windows Server 2022/2025)
• JDK: Zulu 25.0.4-7, x64
• Build: Maven, reuseForks=false, argLine includes --enable-native-access=ALL-UNNAMED
• Project: https://github.com/dfa1/rocksdb-ffm (Java FFM bindings for RocksDB's C API)

Bug description

Bumping maven-surefire-plugin from 3.5.6 to 3.6.0 makes three specific test
classes deterministically kill their forked JVM on windows-latest only.
macOS and Linux (x86_64 and aarch64) are unaffected. The same commit set with
surefire pinned at 3.5.6 (or at 3.6.0-M1) passes on Windows every time.

[ERROR] ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Crashed tests:
  io.github.dfa1.rocksdbffm.CompactionFilterFactoryTest
  io.github.dfa1.rocksdbffm.CompactionFilterTest
  io.github.dfa1.rocksdbffm.EventNotifierTest

This reproduced identically across many independent CI runs — always exactly
these three classes, never a different subset.

What we ruled out

• Not a classic native crash: no hs_err_pid*.log is ever produced,
  despite surefire's own message suggesting a VM crash. If HotSpot's signal
  handler had caught a fault normally, that file would exist.
• Not the "Boot Manifest-JAR" classpath warning: every single fork
  (150+, since reuseForks=false) logs
  Boot Manifest-JAR contains absolute paths in classpath '...', 'other' has different root to the .dumpstream file (a Path.relativize() failure
  across Windows drive roots, falling back to absolute paths in the manifest
  jar). This happens for both passing and crashing forks alike — harmless
  noise, unrelated to the real bug.
• Not simply "any test with a native upcall crashes": other tests in the
  same suite that register FFM upcalls but only ever invoke them
  synchronously on the JVM's own calling thread (e.g. a custom merge
  operator callback, or a test that calls System.exit from a JVM-owned
  thread) pass without issue.

Bisection

We built surefire from source at each candidate commit between
surefire-3.6.0-M1 and surefire-3.6.0 (54 commits) and ran our suite
against each, narrowing by binary search:

Round	Commit (index in the 54)	Result
baseline	3.6.0-M1	pass
1	37f0725 (26)	crash
2	381c404 (12)	crash
3	c633042 (5)	pass
4	90529e9 (8)	crash
5	e759c64 (6)	pass
6 (final)	523e1ce (7)	pass

Index 7 passing and index 8 crashing pins the boundary exactly: commit
90529e9 is the first bad commit in the range.

The regression is isolated to that single commit,
90529e9
/ PR #3374, which
switches StackTraceProvider.getStack() from Thread.currentThread().getStackTrace()
to java.lang.StackWalker.getInstance(options, depth).walk(...) (falling back
to the old path only if the StackWalker call throws).

Suspected mechanism

CompactionFilter/CompactionFilterFactory/EventNotifier register a
native → Java callback that RocksDB invokes from its own background threads
(the compaction thread pool, and flush/compaction event-listener threads) —
not threads the JVM created. The callback path is a JDK 25
Linker.upcallStub() entry point, which the JVM implements as a
generated hidden class/MethodHandle trampoline. The first time such a
background thread fires the upcall, the JVM attaches that foreign thread,
and from the JVM's point of view its call stack starts at that synthetic
hidden-class frame rather than a normal Thread.run()-rooted chain.

Thread.getStackTrace() (still used as the fallback in the patched code)
tolerates that stack shape fine — this project's whole suite passed under
every surefire version before this commit. StackWalker, by contrast,
does extra classification work on each frame specifically to hide
reflection/lambda frames (per the PR's own code comment), which means it's
doing more invasive introspection of frame metadata for exactly the kind of
synthetic/hidden frame an FFM upcall trampoline produces. We suspect that
introspection is hitting a rough edge in HotSpot's stack-walking internals
for hidden-class/MethodHandle frames belonging to a JVM-attached (not
JVM-created) thread, causing a hard, in-process JVM fault severe enough that
the normal crash-dump path (hs_err_pid*.log) never runs.

Reproduction

Not yet minimized to a surefire-only reproducer (would require a small
native library exercising a JDK 25 Linker.upcallStub invoked from a
non-JVM-created thread, run under StackTraceProvider.getStack()'s new
path). Happy to attempt one if useful — our current reproduction is the
full dfa1/rocksdb-ffm CI matrix.

Happy to run additional diagnostics on our CI if that helps narrow this
down further.

[dfa1]

Confirmed empirically: applying stackTraceMaxFrames=0 to the real, published 3.6.0 release (not just the source-built bisection commits) makes all 4 platforms in our CI matrix pass, including windows-latest with the three previously-crashing test classes (CompactionFilterFactoryTest, CompactionFilterTest, EventNotifierTest) all green.

<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>3.6.0</version>
  <configuration>
    <stackTraceMaxFrames>0</stackTraceMaxFrames>
  </configuration>
</plugin>

We've merged this into our project (dfa1/rocksdb-ffm#163) as the interim mitigation rather than pinning back to 3.5.6.

[olamy]

Sounds more like an issue with the JDK not supporting a similar feature with StackWalker than a standard stack trace mechanism.
Well, for the standard use case, StackWalker is just so so so faster.
Maybe a way to disable StackWalker depending on a system property you could activate via argLine?
What do you think?

[dfa1]

yes, looks like a JDK issue as it is not working only on Windows. There is a clear workaround in the meantime.