A cli rascal test runner for use from the maven plugin. by jurgenvinju · Pull Request #2755 · usethesource/rascal

jurgenvinju · 2026-04-13T14:47:43Z

This avoids having to write a Java class file with the JUnitTestRunner annotation, even in project that otherwise have no Java code. It also works for projects that do have Java dependencies.
This closes the abstraction for Rascal as a language (you don't have to understand anymore that it's an interpreted and/or compiled language or how the test runners have been organized (via JUnit or otherwise).
This makes sure that the generic surefire-report XML format is respected for integration with github actions and VScode reporting tools.

To use this add this to your pom.xml and make sure the <srcs> configuration for the rascal-maven-plugin matches what you want to test. You can also add a new <srcs> tag here with a different <configuration> under <execution>:

<execution>
     <id>yourproject-test</id>
     <phase>test</phase>
     <goals>
           <goal>test</goal>
     </goals>
</execution>

CLI command to call the test runner with the same parameters as calling the checker or the compiler or the exec runner
Optionally log in JUnit testsuite report format for picking up by CLI and CI tools
Add test runner statistics for each testsuite/module XML file
Documents the Surefire/JUnit test report XML format as a Rascal abstract grammar.

This does not (yet) work for compiled Rascal code, but the exact same interface could be implemented for generated test code (whether or not we use JUnit for that or not).

codecov · 2026-04-13T14:54:10Z

Codecov Report

❌ Patch coverage is 0% with 140 lines in your changes missing coverage. Please review.
✅ Project coverage is 46%. Comparing base (db87bc1) to head (834276c).

Files with missing lines	Patch %	Lines
...pl/test/infrastructure/JUnitXMLReportListener.java	0%	83 Missing ⚠️
src/org/rascalmpl/shell/RascalTest.java	0%	53 Missing ⚠️
...g/rascalmpl/interpreter/env/GlobalEnvironment.java	0%	3 Missing ⚠️
src/org/rascalmpl/shell/CommandlineParser.java	0%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##              main   #2755    +/-   ##
========================================
- Coverage       46%     46%    -1%     
- Complexity    6723    6727     +4     
========================================
  Files          839     841     +2     
  Lines        66766   66906   +140     
  Branches      9983    9998    +15     
========================================
+ Hits         30781   30796    +15     
- Misses       33598   33730   +132     
+ Partials      2387    2380     -7

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…in the IDE and CI setups

sonarqubecloud · 2026-04-16T12:16:10Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

DavyLandman · 2026-04-17T09:35:58Z

I like this feature 👍 I'll try to allocate time for a review next week.

sonarqubecloud · 2026-06-10T19:09:32Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

DavyLandman · 2026-06-12T08:35:03Z

+	public Map<String, ISourceLocation> moduleFiles() {
+		return moduleLocations.entrySet()
+			.stream()
+			.collect(Collectors.toMap(e -> e.getKey(), e -> URIUtil.assumeCorrectLocation(e.getValue().toASCIIString())));


side-quest: why are we using URIs in that moduleLocations map?

DavyLandman · 2026-06-12T08:36:01Z

        for (int i = 0; i < commandline.length; i++) {
            if (List.of("-help", "--help", "/?", "?", "\\?", "-?", "--?").contains(commandline[i].trim())) {
                printMainHelpMessage(kwTypes);
+                out.flush();


I think this flush should be moved to the printMainHelpMessage?

DavyLandman · 2026-06-12T08:38:43Z

+            RascalShell.setupJavaProcessForREPL();
+
+            var term = RascalShell.connectToTerminal();
+            var monitor = IRascalMonitor.buildConsoleMonitor(term);
+            var err = (monitor instanceof Writer) ?  StreamUtil.generateErrorStream(term, (Writer)monitor) : new PrintWriter(System.err, true);
+            var out = (monitor instanceof PrintWriter) ? (PrintWriter) monitor : new PrintWriter(System.out, false);


I remember surefire had issues jline capturing the output stream using native hooks. And that is why we had to avoid the progress bar feature inside the test.

If this has been tested on this PR, let's also test it on windows.

Can we add this RascalTest to the test suite we run on integration?

DavyLandman · 2026-06-12T08:42:39Z

+                PathConfig pcfg = pcfgCons != null ? new PathConfig(pcfgCons) : new PathConfig();
+                var projectRoot = pcfg.getProjectRoot().getScheme().equals("unknown") ? URIUtil.rootLocation("cwd") : pcfg.getProjectRoot();


I think we should reuse more code from ShellEvaluatorFactory. Some calculate the path config in a more robust way Like for example based on the cwd as a fallback etc.

DavyLandman · 2026-06-12T08:45:31Z

+the test runs of your own DSL, such that they can be integrated
+in Github CI run reports and other UX.
+}
+module lang::xml::\surefire-reports::TestSuites


this module exists, but then we also have JUnitXMLReportListener.java that is again recreating this structure, can we not reuse rascal for the reporting an constructing of the XML?

first versions saved from old branch

af6278b

this works

25ac221

jurgenvinju changed the title ~~first versions saved from old branch~~ A cli rascal test runner for use from the maven plugin. Apr 14, 2026

jurgenvinju marked this pull request as ready for review April 14, 2026 08:24

fixed issues

c7e6187

jurgenvinju self-assigned this Apr 14, 2026

jurgenvinju marked this pull request as draft April 14, 2026 10:20

jurgenvinju added 3 commits April 14, 2026 13:23

made an XML report listener

047b9ef

closed the loc VFS abstraction for the XML test report code

827e001

fixed final issues, adding testsuite/module level statistics for use …

e5baad3

…in the IDE and CI setups

jurgenvinju requested review from DavyLandman and PaulKlint April 16, 2026 12:00

added XML schema for test suites

8bf3c0a

Merge branch 'main' into feat/test-cli

834276c

DavyLandman reviewed Jun 12, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

A cli rascal test runner for use from the maven plugin. #2755

A cli rascal test runner for use from the maven plugin. #2755
jurgenvinju wants to merge 8 commits into
mainfrom
feat/test-cli

jurgenvinju commented Apr 13, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Apr 13, 2026 •

edited

Loading

Uh oh!

sonarqubecloud Bot commented Apr 16, 2026

Uh oh!

DavyLandman commented Apr 17, 2026 •

edited

Loading

Uh oh!

sonarqubecloud Bot commented Jun 10, 2026

Uh oh!

DavyLandman Jun 12, 2026

Uh oh!

DavyLandman Jun 12, 2026

Uh oh!

DavyLandman Jun 12, 2026

Uh oh!

DavyLandman Jun 12, 2026

Uh oh!

DavyLandman Jun 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		PathConfig pcfg = pcfgCons != null ? new PathConfig(pcfgCons) : new PathConfig();
		var projectRoot = pcfg.getProjectRoot().getScheme().equals("unknown") ? URIUtil.rootLocation("cwd") : pcfg.getProjectRoot();

Conversation

jurgenvinju commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sonarqubecloud Bot commented Apr 16, 2026

Quality Gate passed

Uh oh!

DavyLandman commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sonarqubecloud Bot commented Jun 10, 2026

Quality Gate passed

Uh oh!

DavyLandman Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

DavyLandman Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

DavyLandman Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

DavyLandman Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

DavyLandman Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jurgenvinju commented Apr 13, 2026 •

edited

Loading

codecov Bot commented Apr 13, 2026 •

edited

Loading

DavyLandman commented Apr 17, 2026 •

edited

Loading