Add Positron API integration tests and CI workflow by jonvanausdeln · Pull Request #4298 · posit-dev/publisher

jonvanausdeln · 2026-07-21T16:31:32Z

Summary

Adds a Positron-only integration test suite that runs the Publisher extension inside a real Positron build via @posit-dev/positron-test-electron, plus a CI workflow that runs it on every PR and push to main.

Part of the rollout tracked in posit-dev/positron#14531 (pattern established in quarto-dev/quarto#1058).

What's covered

Publisher's real Positron-API surface is interpreter discovery (src/utils/vscode.ts): acquirePositronApi() feature detection plus positron.runtime.getPreferredRuntime() with VSCode fallbacks. The starter tests exercise exactly that against the live API — paths unreachable by the plain VSCode suite or the mock-based contract tests:

extension.test.ts — Positron injects the acquirePositronApi global, the API reports a version, and Publisher activates in the Positron extension host.
interpreter-discovery.test.ts — Publisher's getPythonInterpreterPath() / getRInterpreterPath() resolve the same interpreter Positron reports as its preferred runtime, covering API acquisition, the retry loop, and ~/ runtime-path expansion (Provide 2 distinct fields for runtime path: one full path, one for display positron#12942). The tests wait for Positron's async runtime discovery before asserting, so they measure Publisher's behavior rather than discovery timing.

Harness

scripts/run-positron-tests.mjs — downloads/caches a Positron build (stable by default, POSITRON_CHANNEL=daily supported) and runs the Mocha entry point (src/test/positron/index.ts) inside it. Bundled extensions stay enabled because runtime discovery is provided by the bundled Python and Ark extensions.
Run locally with npm run test-positron (macOS only for now — an upstream positron-test-electron limitation; Windows/Linux follow once extractable archives are published).
esbuild.tests.mjs / .vscode-test.mjs — the plain suite (out/test/suite/) and Positron suite (out/test/positron/) are now separated so npm test doesn't pick up Positron-only tests.
src/test/positron/README.md documents how to run and add tests.

CI

.github/workflows/positron-api-tests.yaml (macos-latest, Python 3.12 + R 4.4 for runtime discovery, date-keyed cache of the Positron download, launched through posit-dev/setup-positron), wired into pull-request.yaml and main.yaml.

Drive-by fix

Raises the scanPythonDependencies (real Python) integration-test timeout from 30s to 120s, matching the scanRPackages block above it — the real-Python scan takes ~25s on its own and flakes under parallel suite load.

Follow-up (not in this PR)

Migrating from the hand-rolled src/@types/positron.d.ts to the published @posit-dev/positron package is tracked in #3647.

Testing

npm test (vanilla VSCode suite) passes with the new output layout
tsc, ESLint, Prettier, and the full Vitest unit suite pass

posit-connect-projects · 2026-07-21T17:30:27Z

Claude finished @jonvanausdeln's task in 1m 13s —— View job

Code Review Complete

Read and analyze the changed files
Review CI workflow configuration
Review test harness script
Review test implementation
Review build configuration changes
Compile review feedback

This is a well-structured PR that adds valuable integration tests for Publisher's Positron API surface. The implementation follows established patterns and the test coverage targets the right code paths that weren't previously testable.

Highlights

Strong test design — The tests exercise the real API boundary (acquirePositronApi, getPreferredRuntime) rather than mocking, which gives much higher confidence that Publisher works correctly inside Positron. The waitForPreferredRuntime helper is a good pattern for handling async runtime discovery.

Clean separation — Keeping Positron tests in src/test/positron/ and updating esbuild.tests.mjs / .vscode-test.mjs to maintain separate output directories is the right approach. This prevents the plain VSCode suite from picking up Positron-only tests.

Good CI integration — The workflow uses date-keyed caching for the Positron download, properly gates on detect-changes, and is included in both PR and main workflows.

Minor Suggestions

interpreter-discovery.test.ts:36 — Infinite loop pattern

The for (;;) loop in waitForPreferredRuntime relies on the Date.now() < deadline assertion to break. While this works, an explicit while (Date.now() < deadline) would make the exit condition clearer at a glance. Not blocking, just a readability consideration.
interpreter-discovery.test.ts:17 — Import path style
```
import { getPythonInterpreterPath, getRInterpreterPath } from "src/utils/vscode";
```
This uses a non-relative path (src/utils/vscode), which works due to the tsconfig paths mapping. Other test files in the codebase use relative paths. Consistency would help, but this is minor since esbuild resolves it correctly.
run-positron-tests.mjs:53 — disableExtensions: false comment

The inline comment explains why extensions are left enabled, which is good. Consider also noting that this means the test run will be slower than the VSCode suite due to loading bundled extensions.
positron-api-tests.yaml:33-34 — Cache key brittleness

The cache key uses positron-${{ env.POSITRON_CHANNEL }}-${{ steps.get-date.outputs.date }}, which creates a new cache entry daily. If Positron releases mid-day, the cache will serve a stale build until the next day. This is probably fine since stable releases are infrequent, but worth being aware of.
integration.test.ts:723 — Timeout increase rationale

The PR description explains the timeout increase from 30s to 120s, but a brief inline comment in the code would help future maintainers understand why this block needs a higher timeout than others.

Questions

macOS-only limitation: The README and workflow comments mention this is macOS-only due to positron-test-electron limitations. Is there a tracking issue for when Windows/Linux support becomes available? Would be good to add a TODO or link for when to revisit this.

Verdict

LGTM — This is solid work that fills an important gap in test coverage. The suggestions above are all minor polish items that shouldn't block merging.

dotNomad

Had a few comments / questions for you including this note about a stack trace in the logs:

Looking at the Run Positron API Tests run in the CI I'm seeing this:

Server: Start requested.
Server State: Starting
Server State: Running
rejected promise not handled within 1 second: Error: GitHubLoginFailed
stack trace: Error: GitHubLoginFailed
	at Ere._authShowWarnings (/Users/runner/work/publisher/publisher/extensions/vscode/.positron-test/stable/2026.07.1-5/darwin-arm64/Positron.app/Contents/Resources/app/extensions/copilot/dist/extension.js:1697:18535)
	at async Ere.getCopilotToken (/Users/runner/work/publisher/publisher/extensions/vscode/.positron-test/stable/2026.07.1-5/darwin-arm64/Positron.app/Contents/Resources/app/extensions/copilot/dist/extension.js:1697:16253)
	at async Ege.getCopilotToken (/Users/runner/work/publisher/publisher/extensions/vscode/.positron-test/stable/2026.07.1-5/darwin-arm64/Positron.app/Contents/Resources/app/extensions/copilot/dist/extension.js:1400:7787)
	at async Jde.updateCachedToken (/Users/runner/work/publisher/publisher/extensions/vscode/.positron-test/stable/2026.07.1-5/darwin-arm64/Positron.app/Contents/Resources/app/extensions/copilot/dist/extension.js:2954:8746)
An unknown error occurred. Please consult the log for more details.
rejected promise not handled within 1 second: Error: GitHubLoginFailed
stack trace: Error: GitHubLoginFailed
	at Ere._authShowWarnings (/Users/runner/work/publisher/publisher/extensions/vscode/.positron-test/stable/2026.07.1-5/darwin-arm64/Positron.app/Contents/Resources/app/extensions/copilot/dist/extension.js:1697:18535)
	at async Ere.getCopilotToken (/Users/runner/work/publisher/publisher/extensions/vscode/.positron-test/stable/2026.07.1-5/darwin-arm64/Positron.app/Contents/Resources/app/extensions/copilot/dist/extension.js:1697:16253)
	at async Ege.getCopilotToken (/Users/runner/work/publisher/publisher/extensions/vscode/.positron-test/stable/2026.07.1-5/darwin-arm64/Positron.app/Contents/Resources/app/extensions/copilot/dist/extension.js:1400:7787)
	at async NS._fetchModels (/Users/runner/work/publisher/publisher/extensions/vscode/.positron-test/stable/2026.07.1-5/darwin-arm64/Positron.app/Contents/Resources/app/extensions/copilot/dist/extension.js:6787:4399)
rejected promise not handled within 1 second: Error: GitHubLoginFailed
stack trace: Error: GitHubLoginFailed
	at Ere._authShowWarnings (/Users/runner/work/publisher/publisher/extensions/vscode/.positron-test/stable/2026.07.1-5/darwin-arm64/Positron.app/Contents/Resources/app/extensions/copilot/dist/extension.js:1697:18535)
	at async Ere.getCopilotToken (/Users/runner/work/publisher/publisher/extensions/vscode/.positron-test/stable/2026.07.1-5/darwin-arm64/Positron.app/Contents/Resources/app/extensions/copilot/dist/extension.js:1697:16253)
	at async Ege.getCopilotToken (/Users/runner/work/publisher/publisher/extensions/vscode/.positron-test/stable/2026.07.1-5/darwin-arm64/Positron.app/Contents/Resources/app/extensions/copilot/dist/extension.js:1400:7787)
	at async Jde.updateCachedToken (/Users/runner/work/publisher/publisher/extensions/vscode/.positron-test/stable/2026.07.1-5/darwin-arm64/Positron.app/Contents/Resources/app/extensions/copilot/dist/extension.js:2954:8746)

Looks like this an issue with the Copilot extension inside of Positron, thought it was worth noting for this.

dotNomad · 2026-07-21T23:15:25Z

+
+- The Positron API is reached through the `acquirePositronApi()` global that
+  Positron injects into the extension host (typed by
+  `src/@types/positron.d.ts`). Publisher's own code feature-detects Positron


Could be worth a note in #3647 to update this ref when we start using the @posit-dev/positron types package.

I'll add a note in #3647

dotNomad · 2026-07-21T23:16:56Z

+
+/**
+ * Positron may report runtime paths with a leading `~/`, which Publisher
+ * expands before use (https://github.com/posit-dev/positron/issues/12942).


It looks like this issue was closed - is this workaround still necessary?

It was fixed (posit-dev/positron#14813), but the fix is only in dailies right now. This suite runs against the stable channel (currently 2026.07.1), CI is testing against a build that predates the fix, and Publisher's production code (utils/vscode.ts) still expands ~/ to stay compatible with pre-2026.08 builds regardless. This test mirrors what Publisher does, not what the newest Positron does, so the helper stays until the expansion leaves utils/vscode.ts. I've updated the comment to make that lifecycle explicit.

dotNomad · 2026-07-21T23:24:17Z

+  });
+
+  const testsRoot = __dirname;
+  for (const file of fs.readdirSync(testsRoot)) {


Should we use { recursive: true } on the readdirSync so we capture any tests in a sub-directory?

Doesn't affect is now, but could be handy if we make sub-directories in the future.

Done — switched to { recursive: true }

dotNomad · 2026-07-21T23:25:48Z

+      const runtime = await api.runtime.getPreferredRuntime(languageId);
+      if (runtime) {
+        return runtime;
+      }


Should we throw if we don't get a runtime back from getPreferredRuntime?

lastError doesn't get set if we have a falsey runtime which could make it confusing when reading the logs.

We could also change the assert.ok string to specify if positron reported no preferred runtime.

I agree, the loop is now an explicit while (Date.now() < deadline), and the failure message distinguishes "getPreferredRuntime threw: …" from "resolved without reporting a preferred runtime", so the empty-result case no longer prints Last error: undefined.

dotNomad · 2026-07-21T23:30:06Z

+    runs-on: macos-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-node@v6


We just updated to use setup-node@v7, would be an easy up here.

Done — rebased onto main and bumped.

dotNomad · 2026-07-21T23:30:39Z

+          cache-dependency-path: "**/package-lock.json"
+      # The interpreter-discovery tests need Python and R installed so
+      # Positron's runtime discovery can find them.
+      - uses: actions/setup-python@v6


Similar note here about using setup-python@v7

Done — rebased onto main and bumped.

dotNomad · 2026-07-21T23:57:16Z

+import Mocha from "mocha";
+
+export function run(): Promise<void> {
+  const mocha = new Mocha({


I'm curious if @posit-dev/positron-test-electron has any plans to create a Positron package similar to @vscode/test-cli that handles the Mocha bootstrapping. We are using it elsewhere in the Publisher codebase and the setup is a bit simpler.

Nothing currently, but I just opened an issue capturing it. posit-dev/positron-test-electron#4

Adds a Positron-only integration test suite that runs the Publisher extension inside a real Positron build via @posit-dev/positron-test-electron, covering the extension's actual Positron API usage (acquirePositronApi + runtime.getPreferredRuntime interpreter discovery) that the plain VSCode suite and the mock-based contract tests cannot reach. - src/test/positron/: Mocha entry point plus starter tests for API acquisition, extension activation, and Python/R interpreter discovery against the live API - scripts/run-positron-tests.mjs: launcher (npm run test-positron; POSITRON_CHANNEL=daily supported; macOS only for now, upstream limitation) - esbuild.tests.mjs/.vscode-test.mjs: separate the plain suite (out/test/suite) from the Positron suite (out/test/positron) - .github/workflows/positron-api-tests.yaml: runs on PRs and main (macos-latest, Python 3.12 + R 4.4, cached Positron download) Also raises the scanPythonDependencies integration-test timeout from 30s to 120s (matching the scanRPackages block) — the real-Python scan takes ~25s alone and flakes under parallel suite load. Part of posit-dev/positron#14531

- Disable the bundled Copilot extension in the test launch so it stops spamming the logs with failed GitHub auth attempts on credential-less CI runners - Make waitForPreferredRuntime distinguish "getPreferredRuntime threw" from "resolved without a runtime" in its failure message, and use an explicit while-deadline loop - Clarify that the ~/ expansion mirror stays for compatibility with Positron builds older than the posit-dev/positron#12942 fix - Discover tests recursively so future subdirectories are picked up - Bump setup-node and setup-python to v7, matching main - Note upstream Windows/Linux support status in the README and launcher

jonvanausdeln · 2026-07-22T16:55:02Z

Looks like this an issue with the Copilot extension inside of Positron, thought it was worth noting for this.

Good catch — that's Positron's bundled Copilot extension (GitHub.copilot-chat) failing GitHub auth on the credential-less runner. We launch with extensions enabled because runtime discovery lives in the bundled Python/Ark extensions, so Copilot was loading along for the ride. Fixed by disabling just it via --disable-extension GitHub.copilot-chat in the launcher; the logs should be clean now

dotNomad

Thank you for doing this and for the updates. This is looking great to merge 🎉

jonvanausdeln marked this pull request as ready for review July 21, 2026 17:29

jonvanausdeln requested a review from a team as a code owner July 21, 2026 17:29

jonvanausdeln mentioned this pull request Jul 21, 2026

Add Positron API integration tests and CI workflow posit-dev/shiny-vscode#115

Merged

dotNomad reviewed Jul 22, 2026

View reviewed changes

jonvanausdeln added 2 commits July 22, 2026 08:46

jonvanausdeln force-pushed the jonv/publisher-extension-api-tests-4cd2ea branch from 404e0f9 to 2a23931 Compare July 22, 2026 15:51

jonvanausdeln mentioned this pull request Jul 22, 2026

Provide a @vscode/test-cli analog that handles Mocha bootstrapping posit-dev/positron-test-electron#4

Open

jonvanausdeln requested a review from dotNomad July 22, 2026 18:37

dotNomad approved these changes Jul 22, 2026

View reviewed changes

jonvanausdeln merged commit b35cf6e into main Jul 22, 2026
46 checks passed

jonvanausdeln deleted the jonv/publisher-extension-api-tests-4cd2ea branch July 22, 2026 19:46

jonvanausdeln mentioned this pull request Jul 27, 2026

test: add Positron API integration tests + CI jonvanausdeln/positron-julia#1

Open

Uh oh!

Conversation

jonvanausdeln commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's covered

Harness

CI

Drive-by fix

Follow-up (not in this PR)

Testing

Uh oh!

posit-connect-projects Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Complete

Highlights

Minor Suggestions

Questions

Verdict

Uh oh!

dotNomad left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonvanausdeln commented Jul 22, 2026

Uh oh!

dotNomad left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jonvanausdeln commented Jul 21, 2026 •

edited

Loading

posit-connect-projects Bot commented Jul 21, 2026 •

edited

Loading