Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,9 @@

## Validation

- Run every Node, npm, and npx command on the `.nvmrc` runtime. npm refuses any
other version (`EBADDEVENGINES`); agent shells may not load nvm, so select it
per command as shown in `docs/test-policy.md`.
- Changed tests must pass `npm run lint:tests`. The behavioral test policy and
explicit migration inventory are owned by `docs/test-policy.md`; affected
selection and coverage commands do not replace required builds or live checks.
Expand Down
22 changes: 16 additions & 6 deletions docs/test-coverage.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ Coverage measures all JavaScript under `src/` and the two hand-maintained instal
scripts, `scripts/auto_refresh.user.js` and
`scripts/coinmarketcap-valuation-helper.user.js`. Generated install artifacts are
mapped back to their original sources, so they do not create another denominator.
The byte-pinned copies of Binance's native webpack factories under
`src/binance-orderbook-trade/chart-storage/` are host code that controlled
fixtures cannot execute; they are mapped like bundled dependencies and receive no
credit, while the hand-written chart-storage modules stay in scope.
Unexecuted source files remain in the report with zero execution credit. Test
files, fixtures, build tooling, and historical installer snapshots are not
production source. The executable scope lives in
Expand Down Expand Up @@ -43,7 +47,12 @@ text receive no credit for executing that source file. Browser collection also
recognizes complete original modules loaded through Blob URLs.

Browser coverage uses one public CDP session per page and caches script bytes when
Chromium parses them. A real reload must go through `reloadPageWithCoverage` in
Chromium parses them. The fixture covers the scenario's first page and every page
it opens with `context.newPage()` before that page can navigate; each page writes
its own capture, and a page closed during the scenario writes it before closing
because Chromium discards the counters with the page. Unload-time handlers and
pages in additional browser contexts therefore stay outside browser coverage.
A real reload must go through `reloadPageWithCoverage` in
`e2e/binance-orderbook/test.js`: it takes a precise checkpoint before discarding
the outgoing document. Raw checkpoints remain in the capture. Only snapshots
with the same session, script ID, URL, and exact source bytes are merged; a new
Expand Down Expand Up @@ -72,11 +81,12 @@ once in the final report. An isolated Node-and-Chromium proof checks known branc
counts so repeated bundled copies cannot hide a branch executed by another copy.

Every selected Node test process must finish its capture. Every production
browser scenario must pass and finish its capture, with IDs matched against the
current Playwright run. Missing, stale, skipped, or retried browser evidence
invalidates completeness. The one explicit collector self-test file uses virtual
code; it must pass but does not contribute production coverage. Source-map and
capture tests validate these boundaries independently.
browser scenario must pass and finish at least one capture, with IDs matched
against the current Playwright run; each CDP session is captured exactly once.
Missing, stale, skipped, or retried browser evidence invalidates completeness.
The explicit collector self-test files use virtual code; they must pass but do
not contribute production coverage. Source-map and capture tests validate these
boundaries independently.

## Repository Gate

Expand Down
7 changes: 6 additions & 1 deletion docs/test-policy.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,12 @@ validation are repository contracts. Other repositories define equivalent checks
for their own entrypoints and infrastructure. Shared agent guidance alone does
not establish that another repository's existing tests or CI already comply.

Use the Node version pinned by `.nvmrc`. The relevant commands are:
Use the Node version pinned by `.nvmrc`. `package.json` `devEngines` makes npm
refuse any other runtime (`EBADDEVENGINES`), and repository CLIs assert the same
version before doing any work. Shells that do not load nvm, including
non-interactive agent shells, select it per command, for example
`source "$NVM_DIR/nvm.sh" && nvm use >/dev/null && npm test`. The relevant
commands are:

| Command | Evidence |
| --- | --- |
Expand Down
8 changes: 5 additions & 3 deletions docs/test-selection.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,8 +143,8 @@ A full run is selected when:
ESLint configuration, the userscript builder, or `scripts/test-*` infrastructure
changes.
- A changed path was deleted, is unavailable, or has no verified consumer and is
not an eligible documentation file. Unmapped JSON, HTML, images, snapshots, and
source files all remain runtime changes.
not an eligible documentation file or Git ignore rule. Unmapped JSON, HTML,
images, snapshots, and source files all remain runtime changes.
- Build mapping, Playwright discovery, or snapshot ownership cannot be verified
globally, or an uncertain dependency cannot be assigned to a test owner.

Expand All @@ -156,7 +156,9 @@ a full run.
`none` is allowed for a valid graph with no changes, or for documentation without
known or uncertain runtime consumers. Eligible documentation is Markdown under
`docs/` and the documented root Markdown names such as `README.md` and `AGENTS.md`.
A Markdown extension alone is not enough to skip a runtime reader.
A Markdown extension alone is not enough to skip a runtime reader. `.gitignore`
files are skipped the same way: ignore rules never change tracked inputs, and a
file they newly expose is classified as its own untracked change.

The selector does not generate userscripts, install them, activate live browser
sessions, or authorize financial actions. Its browser execution is the project's
Expand Down
8 changes: 6 additions & 2 deletions e2e/binance-orderbook/specs/automatic-account-rebalance.pw.js
Original file line number Diff line number Diff line change
Expand Up @@ -628,11 +628,15 @@ for (const allocation of [
}

test('user qualifies again only after independently verified new account activity', async ({ page }) => {
// Given the first automatic episode has completed.
// Given the first automatic episode has completed both transfers.
const api = createAccountRebalanceApi(EXCESS);
await openAutomatic(page, api);
await becomeFlat(page);
await expect.poll(() => episodes(page)).toEqual(['consumed']);
// A bare consumed status also marks the plan before and between transfers.
await expect.poll(() => episodeRecords(page)).toEqual([
{ version: 1, status: 'consumed', outcome: 'automatic_completed' },
]);
expect(transfers(api)).toHaveLength(2);

// When a new server-side order is observed before its later removal.
api.setOrders({ basic: [{ symbol: 'BTCUSDT' }], conditional: [] });
Expand Down
49 changes: 49 additions & 0 deletions e2e/binance-orderbook/specs/coverage-pages.pw.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
import { mkdir, readdir, readFile } from 'node:fs/promises';
import { resolve } from 'node:path';
import { test, expect } from '@playwright/test';

import {
checkpointBrowserCoverage,
coverContextPages,
} from '../../../scripts/test-coverage/collect-browser.mjs';

const ORIGIN = 'https://coverage-pages.test';
const PROBE = '// ==UserScript==\n// @name Coverage pages probe\n// ==/UserScript==\n'
+ 'globalThis.coverageProbeRuns = (globalThis.coverageProbeRuns || 0) + 1;\n';

async function readCaptures(directory) {
const files = (await readdir(directory)).filter(name => name.endsWith('.json'));
return Promise.all(files.map(async name => JSON.parse(await readFile(resolve(directory, name), 'utf8'))));
}

function probeScriptIds(capture) {
return new Set(capture.snapshots.flatMap(snapshot => snapshot.entries)
.filter(entry => entry.source === PROBE).map(entry => entry.scriptId));
}

test('user keeps coverage for every page a scenario opens, reloads, and closes in its context', async ({ page, context }, testInfo) => {
// Given coverage follows the scenario context while its first page runs the probe script.
const directory = testInfo.outputPath('captures');
await mkdir(directory, { recursive: true });
await context.route(ORIGIN + '/**', route => route.fulfill(route.request().url().endsWith('/probe.js')
? { contentType: 'application/javascript', body: PROBE }
: { contentType: 'text/html', body: '<!doctype html><script src="/probe.js"></script>' }));
const finishContextPages = await coverContextPages(context, page, directory, testInfo);
await page.goto(ORIGIN + '/');

// When a second tab opens and reloads its document, and the first tab closes before teardown.
const other = await context.newPage();
await other.goto(ORIGIN + '/');
await checkpointBrowserCoverage(other, 'before-reload');
await other.reload();
await page.close();
await finishContextPages();

// Then each page writes one capture from its own session, including both reloaded documents.
const captures = await readCaptures(directory);
expect(captures.map(capture => capture.testId)).toEqual([testInfo.testId, testInfo.testId]);
expect(new Set(captures.map(capture => capture.sessionId)).size).toBe(2);
expect(captures.map(capture => capture.snapshots.map(snapshot => snapshot.phase).join(' > ')).sort())
.toEqual(['before-reload > finish', 'finish']);
expect(captures.map(capture => probeScriptIds(capture).size).sort()).toEqual([1, 2]);
});
9 changes: 4 additions & 5 deletions e2e/binance-orderbook/test.js
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
import { test as base, expect } from '@playwright/test';
import {
startBrowserCoverage,
checkpointBrowserCoverage,
finishBrowserCoverage,
coverContextPages,
} from '../../scripts/test-coverage/collect-browser.mjs';

import {
Expand All @@ -18,15 +17,15 @@ function jsonAttachment(value) {
}

export const test = base.extend({
sourceCoverage: [async ({ page }, use, testInfo) => {
sourceCoverage: [async ({ page, context }, use, testInfo) => {
const directory = process.env.USERSCRIPTS_BROWSER_COVERAGE_DIRECTORY;
if (!directory) {
await use();
return;
}
await startBrowserCoverage(page);
const finishContextPages = await coverContextPages(context, page, directory, testInfo);
await use();
await finishBrowserCoverage(page, directory, testInfo);
await finishContextPages();
}, { auto: true }],
// Failure evidence belongs to the test runner boundary so every future scenario
// receives the same diagnostics without duplicating cleanup code in each spec.
Expand Down
7 changes: 7 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,13 @@
{
"private": true,
"type": "module",
"devEngines": {
"runtime": {
"name": "node",
"version": "24.16.0",
"onFail": "error"
}
},
"scripts": {
"build:userscripts": "node scripts/build-userscript.mjs",
"build:binance-userscripts": "node scripts/build-userscript.mjs binance-orderbook-trade binance-trading-data binance-coinmarketcap-data binance-strategy27-events binance-strategy29-bollinger binance-strategy31-volume-reversal",
Expand Down
5 changes: 4 additions & 1 deletion playwright.config.js
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,10 @@ export default defineConfig({
fullyParallel: false,
forbidOnly: Boolean(process.env.CI),
retries: 0,
workers: 1,
// Spec files use isolated contexts with routed fixtures, so they can run in parallel while
// tests inside one file stay ordered. Half the logical CPUs is two workers on 4-vCPU CI
// runners, which keeps headroom for the 3s expectation timeout.
workers: '50%',
timeout: 15_000,
snapshotPathTemplate: '{testDir}/{testFilePath}-snapshots/{arg}{ext}',
expect: {
Expand Down
13 changes: 13 additions & 0 deletions scripts/node-runtime.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
import { readFile } from 'node:fs/promises';

/**
* Repository CLIs run only on the exact `.nvmrc` runtime. Call this at module load,
* before any `if (import.meta.main)` guard: older Node versions leave
* `import.meta.main` undefined, so the guard alone would skip all work and exit 0.
*/
export async function assertProjectNodeVersion(actual = process.versions.node) {
const expected = (await readFile(new URL('../.nvmrc', import.meta.url), 'utf8')).trim();
if (actual !== expected) {
throw new Error('Use the project Node version ' + expected + ' from .nvmrc (nvm use); received ' + actual);
}
}
31 changes: 31 additions & 0 deletions scripts/test-coverage/collect-browser.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -76,3 +76,34 @@ export async function finishBrowserCoverage(page, outputDirectory, testInfo) {
snapshots,
}));
}

/**
* Scenarios may open, reload, and close several pages in their own context. Each
* page keeps a separate CDP session because script IDs are unique only within one
* session, and a page's capture is written before it closes because Chromium
* discards its counters with the page, so unload handlers stay outside coverage.
*/
export async function coverContextPages(context, page, outputDirectory, testInfo) {
const open = new Set();
const newPage = context.newPage;
async function cover(target) {
await startBrowserCoverage(target);
open.add(target);
const close = target.close;
target.close = async (options) => {
if (open.delete(target)) await finishBrowserCoverage(target, outputDirectory, testInfo);
return close.call(target, options);
};
}
// A page must be covered before the scenario can navigate it.
context.newPage = async (...args) => {
const created = await newPage.apply(context, args);
await cover(created);
return created;
};
await cover(page);
return async () => {
for (const target of open) await finishBrowserCoverage(target, outputDirectory, testInfo);
open.clear();
};
}
32 changes: 21 additions & 11 deletions scripts/test-coverage/config.mjs
Original file line number Diff line number Diff line change
@@ -1,16 +1,28 @@
import { readdir, readFile } from 'node:fs/promises';
import { readdir } from 'node:fs/promises';
import { dirname, isAbsolute, relative, resolve, sep } from 'node:path';
import { fileURLToPath } from 'node:url';

export const ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '../..');
export const COVERAGE_DIRECTORY = resolve(ROOT, 'test-results/coverage');
export const BRANCH_TARGET = 90;
// This spec validates the collector with virtual code, not production behavior.
export const COVERAGE_SELF_TEST_FILES = ['e2e/binance-orderbook/specs/coverage-merge.pw.js'];
// These specs validate the collector with virtual code, not production behavior.
export const COVERAGE_SELF_TEST_FILES = [
'e2e/binance-orderbook/specs/coverage-merge.pw.js',
'e2e/binance-orderbook/specs/coverage-pages.pw.js',
];
export const HAND_MAINTAINED_SCRIPTS = [
'scripts/auto_refresh.user.js',
'scripts/coinmarketcap-valuation-helper.user.js',
];
/**
* Byte-pinned copies of Binance's native webpack factories. Their bodies are host code
* that controlled fixtures cannot execute, so coverage maps them like bundled
* dependencies; the hand-written chart-storage modules that install them stay in scope.
*/
export const GENERATED_HOST_FACTORIES = [
'src/binance-orderbook-trade/chart-storage/drawing-save-module.js',
'src/binance-orderbook-trade/chart-storage/mirror-module.js',
];

export function relativeSourcePath(value) {
const path = value.startsWith('file:') ? fileURLToPath(value) : value;
Expand All @@ -19,10 +31,15 @@ export function relativeSourcePath(value) {

export function isProductionSource(value) {
const path = relativeSourcePath(value);
return (path.startsWith('src/') && path.endsWith('.js'))
return (path.startsWith('src/') && path.endsWith('.js') && !GENERATED_HOST_FACTORIES.includes(path))
|| HAND_MAINTAINED_SCRIPTS.includes(path);
}

/** Bundled code that artifacts may contain but production coverage never credits. */
export function isVendoredSource(path) {
return path.startsWith('node_modules/') || GENERATED_HOST_FACTORIES.includes(path);
}

export async function productionSourceFiles() {
async function walk(directory) {
const entries = await readdir(resolve(ROOT, directory), { withFileTypes: true });
Expand All @@ -35,10 +52,3 @@ export async function productionSourceFiles() {
return [...await walk('src'), ...HAND_MAINTAINED_SCRIPTS]
.filter(isProductionSource).sort();
}

export async function assertProjectNodeVersion() {
const expected = (await readFile(resolve(ROOT, '.nvmrc'), 'utf8')).trim();
if (process.versions.node !== expected) {
throw new Error('Use the project Node version ' + expected + '; received ' + process.versions.node);
}
}
6 changes: 4 additions & 2 deletions scripts/test-coverage/merge-proof.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,12 @@ import { parse } from 'acorn';
import * as esbuild from 'esbuild';
import { CoverageReport } from 'monocart-coverage-reports';

import { ROOT, assertProjectNodeVersion, relativeSourcePath } from './config.mjs';
import { ROOT, relativeSourcePath } from './config.mjs';
import { composeSourceMap, mapCoverageEntry } from './source-maps.mjs';
import { findArtifactSegments, splitCoverageEntry } from './split-entries.mjs';
import { assertProjectNodeVersion } from '../node-runtime.mjs';

await assertProjectNodeVersion();

const runFile = promisify(execFile);

Expand All @@ -26,7 +29,6 @@ export const PROOF_SOURCE = `export function chooseBranch(nodeBranch) {

/** All synthetic source, raw captures, and reports stay in this test's output. */
export async function createMergeProof(outputDirectory) {
await assertProjectNodeVersion();
await mkdir(outputDirectory, { recursive: true });
const sourceFile = resolve(outputDirectory, 'branch-source.mjs');
const sourcePath = relativeSourcePath(sourceFile);
Expand Down
5 changes: 4 additions & 1 deletion scripts/test-coverage/report.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ export async function buildCoverageReport({ nodeDirectory, browserDirectory, out
});
const capturedTests = new Set();
const capturedBrowserTests = new Set();
const capturedBrowserSessions = new Set();
const browserManifest = browserDirectory === null ? null
: JSON.parse(await readFile(resolve(browserDirectory, 'manifest.json'), 'utf8'));
const unmapped = [];
Expand All @@ -36,7 +37,9 @@ export async function buildCoverageReport({ nodeDirectory, browserDirectory, out
let entries;
if (layer === 'node') capturedTests.add(capture.testFile);
else {
assert.equal(capturedBrowserTests.has(capture.testId), false, 'Duplicate browser capture');
// One scenario may cover several pages; each CDP session is captured once.
assert.equal(capturedBrowserSessions.has(capture.sessionId), false, 'Duplicate browser capture session');
capturedBrowserSessions.add(capture.sessionId);
capturedBrowserTests.add(capture.testId);
}
if (layer === 'browser') {
Expand Down
10 changes: 7 additions & 3 deletions scripts/test-coverage/run.mjs
Original file line number Diff line number Diff line change
@@ -1,11 +1,14 @@
import { spawn } from 'node:child_process';
import { mkdir, mkdtemp, readFile, readdir, writeFile } from 'node:fs/promises';
import { resolve } from 'node:path';
import { ROOT, COVERAGE_DIRECTORY, assertProjectNodeVersion } from './config.mjs';
import { ROOT, COVERAGE_DIRECTORY } from './config.mjs';
import { buildCoverageReport } from './report.mjs';
import { assessBranchCoverage } from './gates.mjs';
import { assertProjectNodeVersion } from '../node-runtime.mjs';
import { exactNodeArgs } from '../test-selection/node-runner.mjs';

await assertProjectNodeVersion();

export async function nodeTestFiles() {
async function walk(directory) {
const entries = await readdir(resolve(ROOT, directory), { withFileTypes: true });
Expand Down Expand Up @@ -39,7 +42,6 @@ export async function runCoverage(mode, { reportOnly = false, requireTarget = fa
if (requireTarget && (reportOnly || mode === 'node')) {
throw new Error('The final coverage target requires a complete gated run');
}
await assertProjectNodeVersion();
const policy = mode === 'all' && !reportOnly
? JSON.parse(await readFile(new URL('./branch-policy.json', import.meta.url), 'utf8'))
: null;
Expand All @@ -56,7 +58,9 @@ export async function runCoverage(mode, { reportOnly = false, requireTarget = fa
reportFile: resolve(runDirectory, 'node-results.txt'),
}), { USERSCRIPTS_NODE_COVERAGE_DIRECTORY: nodeDirectory });
if (browserDirectory !== null) {
await runNode(['./node_modules/@playwright/test/cli.js', 'test',
// Precise block coverage slows every page; parallel workers would push timing-sensitive
// expectations past their limits, so the instrumented browser layer stays serial.
await runNode(['./node_modules/@playwright/test/cli.js', 'test', '--workers=1',
'--reporter=dot,html,./scripts/test-coverage/browser-reporter.mjs'], {
USERSCRIPTS_BROWSER_COVERAGE_DIRECTORY: browserDirectory,
PLAYWRIGHT_HTML_OPEN: 'never',
Expand Down
Loading
Loading