Skip to content

TESTING.md

Latest commit

 

History

History
273 lines (207 loc) · 19.9 KB

TESTING.md

File metadata and controls

273 lines (207 loc) · 19.9 KB
Raw

Testing Strategy

Tools

  • Test runners: Vitest for source tests; Node.js node:test for built CLI, installed-package, and documentation tests
  • Gate command: pnpm verify (runs documentation checks, package boundaries, lint, typecheck, and every test suite)
  • TDD flow: red-green-refactor — write a failing test first, implement the minimal code to pass, then refactor

Package Test Strategy

@cloudburn/rules

Three test layers, all in packages/rules/test/:

Layer File What it verifies
1. Export surface exports.test.ts awsRules is non-empty, preset inclusion policy holds, azureRules/gcpRules are empty
2. Metadata contract rule-metadata.test.ts Catalog fields, IDs, evaluator modes, and declared dataset reads
3. Evaluator behavior {rule-name}.test.ts Full finding payloads for both evaluateLive and evaluateStatic, plus negative cases

Keep catalog invariants and semantic metadata policies in rule-metadata.test.ts. It runs each evaluator with empty resource bags and checks that dataset reads match declarations in both directions, rejecting undeclared reads and unused declarations. Lambda memory recommendations also require inventory for SDK evaluation coverage; an explicit assertion pins that dependency pair. SDK export tests verify declared datasets have registered loaders. Evaluator fixtures cover data-dependent reads and complete findings. Avoid copying every rule's name, description, and dependency list into a second declaration-shaped snapshot; keep exact assertions for decisions such as optional evidence or finding precedence.

For static IaC rules, evaluator coverage must include both Terraform-shaped and CloudFormation-shaped resources. A passing test suite for only one source kind is incomplete.

@cloudburn/sdk

Mock at the provider/parser boundary — do not call real AWS APIs or read real files in unit tests.

Boundary Mock strategy
AWS providers Mock Resource Explorer catalog helpers and hydrators to return fixture data
Terraform parser Mock parseTerraform to return { resources, diagnostics } fixtures
Config loader Mock loadConfig or pass config directly via CloudBurnClient runtime overrides

Test focus areas:

  • runStaticScan — registry + static dataset dependency resolution + parser selection + dataset loading + evaluation
  • runLiveScan — registry + dataset dependency resolution + Resource Explorer catalog + dataset loading + evaluation
  • buildRuleRegistry — mode-aware rule filtering for iac and discovery
  • mergeConfig — per-mode merge behavior and runtime override precedence
  • CloudBurnClient — facade delegates correctly to engine and provider helpers

Split static AWS provider tests into two layers:

  1. Static dataset loader tests per dataset
  2. Orchestration tests in loadAwsStaticResources

When adding an IaC rule or dataset, cover Terraform and CloudFormation normalization in provider tests and positive and negative findings in evaluator tests. Keep representative full scans for package wiring, plus dedicated scope, suppression, diagnostics, configuration, and evaluation-evidence regressions. Add a service-specific scanner test when it exercises a distinct integration behavior; repeating every provider and evaluator expectation at the facade adds maintenance without testing another scanner path.

Split live AWS provider tests into three layers:

  1. Resource Explorer catalog tests
  2. Dataset loader/hydrator tests per service
  3. Orchestration tests in discoverAwsResources

Discovery HTTP integration

packages/sdk/test/discovery-http-integration.test.ts runs the public CloudBurnClient.discover() method with the real catalog, hydrators, AWS serialization/deserialization, and rule evaluation. Only the AWS HTTP transport is intercepted. Synthetic JSON and XML responses cover pagination, catalog deduplication, hydration scope, findings, and denied required evidence. Unexpected requests fail the test before reaching the network. Keep focused provider unit tests for exhaustive service behavior.

The same suite verifies ELB inventory reuse through the public pipeline. It counts load-balancer and target-group metadata calls separately from target-health and metric requests, and checks that unsupported types and failed metric series remain visible in unknown coverage alongside valid findings.

packages/sdk/test/discovery-metric-http-integration.test.ts exercises CloudWatch evidence through the real public discovery pipeline. Partial SageMaker invocation data cannot emit an idle finding, and mixed complete/unknown endpoints retain both valid findings and unknown coverage. Provider tests cover status/pagination/retry behavior and Lambda's rolling windows and weighted duration summaries using synthetic AWS responses and fixed clocks.

Reusable evidence tests run through those same HTTP pipelines: unchanged scans, rule selection, credential sessions, view changes, new catalog membership, partial coverage, strict refresh, and independent cancellation. Lambda tests compare actual CloudWatch request windows with cache provenance and verify dependency-version invalidation. The generic cache suite uses real child processes for concurrent refresh, owner crashes, lease expiry, and old-writer fencing, plus synthetic loaders for corruption, invalidation, bounded eviction, and value isolation. No AWS access is needed.

Pipeline tests gate synthetic catalog pages and hydration responses through CloudBurnClient.discover(). They assert that account evidence starts during catalog preparation, a cached catalog type releases before an unrelated miss, and useful provisional results precede final completion. Empty pages with continuation tokens cannot produce a pass. Cancellation after progress stops transport and later events and rejects the final result. Engine integration coverage retains selected optional evidence, finding precedence, and partial-region evaluation semantics.

cloudburn (CLI)

Command tests (*.command.test.ts) mock the SDK boundary to isolate CLI behavior. Their output assertions also cover formatter integration; keep direct formatter tests for distinct layout and serialization edge cases. Shared root option parsing needs representative command coverage, while mode-specific configuration and restrictions stay with their commands. The separate test/e2e/ suite runs the built executable against real Terraform and CloudFormation files, without mocking the SDK or parsers.

Boundary Mock strategy
CloudBurnClient Mock .scanStatic(), .discover(), and discovery helper methods
builtInRuleMetadata Use real built-in metadata unless explicitly testing the empty state

Test focus areas:

  • Each command produces correct output for a given ScanResult
  • Root and command-local --format resolve to the expected json|table output
  • Config-provided mode formats become the default when --format is absent
  • --config, --enabled-rules, and --disabled-rules pass the expected runtime overrides to the SDK
  • --exit-code sets process.exitCode = 1 when active findings exist
  • --fail-on and configured fail-on apply inclusive severity thresholds with CLI precedence
  • Suppressed IaC findings remain in output but never set process.exitCode = 1
  • Gates without qualifying active findings set process.exitCode = 0
  • discover supported-resource-types, discover init, discover status, config, rules list, and estimate all go through the shared formatter system
  • table output stays human-readable and json output stays machine-readable
  • Runtime errors remain structured JSON on stderr regardless of stdout format

@cloudburn/action

Unit tests mock the GitHub toolkit boundary (@actions/core, @actions/github) and cover input parsing, policy precedence, markdown rendering, annotations, and comment upserts. The test:e2e suite runs the built dist/index.cjs bundle in a child process with INPUT_*/GITHUB_* environment variables against the CLI's e2e fixtures — the same entry the node24 runner executes. Only the shipped JavaScript and WASM are copied to a temporary directory outside the workspace, with global module searches disabled, so Terraform and CloudFormation scans cannot rely on workspace dependencies. It verifies workflow commands on stdout, GITHUB_OUTPUT/GITHUB_STEP_SUMMARY files, the result JSON artifact, and exit status. A build-metafile check rejects bundled AWS SDK and Smithy packages to keep live-discovery dependencies out of the static action. No AWS credentials or real pull requests are used.

@cloudburn/mcp

Unit tests connect an MCP client to the server over an in-memory transport and mock CloudBurnClient. They cover discovery arguments (region target, evidence cache directory, deadline, rule selection, cancellation signal), progress notifications, and error categorization and redaction, which need AWS in a real run.

The test:e2e suite starts the built dist/cli.js over stdio in isolated temporary directories, the way agents launch it, with the CLI's AWS module guard active and no AWS configuration. It calls the tools against the CLI's Terraform and CloudFormation fixtures and reviewed expectations in cases.json, and checks relative-path rejection, config precedence, suppression, and tool errors. It also audits the built plugin folder: version-stamped manifests, exact launcher pins, marketplace entries, the file allowlist and size limits, the README, and skill front matter. Claude Code's claude plugin validate --strict is a manual check because CI does not install Claude Code.

Its test:package suite packs the MCP server, SDK, and rules, installs the archives into an isolated consumer the way npx installs the server, and checks that the packed manifest has only published dependencies and ships only dist/cli.js. It then starts the installed cloudburn-mcp executable over stdio and runs a Terraform scan. Like the CLI's installed-package suite, it is uncached and needs public npm registry access.

@cloudburn/vscode

Transport tests run real child processes to reject incompatible JSON and cancel a CLI subprocess. They cover malformed launcher responses the healthy CLI fixtures cannot reproduce. Editor behavior is owned by the test:e2e suite: it loads the generated dist/vscode/ extension in VS Code 1.100.3 in an initially empty multi-root workspace, verifies automatic scanning on the first plaintext Terraform save without a Terraform language extension, and invokes the extension's real commands with the built CLI. It checks Terraform and CloudFormation JSON/YAML locations, severities, rule links, configuration, suppressions, policy exit code 1, save/fix updates, setting changes, renames, deletion, sibling findings, skipped-file removal, and recovery after a missing CLI. Renaming a file in a never-scanned folder must not start its first scan, while renaming and recreating an external explicit config, or moving the target of a symlinked config away and back, must refresh the scanned folder that references it. Saving a discovered .cloudburn.yml in an ancestor directory must refresh the folder that uses it. A slow launcher signals when the deletion refresh's CLI process starts, then holds it open while a file is saved and save scans are turned off; that refresh must still complete. Overlapping parent/child roots must publish one finding and clear it on save using the outermost folder's settings while retaining sibling results. Clean-machine startup creates the download cache before invoking the editor downloader. The workspace name includes spaces and shell metacharacters to exercise direct argument handling.

Run pnpm exec turbo run test:e2e --filter @cloudburn/vscode. The runner downloads the pinned editor, uses xvfb-run on Linux without a display, and removes temporary workspaces. It clears CI for the editor process so the CLI's config discovery, which is disabled in CI, behaves as it does on a developer machine. CLOUDBURN_VSCODE_EXECUTABLE selects an installed editor for compatibility checks. The task is uncached because it starts an editor process. No AWS access is needed.

Built CLI template tests

pnpm test:e2e builds the CLI and its workspace dependencies, then starts the executable in isolated temporary directories. Fixtures and reviewed finding expectations live in packages/cloudburn/test/e2e/. These tests cover Terraform, CloudFormation YAML and JSON, positive and negative findings, source scope, source locations, configuration, suppression, parser diagnostics, and exit codes. They use no AWS credentials or account resources.

Keep the fixture expectations independent of implementation output. Normalize only irrelevant result ordering; preserve rule IDs, resource IDs, source locations, and diagnostics. Add representative complete command flows here and keep exhaustive rule permutations in SDK/rules tests.

Installed-package tests

pnpm test:packages builds and packs all three workspace packages, installs the archives into a temporary consumer project, then checks the installed CLI executable and SDK ESM/CommonJS exports with real scans. Nothing is published. Installation can access the public npm registry for runtime dependencies, so this suite is uncached and requires registry connectivity. The temporary consumer uses the repository-pinned pnpm version. The uncached task forwards HTTP_PROXY, HTTPS_PROXY, NO_PROXY (including lowercase forms), and NODE_EXTRA_CA_CERTS through Turbo and into subprocesses for registry connectivity. It isolates user npm/pnpm configuration and does not use AWS credentials or contact AWS.

The built CLI template suite installs a module-resolution guard that rejects AWS SDK and Smithy dependencies, including credential providers. Help/version and static fixture scans must complete with the guard active. Installed ESM and CommonJS SDK consumers use the same guard to check static scans, synchronous region helpers, credential-scope Promise identity, and pre-cancelled discovery methods. A separate installed consumer drives real live chunks in both formats with synthetic HTTP responses, checking signed scoped credentials, findings, owned client disposal, and active cancellation. A Hub scan checks capability outcomes, recommendation identity/provenance, and impact on both findings and evaluated resources. Packed manifests must pin exact coordinated internal versions and ship their declared ESM/CommonJS and declaration entry points without workspace-only dependencies. A module-resolution hook also cancels the first AWS import and checks that late import completion never resolves credentials. These structural checks enforce import independence without machine-specific timing gates.

Runtime and type contracts

Export tests should exercise values imported from the public package entry point. Constructing a typed object and asserting its literal fields does not verify an export at runtime: Vitest transpiles tests without checking types. Type contracts are enforced by each package's typecheck script, which compiles src and test through tsconfig.typecheck.json, so a type-level assertion or @ts-expect-error line in a test file fails pnpm typecheck directly without spawning a second compiler during test runs.

Keep consumer-shaped type assertions in the rules purchase contract fixture and the SDK public contracts test so downstream usage stays compiled with the package.

Fixture privacy

Use synthetic AWS account IDs, ARNs, credentials, and responses in committed fixtures. Do not record live account output or add real credentials, real account identifiers, or live AWS workflows to this test pipeline. The synthetic discovery fixtures use 111111111111; this is example data.

Running Tests

# Single package
pnpm turbo run test --filter @cloudburn/rules

# All suites (including built CLI and installed-package checks)
pnpm test

# Built CLI and real template fixtures
pnpm test:e2e

# Installed archives and public package exports
pnpm test:packages

# One discovery integration file
pnpm --filter @cloudburn/sdk exec vitest run test/discovery-http-integration.test.ts

# Repository knowledge checker and its public CLI tests
pnpm docs:check
pnpm docs:test

# Full verification gate
pnpm verify

docs:test exercises scripts/check-docs.mjs through its process exit status and diagnostics with isolated fixtures, and scripts/check-owning-pages.mjs against temporary Git branches and pull request descriptions. The live docs:check validates required entry points, root and package instruction aliases, repository-contained local links, heading fragments, reference-style definitions and uses, code-example exclusion, canonical page reachability, and the root AGENTS.md line budget.

CI and task caching

pnpm release:test executes the extension release guard against local Git fixtures, including matching versions, main ancestry, tag targets, and rejected branch dispatches. Recovery tests select and tag extension-only private releases without querying npm, and exercise the publishing dispatch at the versioned tag. Provider commands are replaced at their process boundary; these tests never upload an extension or need publisher credentials. The extension release workflow runs the real editor suite before packaging and shares one VSIX between its two registry jobs. Registry authentication and actual uploads require a release run and cannot be verified by local fixtures.

The shared validate CI job installs dependencies once, then starts pnpm verify:repo as a background step alongside package verification. Pull requests run pnpm verify:packages --affected --concurrency=2; pushes to main run pnpm verify:packages --concurrency=2. GitHub waits for background steps before cleanup and fails Validation if either lane fails. Documentation checks/tests, release tests, and package boundaries always run. Locally, pnpm verify runs both lanes sequentially and forwards package flags to verify:packages.

The separate Documentation workflow runs only the owning-pages check, so editing a pull request description reruns that check without repeating Validation.

Turbo keeps its concurrency limit of 2 because hosted runners have 4 vCPUs, and running every package's Vitest suite beside the CLI end-to-end tests, lint, and typecheck stretches timer-paced integration tests past their budgets. Turbo selects affected package tasks and shares required builds within the job. A separate smoke job rebuilds the action and runs it through uses: ./packages/action against real fixtures, exercising the same dist/ entry the runner executes.

The actionlint configuration temporarily filters only the unsupported background keyword diagnostic on the CI and release workflows. Other syntax, expression, action, and shell checks remain enabled. Remove the exceptions when actionlint supports GitHub's background-step syntax.

Source tests resolve workspace source directly and can run without dependency builds. The test:inputs transit task propagates upstream source changes into downstream test cache keys without serializing their execution. Built CLI and installed-package suites depend on the CLI build, which depends on SDK/rules builds. Test fixture edits invalidate tests without rebuilding unchanged package output. See the command reference for task dependencies and cache policy.

The action's test:e2e task adds the shared CLI fixtures to its inputs while retaining $TURBO_DEFAULT$ for its own package files. The root affectedUsingTaskInputs flag also selects that task for fixture-only pull requests; cross-package inputs alone invalidate the cache but do not select the action under --affected. Keep these inputs in the action task configuration, without adding an action dependency on the CLI. The MCP package's test:e2e task declares the same fixture inputs in its task configuration.