#93: coverage matrix by miroslavpojer · Pull Request #94 · AbsaOSS/living-doc-toolkit

miroslavpojer · 2026-10-27T11:59:27Z

Overview

Adds coverage-matrix, the third and last command the action runs. It reads one doc-source.json and one ui-tests.json through the shared input validation and writes coverage-matrix.json on the living-doc-utilities 0.8.0 contract. Coverage is counted per acceptance criterion. A criterion that declares aspects or a keyword is one plain row until per-aspect coverage arrives, so the canon's worked example of 1/3 for an aspect criterion is not computed yet.

Entities and rows. Every User Story and every Functionality has an entities[] entry, whatever its state and whether or not it has a counted criterion. User Stories come first, then Functionalities, each in input order. There is no Feature entity. An entity has a row per active or deprecated criterion, in its own order. A row lists the scenario_id of each scenario that links that criterion, each scenario once and in input order. It is covered with at least one scenario and not_covered with none, and carries no aspect and no keyword. A link to a criterion without a row (unknown, planned or in_review) counts nowhere, and in this change it raises no warning. Nothing is sorted.

Summaries. Each entity's summary and the root summary come from CoverageSummary.from_rows, the root's over every row of every entity. planned_summary counts the planned criteria of User Stories and Functionalities by target version, or as backlog when they have none. An in_review criterion is in neither.

--view and --fail-under. --view inner|release (default inner) sets document.view and nothing else: both views have the same rows and counts. --fail-under N is the CLI's gate. The file is written first; then a root coverage_pct below N, or null because nothing is counted, logs one [ERROR] line and exits 1. In the action, that leaves the file in place and sets no output-path. N must be a finite number, as the action's input check already requires: nan or inf is click's usage error, exit 2, with no file written, because no percentage is below nan and the gate would never fail. The gate compares coverage_pct as written, rounded to one decimal place, and the CLI README links the canon's rounding rule.

Envelope with two inputs.

metadata.source is the documentation input's, with each of the tests input's systems it lacks added once.
metadata.source_inputs[] holds the documentation entry, then the tests entry. Each keeps its input's schema_version, producer, run, source and stats.
metadata.run is the CI run, producer.build is null, and the four counts a caller reports are 0.
Every documentation warning, then every tests warning, is forwarded, each naming its own input in context.

Lineage. One table in the package covers both inputs: 162 doc-source paths and 12 ui-tests paths. It maps entity identity, criterion id and state, and scenario_id, and drops everything else with a reason. The tests run assert_complete for both contracts; the command never calls it. The run-time check runs once, before the write, against the documentation entry's selected_stats only. Those stats are computed over what the matrix counts: the User Stories and Functionalities, each with its active and deprecated criteria, and no Feature. Both choices avoid a false FIELD_LOSS, each measured on the fixtures:

With the input's full stats, a documentation file whose criteria are all planned writes no row. It would fail on acceptance_criteria[].id and .state, so a project with only planned work would get no matrix.
Checked against the tests input, a tests file whose scenarios link no counted criterion would fail on scenarios[].scenario_id.

With the counted selection, the fixtures, the full samples and the all-planned file pass, and a builder that empties every row still fails. Because both entity lists map to the same output paths, a criterion lost from one list while the other fills the path is a partial drop. R11 does not detect a partial drop, and the package README says so.

Matching. Matching is by entity id and criterion id only. The deleted matcher's org/repo disambiguation is not rebuilt: input validation stops a repeated entity or criterion id with DUPLICATE_ENTITY_ID / DUPLICATE_AC_ID, and a second documentation file with MULTIPLE_DOCUMENTATION_SOURCES.

Action and pin. main.py loses NOT_AVAILABLE_YET and its refusal, and every page that said a command is not available yet now says all three run. The root requirements.txt, which the action installs, now pins living-doc-utilities==0.8.0 like every package. The root Dependabot pip entry ignores it, as the package entries do. The package is registered in the Makefile, requirements.txt, .github/dependabot.yml, apps/cli and the pages that list packages.

Version impact. Additive, no bump (SPEC.md § Cross-cutting contract changes). It adds a new package, living-doc-service-coverage-matrix 0.1.0, and the CLI's coverage-matrix command with its options and error lines. The action's command: coverage-matrix, which exited 1 with the placeholder "not available yet" refusal from #85, now runs. No input, option, output, exit code or error line of normalize-docs or normalize-tests changes, and no artifact contract changes: the toolkit writes coverage-matrix-v1.0.0 as living-doc-utilities 0.8.0 defines it.

Release Notes

New command coverage-matrix: reads one doc-source.json and one ui-tests.json and writes coverage-matrix.json on the living-doc-utilities 0.8.0 contract. It counts per acceptance criterion and adds a summary per User Story, per Functionality and for the whole matrix, plus a planned-work summary.
--fail-under fails the run when the matrix's coverage percentage is below the number, or there is none; the file is written first. A value that is not a finite number, such as nan or inf, is a usage error.
The action runs every command it declares; no command stops with "not available yet" any more.
coverage-matrix checks its field-lineage table against the documentation input before it writes: a field with data in the counted records that arrives empty stops the run with FIELD_LOSS.
The root requirements.txt pins living-doc-utilities 0.8.0, as every package does.

Framework

Stays inside the frame. The coverage matrix is the toolkit's (F3), and the change mirrors no contract model and commits no schema (F7).

Contract Rules

source is copied from the one documentation input, plus the tests input's systems. The source_inputs[] entries and the producer follow R6 and R7: artifact rules, § The metadata envelope, R6, R7.
Counted states are active and deprecated in both views. document.view changes no count, and in_review and planned criteria have no row. The entity and matrix summaries come from CoverageSummary.from_rows, and planned_summary has total = backlog + targeted: rendering, § Coverage, § Coverage summary, § Planned work summary.
One documentation file and one project per run: pipeline rules, § One project, one pipeline, one documentation source.
The input validation order (MISSING_INPUT for a coverage run without one of its inputs, through DUPLICATE_AC_ID), and forwarded warnings: errors, § Codes, § Forwarded warnings.
Lineage is checked against the one documentation input, with selected_stats computed over the counted records: component checks, § R11.
- Artifact rules, R7, defines selected_stats by the view filter. This command's view filters nothing, and its count rule plays that part. The contract pages do not state that reading yet; it is recorded on Add the coverage-matrix command, which turns doc-source.json and ui-tests.json into coverage-matrix.json #93 (Additional Context) for a later contract change.
R12 checks:
- check 1: no schema is vendored, and make no-vendored-schemas is clean;
- check 2: both inputs are read through read_artifact as contract models, and the matrix is written with write_artifact from one;
- check 3: the written full samples run through the command in both views, input validation included (component checks).
Coverage is per criterion here, where rendering § Coverage computes it per aspect. A row without aspects is valid against AcCoverage, and per-aspect coverage, the keyword and the test-link warnings come in a later change, as Add the coverage-matrix command, which turns doc-source.json and ui-tests.json into coverage-matrix.json #93 states.

No disagreement with the canon was found.

Verification

make qa passed: core 67 tests, normalize-docs 70, normalize-tests 20, coverage-matrix 42 (100 % coverage), CLI 46, root 59. Pylint 10.00 in every package; Black, ruff, mypy and deptry clean; no-vendored-schemas clean.
The fixture pair's matrix:
- US-001: US-001-01 covered (1 scenario), -02 not covered, -04 (deprecated) covered (1); 66.7.
- FUNC-001: -01 covered (3), -02 not covered, -03 covered (1); 66.7.
- FUNC-002 (planned): no row; 0 counted, null.
- Root: 6 counted, 4 covered, 0 partially, 2 not covered, 66.7. planned_summary: total 2, backlog 1, {"1.1.0": 1}.
- Two forwarded warnings, source_input='doc-source-v1.0.0'. The output equals the committed golden with generated_at masked.
The written full samples, in both views: no FIELD_LOSS; six entities (US-010, FUNC-001 and FUNC-003 with no row); 7 counted rows, 3 covered, 42.9. Their planned_summary (4, backlog 2, {"1.6.0": 1, "1.7.0": 1}) and their entity and row ids equal full_sample("coverage-matrix-v1.0.0")'s.
A fresh venv with a plain pip install -r requirements.txt, as the action installs it, runs coverage-matrix through main.py and sets output-path. The new integration.yml scripts ran locally on the repository's venv: the action-run assertions, and the isolation job's outcome check and three replays.

Acceptance criteria

A bare builder.py, lineage.py, service.py or tests/test_*.py path is under packages/services/coverage_matrix/ (src/living_doc_service_coverage_matrix/ for the modules); tests/test_main.py is the root's.

Summary by CodeRabbit

New Features
- Added the coverage-matrix command to the CLI and GitHub Action. It combines documentation and UI-test inputs into a coverage report, with inner and release views.
- Reports include coverage totals, planned work, and source details. An optional threshold can make the command exit with an error when coverage is too low; the report is still written first.
Documentation
- Added usage examples and references for the command, its inputs and outputs, and threshold behavior.

… ui-tests.json into coverage-matrix.json - New package packages/services/coverage_matrix (living-doc-service-coverage-matrix 0.1.0): builder, a lineage table over both inputs, and the service. - One entity per User Story and Functionality, a row per active or deprecated criterion with the scenarios that link it, summaries from CoverageSummary.from_rows, and planned_summary. - The run-time FIELD_LOSS check compares against the documentation input only, with selected_stats over the counted criteria. - The CLI registers coverage-matrix with --doc-input, --tests-input, --output, --view and --fail-under (checked after the write). - The action runs every command it declares: NOT_AVAILABLE_YET and its refusal are gone, and integration.yml runs coverage-matrix. - The root requirements.txt pins living-doc-utilities==0.8.0, and its Dependabot entry ignores it, as the package entries do. Closes #93

…es and improve documentation

coderabbitai · 2026-10-27T11:59:44Z

No actionable comments were generated in the recent review.

Recent review info

Run configuration

Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Organization UI
Review profile: CHILL
Plan: Advanced
Run ID: 0813b41b-95ec-44a7-a150-02232c1d51f9

Commits

Reviewing files that changed from the base of the PR and between 433148b and 36e846b.

Files selected for processing: 34

.claude/agents/test-author.md
.claude/commands/implement-task.md
.github/agents/devops-engineer.agent.md
.github/agents/senior-developer.agent.md
.github/copilot-instructions.md
.github/dependabot.yml
.github/workflows/integration.yml
DEVELOPER.md
Makefile
README.md
SPEC.md
action.yml
apps/cli/README.md
apps/cli/pyproject.toml
apps/cli/src/living_doc_cli/commands/coverage_matrix.py
apps/cli/src/living_doc_cli/main.py
apps/cli/tests/test_cli.py
apps/cli/tests/test_coverage_matrix.py
main.py
packages/services/coverage_matrix/README.md
packages/services/coverage_matrix/pyproject.toml
packages/services/coverage_matrix/src/living_doc_service_coverage_matrix/__init__.py
packages/services/coverage_matrix/src/living_doc_service_coverage_matrix/builder.py
packages/services/coverage_matrix/src/living_doc_service_coverage_matrix/lineage.py
packages/services/coverage_matrix/src/living_doc_service_coverage_matrix/service.py
packages/services/coverage_matrix/tests/conftest.py
packages/services/coverage_matrix/tests/expected/README.md
packages/services/coverage_matrix/tests/expected/coverage-matrix.json
packages/services/coverage_matrix/tests/test_builder.py
packages/services/coverage_matrix/tests/test_lineage.py
packages/services/coverage_matrix/tests/test_service.py
requirements.txt
tests/fixtures/collector-gh/README.md
tests/test_main.py

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

Walkthrough

This PR adds a coverage-matrix service and CLI command. It integrates the command into the composite action, package setup, CI, tests, and documentation.

Changes

Coverage matrix

Layer / File(s)	Summary
Build and validate coverage matrices `packages/services/coverage_matrix/src/...`, `packages/services/coverage_matrix/tests/*`, `packages/services/coverage_matrix/README.md`	The service validates documentation and test inputs, builds criterion rows and coverage summaries, records lineage and metadata, checks for field loss, and writes the artifact. Tests cover builder, service, lineage, and golden output behavior.
Register package and QA targets `Makefile`, `requirements.txt`, `.github/dependabot.yml`, `.claude/commands/implement-task.md`, `.github/agents/*`	The package is added to installation and Makefile targets. Dependabot and per-package QA guidance include the new package and alias.
Add CLI command `apps/cli/src/living_doc_cli/commands/coverage_matrix.py`, `apps/cli/src/living_doc_cli/main.py`, `apps/cli/tests/*`, `apps/cli/README.md`, `apps/cli/pyproject.toml`	The CLI registers coverage-matrix with required inputs, view selection, and an optional finite `--fail-under` threshold. Tests cover artifact output, input errors, threshold handling, and logging.
Enable action and integration runs `main.py`, `action.yml`, `tests/test_main.py`, `.github/workflows/integration.yml`, `tests/fixtures/collector-gh/README.md`	The action entry point accepts coverage-matrix after input validation. Integration checks verify successful artifact generation, output paths, metadata, and the threshold failure after writing.
Update project and command documentation `README.md`, `DEVELOPER.md`, `SPEC.md`, `.github/copilot-instructions.md`, `.claude/agents/test-author.md`	Documentation describes the command’s inputs, output, validation, coverage summaries, and threshold behavior. Maintainer instructions and package references are updated.

Priority: Normal

Estimated code review effort: 4 (Complex) | ~50 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant CLI
  participant coverage_matrix
  participant validate_inputs
  participant build_coverage_matrix
  participant artifact_writer
  CLI->>coverage_matrix: Provide documentation and tests inputs
  coverage_matrix->>validate_inputs: Validate inputs
  validate_inputs-->>coverage_matrix: Return validated inputs
  coverage_matrix->>build_coverage_matrix: Build matrix from inputs
  build_coverage_matrix-->>coverage_matrix: Return matrix result
  coverage_matrix->>artifact_writer: Write after metadata and field-loss checks
  artifact_writer-->>coverage_matrix: Return written artifact
  coverage_matrix-->>CLI: Return result and apply fail-under after writing

Possibly related PRs

AbsaOSS/living-doc-toolkit#90: Adds envelope metadata and warning-forwarding utilities, and defines the UI-test input flow used by this service.
AbsaOSS/living-doc-toolkit#88: Adds shared input validation and run-context utilities used by the coverage-matrix service.

Suggested labels: enhancement

Merge Risk: Minimal · up to 36e84

No actionable merge-blocking risk remains; the change is ready for normal merge checks.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 16.07% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Documentation-only edits and deletions within surv…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Title check		The title identifies the main change, adding the coverage matrix feature, and is concise and relevant.
Description check		The description includes a detailed Overview, Release Notes, Related issue, and the required Framework section. It documents the implementation, behavior, validation, verification, and scope.
Linked Issues check		Issue `#93` requires the coverage-matrix package, command, artifact transformation, validation, lineage checks, metadata, warning forwarding, view handling, fail-under behavior, action and CLI integrati…
Out of Scope Changes check		The changes remain within issue `#93` scope. The package, CLI, action, entry point, fixtures, golden output, documentation, dependency configuration, CI workflow, and repository registration directly su…

Full details: Linked Issues check

Explanation

Issue #93 requires the coverage-matrix package, command, artifact transformation, validation, lineage checks, metadata, warning forwarding, view handling, fail-under behavior, action and CLI integration, tests, documentation, and configuration. The PR summary reports implementation for each requirement. It also reports make qa success, coverage-matrix tests, golden and full-sample checks, and integration workflow checks.

Full details: Out of Scope Changes check

Explanation

The changes remain within issue #93 scope. The package, CLI, action, entry point, fixtures, golden output, documentation, dependency configuration, CI workflow, and repository registration directly support the requested command and integration. The agent and developer guidance updates document the same package and quality checks. No unrelated product behavior is identified.

Full details: Docstring Coverage

Explanation

Docstring coverage is 16.07% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 112 functions across 14 files. (20 skipped: 20 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches: 💡 1

📝 Generate docstrings: 💡

Commit to this branch
Create a new PR

🧪 Generate unit tests (beta)

Commit to this branch
Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{A rabbit checks each row in view
It links the tests to stories too
The matrix lands in files with care
A threshold checks the totals there
Then hops away with carrots to spare}

_{Comment @coderabbitai help to get the list of available commands.}

miroslavpojer added 2 commits October 27, 2026 12:34

Update coverage-matrix command to handle non-finite --fail-under valu…

36e846b

…es and improve documentation

miroslavpojer requested a review from tmikula-dev as a code owner October 27, 2026 11:59

miroslavpojer self-assigned this Oct 27, 2026

miroslavpojer added the enhancement New feature or request label Oct 27, 2026

miroslavpojer merged commit ff45039 into master Oct 27, 2026
27 checks passed

miroslavpojer deleted the feature/93-coverage-matrix branch October 27, 2026 12:13

miroslavpojer mentioned this pull request Oct 27, 2026

Count coverage-matrix per aspect, with the keyword and the test-link warnings #95

Closed

10 tasks

This was referenced Oct 29, 2026

#95: coverage matrix aspects #98

Merged

#99: Add a local integration script that runs the three commands over given input files #100

Merged

#101: Rebuild the repository's pages to one page structure #102

Merged

Navigation Menu

Repository navigation

#93: coverage matrix - #94

#93: coverage matrix#94
miroslavpojer merged 2 commits into
masterfrom
feature/93-coverage-matrix

miroslavpojer commented Oct 27, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Oct 27, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

miroslavpojer commented Oct 27, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Release Notes

Framework

Related

Contract Rules

Verification

Acceptance criteria

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

miroslavpojer commented Oct 27, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Oct 27, 2026 •

edited

Loading