Skip to content

feat(checks): define checks in gulliver.toml and run with gulliver check --all (closes #80) - #105

Merged
laaielichter merged 2 commits into
laaielichter:mainfrom
shaw778:feat/data-checks-config-and-check-all
Nov 11, 2026
Merged

laaielichter merged 2 commits into
laaielichter:mainfrom
shaw778:feat/data-checks-config-and-check-all

Conversation

@shaw778

@shaw778 shaw778 commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes #80.

This PR implements configuration-based data quality checks in gulliver.toml under [checks.<name>], allowing teams to define data checks once and run them by name (gulliver check NAME) or run all configured checks across tables in CI with gulliver check --all.

Key Changes

  1. Configuration (src/gulliver/config.py & src/gulliver/checks.py):

    • Added "checks" to KNOWN_TOP_LEVEL.
    • Defined CheckConfig and added checks: dict[str, CheckConfig] to Settings.
    • Implemented parse_check_config: parses [checks.<name>] tables into CheckSpec.
    • Refuses unknown keys in [checks.<name>] with descriptive ConfigError (matching Budget.from_mapping).
    • Supports flexible shapes:
      • unique: strings "col", composite string "a,b", or nested lists [["a", "b"]].
      • not_null: list or comma-separated string.
      • accepted: table { col = ["a", "b"] }, pipe string { col = "a|b" }, or list ["col=a|b"].
      • range: table { col = [min, max] } (allowing open bounds [0, ""] or ["", 50] or [10]), { col = { min = 0, max = 100 } }, or list ["col=min:max"]. Rejects inverted or non-finite bounds.
      • fresh: table { ts = "24h" } or list ["ts=24h"] (validated via parse_period).
      • min_rows & max_change_pct: non-negative numbers.
      • samples: optional (0-20).
    • Added merge_specs: cleanly layers CLI flags on top of configured checks (overrides or adds to rules).
  2. Execution (src/gulliver/features.py):

    • features.check: resolves named configured checks from settings.checks when name or target matches, merging any explicit CLI flags.
    • features.check_all: executes all configured target checks, collecting per-target results and receipts. Exits with overall status fail (exit code 4) if any check fails, block (exit code 2) if blocked, and ok (exit code 0) when all checks pass.
  3. CLI (src/gulliver/cli.py):

    • gulliver check [NAME] [--all] [FLAGS]:
      • gulliver check NAME: runs configured check NAME (flags override/supplement).
      • gulliver check --all: runs all configured checks and prints a clean summary table (check, target, status, failed, receipt, next_step).
      • Refuses target with --all (parser.error).
      • Refuses check invocation without a target or --all (parser.error).
  4. MCP (src/gulliver/mcp_server.py & src/gulliver/core.py):

    • gulliver_check accepts name: str | None = None alongside target to run a configured check.
    • gulliver_status exposes configured check names under "checks".
  5. Documentation & Tests:

    • Added [checks.events] example to gulliver.example.toml.
    • Updated README.md, llms.txt, skills/gulliver/SKILL.md, and CHANGELOG.md.
    • Added comprehensive tests across tests/test_config.py, tests/test_checks.py, tests/test_features.py, and tests/test_cli_and_mcp.py.
    • 100% clean with ruff check ..

@laaielichter laaielichter left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @shaw778 for this. You posted a plan on #80 before writing code and settled both open questions (composite keys, and run all targets rather than stopping at the first failure), which is exactly what CONTRIBUTING asks for bigger changes. The PR follows that plan closely, and your claim on #80 stands. The test matrix is green and the parser is careful: unknown keys are refused, bounds are checked for finite values and inversion, and the docs, example toml, llms.txt and SKILL.md are all updated.

I merged it onto current main (986a194) in a scratch worktree: clean merge, ruff check . passes, pytest -q gives 864 passed, 10 skipped.

SQL safety: no problem found. Configured checks go through the same features.check -> build_sql -> _gated_run path as flag-driven ones, so budget gating, the read-only gate and receipts all apply. I tried a config with where = "1=1); DROP TABLE users; --" and one with where = "length(read_text('/etc/passwd'))>0". Both came back block and the table was untouched. Config holds no secrets, and samples is capped at 0-20 and stays separately gated. Good.

A few small things need fixing before this can merge; the first two matter most.

  1. A check section with no checks reports "All checks passed" (please fix). With this config, gulliver check --all exits 0 with status ok:

    [checks.users]
    target = "users"

    Passing the same empty spec via flags is refused ("give at least one check"), so this is a false green: a typo such as unique left out, or a section that was never filled in, silently passes in CI. Please refuse a [checks.<name>] section whose spec is empty, ideally in parse_check_config (a ConfigError naming the section), since CheckSpec already has an empty test. A test for it would be great.

  2. --all silently ignores check flags. gulliver check --all --where "1=0" --unique foo runs every check unchanged, and the output shows where: null. You already refuse a positional target with --all; please refuse --where, --unique, --not-null, --accepted, --range, --fresh, --min-rows and --max-change the same way, since someone who passes them expects them to apply. --samples and -c are fine to keep.

  3. Summary next_step when a target errors. If one target errors (say, a misspelt column), the summary says "0 of 2 target(s) failed data checks. Fix the data or pipeline." That is misleading, because nothing failed a data check. Please mention errored and blocked targets in the count, for example "1 errored, 0 failed of 2", and add a test for it. The exit code (1 for error) is already right.

  4. Smaller validation gaps in parse_check_config.

    • accepted = { col = [] } is accepted and then fails every non-null row. Please refuse an empty list.
    • The accepted list form (["col=a|b"]) does not strip values but the table-with-string form does. Please make the two consistent. Whichever you pick, it should match what --accepted does today, which also does not strip, so keeping it unstripped in both is fine.
    • Settings.checks is typed dict[str, Any]. dict[str, CheckConfig] would be clearer, with the import done under TYPE_CHECKING if needed to avoid a cycle.
  5. Docs. AGENTS.md still has the old gulliver_check row and the old pipeline bullet, while llms.txt was updated. They mirror each other, so please update both. A line in the README saying that --all takes no check flags would help once item 2 lands.

  6. Optional. A short test that gulliver_check(name=...) through the MCP tool works would round out the "by name" path for agents. The CLI path is well covered.

Please push the fixes to your branch. For the CHANGELOG line, main now has a new entry from #95 right above yours. It merged clean today, but later merges may need a rebase on main. Nice work again, and thanks for taking an intermediate issue.


Generated by Claude Code

…eck --all (closes laaielichter#80)

Signed-off-by: Masfiquil Islam <shaw778@users.noreply.github.com>
- Refuse check sections with no checks configured (raise ConfigError)
- Refuse empty accepted values lists in check configurations
- Keep accepted table values unstripped to match list and CLI flag behavior
- Refuse check flags (--where, --unique, etc.) when --all is passed
- Detail errored and blocked counts in check_all next_step summary
- Type Settings.checks as dict[str, CheckConfig]
- Update AGENTS.md and README.md docs to match llms.txt and --all behavior
- Add comprehensive unit and CLI validation tests

Signed-off-by: Masfiquil Islam <shaw778@users.noreply.github.com>
@shaw778
shaw778 force-pushed the feat/data-checks-config-and-check-all branch from 299ba72 to 6f16def Compare November 10, 2026 21:08
@shaw778

shaw778 commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

Thank you @laaielichter for the thorough review and guidance! I have rebased onto main and addressed all feedback items:

  1. Refuse empty [checks.<name>] sections: In parse_check_config, we now check if spec.is_empty(): raise ConfigError(...), ensuring sections lacking checks fail with a clear configuration error. Tested in tests/test_config.py.
  2. Refuse check flags when --all is passed: gulliver check --all now checks and rejects any passed check flags (--where, --unique, --not-null, --accepted, --range, --fresh, --min-rows, --max-change) with parser.error(...). Added CLI validation tests in tests/test_features.py.
  3. Summary next_step when a target errors or blocks: check_all now explicitly counts and details errored and blocked targets alongside failed targets (e.g. "1 errored, 0 failed of 2 target(s)"). Added test in tests/test_features.py.
  4. Validation gaps in parse_check_config:
    • Refuse empty lists for accepted values (ConfigError when col = []).
    • Kept accepted table-with-string values unstripped to match the list form and CLI flag semantics.
    • Typed Settings.checks as dict[str, CheckConfig] using TYPE_CHECKING import.
  5. Documentation & Changelog:
  6. MCP coverage: The existing test_cli_and_mcp.py test suite covers gulliver_check(name=...) and unknown check handling through call_tool.

All 705 tests pass locally and ruff check . is 100% clean.

@laaielichter laaielichter left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @shaw778, this is a thorough round: every point from my last review is fixed in 6f16def, with tests.

What I checked. I merged 6f16def onto current main (c48adf8) in a scratch worktree (a fast-forward, no conflict): ruff check . passes and pytest -q gives 883 passed, 10 skipped. Your own CI and CodeQL runs for this commit are green too. By hand, with a gulliver.toml:

  • A [checks.users] section with only target = "users" is now refused with [checks.users] has no checks configured; give at least one check (...), so there is no false green.
  • accepted = { country = [] } is refused; the table-with-string form is no longer stripped, so it matches the list form and --accepted.
  • gulliver check --all --unique x is refused with do not pass check flags (...) when using --all; --samples and -c still work.
  • When targets error, the summary reads 2 errored, 0 failed of 2 target(s). Fix the data or pipeline., and errored_targets and blocked_targets are in the JSON.
  • Settings.checks is typed dict[str, CheckConfig] (with TYPE_CHECKING), and AGENTS.md, README and llms.txt now say the same thing.

The earlier SQL-safety result still holds: configured checks go through features.check and the same gated path, and samples stays capped and separately gated.

Two small notes, neither blocks the merge:

  • The CHANGELOG line is the second line of ### Added, right under the #97 line. It merges cleanly now. If another PR adds a line at the top of that list first, a git merge origin/main and keeping both lines is all it needs.
  • Still optional, as before: a short test of gulliver_check(name=...) through the MCP tool.

Thank you for taking an intermediate issue, and for settling the two design questions on #80 before writing code.


Generated by Claude Code

@laaielichter
laaielichter merged commit ece385e into laaielichter:main Nov 11, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Data checks in gulliver.toml: define them once, run them all with gulliver check --all

2 participants