Repository navigation
feat(checks): define checks in gulliver.toml and run with gulliver check --all (closes #80) - #105
Conversation
laaielichter
left a comment
There was a problem hiding this comment.
Thank you @shaw778 for this. You posted a plan on #80 before writing code and settled both open questions (composite keys, and run all targets rather than stopping at the first failure), which is exactly what CONTRIBUTING asks for bigger changes. The PR follows that plan closely, and your claim on #80 stands. The test matrix is green and the parser is careful: unknown keys are refused, bounds are checked for finite values and inversion, and the docs, example toml, llms.txt and SKILL.md are all updated.
I merged it onto current main (986a194) in a scratch worktree: clean merge, ruff check . passes, pytest -q gives 864 passed, 10 skipped.
SQL safety: no problem found. Configured checks go through the same features.check -> build_sql -> _gated_run path as flag-driven ones, so budget gating, the read-only gate and receipts all apply. I tried a config with where = "1=1); DROP TABLE users; --" and one with where = "length(read_text('/etc/passwd'))>0". Both came back block and the table was untouched. Config holds no secrets, and samples is capped at 0-20 and stays separately gated. Good.
A few small things need fixing before this can merge; the first two matter most.
-
A check section with no checks reports "All checks passed" (please fix). With this config,
gulliver check --allexits 0 with statusok:[checks.users] target = "users"
Passing the same empty spec via flags is refused ("give at least one check"), so this is a false green: a typo such as
uniqueleft out, or a section that was never filled in, silently passes in CI. Please refuse a[checks.<name>]section whose spec is empty, ideally inparse_check_config(aConfigErrornaming the section), sinceCheckSpecalready has an empty test. A test for it would be great. -
--allsilently ignores check flags.gulliver check --all --where "1=0" --unique fooruns every check unchanged, and the output showswhere: null. You already refuse a positional target with--all; please refuse--where,--unique,--not-null,--accepted,--range,--fresh,--min-rowsand--max-changethe same way, since someone who passes them expects them to apply.--samplesand-care fine to keep. -
Summary
next_stepwhen a target errors. If one target errors (say, a misspelt column), the summary says "0 of 2 target(s) failed data checks. Fix the data or pipeline." That is misleading, because nothing failed a data check. Please mention errored and blocked targets in the count, for example "1 errored, 0 failed of 2", and add a test for it. The exit code (1 for error) is already right. -
Smaller validation gaps in
parse_check_config.accepted = { col = [] }is accepted and then fails every non-null row. Please refuse an empty list.- The
acceptedlist form (["col=a|b"]) does not strip values but the table-with-string form does. Please make the two consistent. Whichever you pick, it should match what--accepteddoes today, which also does not strip, so keeping it unstripped in both is fine. Settings.checksis typeddict[str, Any].dict[str, CheckConfig]would be clearer, with the import done underTYPE_CHECKINGif needed to avoid a cycle.
-
Docs.
AGENTS.mdstill has the oldgulliver_checkrow and the old pipeline bullet, whilellms.txtwas updated. They mirror each other, so please update both. A line in the README saying that--alltakes no check flags would help once item 2 lands. -
Optional. A short test that
gulliver_check(name=...)through the MCP tool works would round out the "by name" path for agents. The CLI path is well covered.
Please push the fixes to your branch. For the CHANGELOG line, main now has a new entry from #95 right above yours. It merged clean today, but later merges may need a rebase on main. Nice work again, and thanks for taking an intermediate issue.
Generated by Claude Code
…eck --all (closes laaielichter#80) Signed-off-by: Masfiquil Islam <shaw778@users.noreply.github.com>
- Refuse check sections with no checks configured (raise ConfigError) - Refuse empty accepted values lists in check configurations - Keep accepted table values unstripped to match list and CLI flag behavior - Refuse check flags (--where, --unique, etc.) when --all is passed - Detail errored and blocked counts in check_all next_step summary - Type Settings.checks as dict[str, CheckConfig] - Update AGENTS.md and README.md docs to match llms.txt and --all behavior - Add comprehensive unit and CLI validation tests Signed-off-by: Masfiquil Islam <shaw778@users.noreply.github.com>
299ba72 to
6f16def
Compare
|
Thank you @laaielichter for the thorough review and guidance! I have rebased onto
All 705 tests pass locally and |
laaielichter
left a comment
There was a problem hiding this comment.
Thank you @shaw778, this is a thorough round: every point from my last review is fixed in 6f16def, with tests.
What I checked. I merged 6f16def onto current main (c48adf8) in a scratch worktree (a fast-forward, no conflict): ruff check . passes and pytest -q gives 883 passed, 10 skipped. Your own CI and CodeQL runs for this commit are green too. By hand, with a gulliver.toml:
- A
[checks.users]section with onlytarget = "users"is now refused with[checks.users] has no checks configured; give at least one check (...), so there is no false green. accepted = { country = [] }is refused; the table-with-string form is no longer stripped, so it matches the list form and--accepted.gulliver check --all --unique xis refused withdo not pass check flags (...) when using --all;--samplesand-cstill work.- When targets error, the summary reads
2 errored, 0 failed of 2 target(s). Fix the data or pipeline., anderrored_targetsandblocked_targetsare in the JSON. Settings.checksis typeddict[str, CheckConfig](withTYPE_CHECKING), and AGENTS.md, README and llms.txt now say the same thing.
The earlier SQL-safety result still holds: configured checks go through features.check and the same gated path, and samples stays capped and separately gated.
Two small notes, neither blocks the merge:
- The CHANGELOG line is the second line of
### Added, right under the #97 line. It merges cleanly now. If another PR adds a line at the top of that list first, agit merge origin/mainand keeping both lines is all it needs. - Still optional, as before: a short test of
gulliver_check(name=...)through the MCP tool.
Thank you for taking an intermediate issue, and for settling the two design questions on #80 before writing code.
Generated by Claude Code
Summary
Closes #80.
This PR implements configuration-based data quality checks in
gulliver.tomlunder[checks.<name>], allowing teams to define data checks once and run them by name (gulliver check NAME) or run all configured checks across tables in CI withgulliver check --all.Key Changes
Configuration (
src/gulliver/config.py&src/gulliver/checks.py):"checks"toKNOWN_TOP_LEVEL.CheckConfigand addedchecks: dict[str, CheckConfig]toSettings.parse_check_config: parses[checks.<name>]tables intoCheckSpec.[checks.<name>]with descriptiveConfigError(matchingBudget.from_mapping).unique: strings"col", composite string"a,b", or nested lists[["a", "b"]].not_null: list or comma-separated string.accepted: table{ col = ["a", "b"] }, pipe string{ col = "a|b" }, or list["col=a|b"].range: table{ col = [min, max] }(allowing open bounds[0, ""]or["", 50]or[10]),{ col = { min = 0, max = 100 } }, or list["col=min:max"]. Rejects inverted or non-finite bounds.fresh: table{ ts = "24h" }or list["ts=24h"](validated viaparse_period).min_rows&max_change_pct: non-negative numbers.samples: optional (0-20).merge_specs: cleanly layers CLI flags on top of configured checks (overrides or adds to rules).Execution (
src/gulliver/features.py):features.check: resolves named configured checks fromsettings.checkswhennameortargetmatches, merging any explicit CLI flags.features.check_all: executes all configured target checks, collecting per-target results and receipts. Exits with overall statusfail(exit code 4) if any check fails,block(exit code 2) if blocked, andok(exit code 0) when all checks pass.CLI (
src/gulliver/cli.py):gulliver check [NAME] [--all] [FLAGS]:gulliver check NAME: runs configured checkNAME(flags override/supplement).gulliver check --all: runs all configured checks and prints a clean summary table (check,target,status,failed,receipt,next_step).--all(parser.error).--all(parser.error).MCP (
src/gulliver/mcp_server.py&src/gulliver/core.py):gulliver_checkacceptsname: str | None = Nonealongsidetargetto run a configured check.gulliver_statusexposes configured check names under"checks".Documentation & Tests:
[checks.events]example togulliver.example.toml.README.md,llms.txt,skills/gulliver/SKILL.md, andCHANGELOG.md.tests/test_config.py,tests/test_checks.py,tests/test_features.py, andtests/test_cli_and_mcp.py.ruff check ..