Repository navigation
fix(incremental): refuse a --time-column or --unique-key that does not exist (#157) - #165
Conversation
Check the time column against the source table and the unique key against the model's output columns (case-insensitive) before generating advice. Column names are read with LIMIT 0 through the guarded path. Without a connection, or when the columns cannot be read, nothing is checked and a note says so. Closes laaielichter#157
|
Ran locally before opening: |
laaielichter
left a comment
There was a problem hiding this comment.
Approved on eeebf15. Thank you, @Ipsita-Mondal-30, a second solid PR. Reading names with SELECT * ... LIMIT 0 through _gated_run, and turning every "cannot check" case into a note instead of a failure, is what the issue asked for.
What I checked, on this head merged with current main (no conflicts with main):
ruff check .clean; fullpytest892 passed, 10 skipped.- Before (main):
--time-column nosuchgivesstatus: ok, exit 0, with a failing savings line;--unique-key nosuchgivesstatus: ok. After: exit 1,Time column 'nosuch' is not a column of events.withtslisted, andUnique key column(s) not returned by the model: nosuchwithuser_id, last_ts, nlisted. A bad name in--unique-key user_id,nosuchis caught too. - Valid names,
TS/USER_IDin other case, andevents.tsgive the same output as main. - A missing table and a dbt
{{ ref() }}model still return their advice (the missing table with a note that the check was skipped). - Mutation check: with
src/gulliver/features.pyreverted to main, the three refusal and note tests fail. - The diff is limited to
features.py,tests/test_features.pyandCHANGELOG.md.
One small non-blocking suggestion, which can be a follow-up or an extra commit if you like: a quoted name is compared with its quotes, so a column that does exist is refused. --unique-key '"user_id"' now gives Unique key column(s) not returned by the model: "user_id" (main returned ok), and --time-column '"ts"' says '"ts"' is not a column of events. Stripping one pair of surrounding double quotes before the comparison (and a test for it) would fix both. I am not holding the PR for it.
Ready to merge from my side. It still needs the maintainer to approve the first-time CI runs (action_required) and the 14 checks have to pass. I will add the "Thanks to @Ipsita-Mondal-30" credit to the CHANGELOG line myself. #164 adds a line at the same spot in CHANGELOG.md, so whichever of the two lands second will need main merged and both lines kept.
Generated by Claude Code
|
@Ipsita-Mondal-30 a heads-up: #164 landed just now and added its CHANGELOG line in Generated by Claude Code |
|
Good news: you no longer need to resolve the CHANGELOG conflict. We changed the rule so contributors don't edit CHANGELOG.md any more; we add your entry with credit when we merge. Easiest fix: drop your CHANGELOG.md change (or merge main and take main's version of the file) and push; the rest of the PR is unchanged. Generated by Claude Code |
|
Hi @Hayabusa-YL, welcome to Gulliver, and thanks for the careful write-up on #140. #165 is not free to take over: #157 is assigned to @Ipsita-Mondal-30, and #165 is her open PR for it. Please don't open a second PR for #157 or #165. A free issue to start with is #159, a good first issue with no code needed: translate the README quick start into a language of your choice. Comment on #159 with the language you take and it is yours. If you would rather write code, #125 (lint rule GV019, with a same-rows autofix) is free too. Contributing guide: CONTRIBUTING.md. Setup is If Gulliver is useful to you, a ⭐ helps other people find it. Generated by Claude Code |
|
Merged main into the branch and took main's CHANGELOG.md, so this PR no longer touches it and the conflict is gone. Full suite (1089 passed, 10 skipped) and ruff pass locally. Could you approve the workflow run again when you have a moment? |
laaielichter
left a comment
There was a problem hiding this comment.
Re-verified on d25017d, merged with current main, CHANGELOG.md taken from main:
ruff check .clean; fullpytest1089 passed, 10 skipped.- Repro of #157 on main and on this head:
--time-column nosuchand--unique-key nosuchnow givestatus: errorwith the columns listed. Valid names, and mixed case such asTS/USER_ID, give the same output as main. - Mutation check: with
features.pyreverted to main, the three new tests fail (unknown time column, unknown unique key, skipped-check note). - All 14 checks are green on this head.
Still open from my earlier review, non-blocking: a quoted name such as --time-column '"ts"' is refused although main accepted it. Stripping one pair of surrounding double quotes would fix that, as a follow-up or a small extra commit.
Thanks, @Ipsita-Mondal-30. Merging now; the CHANGELOG credit goes in with the credits PR.
Generated by Claude Code
docs(changelog): credit @Ipsita-Mondal-30 for #157 (#165)
Closes #157
gulliver incrementalcopied--time-columnand--unique-keyinto the generated model without checking them, and returnedstatus: okwith exit 0.Change (
src/gulliver/features.py)_check_incremental_namesruns after the strategy is known and before the savings query:--time-columnmust be a column of the source table (time_column_source). Otherwise:status: error,Time column 'nosuch' is not a column of events., and anext_steplisting the timestamp-like columns (or all columns if none look like timestamps). Exit code 1.--unique-keyname must be an output column of the model. Otherwise:Unique key column(s) not returned by the model: nosuch, with the model's columns innext_step.SELECT * ... LIMIT 0through_gated_run, so the extra queries go through the same guarded path as the savings queries.features.incremental(None, ...)): nothing is checked, output unchanged. If the source or the rendered model cannot be queried (for example a dbt model that does not run as plain SQL), the check is skipped and anotesentry says so.Tests (
tests/test_features.py): unknown time column, unknown unique key, mixed case, valid names equal to the offline advice, no connection, and the skipped-check note. CHANGELOG line added.pytest(892 passed, 10 skipped) andruff check .pass.