Skip to content

Fix missing not_columns entries in write filters - #1228

Draft
ebhills with Copilot wants to merge 2 commits into
mainfrom
copilot/wrangle-dont-error-on-nonexistent-columns
Draft

ebhills with Copilot wants to merge 2 commits into
mainfrom
copilot/wrangle-dont-error-on-nonexistent-columns

Conversation

Copilot AI commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Linked issue

What changes

write filters were treating a missing not_columns name as a hard failure, even though the expected behavior is to ignore absent columns. This patch makes the recipe dataframe filter treat unknown not_columns entries as a no-op while preserving normal wildcard and regex exclusion behavior.

  • Updated wrangles.recipe._filter_dataframe to skip names that do not resolve to real columns before applying exclusions.
  • Kept valid not_columns selection behavior unchanged for real column names, wildcard patterns, and regex matches.
  • Added a regression test covering a write recipe that includes one missing not_columns entry alongside a valid exclusion.
write:
  - dataframe:
      not_columns:
        - missing_col
        - col2

This now correctly returns the remaining columns instead of raising an error.

How it was verified

  • python3 -m pytest -c pytest-local.ini tests/recipes/test_read.py tests/recipes/test_write.py -q

Compatibility and risk

Low risk. The change only relaxes the failure mode for missing not_columns entries; valid exclusions still apply, and the behavior is directly covered by focused regression tests. Safe rollback is a revert of the filter change if needed.

Ready-for-review checklist

  • One human delivery owner is assigned
  • The linked issue and intended milestone are correct
  • The branch is current with main and has no merge conflicts
  • Focused tests pass
  • New or changed behavior has direct test coverage
  • Documentation/schema/configuration is updated where applicable
  • The PR contains no unrelated changes
  • The PR description reflects the branch's current scope and latest validation
  • One primary reviewer is requested only when this PR is ready

See the pull request workflow.

Copilot AI linked an issue Oct 11, 2026 that may be closed by this pull request
Co-authored-by: ebhills <53243273+ebhills@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix not_columns to ignore nonexistent columns without error Fix missing not_columns entries in write filters Oct 11, 2026
Copilot AI requested a review from ebhills October 11, 2026 00:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

not_columns should not error

2 participants