Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
102 commits
Select commit Hold shift + click to select a range
6790dd3
parser,ast: support column masking grammar
tiancaiamao Jan 22, 2026
b2a6e6a
meta: update meta data for column masking
tiancaiamao Jan 22, 2026
7da4953
meta,session: create mysql.tidb_masking_policy for column masking
tiancaiamao Jan 22, 2026
eaeae4a
infoschema: infoSchema integration for column masking
tiancaiamao Jan 22, 2026
c8f5249
ddl: implement CREATE/ALTER/DROP masking policy DDL operations
tiancaiamao Jan 23, 2026
3391ca8
*: add 'show masking policies' statement
tiancaiamao Jan 23, 2026
9fd16d1
planner: apply masking in projection
tiancaiamao Feb 4, 2026
2432af0
expression: add masking builtins
tiancaiamao Feb 4, 2026
b22081c
*: add dynamic privilege and errer code for column masking
tiancaiamao Feb 4, 2026
48a0643
masking: support RESTRICT ON and masking_type updates
tiancaiamao Feb 24, 2026
f665959
planner: add masking tests for blob and clob columns
tiancaiamao Feb 24, 2026
9917a88
ddl, privilege: cover ALTER MASKING POLICY expression/restrict privil…
tiancaiamao Feb 24, 2026
3cae632
ddl: test current_user/current_role operators in masking policy
tiancaiamao Feb 24, 2026
6f1f992
executor: cover masking_type and restrict_on in SHOW MASKING POLICIES
tiancaiamao Feb 24, 2026
50104d9
expression: add integration coverage for blob/clob masking functions
tiancaiamao Feb 24, 2026
faa2f86
planner: test current_user/current_role operator behavior in masking
tiancaiamao Feb 24, 2026
3339418
privilege: add integration coverage for RESTRICT ON runtime denial
tiancaiamao Feb 24, 2026
9b05063
docs: add column-level masking design and diagrams
tiancaiamao Feb 27, 2026
342c2f0
docs: add column-level masking design and diagrams
tiancaiamao Feb 27, 2026
ce32668
Merge master into pr/phase2-meta
tiancaiamao Feb 27, 2026
35f4ab8
Merge commit '2f82da0171' into pr/phase3-system-table
tiancaiamao Feb 27, 2026
4185498
Merge commit '2f82da0171' into pr/phase4-infoschema
tiancaiamao Feb 27, 2026
1d0b794
meta,parser: fix masking policy review issues
tiancaiamao Feb 27, 2026
ac31dba
make bazel_prepare
tiancaiamao Feb 27, 2026
949d223
meta,errno: add dedicated masking policy error codes
tiancaiamao Feb 28, 2026
2d9aeb1
meta: include masking policy id in existence errors
tiancaiamao Feb 28, 2026
3466cb7
meta: avoid GenWithStack in masking policy existence checks
tiancaiamao Feb 28, 2026
f796305
meta: align masking policy metadata with latest schema
tiancaiamao Feb 28, 2026
aa15e94
meta: include policy id in GetMaskingPolicy error
tiancaiamao Feb 28, 2026
1c9cde2
Merge remote-tracking branch 'tiancaiamao/pr/phase2-meta' into pr/pha…
tiancaiamao Feb 28, 2026
792b17a
meta: align tidb_masking_policy system table schema
tiancaiamao Feb 28, 2026
f89e569
Merge origin/master into pr/phase3-system-table
tiancaiamao Mar 1, 2026
7fa2969
Merge origin/master into pr/phase4-infoschema
tiancaiamao Mar 1, 2026
f711cb9
parser: regenerate keywords after go generate
tiancaiamao Mar 2, 2026
babd1fe
executor: fix infoschema builder call in slow query test
tiancaiamao Mar 2, 2026
49f8aed
executor: fix infoschema builder call in slow query test
tiancaiamao Mar 2, 2026
b059607
Revert "executor: fix infoschema builder call in slow query test"
tiancaiamao Mar 2, 2026
73885a5
executor: align slow query test with infoschema builder args
tiancaiamao Mar 2, 2026
bd97897
executor: fix slow query builder args in helper
tiancaiamao Mar 2, 2026
0b665bc
importer: make query chunk checksum test table-id agnostic
tiancaiamao Mar 2, 2026
7bdb974
br: sync system table monitor with bootstrap version 255
tiancaiamao Mar 2, 2026
2d502fa
importer: make query chunk checksum test table-id agnostic
tiancaiamao Mar 2, 2026
4317a6b
br: sync system table monitor with bootstrap version 255
tiancaiamao Mar 2, 2026
73cbf2f
dxf/importinto: make collect-conflicts checksum test table-id agnostic
tiancaiamao Mar 2, 2026
2e91811
Merge branch 'master' into demo
tiancaiamao Mar 5, 2026
543acc8
Merge branch 'pr/phase3-system-table' into demo
tiancaiamao Mar 5, 2026
1044121
Merge branch 'pr/phase4-infoschema' into demo
tiancaiamao Mar 5, 2026
5cad50e
Merge branch 'pr/docs-column-level-masking-design-20260227' into demo
tiancaiamao Mar 5, 2026
4f3ab64
pkg/ddl, infoschema, tests: harden column masking policy and expand c…
tiancaiamao Mar 6, 2026
f72cc51
tests/integrationtest, .agents: add more masking P1 intersection cove…
tiancaiamao Mar 6, 2026
ad48f96
ddl, expression: fix gosec and sync generated thread-safe funcs
tiancaiamao Mar 10, 2026
b418ae9
tests/integrationtest, .agents: add more masking P1 intersection cove…
tiancaiamao Mar 6, 2026
d9a8f17
fix CI: add masking functions to knownGood list and remove duplicate …
tiancaiamao Mar 11, 2026
5bff686
tests, errdoc: sync mask_date expectations and error codes
tiancaiamao Mar 11, 2026
01dc53c
agents: add license headers for validation scripts
tiancaiamao Mar 11, 2026
8968006
masking: persist restrict_on in system table and refresh via mysql so…
tiancaiamao Mar 11, 2026
cf3633c
test: fix bootstrap schema id check and show privileges output
tiancaiamao Mar 11, 2026
4c1b787
build: include masking policy loader in infoschema bazel target
tiancaiamao Mar 11, 2026
7149075
ddl: avoid dynamic SQL concatenation in masking policy loader
tiancaiamao Mar 11, 2026
d8e2cbb
tests/realtikv: tolerate missing table during nocache cleanup
tiancaiamao Mar 11, 2026
6f79ae8
tests/realtikv: add missing bazel deps for new cleanup imports
tiancaiamao Mar 11, 2026
7e72d49
Merge origin/master into demo
tiancaiamao Mar 11, 2026
5ef21b7
infoschema: avoid reload lock inversion with masking policy load
tiancaiamao Mar 11, 2026
8b5a20a
masking: fix enum status decode and avoid reload recursive lock
tiancaiamao Mar 11, 2026
e1f96c6
ddl: fix masking modify-column guard test expectations
tiancaiamao Mar 11, 2026
f900025
infoschema: refresh masking policy diffs from system table
tiancaiamao Mar 11, 2026
0e2b548
test: update expected mysql system table count to 61
tiancaiamao Mar 12, 2026
36d4964
sessiontxn: fix nil pointer in GetTxnInfoSchema
tiancaiamao Mar 12, 2026
7b061d9
ddl: fix masking policy uniqueness to be database-scoped
tiancaiamao Mar 12, 2026
a9b9f4a
Merge remote-tracking branch 'origin/master' into demo
tiancaiamao Mar 27, 2026
9abd5b7
docs: sync column masking design doc from #67287
tiancaiamao Mar 27, 2026
cd7faf8
testplan: sync column masking P0 scenario matrix from #67287
tiancaiamao Mar 27, 2026
59acd1a
fix: Update masking builtin function signatures to match implementation
tiancaiamao Mar 16, 2026
03b35b8
fix: update masking policy names in sys table after RENAME TABLE
tiancaiamao Mar 16, 2026
6c0f14d
planner, executor: apply masking policy in BatchPointGet path
tiancaiamao Mar 16, 2026
812818d
fix: apply masking policy in PointGet path
tiancaiamao Mar 16, 2026
4dcab91
fix: respect masking policy at_result semantics
tiancaiamao Mar 16, 2026
e10b8df
ddl: fail closed on invalid masking policy expressions
tiancaiamao Mar 18, 2026
0151dea
planner: fix panic for masking restrict insert-select
tiancaiamao Mar 18, 2026
c5b2e44
fix: finish remaining masking fixes from #67287
tiancaiamao Mar 27, 2026
06518e4
Merge branch 'master' into demo
tiancaiamao Mar 30, 2026
5cb13a0
chore: update bazel file
tiancaiamao Mar 30, 2026
c671ebf
fix masking policy planning and integration regressions
tiancaiamao Mar 30, 2026
18f4879
fix: address next-gen CI regressions in masking policy branch
tiancaiamao Mar 30, 2026
86f8020
test: fix flaky expectations in unit and integration checks
tiancaiamao Mar 30, 2026
cec2d62
test: fix ci regressions on demo masking branch
tiancaiamao Mar 30, 2026
d713f18
chore: update bazel file
tiancaiamao Mar 30, 2026
1b940c8
fix masking policy CI regressions
tiancaiamao Mar 30, 2026
5f3fb70
tests: make initstats ID assertion dynamic and relax rg test timeout
tiancaiamao Mar 31, 2026
ef23942
chore: update bazel file
tiancaiamao Mar 31, 2026
359263e
test: stabilize traceevent flight recorder category assertion
tiancaiamao Mar 31, 2026
ff4e873
planner, privilege: fix CTE masking policy to use original values in …
tiancaiamao Apr 1, 2026
0bbc93c
skills: add CTE masking test to column-masking-auto-validation
tiancaiamao Apr 1, 2026
65ff934
planner: fix CTE inline merge to not set buildingCTE flag
tiancaiamao Apr 1, 2026
8893763
planner: fix CTE inline merge to preserve b.outerCTEs for buildDistinct
tiancaiamao Apr 1, 2026
0d346b5
chore: remove erroneous bazel symlink file
tiancaiamao Apr 1, 2026
6578b1e
Merge branch 'master' into demo
tiancaiamao Apr 9, 2026
085cca2
WIP: fix column masking CTE issues - SET ROLE cache invalidation + CT…
tiancaiamao Apr 9, 2026
01a6eba
fix: apply masking policy for CTE consumption and SET ROLE cache inva…
tiancaiamao Apr 9, 2026
a3dc083
WIP: fix CTE masking - adjustCTEPlanOutputName preserves OrigTblName,…
tiancaiamao Apr 9, 2026
0aff74b
fix: CTE masking policy and mask_partial function
tiancaiamao Apr 9, 2026
c5d72e7
ddl, infoschema: fix masking policy uniqueness to table scope
tiancaiamao Apr 10, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .agents/skills/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,13 @@ This repository stores TiDB's repo-level skills under `.agents/skills`.
- Put shared repository skills here.
- Keep skill-specific references under each skill folder (for example: `tidb-test-guidelines/references/`).

## Skill Index

- `tidb-test-guidelines`: test placement and writing conventions.
- `column-masking-auto-validation`: run column masking validation and generate scenario-level report automatically.

## Workflow Skills

Current operational workflow skills:

- `tidb-verify-profile`: choose WIP/Ready/Heavy validation scope before running checks.
Expand Down
59 changes: 59 additions & 0 deletions .agents/skills/column-masking-auto-validation/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
---
name: column-masking-auto-validation
description: Run TiDB column masking automated validation and generate a human-readable report from a fixed scenario matrix. Use when you need repeatable feature validation + report generation.
---

# Column Masking Auto Validation

## Purpose

This skill provides a repeatable workflow to validate the column masking feature and generate a human-readable report automatically.

The output is:

- one execution artifact directory (`artifacts/column-masking/<timestamp>/`)
- one markdown report with scenario-level status and evidence

## What this skill includes

- feature test plan (human-facing): `references/column-masking-test-plan.md`
- report template: `references/report-template.md`
- scenario matrix (P0): `references/p0-scenario-matrix.json`
- scenario matrix (P1): `references/p1-scenario-matrix.json`
- automation scripts:
- `scripts/run_validation.sh`
- `scripts/generate_report.py`

## Standard workflow

1. Run validation:

```bash
./.agents/skills/column-masking-auto-validation/scripts/run_validation.sh
```

2. For stricter CI-like checks, include prepare/lint:

```bash
./.agents/skills/column-masking-auto-validation/scripts/run_validation.sh --with-bazel-prepare --with-lint
```

3. Read generated report:

- `artifacts/column-masking/<timestamp>/column-masking-report.md`

## Optional modes

- Generate report from an existing artifacts directory:

```bash
./.agents/skills/column-masking-auto-validation/scripts/run_validation.sh --skip-tests --artifacts-dir <existing_artifacts_dir>
```

## Notes

- The script uses repository-approved test commands (targeted unit tests + integration tests).
- The final human-facing outputs are:
- the test plan (`references/column-masking-test-plan.md`)
- the generated report in artifacts (created every run)
- Do not maintain static conclusion documents under `docs/design` for this feature.
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
# Column Masking Test Plan

## 1. Goal and Scope

This plan validates the TiDB column masking feature for:

- correctness
- security and permission boundaries
- DDL lifecycle consistency
- observability and metadata consistency
- compatibility across key SQL/runtime features
- upgrade/downgrade impact

Primary feature scope:

- `CREATE/ALTER/DROP MASKING POLICY`
- `AT RESULT` semantics (compute on original values, return masked values)
- `RESTRICT ON` runtime controls
- masking functions (`MASK_FULL`, `MASK_PARTIAL`, `MASK_NULL`, `MASK_DATE`)
- visibility surfaces (`SHOW MASKING POLICIES`, `SHOW CREATE TABLE`, `mysql.tidb_masking_policy`)

Out of scope for this plan:

- standalone log-redaction feature validation
- full toolchain performance benchmark framework (only targeted gates here)

## 2. Test Strategy

Use automated tests as default:

- unit tests for function logic and planner/DDL internals
- integration tests for SQL-visible behavior and end-to-end semantics

Execution policy:

- prioritize deterministic targeted tests
- avoid one-off ad-hoc scripts for behavioral validation
- generate a scenario-level report automatically from the skill

## 3. Coverage Model

### 3.1 Priority Levels

- `P0`: must-have for GA decision
- `P1`: strong recommendation before GA
- `P2`: ecosystem/toolchain and longer-path validation

### 3.2 P0 Scenario Groups

- `P0-DDL`: lifecycle, constraints, unsupported objects/columns, binding stability
- `P0-AUTH`: dynamic privileges and identity-function semantics
- `P0-CORE`: `AT RESULT` correctness in query pipeline
- `P0-RES`: `RESTRICT ON` deny/allow behavior
- `P0-FUNC`: masking builtin behavior and boundaries
- `P0-OBS`: metadata and `SHOW` consistency

### 3.3 Test Surfaces

- Integration:
- `tests/integrationtest/t/privilege/column_masking_policy.test`
- `tests/integrationtest/t/privilege/column_masking_cte.test`
- Unit:
- `pkg/ddl/masking_policy_test.go`
- `pkg/planner/core/masking_policy_projection_test.go`
- `pkg/planner/core/masking_policy_restrict_test.go`
- `pkg/planner/core/masking_policy_expr_cache_test.go`
- `pkg/executor/show_test.go`
- Integration (masking builtins):
- `tests/integrationtest/t/expression/builtin.test`

### 3.4 P1 Scenario Groups

- `P1-CACHE`: prepared statement / schema cache invalidation after policy and column metadata changes
- `P1-COMPAT`: partition tables and transaction mode intersections
- `P1-RES`: `RESTRICT ON` behavior with prepared DML statements

## 4. Compatibility and Intersections

This plan explicitly tracks intersections with:

- SQL operators and clauses (`WHERE`, `JOIN`, `GROUP BY`, `ORDER BY`, projection expressions)
- prepared statements and plan/schema cache invalidation
- transaction modes (pessimistic/optimistic)
- partitioned tables and index access paths
- observability features (`SHOW`, statement summary, slow log)
- management/tooling surfaces listed in TiDB basic features

Reference:

- https://docs.pingcap.com/zh/tidb/stable/basic-features/
- https://docs.pingcap.com/zh/tidb/stable/basic-features/#管理可视化和工具

## 5. Performance and Stability Gates

### 5.1 Performance Gates

- single-column masking query: latency/QPS regression gate
- multi-column masked projection: latency/CPU regression gate
- `RESTRICT ON` rejection path: stable rejection latency and error behavior
- masking builtin micro-benchmark trend monitoring

### 5.2 Stability Gates

- long-run mixed workload (reads + restricted writes + policy changes)
- high-frequency policy toggles and updates (no cache stale behavior)
- large policy-metadata volume operations

## 6. Version and Lifecycle Validation

Required paths:

- upgrade from pre-feature versions to feature versions
- patch upgrade within feature-enabled versions
- rolling upgrade with mixed binaries
- downgrade strategy validation (block or pre-cleanup, per product definition)
- BR/PITR restore with policy metadata and identity semantics checks

## 7. Known Constraints / N-A Areas

- `CREATE TABLE ... SELECT` (`CTAS`) may be unsupported in the current branch; mark related scenario `N/A` until implemented.
- if implementation has known open behavior (for example some rename edge paths), mark as `PARTIAL` with evidence in the generated report.

## 8. Acceptance Criteria (GA-Facing)

Minimum for GA decision:

1. all `P0` scenarios are automated and mapped to concrete test surfaces
2. generated report clearly shows `PASS/FAIL/PARTIAL/NOT_COVERED/N/A` by scenario
3. high-risk `P1` intersections (cache, transaction, partition, prepare) are covered and pass
4. no unresolved correctness/security blockers in report open items

## 9. Execution and Reporting Workflow

Run using this skill:

```bash
./.agents/skills/column-masking-auto-validation/scripts/run_validation.sh
```

Optional strict mode:

```bash
./.agents/skills/column-masking-auto-validation/scripts/run_validation.sh --with-bazel-prepare --with-lint
```

Output model:

- human-facing plan: this document
- machine-executed evidence: artifacts logs and step records
- human-facing result: autogenerated report in artifacts directory (`column-masking-report.md`)
Loading
Loading