Skip to content

Commit 12dfe32

Browse files
andreyorstandreyorst
committed
add de-identification section to SOF
1 parent fae5bbc commit 12dfe32

4 files changed

Lines changed: 345 additions & 0 deletions

File tree

‎SUMMARY.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -376,6 +376,7 @@
376376
* [Defining flat views with view definitions](modules/sql-on-fhir/defining-flat-views-with-view-definitions.md)
377377
* [Migrate to the spec-compliant ViewDefinition format](modules/sql-on-fhir/migrate-to-the-spec-compliant-viewdefinition-format.md)
378378
* [Query data from flat views](modules/sql-on-fhir/query-data-from-flat-views.md)
379+
* [De-identification](modules/sql-on-fhir/de-identification.md)
379380
* [Reference](modules/sql-on-fhir/reference.md)
380381
* [Integration Toolkit](modules/integration-toolkit/README.md)
381382
* [C-CDA / FHIR Converter](modules/integration-toolkit/ccda-converter/README.md)

‎docs/SUMMARY.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -383,6 +383,7 @@
383383
* [Defining flat views with view definitions](modules/sql-on-fhir/defining-flat-views-with-view-definitions.md)
384384
* [Migrate to the spec-compliant ViewDefinition format](modules/sql-on-fhir/migrate-to-the-spec-compliant-viewdefinition-format.md)
385385
* [Query data from flat views](modules/sql-on-fhir/query-data-from-flat-views.md)
386+
* [De-identification](modules/sql-on-fhir/de-identification.md)
386387
* [Reference](modules/sql-on-fhir/reference.md)
387388
* [Integration Toolkit](modules/integration-toolkit/README.md)
388389
* [C-CDA / FHIR Converter](modules/integration-toolkit/ccda-converter/README.md)

‎docs/modules/sql-on-fhir/README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,12 @@ Once your flat view is defined and materialized, you can query data from it usin
2222

2323
See [Query data from flat views](./query-data-from-flat-views.md).
2424

25+
## De-identification
26+
27+
ViewDefinition columns can be annotated with de-identification methods to transform sensitive data during SQL generation. Supported methods include redact, cryptoHash, dateshift, encrypt, substitute, perturb, and custom PostgreSQL functions.
28+
29+
See [De-identification](./de-identification.md).
30+
2531
## SQL on FHIR reference
2632

2733
To dive deeper into the nuances of using SQL on FHIR in Aidbox, consult the reference page.

‎docs/modules/sql-on-fhir/de-identification.md‎

Lines changed: 337 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,337 @@
1+
# De-identification
2+
3+
Aidbox supports per-column de-identification in ViewDefinitions via a FHIR extension. When a column has a de-identification extension, the SQL compiler wraps the column expression with a PostgreSQL function that transforms the value before it reaches the output.
4+
5+
This works with all ViewDefinition operations: `$run`, `$sql`, and `$materialize`.
6+
7+
## Extension format
8+
9+
Add the de-identification extension to any column in the `select` array:
10+
11+
```json
12+
{
13+
"name": "birth_date",
14+
"path": "birthDate",
15+
"extension": [
16+
{
17+
"url": "http://health-samurai.io/fhir/core/StructureDefinition/de-identification",
18+
"extension": [
19+
{"url": "method", "valueCode": "dateshift"},
20+
{"url": "dateShiftKey", "valueString": "my-secret-key"}
21+
]
22+
}
23+
]
24+
}
25+
```
26+
27+
The extension uses sub-extensions for the method and its parameters. The `method` sub-extension is required and specifies which de-identification method to apply.
28+
29+
## Methods
30+
31+
### redact
32+
33+
Replaces the value with NULL. No parameters.
34+
35+
```json
36+
{"url": "method", "valueCode": "redact"}
37+
```
38+
39+
### cryptoHash
40+
41+
Replaces the value with its HMAC-SHA256 hash (hex-encoded). Deterministic — same input always produces the same hash. One-way, cannot be reversed.
42+
43+
| Parameter | Type | Required | Description |
44+
|-----------|------|----------|-------------|
45+
| cryptoHashKey | string | yes | HMAC secret key |
46+
47+
```json
48+
[
49+
{"url": "method", "valueCode": "cryptoHash"},
50+
{"url": "cryptoHashKey", "valueString": "my-hash-key"}
51+
]
52+
```
53+
54+
### dateshift
55+
56+
Shifts date and dateTime values by a deterministic offset derived from the resource id. All dates within the same resource shift by the same number of days, preserving temporal relationships. The offset range is -50 to +50 days.
57+
58+
Year-only values (`"2000"`) and year-month values (`"2000-06"`) cannot be shifted meaningfully and are replaced with NULL.
59+
60+
| Parameter | Type | Required | Description |
61+
|-----------|------|----------|-------------|
62+
| dateShiftKey | string | yes | HMAC key used to compute the per-resource offset |
63+
64+
```json
65+
[
66+
{"url": "method", "valueCode": "dateshift"},
67+
{"url": "dateShiftKey", "valueString": "my-shift-key"}
68+
]
69+
```
70+
71+
### birthDateSafeHarbor
72+
73+
Intended **only for `Patient.birthDate`**. Behaves like `dateshift` but returns NULL when the birth date implies the patient is over 89 years old, per HIPAA Safe Harbor rule 45 CFR 164.514(b)(2)(i)(C).
74+
75+
Applying this method to any other date column is semantically incorrect — the function computes `age(current_date, input)` and treats the input as a birth date. Use plain `dateshift` for non-birth-date fields.
76+
77+
Because the function depends on `current_date`, it is marked `STABLE` rather than `IMMUTABLE`. The age cutoff re-evaluates on every query.
78+
79+
| Parameter | Type | Required | Description |
80+
|-----------|------|----------|-------------|
81+
| dateShiftKey | string | yes | HMAC key used to compute the per-resource offset |
82+
83+
```json
84+
[
85+
{"url": "method", "valueCode": "birthDateSafeHarbor"},
86+
{"url": "dateShiftKey", "valueString": "my-shift-key"}
87+
]
88+
```
89+
90+
### encrypt
91+
92+
AES-128-CBC encrypts the value and returns a base64-encoded string. Reversible with the key. Uses a zero initialization vector for deterministic output — same plaintext always produces the same ciphertext.
93+
94+
| Parameter | Type | Required | Description |
95+
|-----------|------|----------|-------------|
96+
| encryptKey | string | yes | Hex-encoded AES-128 key (32 hex characters = 16 bytes) |
97+
98+
```json
99+
[
100+
{"url": "method", "valueCode": "encrypt"},
101+
{"url": "encryptKey", "valueString": "0123456789abcdef0123456789abcdef"}
102+
]
103+
```
104+
105+
### substitute
106+
107+
Replaces the value with a fixed string.
108+
109+
| Parameter | Type | Required | Description |
110+
|-----------|------|----------|-------------|
111+
| replaceWith | string | yes | Replacement value |
112+
113+
```json
114+
[
115+
{"url": "method", "valueCode": "substitute"},
116+
{"url": "replaceWith", "valueString": "REDACTED"}
117+
]
118+
```
119+
120+
### perturb
121+
122+
Adds random noise to numeric values. The result is non-deterministic — each query produces different output.
123+
124+
| Parameter | Type | Required | Description |
125+
|-----------|------|----------|-------------|
126+
| span | decimal | no | Noise magnitude. Default: 1.0 |
127+
| rangeType | code | no | `fixed` (absolute noise) or `proportional` (relative to value). Default: `fixed` |
128+
| roundTo | integer | no | Decimal places to round to. 0 means integer. Default: 0 |
129+
130+
With `fixed` range type, noise is in the range ±span/2. With `proportional`, noise is ±(span × value)/2. Any other `rangeType` value raises a SQL error.
131+
132+
```json
133+
[
134+
{"url": "method", "valueCode": "perturb"},
135+
{"url": "span", "valueDecimal": 10},
136+
{"url": "rangeType", "valueCode": "fixed"},
137+
{"url": "roundTo", "valueInteger": 0}
138+
]
139+
```
140+
141+
### custom_function
142+
143+
Applies a user-provided PostgreSQL function. The function must already exist in the database. Its first argument is the column value cast to text. An optional second argument can be passed via `custom_arg`.
144+
145+
| Parameter | Type | Required | Description |
146+
|-----------|------|----------|-------------|
147+
| custom_function | string | yes | PostgreSQL function name. Must match `^[a-zA-Z][a-zA-Z0-9_.]*$` |
148+
| custom_arg | string, integer, decimal, boolean, or code | no | Optional second argument |
149+
150+
```json
151+
[
152+
{"url": "method", "valueCode": "custom_function"},
153+
{"url": "custom_function", "valueString": "left"},
154+
{"url": "custom_arg", "valueInteger": 4}
155+
]
156+
```
157+
158+
This example uses the built-in PostgreSQL `left` function to keep only the first 4 characters (e.g. extracting just the year from a date string).
159+
160+
## Example ViewDefinition
161+
162+
A complete ViewDefinition that de-identifies Patient data:
163+
164+
```json
165+
{
166+
"resourceType": "ViewDefinition",
167+
"id": "deident-patients",
168+
"name": "deident_patients",
169+
"status": "active",
170+
"resource": "Patient",
171+
"select": [{
172+
"column": [{
173+
"name": "id",
174+
"path": "id",
175+
"extension": [{
176+
"url": "http://health-samurai.io/fhir/core/StructureDefinition/de-identification",
177+
"extension": [{
178+
"url": "method",
179+
"valueCode": "cryptoHash"
180+
}, {
181+
"url": "cryptoHashKey",
182+
"valueString": "patient-hash-key"
183+
}]
184+
}]
185+
}, {
186+
"name": "gender",
187+
"path": "gender"
188+
}, {
189+
"name": "birth_date",
190+
"path": "birthDate",
191+
"extension": [{
192+
"url": "http://health-samurai.io/fhir/core/StructureDefinition/de-identification",
193+
"extension": [{
194+
"url": "method",
195+
"valueCode": "dateshift"
196+
}, {
197+
"url": "dateShiftKey",
198+
"valueString": "date-shift-key"
199+
}]
200+
}]
201+
}]
202+
}, {
203+
"forEach": "name",
204+
"select": [{
205+
"column": [{
206+
"name": "use",
207+
"path": "use"
208+
}, {
209+
"name": "family",
210+
"path": "family",
211+
"extension": [{
212+
"url": "http://health-samurai.io/fhir/core/StructureDefinition/de-identification",
213+
"extension": [{
214+
"url": "method",
215+
"valueCode": "redact"
216+
}]
217+
}]
218+
}]
219+
}]
220+
}, {
221+
"forEach": "address",
222+
"select": [{
223+
"column": [{
224+
"name": "state",
225+
"path": "state"
226+
}, {
227+
"name": "postal_code",
228+
"path": "postalCode",
229+
"extension": [{
230+
"url": "http://health-samurai.io/fhir/core/StructureDefinition/de-identification",
231+
"extension": [{
232+
"url": "method",
233+
"valueCode": "substitute"
234+
}, {
235+
"url": "replaceWith",
236+
"valueString": "000"
237+
}]
238+
}]
239+
}]
240+
}]
241+
}]
242+
}
243+
```
244+
245+
In this example:
246+
247+
- `id` is replaced with a consistent hash
248+
- `gender` passes through unchanged (no extension)
249+
- `birthDate` is shifted by a deterministic offset per patient
250+
- `name.family` is redacted (NULL)
251+
- `name.use` passes through (code values are not PHI)
252+
- `address.state` passes through (safe at state level)
253+
- `address.postalCode` is replaced with "000"
254+
255+
## Materialization restriction
256+
257+
A ViewDefinition that contains any de-identification extension can only be materialized as a `table`. Attempting to materialize as `view` or `materialized-view` returns HTTP 422 with an OperationOutcome:
258+
259+
> ViewDefinitions with de-identification extensions can only be materialized as 'table'. Views and materialized views expose cryptographic keys in PostgreSQL system catalogs.
260+
261+
This restriction exists because PostgreSQL stores the full view definition (including the compiled SQL with embedded keys) in `pg_views.definition` and `pg_matviews.definition`. Any user with `SELECT` on those catalogs would see the `cryptoHashKey`, `dateShiftKey`, or `encryptKey` values in plaintext. Tables materialize the transformed data only, leaving the keys inside the ViewDefinition resource itself (which is access-controlled).
262+
263+
`$run` and `$sql` are unaffected — they return data or SQL strings directly without storing anything in system catalogs.
264+
265+
## Pre-built ViewDefinitions
266+
267+
Aidbox ships an IG package `io.health-samurai.de-identification.r4` with ready-made Safe Harbor ViewDefinitions for common FHIR R4 resource types:
268+
269+
| Resource | Use |
270+
|----------|-----|
271+
| Patient | Uses `birthDateSafeHarbor` on `birthDate`, cryptoHash on `id`, redact on name/address identifiers |
272+
| Encounter, Condition, Observation | `dateshift` on clinical dates, cryptoHash on references |
273+
| Claim, ExplanationOfBenefit | `dateshift` on billable periods |
274+
| AllergyIntolerance, DiagnosticReport, MedicationRequest, MedicationDispense, MedicationAdministration, Immunization, Procedure, Specimen, DocumentReference | Same general approach |
275+
| Practitioner, Location | Identifier redaction |
276+
277+
Install the package via FHIR package management and use these ViewDefinitions directly, or copy and customize them. Every cryptographic key parameter in the pre-built VDs is blank (`""`) — you must set real keys before using them for actual de-identification.
278+
279+
## Using the UI
280+
281+
The ViewDefinition builder in Aidbox UI includes a de-identification picker on each column. Click the shield icon next to a column's path to open the configuration popover.
282+
283+
When a de-identification method is configured, the shield icon turns blue. Hovering shows the current method name.
284+
285+
## Writing custom PostgreSQL functions
286+
287+
Custom functions referenced via `custom_function` must:
288+
289+
- Accept `text` as the first argument (the column value)
290+
- Optionally accept a second argument of any type (passed via `custom_arg`)
291+
- Already exist in the database before the ViewDefinition is executed
292+
293+
Example:
294+
295+
```sql
296+
CREATE OR REPLACE FUNCTION my_mask(value text)
297+
RETURNS text LANGUAGE sql IMMUTABLE PARALLEL SAFE AS $$
298+
SELECT CASE
299+
WHEN value IS NULL THEN NULL
300+
ELSE left(value, 1) || repeat('*', greatest(length(value) - 1, 0))
301+
END;
302+
$$;
303+
```
304+
305+
Then reference it in a column:
306+
307+
```json
308+
{
309+
"url": "http://health-samurai.io/fhir/core/StructureDefinition/de-identification",
310+
"extension": [
311+
{"url": "method", "valueCode": "custom_function"},
312+
{"url": "custom_function", "valueString": "my_mask"}
313+
]
314+
}
315+
```
316+
317+
## Security considerations
318+
319+
### Key management
320+
321+
Cryptographic keys (`cryptoHashKey`, `dateShiftKey`, `encryptKey`) are stored as plaintext strings inside the ViewDefinition resource. Anyone with read access to the ViewDefinition can see the keys.
322+
323+
Restrict access to ViewDefinition resources using [AccessPolicy](../../access-control/authorization/README.md) to ensure only authorized users can view or modify de-identification configurations.
324+
325+
### SQL injection prevention
326+
327+
The `custom_function` parameter is validated against `^[a-zA-Z][a-zA-Z0-9_.]*$` — only letters, digits, underscores, and dots are allowed. This validation happens both in the Aidbox UI and in the SQL compiler. String arguments passed via `custom_arg` are safely escaped by the SQL generator.
328+
329+
### Encryption limitations
330+
331+
The `encrypt` method uses AES-128-CBC with a zero initialization vector. This makes encryption deterministic — the same plaintext always produces the same ciphertext, which is useful for consistent de-identification but leaks frequency information. This is not suitable for general-purpose encryption.
332+
333+
See also:
334+
335+
- [Defining flat views with view definitions](./defining-flat-views-with-view-definitions.md)
336+
- [$run operation](./operation-run.md)
337+
- [$materialize operation](./operation-materialize.md)

0 commit comments

Comments
 (0)