Skip to content

Commit 0eb4b88

Browse files
andreyorstandreyorst
authored
Merge pull request #5 from HealthSamurai/de-identification
add de-identification section to SOF
2 parents 3476258 + daa6353 commit 0eb4b88

6 files changed

Lines changed: 345 additions & 4 deletions

File tree

‎SUMMARY.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -327,6 +327,7 @@
327327
* [Defining flat views with view definitions](modules/sql-on-fhir/defining-flat-views-with-view-definitions.md)
328328
* [Migrate to the spec-compliant ViewDefinition format](modules/sql-on-fhir/migrate-to-the-spec-compliant-viewdefinition-format.md)
329329
* [Query data from flat views](modules/sql-on-fhir/query-data-from-flat-views.md)
330+
* [De-identification](modules/sql-on-fhir/de-identification.md)
330331
* [Reference](modules/sql-on-fhir/reference.md)
331332
* [Integration Toolkit](modules/integration-toolkit/README.md)
332333
* [C-CDA / FHIR Converter](modules/integration-toolkit/ccda-converter/README.md)

‎assets/deident-viewdefinition-builder.png‎

195 KB
Loading

‎docs/modules/sql-on-fhir/README.md‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,8 @@ description: Create flat SQL views from FHIR resources using ViewDefinitions for
44

55
# SQL on FHIR
66

7-
{% hint style="info" %}
8-
**SQL on FHIR** engine is currently in **preview**
7+
{% hint style="warning" %}
8+
Starting from version **2604**, SQL on FHIR requires **fhir-schema mode** (`fhir.validation.fhir-schema-validation=true`). ViewDefinitions are stored in the FHIR Artifact Registry (FAR), which is only available in fhir-schema mode. Without it, ViewDefinition CRUD, `$run`, `$sql`, and `$materialize` operations will not work.
99
{% endhint %}
1010

1111
Performing analysis on FHIR data requires extracting data from deeply nested structures of resources, which may be cumbersome in some cases. To address this problem, Aidbox implements [SQL on FHIR](https://build.fhir.org/ig/FHIR/sql-on-fhir-v2/index.html) specification allowing users to create flat views of their resources in a simple, straightforward way
@@ -22,6 +22,12 @@ Once your flat view is defined and materialized, you can query data from it usin
2222

2323
See [Query data from flat views](./query-data-from-flat-views.md).
2424

25+
## De-identification
26+
27+
Starting from version **2604**, ViewDefinition columns can be annotated with de-identification methods to transform sensitive data during SQL generation. Supported methods include redact, cryptoHash, dateshift, encrypt, substitute, perturb, and custom PostgreSQL functions.
28+
29+
See [De-identification](./de-identification.md).
30+
2531
## SQL on FHIR reference
2632

2733
To dive deeper into the nuances of using SQL on FHIR in Aidbox, consult the reference page.

‎docs/modules/sql-on-fhir/de-identification.md‎

Lines changed: 334 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,334 @@
1+
# De-identification
2+
3+
Starting from version **2604**, Aidbox supports per-column de-identification in ViewDefinitions via a FHIR extension. When a column has a de-identification extension, the SQL compiler wraps the column expression with a PostgreSQL function that transforms the value before it reaches the output.
4+
5+
This works with all ViewDefinition operations: `$run`, `$sql`, and `$materialize`.
6+
7+
{% hint style="info" %}
8+
Requires **fhir-schema mode**.
9+
{% endhint %}
10+
11+
## Extension format
12+
13+
Add the de-identification extension to any column in the `select` array:
14+
15+
```json
16+
{
17+
"name": "birth_date",
18+
"path": "birthDate",
19+
"extension": [
20+
{
21+
"url": "http://health-samurai.io/fhir/core/StructureDefinition/de-identification",
22+
"extension": [
23+
{"url": "method", "valueCode": "dateshift"},
24+
{"url": "dateShiftKey", "valueString": "my-secret-key"}
25+
]
26+
}
27+
]
28+
}
29+
```
30+
31+
The extension uses sub-extensions for the method and its parameters. The `method` sub-extension is required and specifies which de-identification method to apply.
32+
33+
## Methods
34+
35+
### redact
36+
37+
Replaces the value with NULL. No parameters.
38+
39+
```json
40+
{"url": "method", "valueCode": "redact"}
41+
```
42+
43+
### cryptoHash
44+
45+
Replaces the value with its HMAC-SHA256 hash (hex-encoded). Deterministic — same input always produces the same hash. One-way, cannot be reversed.
46+
47+
| Parameter | Type | Required | Description |
48+
|-----------|------|----------|-------------|
49+
| cryptoHashKey | string | yes | HMAC secret key |
50+
51+
```json
52+
[
53+
{"url": "method", "valueCode": "cryptoHash"},
54+
{"url": "cryptoHashKey", "valueString": "my-hash-key"}
55+
]
56+
```
57+
58+
### dateshift
59+
60+
Shifts date and dateTime values by a deterministic offset derived from the resource id. All dates within the same resource shift by the same number of days, preserving temporal relationships. The offset range is -50 to +50 days.
61+
62+
Year-only values (`"2000"`) and year-month values (`"2000-06"`) cannot be shifted meaningfully and are replaced with NULL.
63+
64+
| Parameter | Type | Required | Description |
65+
|-----------|------|----------|-------------|
66+
| dateShiftKey | string | yes | HMAC key used to compute the per-resource offset |
67+
68+
```json
69+
[
70+
{"url": "method", "valueCode": "dateshift"},
71+
{"url": "dateShiftKey", "valueString": "my-shift-key"}
72+
]
73+
```
74+
75+
### birthDateSafeHarbor
76+
77+
Intended **only for `Patient.birthDate`**. Behaves like `dateshift` but returns NULL when the birth date implies the patient is over 89 years old, per HIPAA Safe Harbor rule [45 CFR 164.514(b)(2)(i)(C)](https://www.ecfr.gov/current/title-45/subtitle-A/subchapter-C/part-164/subpart-E/section-164.514).
78+
79+
Applying this method to any other date column is semantically incorrect — the function computes `age(current_date, input)` and treats the input as a birth date. Use plain `dateshift` for non-birth-date fields.
80+
81+
Because the function depends on `current_date`, it is marked [`STABLE`](https://www.postgresql.org/docs/current/xfunc-volatility.html) rather than `IMMUTABLE` — PostgreSQL guarantees it returns the same result within a single transaction, but the result may differ between transactions as the current date changes. This means the age cutoff re-evaluates on every query.
82+
83+
| Parameter | Type | Required | Description |
84+
|-----------|------|----------|-------------|
85+
| dateShiftKey | string | yes | HMAC key used to compute the per-resource offset |
86+
87+
```json
88+
[
89+
{"url": "method", "valueCode": "birthDateSafeHarbor"},
90+
{"url": "dateShiftKey", "valueString": "my-shift-key"}
91+
]
92+
```
93+
94+
### encrypt
95+
96+
AES-128-CBC encrypts the value and returns a base64-encoded string. Reversible with the key. Uses a zero initialization vector for deterministic output — same plaintext always produces the same ciphertext.
97+
98+
| Parameter | Type | Required | Description |
99+
|-----------|------|----------|-------------|
100+
| encryptKey | string | yes | Hex-encoded AES-128 key (32 hex characters = 16 bytes) |
101+
102+
```json
103+
[
104+
{"url": "method", "valueCode": "encrypt"},
105+
{"url": "encryptKey", "valueString": "0123456789abcdef0123456789abcdef"}
106+
]
107+
```
108+
109+
{% hint style="warning" %}
110+
**Key management.** Cryptographic keys (`cryptoHashKey`, `dateShiftKey`, `encryptKey`) are stored as plaintext strings inside the ViewDefinition resource. Anyone with read access to the ViewDefinition can see the keys. Restrict access using [AccessPolicy](../../access-control/authorization/README.md).
111+
{% endhint %}
112+
113+
{% hint style="warning" %}
114+
**Encryption limitations.** `encrypt` uses AES-128-CBC with a zero initialization vector. Encryption is deterministic — same plaintext always produces same ciphertext, which is useful for consistent de-identification but leaks frequency information. Not suitable for general-purpose encryption.
115+
{% endhint %}
116+
117+
### substitute
118+
119+
Replaces the value with a fixed string.
120+
121+
| Parameter | Type | Required | Description |
122+
|-----------|------|----------|-------------|
123+
| replaceWith | string | yes | Replacement value |
124+
125+
```json
126+
[
127+
{"url": "method", "valueCode": "substitute"},
128+
{"url": "replaceWith", "valueString": "REDACTED"}
129+
]
130+
```
131+
132+
### perturb
133+
134+
Adds random noise to numeric values. The result is non-deterministic — each query produces different output.
135+
136+
| Parameter | Type | Required | Description |
137+
|-----------|------|----------|-------------|
138+
| span | decimal | no | Noise magnitude. Default: 1.0 |
139+
| rangeType | code | no | `fixed` (absolute noise) or `proportional` (relative to value). Default: `fixed` |
140+
| roundTo | integer | no | Decimal places to round to. 0 means integer. Default: 0 |
141+
142+
With `fixed` range type, noise is in the range `±span/2`. With `proportional`, noise is `±(span × value)/2`. Any other `rangeType` value raises a SQL error.
143+
144+
```json
145+
[
146+
{"url": "method", "valueCode": "perturb"},
147+
{"url": "span", "valueDecimal": 10},
148+
{"url": "rangeType", "valueCode": "fixed"},
149+
{"url": "roundTo", "valueInteger": 0}
150+
]
151+
```
152+
153+
### custom_function
154+
155+
Applies a user-provided PostgreSQL function. The function must already exist in the database. Its first argument is the column value cast to text. An optional second argument can be passed via `custom_arg`.
156+
157+
| Parameter | Type | Required | Description |
158+
|-----------|------|----------|-------------|
159+
| custom_function | string | yes | PostgreSQL function name. Must match `^[a-zA-Z][a-zA-Z0-9_.]*$` |
160+
| custom_arg | any primitive | no | Optional second argument, passed as a FHIR sub-extension using the appropriate `value[x]` type: `valueString`, `valueInteger`, `valueDecimal`, `valueBoolean`, or `valueCode` |
161+
162+
```json
163+
[
164+
{"url": "method", "valueCode": "custom_function"},
165+
{"url": "custom_function", "valueString": "left"},
166+
{"url": "custom_arg", "valueInteger": 4}
167+
]
168+
```
169+
170+
This example uses the built-in PostgreSQL `left` function to keep only the first 4 characters (e.g. extracting just the year from a date string).
171+
See also: [Writing custom PostgreSQL functions](#writing-custom-postgresql-functions)
172+
173+
## Example ViewDefinition
174+
175+
A complete ViewDefinition that de-identifies Patient data:
176+
177+
```json
178+
{
179+
"resourceType": "ViewDefinition",
180+
"id": "deident-patients",
181+
"name": "deident_patients",
182+
"status": "active",
183+
"resource": "Patient",
184+
"select": [{
185+
"column": [{
186+
"name": "id",
187+
"path": "id",
188+
"extension": [{
189+
"url": "http://health-samurai.io/fhir/core/StructureDefinition/de-identification",
190+
"extension": [{
191+
"url": "method",
192+
"valueCode": "cryptoHash"
193+
}, {
194+
"url": "cryptoHashKey",
195+
"valueString": "patient-hash-key"
196+
}]
197+
}]
198+
}, {
199+
"name": "gender",
200+
"path": "gender"
201+
}, {
202+
"name": "birth_date",
203+
"path": "birthDate",
204+
"extension": [{
205+
"url": "http://health-samurai.io/fhir/core/StructureDefinition/de-identification",
206+
"extension": [{
207+
"url": "method",
208+
"valueCode": "dateshift"
209+
}, {
210+
"url": "dateShiftKey",
211+
"valueString": "date-shift-key"
212+
}]
213+
}]
214+
}]
215+
}, {
216+
"forEachOrNull": "name",
217+
"select": [{
218+
"column": [{
219+
"name": "family",
220+
"path": "family",
221+
"extension": [{
222+
"url": "http://health-samurai.io/fhir/core/StructureDefinition/de-identification",
223+
"extension": [{
224+
"url": "method",
225+
"valueCode": "redact"
226+
}]
227+
}]
228+
}]
229+
}]
230+
}, {
231+
"forEachOrNull": "address",
232+
"select": [{
233+
"column": [{
234+
"name": "postal_code",
235+
"path": "postalCode",
236+
"extension": [{
237+
"url": "http://health-samurai.io/fhir/core/StructureDefinition/de-identification",
238+
"extension": [{
239+
"url": "method",
240+
"valueCode": "substitute"
241+
}, {
242+
"url": "replaceWith",
243+
"valueString": "000"
244+
}]
245+
}]
246+
}]
247+
}]
248+
}]
249+
}
250+
```
251+
252+
In this example:
253+
254+
- `id` is replaced with a consistent hash
255+
- `gender` passes through unchanged (no extension)
256+
- `birthDate` is shifted by a deterministic offset per patient
257+
- `name.family` is redacted (NULL)
258+
- `address.postalCode` is replaced with "000"
259+
260+
The result from the `$run` operation would look like this:
261+
262+
| `id` | `gender` | `birth_date` | `family` | `postal_code` |
263+
|------------------------------------------------------------------|----------|--------------|----------|---------------|
264+
| a9c063ce560ab35c2156d4bf153457d8c7b0ad6325c1c4112b34eb7147aaa8f9 | female | 1985-03-02 | null | 000 |
265+
| 6e7dfba4a51c359ead0afd9e3ff542c9417505957bf374e510eb37ec020fbc12 | male | 1952-11-12 | null | 000 |
266+
| 27fb6fd29c5657c1a122aa1ae28cdfc5e10b202c6dc7d498cec72609b3a1b447 | female | 1930-05-08 | null | 000 |
267+
268+
## Materialization restriction
269+
270+
A ViewDefinition that contains any de-identification extension can only be materialized as a `table`. Attempting to materialize as `view` or `materialized-view` returns HTTP 422 with an OperationOutcome:
271+
272+
This restriction exists because PostgreSQL stores the full view definition (including the compiled SQL with embedded keys) in `pg_views.definition` and `pg_matviews.definition`. Any user with `SELECT` on those catalogs would see the `cryptoHashKey`, `dateShiftKey`, or `encryptKey` values in plaintext. Tables materialize the transformed data only, leaving the keys inside the ViewDefinition resource itself (which is access-controlled).
273+
274+
`$run` and `$sql` are unaffected — they return data or SQL strings directly without storing anything in system catalogs.
275+
276+
## Pre-built ViewDefinitions
277+
278+
The IG package [`io.health-samurai.de-identification.r4`](https://get-ig.org/io.health-samurai.de-identification.r4) provides ready-made Safe Harbor ViewDefinitions for common FHIR R4 resource types. Install it via [Artefact Registry](../../artifact-registry/artifact-registry-overview.md) ("FHIR packages" in Aidbox UI):
279+
280+
| Resource | Use |
281+
|----------|-----|
282+
| Patient | Uses `birthDateSafeHarbor` on `birthDate`, cryptoHash on `id`, redact on name/address identifiers |
283+
| Encounter, Condition, Observation | `dateshift` on clinical dates, cryptoHash on references |
284+
| Claim, ExplanationOfBenefit | `dateshift` on billable periods |
285+
| AllergyIntolerance, DiagnosticReport, MedicationRequest, MedicationDispense, MedicationAdministration, Immunization, Procedure, Specimen, DocumentReference | Same general approach |
286+
| Practitioner, Location | Identifier redaction |
287+
288+
Install the package via FHIR package management and use these ViewDefinitions directly, or copy and customize them. Every cryptographic key parameter in the pre-built VDs is blank (`""`) — you must set real keys before using them for actual de-identification.
289+
290+
## Using the UI
291+
292+
The ViewDefinition builder in Aidbox UI includes a de-identification picker on each column. Click the shield icon next to a column's path to open the configuration popover.
293+
294+
![De-identification picker in the ViewDefinition Builder](../../../assets/deident-viewdefinition-builder.png)
295+
296+
When a de-identification method is configured, the shield icon turns blue. Hovering shows the current method name.
297+
298+
## Writing custom PostgreSQL functions
299+
300+
Custom functions referenced via `custom_function` must:
301+
302+
- Accept `text` as the first argument (the column value)
303+
- Optionally accept a second argument of any type (passed via `custom_arg`)
304+
- Already exist in the database before the ViewDefinition is executed
305+
306+
Example:
307+
308+
```sql
309+
CREATE OR REPLACE FUNCTION my_mask(value text)
310+
RETURNS text LANGUAGE sql IMMUTABLE PARALLEL SAFE AS $$
311+
SELECT CASE
312+
WHEN value IS NULL THEN NULL
313+
ELSE left(value, 1) || repeat('*', greatest(length(value) - 1, 0))
314+
END;
315+
$$;
316+
```
317+
318+
Then reference it in a column:
319+
320+
```json
321+
{
322+
"url": "http://health-samurai.io/fhir/core/StructureDefinition/de-identification",
323+
"extension": [
324+
{"url": "method", "valueCode": "custom_function"},
325+
{"url": "custom_function", "valueString": "my_mask"}
326+
]
327+
}
328+
```
329+
330+
See also:
331+
332+
- [Defining flat views with view definitions](./defining-flat-views-with-view-definitions.md)
333+
- [$run operation](./operation-run.md)
334+
- [$materialize operation](./operation-materialize.md)

‎docs/modules/sql-on-fhir/operation-materialize.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ description: Materializing SQL-on-FHIR ViewDefinitions as database tables, views
88
{% endhint %}
99

1010
{% hint style="warning" %}
11-
When running Aidbox not in FHIRSchema mode, please be aware that input parameters for the `$materialize` operation are not validated against FHIR specifications.
11+
Requires **fhir-schema mode**.
1212
{% endhint %}
1313

1414
{% hint style="info" %}

‎docs/modules/sql-on-fhir/operation-run.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ description: Using the $run operation to execute SQL-on-FHIR ViewDefinitions
44
# $run operation
55

66
{% hint style="warning" %}
7-
When running Aidbox not in FHIRSchema mode, please be aware that input parameters for the `$run` operation are not validated against FHIR specifications.
7+
Requires **fhir-schema mode**.
88
{% endhint %}
99

1010
{% hint style="info" %}

0 commit comments

Comments
 (0)