Skip to content

transformers.md

Latest commit

 

History

History
1588 lines (1234 loc) · 88 KB

transformers.md

File metadata and controls

1588 lines (1234 loc) · 88 KB
Raw

🔀 Transformers

transformer diagram

pgstream supports column value transformations to anonymize or mask sensitive data during replication and snapshots. This is particularly useful for compliance with data privacy regulations.

pgstream integrates with existing transformer open source libraries, such as greenmask, neosync and go-masker, to leverage a large amount of transformation capabilities, as well as having support for custom transformations.

⚠️ Most transformers do not preserve uniqueness. Applying one to a column covered by a unique index or primary key produces duplicate key violations. See Uniqueness and unique indexes.

Uniqueness and unique indexes

Anonymization is lossy by design, and that conflicts directly with unique constraints. Masking 609898123456 with masking type id yields 609898****: the transformer keeps a 6 character prefix, so every source value sharing that prefix becomes the same masked value. On a column covered by a unique index the load then fails with duplicate key value violates unique constraint, part way through the data, long after the run started.

Each transformer declares how it behaves with respect to uniqueness:

Uniqueness Meaning Validation
preserved Distinct inputs always produce distinct outputs. Safe on unique columns. passes
not_guaranteed Output is random, hashed, or driven by user supplied logic. Duplicates are possible, at the birthday bound of the output space (which usually depends on parameters). warns
lossy Distinct inputs are mapped to the same output by construction: partial masks, fixed literals, small value sets, name dictionaries. errors
Transformer Uniqueness
encrypted_aes_siv preserved
fpe_ff1 preserved
email not_guaranteed
greenmask_date not_guaranteed
greenmask_float not_guaranteed
greenmask_integer not_guaranteed
greenmask_string not_guaranteed
greenmask_unix_timestamp not_guaranteed
greenmask_utc_timestamp not_guaranteed
greenmask_uuid not_guaranteed
hstore not_guaranteed
json not_guaranteed
neosync_email not_guaranteed
neosync_string not_guaranteed
pg_anonymizer not_guaranteed
phone_number not_guaranteed
string not_guaranteed
template not_guaranteed
greenmask_boolean lossy
greenmask_choice lossy
greenmask_firstname lossy
lookup_choice lossy
literal_string lossy
masking lossy
neosync_firstname lossy
neosync_fullname lossy
neosync_lastname lossy

On an array column the classification describes the whole array transform, which depends on the configured generator. See Array columns.

When a source Postgres URL is configured, pgstream reads the unique indexes, unique constraints and primary keys of every table in the transformation rules and checks them against the configured transformers. Columns with no rule, or with a noop rule, keep their original value and are never flagged. Run the check on its own with:

pgstream validate rules -c pg2pg.yaml

With a Postgres target, a lossy transformer on a covered column fails the rules with transformation rules break a unique index, because the target recreates the source's indexes and the load would hit a duplicate key. With any other target (Kafka, Elasticsearch/OpenSearch, webhooks) there is no unique index to violate, so the same finding is reported as a warning and does not block the pipeline. A not_guaranteed transformer is always a warning.

To keep a lossy transformer on a covered column anyway — because the target does not enforce that index, or because the values are known not to collide — set allow_uniqueness_loss on that column rule:

column_transformers:
  pms_patient_id:
    name: masking
    parameters:
      type: id
    allow_uniqueness_loss: true

Keeping a column unique

If you need an anonymized column to stay unique, use encrypted_aes_siv: it is deterministic, so equality relationships survive across rows, tables and runs, and no two distinct inputs share a token. Its output is a base64 token rather than a value in the original format.

Use fpe_ff1 when the column must also keep its format. For example, a phone number must stay a phone number, and a code must still fit a varchar(12) column. fpe_ff1 encrypts with a format-preserving algorithm. The output has the same length and the same characters as the input. Two different inputs always give two different outputs.

⚠️ encrypted_aes_siv and fpe_ff1 are pseudonymization, not anonymization. The output is reversible by anyone holding key_hex, so it remains personal data, and because tokens are stable they can be correlated across tables and across successive dumps. Treat key_hex as a secret: inject it from a secret store rather than committing it in the rules file, use a different key per environment, and set associated_data (for example schema.table.column) so tokens cannot be correlated between columns.

What the check does not cover

  • Expression indexes. A unique index over an expression, such as lower(email), cannot be resolved to a column, so pgstream reports a warning naming the index and leaves it to you to verify.
  • Target-only indexes. The check reads the source catalog. A unique index that exists only on the target is not seen.
  • Indexes created after startup. Rules are validated once when the pipeline starts. A unique index added to the source later is not re-checked.
  • Exclusion constraints with equality semantics (EXCLUDE (email WITH =)) are not treated as unique indexes.

If a load fails with duplicate key value violates unique constraint on a transformed column, run pgstream validate rules against the source to see which rules the check flags.

Array columns

A transformation rule on a one-dimensional array column applies the transformer to each element. The rule reads like a rule on a scalar column:

column_transformers:
  emails: # emails is varchar(500)[]
    name: email
    parameters:
      replacement_domain: "@example.com.invalid"

pgstream decodes the array before it applies the transformer, so each element reaches the transformer as the value of the element type. An int4[] element reaches the transformer as an integer, not as text. The same rules apply to text and to text[]: a transformer accepts an array column if it accepts the element type. This covers extension types, so a transformer that accepts citext also accepts citext[].

Array columns need a source PostgreSQL connection, because pgstream reads the column type from the source catalog.

Generators

The optional array_options block selects how the elements of the new array are produced:

column_transformers:
  emails:
    name: email
    array_options:
      generator: random
      min_count: 0
      max_count: 12
Generator Behavior
map Applies the transformer to each source element, in order. The array keeps its length. This is the default.
random Emits between min_count and max_count elements. Each element is the transform of a source element selected at random, with repeats.

min_count and max_count are required with the random generator. They must be non-negative, min_count must not be greater than max_count, and max_count must not be greater than 10000. The limit protects against a mistyped value, which pgstream would otherwise apply to every row. These parameters are not valid with the map generator. pgstream rejects the rules at startup if these conditions are not met, and names the schema, table and column.

The random generator never invents a value. Each new element is the transform of a value that is in the row. An empty source array stays empty.

NULL values and errors

  • A NULL array stays NULL. pgstream does not call the transformer.
  • A NULL element stays NULL. pgstream does not call the transformer for that element.
  • A transformer that returns no value produces a NULL element in the same position. The array keeps its length.
  • The first element that fails makes the whole column fail. The on_error policy then applies to the whole column: fail stops the run, null sets the whole column to NULL, and pass-through restores the whole original array.

Limitations

  • Multi-dimensional arrays are not supported. pgstream rejects a rule at startup when the column is declared with more than one dimension. PostgreSQL does not enforce the declared number of dimensions, so a column declared as one-dimensional can still hold a multi-dimensional value. On the replication path this value is a per-row error, which the on_error policy handles. On the snapshot path pgstream cannot detect it, because the source driver returns the elements already flattened, and it writes a one-dimensional array to the target.
  • literal_string and pg_anonymizer write into each element. On an array column these transformers write into every element instead of into the column as a whole. This is different from the behaviour before pgstream supported array columns. pgstream validate rules reports a warning for each of these rules.
  • pg_anonymizer sends one query for each element. A wide array multiplies the queries that pgstream sends to the source database for the row. pgstream validate rules reports a warning for each rule that uses this transformer on an array column.
  • Dynamic parameters are not paired element by element. A dynamic_parameters sibling that is itself an array falls back to the parameter default.

Uniqueness

The uniqueness check reads the classification of the whole array transform, not of the transformer the rule names:

  • With map, the array keeps the classification of the transformer. fpe_ff1 on a text[] column stays preserved, because mapping a transformer that preserves uniqueness over an array of the same length also preserves uniqueness.
  • With random, the array is always lossy, whatever the transformer guarantees. Two different source arrays that share an element can produce the same output, and min_count: 0 lets any row produce an empty array. On a PostgreSQL target this is an error. Set allow_uniqueness_loss on the column to override it.

Supported transformers

PostgreSQL Anonymizer

pg_anonymizer

Description: Integrates with the PostgreSQL Anonymizer extension to provide advanced data anonymization using built-in anonymizer functions.

⚠️ This transformer requires a PostgreSQL database connection to execute transformations, which may impact performance compared to other transformers in this document that generate values locally without database queries.

Supported PostgreSQL types
Dependent on anonymizer function
Parameter Type Default Required Values
anon_function string N/A Yes Any valid anon.* function
postgres_url string N/A Yes PostgreSQL connection URL
salt string "" No Salt for deterministic functions
hash_algorithm string sha256 No Algorithm for anon.digest. One of md5, sha224, sha256, sha384, sha512
interval string N/A No Time interval for anon.dnoise function
ratio float N/A No Noise ratio for anon.noise function
sigma float N/A No Blur sigma for anon.image_blur function
mask string N/A No Mask character for anon.partial function
mask_prefix_count int 0 No Prefix count for anon.partial function
mask_suffix_count int 0 No Suffix count for anon.partial function
min string N/A No Minimum value for anon.random_*_between functions
max string N/A No Maximum value for anon.random_*_between functions
range string "" No Range for anon.random_in_* functions
locale string en_US No Locale for dummy supported functions. One of ar_SA, en_US, fr_FR, ja_JP, pt_BR, zh_CN, zh_TW
count int 0 No Count parameter for functions like anon.random_string and anon.lorem_ipsum
unit string paragraphs No Unit for anon.lorem_ipsum function. One of characters, words, paragraphs
prefix string "" No Prefix for anon.random_phone function

Notes:

  • The transformer executes functions directly in PostgreSQL, ensuring compatibility with all anonymizer features
  • Deterministic functions (pseudo_*, hash, digest) produce consistent output for the same input
  • Functions that don't require parameters (like anon.fake_*()) can be used without additional configuration

Prerequisites:

  • PostgreSQL Anonymizer extension must be installed and enabled on the source (or the configured url)
  • Extension must be loaded in shared_preload_libraries
  • Run SELECT anon.init();in order to use the faking functions

Supported Functions:

Unsupported Functions:

Example Configurations:

transformations:
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        first_name:
          name: pg_anonymizer
          parameters:
            anon_function: anon.fake_first_name()
        id:
          name: pg_anonymizer
          parameters:
            anon_function: anon.digest
            salt: salt
            hash_algorithm: md5
        phone:
          name: pg_anonymizer
          parameters:
            anon_function: anon.random_phone
            prefix: "+1-555-"
        api_key:
          name: pg_anonymizer
          parameters:
            anon_function: anon.random_string
            count: 32
        content:
          name: pg_anonymizer
          parameters:
            anon_function: anon.lorem_ipsum
            unit: "words"
            count: 50
        status:
          name: pg_anonymizer
          parameters:
            anon_function: anon.random_in
            range: "ARRAY['active', 'inactive', 'pending']"

Input-Output Examples:

Input Value Function Configuration Output Value
John anon_function: anon.fake_first_name() Michael (random)
john@test.com anon_function: anon.pseudo_email, salt: "key123" alice@test.com (deterministic)
1234567890 anon_function: anon.partial, mask: "*", mask_prefix_count: 3, mask_suffix_count: 3 123****890
sensitive_data anon_function: anon.digest, salt: "key", hash_algorithm: "sha256" a1b2c3d4e5f6... (hash)
100.50 anon_function: anon.noise, ratio: 0.1 95.23 (with 10% noise)
2023-01-15 anon_function: anon.dnoise, interval: "1 day" 2023-01-16 (±1 day noise)
password123 anon_function: anon.hash ef92b778bafe771e89245b89ecbc08a4...
Alice Smith anon_function: anon.pseudo_first_name, salt: "s1" Bob Smith (deterministic)
user@company.com anon_function: anon.partial_email u***@company.com
42 anon_function: anon.random_int_between(1, 100) 73 (random between 1-100)
/path/image.jpg anon_function: anon.image_blur, sigma: 2.5 Blurred image data
Any value anon_function: anon.fake_company() Acme Corporation (random)
25 anon_function: anon.random_int_between, min: "18", max: "65" 42 (random between 18-65)
Any value anon_function: anon.random_in, range: "ARRAY['A', 'B', 'C']" B (random from array)
Any value anon_function: anon.lorem_ipsum, unit: "words", count: 5 Lorem ipsum dolor sit amet
Any value anon_function: anon.random_string, count: 8 aB3xY9z1 (random string)
Any value anon_function: anon.random_phone, prefix: "+1-555-" +1-555-123-4567
John anon_function: anon.fake_first_name_locale, locale: "fr_FR" Pierre (French name)

Greenmask

greenmask_boolean

Description: Generates random or deterministic boolean values (true or false).

Uniqueness: lossy. The output space has two values. Cannot be used on a column covered by a unique index. See Uniqueness and unique indexes.

Supported PostgreSQL types
boolean
Parameter Type Default Required Values
generator string random No random,deterministic

Example Configuration:

transformations:
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        is_active:
          name: greenmask_boolean
          parameters:
            generator: deterministic

Input-Output Examples:

Input Value Configuration Parameters Output Value
true generator: deterministic false
false generator: deterministic true
true generator: random true or false (random)
greenmask_choice

Description: Randomly selects a value from a predefined list of choices.

Uniqueness: lossy. Any table with more rows than choices produces duplicates. Cannot be used on a column covered by a unique index. See Uniqueness and unique indexes.

Supported PostgreSQL types
text, varchar, char, bpchar, user-defined enum, and arrays of these

choices is optional for an enum column. If you do not set it, pgstream uses the labels of the enum. If you set it, pgstream compares each value with those labels. A wrong value stops the run at startup, not at each insert.

A domain over an enum resolves to the enum. An array of an enum resolves to the enum too, so an array column receives the same choices and transforms each element.

greenmask_choice is the only type-specific transformer for an enum column. It is the only one that you can limit to the values of the enum. literal_string and pg_anonymizer accept all types. They also apply to an enum column. For literal_string, use a literal that is a valid label.

The default choices have three limits:

  • A source Postgres URL is necessary. Without it, pgstream does not validate the rules against a catalog. A rule without choices then stops the run at startup with the message greenmask_choice: choices must not be empty. Set choices for a pipeline that has no Postgres source, for example a pipeline with a Kafka source.
  • pgstream reads the labels one time, at startup. If you add or rename a label on the source, pgstream uses the new label only after a restart. Before the restart, a replicated ALTER TYPE ... RENAME VALUE makes pgstream write a label that the target refuses. At startup, pgstream writes the labels in use to the log for each column with default choices.
  • The default is the full set of labels. pgstream does not read a CHECK constraint on a domain or on a table. It does not apply such a constraint. Set choices for a column with a CHECK constraint.

⚠️ The transformer can select the label that the source row already contains. It selects from all the choices and does not remove the source value from the set. With generator: random, approximately 1 row in N keeps its source value, where N is the number of choices. With generator: deterministic, each label maps to one fixed label, so a label that maps to itself keeps its value in every row. The transformers that select from a fixed dictionary, for example greenmask_firstname, operate in the same way.

⚠️ Use generator: random for an enum column. The deterministic generator maps each label to one fixed label and uses no secret key. The target contains all labels of the enum. A person who reads the target can find the source label of each value. pgstream writes a warning to the log when a rule uses deterministic with default enum choices.

ℹ️ The transformer writes the value as a string. This is a change for the Kafka, webhook, Elasticsearch and OpenSearch targets. Before this change, greenmask_choice wrote a byte array, and these targets encoded that byte array as base64 in JSON. Update the consumers that decode base64. A search index also contains base64 in the documents from before this change.

Parameter Type Default Required Values
generator string random No random,deterministic
choices string[] the enum's labels, if an enum Yes, unless an enum column N/A

transformers-definition.json shows choices as always required. This file describes the transformer, not the column that you configure it on.

Example Configuration:

transformations:
  table_transformers:
    - schema: public
      table: orders
      column_transformers:
        status:
          name: greenmask_choice
          parameters:
            generator: random
            choices: ["pending", "shipped", "delivered", "cancelled"]
        # an enum column does not need choices; pgstream uses the enum labels
        mood:
          name: greenmask_choice

Input-Output Examples:

Input Value Configuration Parameters Output Value
pending generator: random shipped (random)
shipped generator: deterministic pending
delivered generator: random cancelled (random)
greenmask_date

Description: Generates random or deterministic dates within a specified range.

Supported PostgreSQL types
date, timestamp, timestamptz
Parameter Type Default Required Values
generator string random No random,deterministic
min_value string (yyyy-MM-dd) N/A Yes N/A
max_value string (yyyy-MM-dd) N/A Yes N/A

Example Configuration:

transformations:
  table_transformers:
    - schema: public
      table: events
      column_transformers:
        event_date:
          name: greenmask_date
          parameters:
            generator: random
            min_value: "2020-01-01"
            max_value: "2025-12-31"

Input-Output Examples:

Input Value Configuration Parameters Output Value
2023-01-01 generator: random, min_value: 2020-01-01, max_value: 2025-12-31 2021-05-15 (random)
2022-06-15 generator: deterministic 2020-01-01
greenmask_firstname

Description: Generates random or deterministic first names, optionally filtered by gender.

Uniqueness: lossy. Names come from a fixed dictionary and repeat well before a table of any size is exhausted. Cannot be used on a column covered by a unique index. See Uniqueness and unique indexes.

Supported PostgreSQL types
text, varchar, char, bpchar
Parameter Type Default Required Values Dynamic
generator string random No random,deterministic No
gender string Any No Any,Female,Male Yes

gender can also be a dynamic parameter, referring to some other column. Please see the below example config.

Example Configuration:

transformations:
  table_transformers:
    - schema: public
      table: employees
      column_transformers:
        first_name:
          name: greenmask_firstname
          parameters:
            generator: deterministic
          dynamic_parameters:
            gender:
              column: sex

Input-Output Examples:

Input Name Configuration Parameters Output Name
John preserve_gender: true Michael
Jane preserve_gender: true Emily
Alex preserve_gender: false Jordan
Chris generator: random Taylor
greenmask_float

Description: Generates random or deterministic floating-point numbers within a specified range.

Supported PostgreSQL types
real, double precision, numeric

⚠️ Set min_value and max_value for a numeric column. These parameters are required there. The default range covers all float32 values. With the default range, the transformer writes the same constant value in each row.

The transformer converts the numeric value to a float64 and uses that float64 as the seed for the generator. The output is also a float64. If the source value has more digits than a float64 holds, the transformer rounds it. If the source value is outside the float64 range, the transformer uses the largest float64 value. These changes apply to the seed only, because the transformer discards the source value.

pgstream compares the range with the column when it validates the rules. If the range does not fit a numeric(p,s) column, the run stops. A numeric column without a precision holds any value. For such a column, pgstream checks only that you set both bounds.

Parameter Type Default Required Values
generator string random No random,deterministic
min_value float -3.40282346638528859811704183484516925440e+38 No N/A
max_value float 3.40282346638528859811704183484516925440e+38 No N/A
greenmask_integer

Description: Generates random or deterministic integers within a specified range.

Supported PostgreSQL types
smallint, integer, bigint, real, double precision, numeric
Parameter Type Default Required Values
generator string random No random,deterministic
size int 4 No 2,4
min_value int -2147483648 No N/A
max_value int 2147483647 No N/A

Example Configuration:

transformations:
  table_transformers:
    - schema: public
      table: products
      column_transformers:
        stock_quantity:
          name: greenmask_integer
          parameters:
            generator: random
            min_value: 1
            max_value: 1000
greenmask_string

Description: Generates random or deterministic strings with customizable length and character set.

Supported PostgreSQL types
text, varchar, char, bpchar
Parameter Type Default Required Values
generator string random No random,deterministic
symbols string abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890 No N/A
min_length int 1 No N/A
max_length int 100 No N/A

Example Configuration:

transformations:
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        username:
          name: greenmask_string
          parameters:
            generator: random
            min_length: 5
            max_length: 15
            symbols: "abcdefghijklmnopqrstuvwxyz1234567890"
greenmask_unix_timestamp

Description: Generates random or deterministic unix timestamps.

Supported PostgreSQL types
smallint, integer, bigint, real, double precision, numeric
Parameter Type Default Required Values
generator string random No random,deterministic
min_value string N/A Yes N/A
max_value string N/A Yes N/A
greenmask_utc_timestamp

Description: Generates random or deterministic UTC timestamps.

Supported PostgreSQL types
timestamp
Parameter Type Default Required Values
generator string random No random,deterministic
truncate_part string "" No nanosecond,microsecond,millisecond,second,minute,hour,day,month,year
min_timestamp string (RFC3339) N/A Yes N/A
max_timestamp string (RFC3339) N/A Yes N/A
greenmask_uuid

Description: Generates random or deterministic UUIDs.

Supported PostgreSQL types
uuid,text, varchar, char, bpchar
Parameter Type Default Required Values
generator string random No random,deterministic

Neosync

neosync_email

Description: Anonymizes email addresses while optionally preserving length and domain.

Supported PostgreSQL types
text, varchar, char, bpchar, citext
Parameter Type Default Required Values
preserve_length bool false No
preserve_domain bool false No
excluded_domains string[] N/A No
max_length int 100 No
email_type string uuidv4 No uuidv4,fullname,any
invalid_email_action string 100 No reject,passthrough,null,generate
seed int Rand No

Example Configuration:

transformations:
  table_transformers:
    - schema: public
      table: customers
      column_transformers:
        email:
          name: neosync_email
          parameters:
            preserve_length: true
            preserve_domain: true

Input-Output Examples:

Input Email Configuration Parameters Output Email
john.doe@example.com preserve_length: true, preserve_domain: true abcd.efg@example.com
jane.doe@company.org preserve_length: false, preserve_domain: true random@company.org
user123@gmail.com preserve_length: true, preserve_domain: false abcde123@random.com
invalid-email invalid_email_action: passthrough invalid-email
invalid-email invalid_email_action: null NULL
invalid-email invalid_email_action: generate generated@random.com
neosync_firstname

Description: Generates anonymized first names while optionally preserving length.

Uniqueness: lossy. Names come from a fixed dictionary and repeat well before a table of any size is exhausted. Cannot be used on a column covered by a unique index. See Uniqueness and unique indexes.

Supported PostgreSQL types
text, varchar, char, bpchar
Parameter Type Default Required
preserve_length bool false No
max_length int 100 No
seed int Rand No

Example Configuration:

transformations:
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        first_name:
          name: neosync_firstname
          parameters:
            preserve_length: true
neosync_lastname

Description: Generates anonymized last names while optionally preserving length.

Uniqueness: lossy. Names come from a fixed dictionary and repeat well before a table of any size is exhausted. Cannot be used on a column covered by a unique index. See Uniqueness and unique indexes.

Supported PostgreSQL types
text, varchar, char, bpchar
Parameter Type Default Required
preserve_length bool false No
max_length int 100 No
seed int Rand No

Example Configuration:

transformations:
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        last_name:
          name: neosync_lastname
          parameters:
            preserve_length: true
neosync_fullname

Description: Generates anonymized full names while optionally preserving length.

Uniqueness: lossy. Names come from fixed dictionaries, so even first/last combinations repeat well before a large table is exhausted. Cannot be used on a column covered by a unique index. See Uniqueness and unique indexes.

Supported PostgreSQL types
text, varchar, char, bpchar
Parameter Type Default Required
preserve_length bool false No
max_length int 100 No
seed int Rand No

max_length must be greater than 2. If preserve_length is set to true, generated value can might be longer than max_length, depending on the input length.

Example Configuration:

transformations:
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        full_name:
          name: neosync_fullname
          parameters:
            preserve_length: true
neosync_string

Description: Generates anonymized strings with customizable length.

Supported PostgreSQL types
text, varchar, char, bpchar
Parameter Type Default Required
preserve_length bool false No
min_length int 1 No
max_length int 100 No
seed int Rand No

Example Configuration:

transformations:
  table_transformers:
    - schema: public
      table: logs
      column_transformers:
        log_message:
          name: neosync_string
          parameters:
            min_length: 10
            max_length: 50

Xata

template

Description: Transforms the data using go templates

Supported PostgreSQL types
All types with a string representation
Parameter Type Default Required
template string N/A Yes

This transformer can be used for any Postgres type as long as the given template produces a value with correct syntax for that column type. e.g It can be "5-10-2021" for a date column, or "3.14159265" for a double precision one.

.GetValue renders the same text from a snapshot and from replication. During a snapshot the value is encoded as Postgres text for the column type, so a numeric renders as decimal text, a uuid as a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11, a date as 2024-02-29, a bytea as \xdeadbeef, an array as {a,b}, and so on. Values from other columns, read with .GetDynamicValue, keep the Go time.Time type during a snapshot so the sprig date functions can format them.

Template transformer supports a bunch of useful functions. Use .GetValue to refer to the value to be transformed. Use .GetDynamicValue "<column_name>" to refer to some other column. Other than the standard go template functions, there are many useful helper functions supported to be used with template transformer, thanks to greenmask's huge set of core functions including masking function by go-masker and various random data generator functions powered by the open source library faker. Also, template transformer has support for the open source library sprig which has many useful helper functions.

With the below example config pgstream masks values in the column email of the table users, using go-masker's email masking function. But first, this template checks if there's a non-empty value to be used in the column email. If not, it simply looks for another column named secondary_email and uses that instead. Then we have another check to see if it's a @xata email or not. Finally masking the value, only if it's not a @xata email, passing it without a mask otherwise.

Example Configuration:

transformations:
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        email:
          name: template
          parameters:
            template: >
              {{ $email := "" }}
              {{- if and (ne .GetValue nil) (isString .GetValue) (gt (len .GetValue) 0) -}}
                {{ $email = .GetValue }}
              {{- else -}}
                {{ $email = .GetDynamicValue "secondary_email" }}
              {{- end -}}
              {{- if (not (contains "@xata" $email)) -}}
                {{ masking "email" $email }}
              {{- else -}}
                {{ $email }}
              {{- end -}}
masking

Description: Masks string values using the provided masking function.

Uniqueness: lossy. Every type replaces part of the value with *, so all values sharing the untouched part collapse to the same output — type: id keeps only a 6 character prefix. Cannot be used on a column covered by a unique index. See Uniqueness and unique indexes.

A custom mask that would cover zero characters (for example mask_begin: 3 with mask_end: 3, or unmask_begin: 0) is rejected at startup: it leaves values completely unmasked, which silently defeats anonymization. Use the noop transformer if passing a column through untouched is what you want.

Supported PostgreSQL types
text, varchar, char, bpchar

Parameter Details:

Parameter Type Default Required Values
type string default No custom, password, name, address, email, mobile, tel, id, credit_card, url, default

Example Configuration:

transformations:
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        email:
          name: masking
          parameters:
            type: email

Input-Output Examples:

Input Value Configuration Parameters Output Value
aVeryStrongPassword123 type: password ************
john.doe@example.com type: email joh****e@example.com
Sensitive Data type: default **************

With custom type, the masking function is defined by the user, by providing beginning and end indexes for masking. If the input is shorter than the end index, the rest of the string will all be masked. See the third example below.

transformations:
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        email:
          name: masking
          parameters:
            type: custom
            mask_begin: "4"
            mask_end: "12"

Input-Output Examples:

Input Value Output Value
1234567812345678 1234********5678
sensitive@example.com sens********ample.com
sensitive sens*****

If the begin index is not provided, it defaults to 0. If the end is not provided, it defaults to input length.

transformations:
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        email:
          name: masking
          parameters:
            type: custom
            mask_end: "5"
Input Value Output Value
1234567812345678 *****67812345678
sensitive@example.com *****tive@example.com
sensitive *****tive

Alternatively, since input length may vary, user can provide relative beginning and end indexes, as percentages of the input length.

transformations:
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        email:
          name: masking
          parameters:
            type: custom
            mask_begin: "15%"
            mask_end: "85%"
Input Value Output Value
1234567812345678 12***********678
sensitive@example.com sen***************com
sensitive s******ve

Alternatively, user can provide unmask begin and end indexes. In that case, the specified part of the input will remain unmasked, while all the rest is masked. Mask and unmask parameters cannot be provided at the same time.

transformations:
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        email:
          name: masking
          parameters:
            type: custom
            unmask_end: "3"
Input Value Output Value
1234567812345678 123*************
sensitive@example.com sen******************
sensitive sen******
json

Description: Transforms json data with set and delete operations

Supported PostgreSQL types
json, jsonb
Parameter Type Default Required
operations array N/A Yes

Parameter for each operation:

Parameter Type Default Required Values
operation string N/A Yes set, delete
path string N/A Yes sjson syntax*
skip_not_exist boolean true No true, false
error_not_exist boolean false No true, false
value string N/A Yes** Any valid JSON representation
value_template string N/A Yes** Any template with valid syntax

*Paths should follow sjson syntax

**Either value or value_template must be provided if the operation is set. If both are provided, value_template takes precedence.

JSON transformer can be used for Postgres types json and jsonb. This transformer executes a list of given operations on the json data to be transformed.

All operations must be either set or delete. set operations support literal values as well as templates, making use of sprig and greenmask's function sets. See template transformer section for more details. Also, like the template transformer, .GetValue and .GetDynamicValue functions are supported. Unlike template transformer, here .GetValue refers to the value at given path, rather than the entire JSON object; whereas .GetDynamicValue is again used for referring to other columns. delete operations simply delete the object at the given path. Execution of an operation will be skipped if the given path does not exists and the parameter skip_not_exist is set to true, which is also the default behavior. Execution of an operation errors out if the given path does not exists and the parameter error_not_exist - which is false by default - is set to true; unless the operation is skipped already.

JSON transformer uses sjson library for executing the operations. Operation paths should follow the synxtax rules of sjson

With the below config pgstream transforms the json values in the column user_info_json of the table users by:

  • First, traversing all the items in the array named purchases, and for each element, setting value to "-" for key "item".
  • Then, deleting the object named "country" under the top-level object "address".
  • Completely masking the "city" value under object "address", using go-masker's default masking function supported by pgstream's templating.
  • Finally, setting the user's lastname after fetching it from some other column named lastname, using dynamic values support. Assuming there's such column, having the lastname info for users.

Example input-output is given below the config.

Example Configuration:

transformations:
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        user_info_json:
          name: json
          parameters:
            operations:
              - operation: set
                path: "purchases.#.item"
                value: "-"
                error_not_exist: true
              - operation: delete
                path: "address.country"
              - operation: set
                path: "address.city"
                value_template: '"{{ masking "default" .GetValue }}"'
              - operation: set
                path: "user.lastname"
                value_template: '"{{ .GetDynamicValue "lastname" }}"'

For input JSON value,

{
  "user": {
    "firstname": "john",
    "lastname": "unknown"
  },
  "residency": {
    "city": "some city",
    "country": "some country"
  },
  "purchases": [
    {
      "item": "book",
      "price": 10
    },
    {
      "item": "pen",
      "price": 2
    }
  ]
}

the JSON transformer with above config produces output:

{
  "user": {
    "firstname": "john",
    "lastname": "doe"
  },
  "residency": {
    "city": "*********"
  },
  "purchases": [
    {
      "item": "-",
      "price": 10
    },
    {
      "item": "-",
      "price": 2
    }
  ]
}
hstore

Description: Transforms hstore data with set and delete operations

Supported PostgreSQL types
hstore
Parameter Type Default Required
operations array N/A Yes

Parameter for each operation:

Parameter Type Default Required Values
operation string N/A Yes set, delete
key string N/A Yes
skip_not_exist boolean true No true, false
error_not_exist boolean false No true, false
value string, null N/A Yes* Any string or null
value_template string N/A Yes* Any template with valid syntax

*Either value or value_template must be provided if the operation is set. If both are provided, value_template takes precedence.

Hstore transformer can be used for Postgres type hstore. This transformer executes a list of given operations on the hstore data to be transformed.

All operations must be either set or delete. set operations support literal values as well as templates, making use of sprig and greenmask's function sets. See template transformer section for more details. Also, like the template transformer, .GetValue and .GetDynamicValue functions are supported. Unlike template transformer, here .GetValue refers to the value for the given key, rather than the entire Hstore object; whereas .GetDynamicValue is again used for referring to other columns. delete operations simply delete the pair with the given key. Execution of an operation will be skipped if the given key does not exists and the parameter skip_not_exist is set to true, which is also the default behavior. Execution of an operation errors out if the given key does not exists and the parameter error_not_exist - which is false by default - is set to true; unless the operation is skipped already.

A limitation to be aware of: When using hstore transformer templates, you cannot set a value to the string literal "<no value>". This is because Go templates produce "<no value>" as output when the result is nil, creating an ambiguity. In such cases, pgstream will interpret it as nil and set the hstore value to NULL rather than storing the actual string "<no value>".

With the below config pgstream transforms the hstore values in the column attributes of the table users by:

  • First, updating the value for key "email" to the masked version of it, using email masking function. If the key "email" is not found, it simply ignores it, since error_not_exist is not set to true explicitly and it is false by default.
  • Then, deleting the pair where the key is "public_key". If there's no such key, errors out, because the parameter "error_not_exist" is set to true.
  • Completely masking the value for key "private_key", using go-masker's default masking function supported by pgstream's templating.
  • Finally, updating the value for key "newKey" to "newValue". Since "error_not_exist" is false by default, and there is no such key in the example below, this operation will be done by adding a new key-value pair.

Example input-output is given below the config.

Example Configuration:

transformations:
  validation_mode: relaxed
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        attributes:
          name: hstore
          parameters:
            operations:
              - operation: set
                key: "email"
                value_template: '{{masking "email" .GetValue}}'
              - operation: delete
                key: "public_key"
                error_not_exist: true
              - operation: set
                key: "private_key"
                value_template: '{{masking "default" .GetValue}}'
              - operation: set
                key: "newKey"
                value: "newValue"

For input hstore value,

  public_key => 12345,
  private_key => 12345abcdefg,
  email => user@email.com

the hstore transformer with above config produces output:

  private_key => ************,
  email => use***@email.com
  newKey => newValue
literal_string

Description: Transforms all values into the given constant value.

Uniqueness: lossy. Every value becomes the same literal. Cannot be used on a column covered by a unique index. See Uniqueness and unique indexes.

Supported PostgreSQL types
All types with a string representation
Parameter Type Default Required
literal string N/A Yes

Below example makes all values in the JSON column log_message to become {'error': null}. This transformer can be used for any Postgres type as long as the given string literal has the correct syntax for that type. e.g It can be "5-10-2021" for a date column, or "3.14159265" for a double precision one.

Example Configuration:

transformations:
  table_transformers:
    - schema: public
      table: logs
      column_transformers:
        log_message:
          name: literal_string
          parameters:
            literal: "{'error': null}"
phone_number

Description: Generates anonymized phone numbers with customizable length.

Supported PostgreSQL types
text, varchar, char, bpchar
Parameter Type Default Required Values Dynamic
prefix string "" No N/A Yes
min_length int 6 No N/A No
max_length int 10 No N/A No
generator string random No random, deterministic No

If the prefix is set, this transformer will always generate phone numbers starting with the prefix. prefix can also be a dynamic parameter, referring to some other column. Please see the below example config.

Example Configuration:

transformations:
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        phone:
          name: phone_number
          parameters:
            min_length: 9
            max_length: 12
            generator: deterministic
          dynamic_parameters:
            gender:
              column: country_code
email

Description: Anonymizes email addresses while optionally excluding domain to anonymize.

Supported PostgreSQL types
text, varchar, char, bpchar, citext
Parameter Type Default Required Values
replacement_domain string "@example.com" No
exclude_domain string "" No
salt string "defaultsalt" No

Example Configuration:

transformations:
  table_transformers:
    - schema: public
      table: customers
      column_transformers:
        email:
          name: email
          parameters:
            exclude_domain: "example.com"
            salt: "helloworld"

Input-Output Examples:

Input Email Configuration Parameters Output Email
john.doe@company.org exclude_domain: "company.org", salt: "helloworld" john.doe@company.org
jane.doe@company.org exclude_domain: "exclude.com", salt: "helloworld" T79P9zlFWzmT0yCUDMEE7S@example.com
jane.doe@company.org exclude_domain: "exclude.com", salt: "helloworld", replacement_domain: "@random.com" 6EIWw5lEa8nsY9JDOm5@random.com
invalid-email exclude_domain: "exclude.com", salt: "helloworld" 1fk5VLgTeoRQCCvqXFoToC1@example.com
invalid-email exclude_domain: "exclude.com", salt: "helloworld", replacement_domain: "@random.com" 6EIWw5lEa8nsY9JDOm5@random.com
encrypted_aes_siv

Description: Encrypts values with AES-SIV (RFC 5297), a deterministic authenticated encryption scheme. The same input, key and associated data always produce the same token, so equality relationships between column values are preserved across rows, tables and runs — while remaining reversible by holders of the key, unlike hashing. Tokens are authenticated: tampered or forged values fail decryption. The output is the ciphertext encoded as unpadded base64url, safe for URLs and file names.

Uniqueness: preserved. Encryption is reversible with the key, so two distinct plaintexts cannot share a token. This is the only transformer that never collides on a column covered by a unique index. Note that being reversible makes it pseudonymization rather than anonymization — see Keeping a column unique.

Supported PostgreSQL types
text, varchar, char, bpchar, bytea

Note on length-constrained columns: the token is always longer than the input — ceil(4 × (input_length + 16) / 3) characters (36 for an 11-character input, 22 minimum). Length-constrained columns (varchar(n), char(n)) must be wide enough to hold the expanded token or writes to the target will fail; prefer text columns.

For bytea columns the raw bytes are encrypted (the transformer normalizes the hex-text form delivered during replication and the raw bytes delivered during snapshots to the same plaintext), and the token is stored as the ASCII bytes of the base64url text.

Parameter Type Default Required
key_hex string N/A Yes
associated_data string "" No

key_hex is the 64-byte AES-SIV key, hex-encoded (128 characters), e.g. generated with openssl rand -hex 64. AES-SIV requires the full 64-byte key (RFC 5297); shorter keys are rejected.

associated_data is authenticated but not encrypted: a token minted with one associated data value fails decryption under another. Use it to bind tokens to a context (such as a table/column name) so they cannot be replayed elsewhere.

Security note: the encryption is deterministic by design — equal inputs produce equal tokens, which reveals equality (and only equality) of the underlying values. Anyone holding the key can decrypt the tokens, so the anonymization guarantee is key custody: load the key from a secret manager and never store it alongside the transformed data.

Example Configuration:

transformations:
  table_transformers:
    - schema: public
      table: orders
      column_transformers:
        file_path:
          name: encrypted_aes_siv
          parameters:
            # example key only — generate your own and load it from a secret store
            key_hex: "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f"
            associated_data: "public.orders.file_path"

Input-Output Examples:

Input Configuration Parameters Output
hello world key_hex: "000102…3e3f", no associated_data Hc5d96xIxu2ute1RbFuenEftGxw-P__m1Vv_
hello world key_hex: "000102…3e3f", associated_data: "public.orders.file_path" (example above) AsC2hoq-Y9V0iqK6JmNIxq0Fsr3SFPo27QNq

Every run with the same key and parameters produces the same output. Tokens can be decrypted with any RFC 5297 AES-SIV implementation, for example Tink's daead/subtle package in Go.

lookup_choice

Description: Replaces a value with one taken from a column of another table, for example a foreign key column pointing at a lookup table. The values are read from the source database once, when the pipeline starts, so the configuration does not have to be regenerated when the lookup table's contents change. It is the live-table counterpart of greenmask_choice, which chooses from a list written into the configuration.

Uniqueness: lossy. Any table with more rows than the lookup column has values produces duplicates, in both generator modes. Cannot be used on a column covered by a unique index. See Uniqueness and unique indexes.

Supported PostgreSQL types
text, varchar, char, bpchar, citext, bytea, boolean, int2, int4, int8, float4, float8, uuid, date, timestamp, timestamptz

The type comes from the lookup column: the transformer asks PostgreSQL what it is and reports the column types its values can be written to, so a rule pointing a column at a lookup column of an incompatible type is rejected on startup. A narrower integer or float is accepted for a wider column (an int4 lookup key can fill an int8 foreign key). A lookup column of any other type is rejected rather than silently skipping the check.

Parameter Type Default Required Values
lookup_table string N/A Yes N/A
lookup_column string N/A Yes N/A
generator string random No random, deterministic
max_values integer 100000 No N/A
ignore_values any[] [] No N/A
postgres_url string N/A Yes N/A

lookup_table is schema qualified, e.g. public.countries; an unqualified name is read from the public schema. Both names are quoted as written, so Public.Countries looks for a case-sensitive "Countries".

postgres_url is required, but the PostgreSQL parser fills it in with the URL of the source database being read, so it only has to be written out when the source is not PostgreSQL.

max_values caps how many values are read. The load fails if the lookup column holds more, rather than truncating the list, because a truncated list would silently change which value every row is mapped to. Raise it if the table really is that large and the memory cost is acceptable.

ignore_values removes values from the list after it is read, for placeholder rows such as an "unknown" id. An entry that matches nothing is an error, so a typo or a value written in a form the column never produces is reported rather than quietly leaving the row in the choice set. If it excludes every value, or the lookup column is empty, the pipeline fails to start rather than writing the same value into every row.

generator: deterministic picks the value from a hash of the incoming value, so every row that pointed at the same original value still points at one single new value. generator: random picks independently for each row and destroys that grouping. Deterministic mode is rejected for timestamp and timestamptz lookup columns, because a snapshot and a replication event deliver a timestamp in forms that cannot be reduced to the same hash input, so the same row would be mapped differently either side of the cutover.

Security note: the deterministic mapping is an unsalted hash over a value set that is usually small and often public. Anyone holding the transformed data and a guess at the lookup table can compute the same hashes and recover much of the original mapping. Deterministic mode preserves structure; it does not hide the values it maps from. The same is true of greenmask_choice.

⚠️ A reference can be left dangling, and the rows are dropped rather than reported. The values are read from the lookup table in the source database. If that table's own key column is itself transformed, or the lookup row is deleted after the pipeline started, the value chosen here will not exist in the target and the foreign key constraint rejects it. Nothing checks this for you. During a snapshot the constraints are restored after the data, so the failure arrives at the end of the load as a constraint violation that names the constraint rather than this rule. Under replication a constraint violation is not retried: in the default mode the row is dropped with a DATALOSS log line and the checkpoint advances, so the pipeline keeps running without it, and with deterministic every row sharing that original value is dropped too. Under strict_mode the pipeline stops instead. Leave the lookup table's key column untransformed, with noop if the validation mode requires a rule for it.

⚠️ The mapping only holds while the lookup column does. The value is chosen by position in the loaded list, so inserting or deleting a single row in the lookup table — or editing ignore_values — remaps almost every input the next time the pipeline starts. Deterministic mode is reproducible across restarts for a fixed lookup set; it is not stable across changes to it. New rows in the lookup table are not picked up until a restart.

⚠️ The whole column is loaded into memory, once per rule. There is no paging, and each column rule runs its own query: three columns reading the same lookup table load it three times. max_values caps each of those loads at 100000 values by default, so a rule pointed at a large table fails on startup instead of growing until the kernel intervenes. The read is given 30 seconds, so a locked or unreachable lookup table fails startup instead of hanging it.

Example Configuration:

transformations:
  table_transformers:
    - schema: public
      table: addresses
      column_transformers:
        country_id:
          name: lookup_choice
          parameters:
            lookup_table: public.countries
            lookup_column: id
            generator: deterministic
            ignore_values: [0, -1]

Input-Output Examples:

Given a public.countries table whose id column holds 1, 2, 3:

Input Value Configuration Parameters Output Value
7 generator: random 3 (random)
7 generator: deterministic 2
7 generator: deterministic 2 (again, next run)
8 generator: deterministic 1
fpe_ff1

Description: This transformer encrypts values with FF1 (NIST SP 800-38G). FF1 is a format-preserving encryption algorithm. The output contains the same characters as the input, and it has the same length. An encrypted phone number is still a phone number. An encrypted product code still fits a varchar(n) column. The transformer copies the characters that are not in the alphabet to the same positions in the output.

Uniqueness: preserved. FF1 maps each input to one different output. Two different inputs cannot give the same output. Use this transformer when a unique column must keep its format. Use encrypted_aes_siv when the format is not important.

Supported PostgreSQL types
text, varchar, char, bpchar
Parameter Type Default Required Values
key_hex string N/A Yes 32, 48 or 64 hexadecimal characters
associated_data string "" No Any string
alphabet string digits No digits, letters, alphanumeric, or a set of characters
passthrough string keep No keep, error
keep_prefix int 0 No Number of characters in the alphabet to keep at the start
keep_suffix int 0 No Number of characters in the alphabet to keep at the end
min_length int 0 No 0 sets the minimum that FF1 permits for the alphabet
preserve_from string "" No A delimiter. The transformer keeps the text from its last occurrence.

key_hex is the AES key in hexadecimal. Use 32, 48 or 64 characters for a key of 128, 192 or 256 bits. To make a key, run openssl rand -hex 32. The encrypted_aes_siv transformer is different, because it needs a key of 64 bytes.

associated_data is the FF1 tweak. The tweak is not secret. It makes the output different for each column when you use the same key. Set it to schema.table.column.

alphabet is the set of characters that the transformer encrypts. There are three named sets: digits (0-9), letters (a-zA-Z) and alphanumeric (0-9a-zA-Z). Any other value is a set of characters that you write out. Such a set must contain two characters or more, and it must not contain a character two times. The transformer does not encrypt the characters that are not in the alphabet.

passthrough controls the characters that are not in the alphabet. The value keep copies these characters to the output. Characters such as -, + and the space stay in a phone number. The value error rejects the value.

keep_prefix and keep_suffix keep a number of characters without a change. Use them to keep a country code or a check digit. These parameters count only the characters in the alphabet. Separators do not increase the count, because the transformer does not encrypt them. For example, set keep_prefix: 4 for a digits column. The transformer then keeps the country code and the operator code of +36301234567, +36 30 123 4567 and +36-30-123-4567. The three formats give the same digits in the output. The transformer rejects a value that has too few characters. Refer to the first caution below.

preserve_from keeps the end of the value without a change. The part that the transformer keeps starts at the last occurrence of the delimiter. Use this parameter when a delimiter marks the part to keep. keep_suffix keeps a fixed number of characters, but preserve_from keeps a variable number. In an email address, preserve_from: "@" keeps the full domain. preserve_from: "." keeps only the top-level domain, and it encrypts the domain name. If the value does not contain the delimiter, the transformer encrypts the full value. The delimiter must not contain a character from the alphabet. If it does, pgstream rejects the configuration when it starts. The transformer copies only the characters that are not in the alphabet to the same position in the output. This condition is necessary to keep the outputs unique.

⚠️ The transformer rejects short values. FF1 does not encrypt a set of possible values that is smaller than 106, because an attacker can try all of them. For digits, a value must contain 6 characters or more from the alphabet. For letters and alphanumeric, it must contain 4 characters or more. Count the characters after you remove the prefix, the suffix, and the characters that are not in the alphabet. The transformer returns an error for a shorter value. It does not send the original value to the target, because this makes the data visible. The error goes to the on_error policy, where you select fail, pass-through or null. Be careful with pass-through, because it writes the original value to the target. min_length increases the minimum, for example to 9 digits. It cannot decrease the minimum.

⚠️ The transformer keeps the format, but the output is not realistic. A phone number stays a possible phone number, and a code stays a valid code. But with alphabet: letters, a company name becomes a random text such as Dmkh VjksJwk. The output has the shape of a name and it is unique, but it is not a real company name.

Email addresses. With alphabet: alphanumeric and passthrough: keep, the transformer encrypts the local part, the domain and the top-level domain. The address john.doe@example.com becomes FdSQ.v6s@udengIL.0Td. The top-level domain is not valid, and a mail server cannot deliver to this address. Use preserve_from to correct this:

preserve_from Result What the transformer encrypts
unset FdSQ.v6s@udengIL.0Td The full address. The top-level domain is not valid.
"." lHtC.Mr7@1uH9RP1.com The local part and the domain name. The transformer keeps the top-level domain, thus the address has a correct format.
"@" 69pe.Nzy@example.com The local part only. The address is valid, but the target shows the domain.

Select the option that agrees with your data. Do not use preserve_from: "@" when the domain is sensitive, because the target then contains each source domain. All three options keep the outputs unique. Use neosync_email with preserve_domain: true when a mail server must deliver to the address. That transformer makes a new local part, but its uniqueness is not_guaranteed. Two different addresses can give the same output.

Security note: The encryption is deterministic. Equal inputs give equal outputs. This shows which values are equal, but it shows no other data. A person who has the key can decrypt the values. Keep the key in a secret manager, and do not store it with the transformed data.

Example Configuration:

transformations:
  table_transformers:
    - schema: public
      table: persons
      column_transformers:
        phone_number:
          name: fpe_ff1
          parameters:
            # example key only — generate your own and load it from a secret store
            key_hex: "000102030405060708090a0b0c0d0e0f"
            associated_data: "public.persons.phone_number"
            alphabet: digits
            passthrough: keep
            # keep 4 digits: the country code and the operator code.
            # separators do not increase this count.
            keep_prefix: 4

Input-Output Examples:

Input Configuration Parameters Output
1234567890 key_hex: "000102…0e0f", no associated_data 5102547240
1234567890 key_hex: "000102…0e0f", associated_data: "public.persons.phone_number" 0453276999
+36301234567 as the example above (keep_prefix: 4) +36303497985
+36 30 123 4567 as the example above (keep_prefix: 4) +36 30 349 7985
(301) 555-0123 as the example above (keep_prefix: 4) (301) 545-8564
Acme Trading key_hex: "000102…0e0f", alphabet: letters Dmkh VjksJwk
john.doe@example.com key_hex: "000102…0e0f", alphabet: alphanumeric FdSQ.v6s@udengIL.0Td
john.doe@example.com key_hex: "000102…0e0f", alphabet: alphanumeric, preserve_from: "." lHtC.Mr7@1uH9RP1.com
john.doe@example.com key_hex: "000102…0e0f", alphabet: alphanumeric, preserve_from: "@" 69pe.Nzy@example.com

Each run gives the same output for the same key and the same parameters. Any NIST FF1 implementation can decrypt the values. It needs the key, the tweak and the alphabet.

Transformation rules

The rules for the transformers are defined in a dedicated yaml file with the following format:

transformations:
  infer_from_security_labels: false # whether to infer anonymization rules from Postgres `anon` security labels. Requires a live connection to the source database. Defaults to false
  dump_inferred_rules: false # if set, dumps the inferred anonymization rules to a file in YAML format for debugging purposes. The file will be named `inferred_anon_transformation_rules.yaml` and will be created in the directory where pgstream is run. Defaults to false
  validation_mode: <validation_mode> # Validation mode for the transformation rules. Can be one of strict, relaxed or table_level. Defaults to relaxed if not provided.
  table_transformers: # List of table transformations
    - schema: <schema_name> # Name of the table schema
      table: <table_name> # Name of the table
      validation_mode: <validation_mode> # To be used when the global validation_mode is set to `table_level`. Can be one of strict or relaxed
      column_transformers: # List of column transformations
        <column_name>: # Name of the column to which the transformation will be applied
          name: <transformer_name> # Name of the transformer to be applied to the column. If no transformer needs to be applied on strict validation mode, it can be left empty or use `noop`
          allow_uniqueness_loss: false # Whether to allow a transformer that can produce duplicates on a column covered by a unique index. Defaults to false. See "Uniqueness and unique indexes"
          parameters: # Transformer parameters as defined in the supported transformers documentation
            <transformer_parameter>: <transformer_parameter_value>
          array_options: # Only valid on an array column. If omitted, the transformer is applied to each element in order. See "Array columns"
            generator: <map|random> # How the elements of the new array are produced. Defaults to map
            min_count: <min_count> # Smallest number of elements to emit. Required with the random generator
            max_count: <max_count> # Largest number of elements to emit. Required with the random generator

When the infer_from_security_labels option is enabled, the table transformers will be parsed from the source Postgres SECURITY LABELS for the anon extension. If the option is not enabled, the table transformers need to be explicitly provided.

Below is a complete example of a transformation rules YAML file:

transformations:
  infer_from_security_labels: false # whether to infer anonymization rules from Postgres `anon` security labels. Requires a live connection to the source database. Defaults to false
  dump_inferred_rules: false # if set, dumps the inferred anonymization rules to a file in YAML format for debugging purposes. The file will be named `inferred_anon_transformation_rules.yaml` and will be created in the directory where pgstream is run. Defaults to false
  validation_mode: table_level
  table_transformers:
    - schema: public
      table: users
      validation_mode: strict
      column_transformers:
        email:
          name: neosync_email
          parameters:
            preserve_length: true
            preserve_domain: true
        first_name:
          name: greenmask_firstname
          parameters:
            gender: Male
        username:
          name: greenmask_string
          parameters:
            generator: random
            min_length: 5
            max_length: 15
            symbols: "abcdefghijklmnopqrstuvwxyz1234567890"
    - schema: public
      table: orders
      validation_mode: relaxed
      column_transformers:
        status:
          name: greenmask_choice
          parameters:
            generator: random
            choices: ["pending", "shipped", "delivered", "cancelled"]
        order_date:
          name: greenmask_date
          parameters:
            generator: random
            min_value: "2020-01-01"
            max_value: "2025-12-31"

Validation mode can be set to strict or relaxed for all tables at once. Or it can be determined for each table individually, by setting the higher level validation_mode parameter to table_level. When it is set to strict, pgstream will throw an error if any of the columns in the table do not have a transformer defined. When set to relaxed, pgstream will skip any columns that do not have a transformer defined. Also in strict mode, all snapshot tables must be provided in the transformation config. For details on how to use and configure the transformer, check the transformer tutorial.