feat(data): add PostgresSource for live config reads#900
Draft
zulkhair wants to merge 5 commits into
Draft
Conversation
…ray reads json_agg returns json, so coalescing it with a '[]'::jsonb fallback errored (SQLSTATE 42846) on every non-empty config table. Use a '[]'::json fallback so array kinds read back; output text is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The data plugin loads each config kind from build-time embedded JSON (
EmbedSource). This addsPostgresSource, an alternativeSourcethat reads config from a Postgres database, selected at runtime by theCONFIG_DB_DSNenv var — so a deployment can change config without rebuilding and redeploying.When
CONFIG_DB_DSNis set the plugin reads config only from Postgres — fail-loud: a missing table is an error, with no silent fallback to embedded JSON. When it's unset, the embedded path is unchanged.Table contract
Each config kind is read from its own table, where one row holds one record as JSON in a
doc jsonbcolumn:SELECT coalesce(json_agg(doc ORDER BY id), '[]'::json)::text FROM <table>— rows aggregated into a JSON array, ordered byidfor a stable content hash.SELECT doc::text FROM <table> LIMIT 1— one JSON object. A kind opts into this by implementing theSingletonmarker interface (SingleObject()).The table name for each kind — and whether it's a singleton — is registered when the kind registers:
Register[T]callsRegisterKind(JSONFile(), Name())andRegisterSingleton(...)forSingletonkinds. So a kind reads from a table named after itsName(), and shardmain.gostays unchanged.Design notes
EmbedSource: it ignores the requested content hash and returns current rows, so a snapshot restore resumes on current config (Reconcile's warn path) instead of erroring (the panic path).PickSourcecentralizes the choice (CONFIG_DB_DSNset →PostgresSource, elseEmbedSource); a set-but-unusable DSN panics — fail loud rather than silently serve stale embedded config.configReaderinterface seam makesFetchunit-testable without a live database.pgxpool; the pool connects lazily (a malformed DSN fails at construction, connectivity fails on firstFetch).Testing
pkg/plugin/data/system/postgres_internal_test.go(fakeconfigReader, no database): table-name resolution + registrar override, single-object vs array read shapes, rows +sha256hash return, single-version (requested hash ignored), and error propagation.Compatibility
Additive. The default path (no
CONFIG_DB_DSN) is the unchangedEmbedSource. New dependencies:jackc/pgx/v5+pgxpool.