feat: cloud object stores, credential chains, and config secret injection - #7
Open
manudiv16 wants to merge 3 commits into
Open
feat: cloud object stores, credential chains, and config secret injection#7manudiv16 wants to merge 3 commits into
manudiv16 wants to merge 3 commits into
Conversation
GCS/Azure object stores + credential chain docs (PR #8): - GCS backend with ADC / Workload Identity Federation - Azure backend (az:// and abfs://) with DefaultAzureCredential chain - S3 default credential chain docs (env -> IMDS -> IRSA) Config secret injection (PR #9): - Secret newtype for credential fields: plain string or { file = "path" } TOML forms, Debug redacted, expose() accessors - K2I_* env var overrides with warnings on invalid numeric/enum values and unrecognized variables (typo detection) - Mutex-serialized env-var tests - docs/kubernetes.md: projected volumes, env injection, Secrets Store CSI Driver, full variable table - azure_access_key uses Secret for consistency Closes #1
manudiv16
force-pushed
the
feat/cloud-object-stores-credential-chains
branch
from
July 25, 2026 17:46
e5d1077 to
919957a
Compare
manudiv16
changed the title
The in-bucket warehouse prefix was applied inside generate_file_path, but
that path is also what goes to the catalog, the transaction log, and the
read path — all of which resolve it by joining against warehouse_path.
For a warehouse of s3://bucket/warehouse the catalog recorded
s3://bucket/warehouse/warehouse/data/... while the upload landed at
s3://bucket/warehouse/data/..., so every committed file was unreadable.
Apply the prefix in a dedicated storage_path() used only by upload_file,
keeping every externally-visible path warehouse-relative.
Also fixes three defects in the env-override layer:
- K2I_MONITORING_LOG_FORMAT never took effect: the tracing subscriber is
configured before the config is loaded and parsed the TOML directly.
- K2I_RPC_ENABLED read any unrecognized value as false, so =yes silently
disabled an RPC server the TOML had enabled.
- The cloud object-store fields had no overrides at all, including the
Azure-required azure_storage_account_name.
And suppresses the spurious unrecognized-variable warnings for K2I_E2E_*
and the other harness variables that share the engine's environment.
Docs: docs/configuration.md promised ${VAR} shell substitution, which does
not exist — following it would have authenticated with the literal string.
Replaced with the two real mechanisms. README.md, docs/architecture.md and
docs/configuration.md still described GCS/Azure as unwired.
Verified locally: 319 tests (up from 289), clippy -D warnings, fmt,
cargo-audit, cargo-semver-checks, the Docker Kafka integration tests, and
all five docker/e2e suites.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… for real Three gaps remained after the warehouse-prefix fix. Secret redacted Debug but not Serialize, and Config derives Serialize, so any code that dumped or echoed the configuration would emit credentials in the clear. Redact in both, following the secrecy crate's convention that emitting a secret must be a conscious act. A serialized Config no longer round-trips; that trade-off is deliberate and documented, since a visibly broken credential beats a silently leaked one. Cloud warehouse settings that cannot be derived from the path are now checked in Config::validate rather than at writer construction. Azure needs a storage account name that neither URL form carries; previously a long-running ingest reported healthy and failed minutes later on its first flush. Warehouse-path parsing now lives in one place and is shared with the writer, so validation and store construction cannot drift. The prefix fix was only unit-tested. Added container-backed S3 round-trip tests (MinIO) for a prefixed warehouse, a multi-segment prefix, and a bucket-root warehouse, each asserting that warehouse_path joined with the reported path resolves to a real object and that nothing landed at the doubled-prefix or bucket-root locations. Verified they fail against the previous behaviour with the exact production symptom: a 404 on s3://bucket/warehouse/prod/warehouse/prod/data/... Those tests would not have run in CI: every container-backed test is #[ignore = "requires Docker"], and the integration job never passed --include-ignored, so it provisioned Docker and ran no Docker test at all, including the pre-existing Kafka ones. Fixed, and the workflow now also triggers on docs/ and config/ since a test asserts every K2I_* variable is documented. Verified locally: 326 tests, the CI integration command (13 tests incl. 5 container-backed), clippy -D warnings, fmt, cargo-audit, cargo-semver-checks, and all five docker/e2e suites. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mirror of osodevops#8 for CI runs. Combined cloud object stores + config secret injection. See osodevops#8 for full description.