Skip to content

Fail migrations when named Azure Blob sources are missing - #262

Merged
Mark Brown (markjbrown) merged 2 commits into
mainfrom
markjbrown-missing-azure-blob-source
Oct 31, 2026
Merged

Mark Brown (markjbrown) merged 2 commits into
mainfrom
markjbrown-missing-azure-blob-source

Conversation

@markjbrown

@markjbrown Mark Brown (markjbrown) commented Oct 31, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Fixes #250.

  • Replace the silent empty-source result for a missing named Azure blob with an actionable failure that reaches the CLI as exit code 1, without Data transfer complete.
  • Preserve valid JSON and valid empty [] input, cancellation, stream seeking, and disposal.
  • Report safe container/blob context and HTTP status for source initialization, access, open, and deferred-read failures without exposing connection strings, signed URL queries, SDK response bodies, or inner exception details.
  • Add 50 deterministic offline regressions covering the source, real JSON formatter/file-sink pipeline, command handler, diagnostics, and stream behavior. Document the source-failure contract.

Destination-file atomicity and rollback remain separate in #251. This PR does not change Core, JSON, or Common production code, and does not guarantee preservation of destination bytes after a source failure.

Copilot review follow-up

Both findings were reproduced and fixed in 2858d2ababa5321773bba8a0c1c712c1c4bf22b8:

  • Withhold fallback locations containing possible user-info, covering malformed HTTP URLs, non-HTTP URLs, protocol-relative URLs, and opaque locations.
  • Sanitize nonempty aggregates containing only recognized read failures, including nested transport retries. Keep safe HTTP status information while omitting original messages and inner exceptions. Cancellation-containing, empty, and unexpected-error aggregates remain unchanged.

Added 20 regression cases; all 50 Azure Blob tests pass locally and in CI. Both review threads are resolved.

Verification

Current validated head: 2858d2ababa5321773bba8a0c1c712c1c4bf22b8, based on 81a625be548effbce3ce1d1b4074581fcc9ecaff containing the merged .NET 10 upgrade.

  • Current-head Build/Test CI: successful run 41222325961. Downloaded TRX confirms 1,403 passed, 0 failed, 9 skipped, including all 50 Azure Blob regressions. Skips are 7 unconfigured Cosmos integration tests, 1 unconfigured SQL test, and the existing CSV current-culture test when the Linux environment has an empty culture name.
  • Current-head package/container/native smoke CI: successful run 41222325991, including six runtime builds, six native smoke jobs, and the non-publishing container smoke job.
  • Current-head CodeQL: successful run 41222324261.
  • Local review validation: all 50 offline Azure Blob tests passed on pinned SDK 10.0.403/runtime 10.0.14. Before the follow-up fix, six unsafe-location cases leaked the sentinel and three aggregate pipeline cases bypassed sanitization; all now pass.
  • Original implementation (548e22b): before the fix, the missing-blob source threw no exception and the real JSON/file-sink + command-handler pipeline returned 0; after the fix, it fails with exit code 1. The Windows whole-solution run passed 1,384 tests with 8 unconfigured-service skips. Initial Build/Test and Validate runs also passed.
  • Independent packaged CLI check of 548e22b: the coordinating session tested the self-contained Windows CLI and real Azure SDK against a localhost-only HTTP fixture. All 12 cases passed: missing 404, denied 403, disappearance after HEAD, and malformed JSON fail nonzero; valid JSON and empty arrays succeed. Checks include new/existing destinations and a SAS sentinel that never appears in logs. No live Azure account or credentials were used.

Limits

No live-cloud or emulator integration run is claimed. Existing package vulnerability warnings (System.Security.Cryptography.Pkcs 9.0.0, NU1902/NU1903) and pre-existing nullable/analyzer warnings remain unsuppressed. No test exclusions were broadened. No release, deploy, or publication workflow was run.

Fail named missing blobs instead of completing an empty migration. Preserve safe source context across access and streaming errors, and add offline JSON pipeline and CLI regressions.

Relates to #250

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Credential redaction remains incomplete because malformed or non-HTTP URLs can expose user-info, while aggregate transport failures can bypass sanitization and leak secrets through exception logging.

2 open issues
What changed in this PR

This PR addresses #250 by making missing named Azure Blob sources fail migrations instead of appearing as successful empty input. Destination atomicity and rollback remain separate under #251.

Changes:

  • Adds source-failure propagation and sanitized diagnostics for initialization, access, opening, and streaming.
  • Adds 30 offline regression cases covering pipeline outcomes, CLI reporting, diagnostics, cancellation, and stream behavior.
  • Documents the source-failure contract and preserves valid empty JSON input.
File Description
Extensions/​AzureBlob/​README.md Documents failure behavior and rollback limitations.
Extensions/​AzureBlob/​Cosmos.DataTransfer.AzureBlobStorage/​Cosmos.DataTransfer.AzureBlobStorage.csproj Exposes internals to the test assembly.
Extensions/​AzureBlob/​Cosmos.DataTransfer.AzureBlobStorage/​AzureBlobDataSource.cs Adds explicit failures, diagnostic sanitization, and a read-stream wrapper.
Extensions/​AzureBlob/​Cosmos.DataTransfer.AzureBlobStorage.UnitTests/​Cosmos.DataTransfer.AzureBlobStorage.UnitTests.csproj Defines the offline test project and dependencies.
Extensions/​AzureBlob/​Cosmos.DataTransfer.AzureBlobStorage.UnitTests/​AzureBlobSourceFailureTests.cs Adds source, pipeline, CLI, and stream regressions.
CosmosDbDataMigrationTool.sln Registers the Azure Blob test project.

Review effort: Balanced

Redact possible URL user-info in unsafe location fallbacks and sanitize nested aggregates of read failures. Preserve cancellation and unexpected aggregate exceptions; add 20 offline regressions.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@markjbrown
Mark Brown (markjbrown) merged commit 971d2b5 into main Oct 31, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Missing named Azure Blob source can be reported as a successful empty migration

2 participants