Skip to content

Add universe domain support for sovereign cloud environments - #2373

Merged
kubernetes-prow[bot] merged 2 commits into
kubernetes-sigs:masterfrom
rvagner78:upstream-universe-domain
Aug 7, 2026
Merged

Add universe domain support for sovereign cloud environments#2373
kubernetes-prow[bot] merged 2 commits into
kubernetes-sigs:masterfrom
rvagner78:upstream-universe-domain

Conversation

@rvagner78

@rvagner78 rvagner78 commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/kind api-change
/kind bug
/kind cleanup
/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

What this PR does / why we need it:

The GCE PD CSI driver currently hardcodes googleapis.com in all API endpoint construction — compute resource paths, Cloud Resource Manager tag endpoints, and credential handling. This prevents the driver from operating in Google Cloud environments that use a non-default universe domain, such as Google Cloud Dedicated (GCD), GCP's sovereign cloud offering.

This PR is a proposal how to make the driver universe-domain-aware:

  • Replaces google.DefaultTokenSource with google.FindDefaultCredentials, which exposes the universe domain from the credential
  • Replaces hardcoded //compute.googleapis.com/... format strings with a computeParentPath() helper that interpolates the correct universe domain
  • Replaces the hardcoded cloudresourcemanager.googleapis.com constant with a resourceManagerHost() helper for tag value and tag binding clients
  • Introduces authOpts() to centralize client option construction — uses WithAuthCredentialsJSON when credential JSON is available, falls back to WithHTTPClient for alternate token sources

On standard GCP the behavior is unchanged — when the credential returns the default universe domain (googleapis.com) or empty string, all endpoints resolve identically. No additional configuration is needed from the cluster operator; the universe domain is determined from the credential itself.

Which issue(s) this PR fixes:

None

Special notes for your reviewer:
The universe domain is a DNS suffix that organizes Google Cloud API endpoints. Standard GCP uses googleapis.com. Sovereign cloud deployments (e.g. Google Cloud Dedicated) use a different universe domain so that all API traffic stays within the sovereign boundary. See https://cloud.google.com/dedicated/docs/overview for details.

Does this PR introduce a user-facing change?:

Add universe domain support to allow the GCE PD CSI driver to operate in sovereign cloud environments with non-default universe domains (e.g. Google Cloud Dedicated)

Adds custom universe domain support in order to allow operation
in environments with non-default universe domains, such as
Google Cloud Dedicated, GCP's sovereign cloud offering.

The universe domain can be determined from the credential, and
replaces the standard googleapis.com resource path.
@kubernetes-prow kubernetes-prow Bot added kind/feature Categorizes issue or PR as related to a new feature. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jul 27, 2026
@kubernetes-prow

Copy link
Copy Markdown
Contributor

Welcome @rvagner78!

It looks like this is your first PR to kubernetes-sigs/gcp-compute-persistent-disk-csi-driver 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/gcp-compute-persistent-disk-csi-driver has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@kubernetes-prow kubernetes-prow Bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 27, 2026
@kubernetes-prow

Copy link
Copy Markdown
Contributor

Hi @rvagner78. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kubernetes-prow kubernetes-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jul 27, 2026
@kubernetes-prow
kubernetes-prow Bot requested review from hajiler and hime July 27, 2026 12:56
@jsafrane

Copy link
Copy Markdown
Contributor

/ok-to-test

@kubernetes-prow kubernetes-prow Bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 27, 2026
@linux-foundation-easycla

linux-foundation-easycla Bot commented Jul 27, 2026

Copy link
Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.

  • ✅ login: patrickdillon / name: Patrick Dillon (552bada)
  • ✅ login: rvagner78 / name: Radomir Vagner (f5c8c3f)

@kubernetes-prow kubernetes-prow Bot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jul 27, 2026
@jsafrane

Copy link
Copy Markdown
Contributor

/test all

@jsafrane

jsafrane commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Cross referencing the same problem in the cloud provider kubernetes/cloud-provider-gcp#1227

@patrickdillon

patrickdillon commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Cross referencing the same problem in the cloud provider kubernetes/cloud-provider-gcp#1227

kubernetes/cloud-provider-gcp#1250 is the accepted solution

@rvagner78
rvagner78 marked this pull request as ready for review July 31, 2026 12:59
@kubernetes-prow kubernetes-prow Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 31, 2026
@sunnylovestiramisu

Copy link
Copy Markdown
Contributor

Translate this to a term we know better: this is for support of Trusted Partner Cloud, but our driver should have already supported TPC. Is this some bugs you see?

@patrickdillon

Copy link
Copy Markdown
Contributor

Translate this to a term we know better: this is for support of Trusted Partner Cloud, but our driver should have already supported TPC. Is this some bugs you see?

@sunnylovestiramisu Thanks for taking a look! Yes this proposes to add support for TPC (or Google Cloud Dedicated) by enabling support for alternate Universe Domains. I do not believe there is support in the driver for alternate Universe Domains/TPC for two primary reasons:

First, the driver constructs API endpoints, and those are hard-coded to googleapis.com:

resourceManagerHostSubPath = "cloudresourcemanager.googleapis.com"
// zonalOrRegionalComputeParentPathFmt is the string format for the full path of compute resource.
// belonging to a zone or a region
zonalOrRegionalComputeParentPathFmt = "//compute.googleapis.com/projects/%s/%s/%s/%s/%d"
// globalComputeParentPathFmt is the string format for the full path of global compute resource.
globalComputeParentPathFmt = "//compute.googleapis.com/projects/%s/global/%s/%d"

In a TPC-environment with an alternate universe domain, that endpoint will not be accessible and calls to those endpoints will fail.

Second, the GCP SDK requires that callers opt-in and explicitly set the universe domain so that it matches the credentials. In a TPC environment, the credentials will include an alternate universe domain and therefore the client calls must also set that universe domain. Technically that universe domain could be set with the GOOGLE_CLOUD_UNIVERSE_DOMAIN, so that could happen outside of this code. That said, explicitly setting WithUniverseDomain handles this better. Also, in my testing of the Berlin GCD environment, oauth endpoints were not available for the alternate universe domain, which required using the JSON authentication methods, which use self-signed JWTs instead of oauth endpoints.

@sunnylovestiramisu

sunnylovestiramisu commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Can you try to configure the --compute-endpoint to your customized endpoint and see if the driver is working in TPC?

But I am not sure about the cloudresourcemanager.googleapis.com one hmm. I checked that we did tested the TPC in TPC Staging internally at Google and it is working. I wonder if we missed something.

@patrickdillon

patrickdillon commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Can you try to configure the --compute-endpoint to your customized endpoint and see if the driver is working in TPC?

But I am not sure about the cloudresourcemanager.googleapis.com one hmm. I checked that we did tested the TPC in TPC Staging internally at Google and it is working. I wonder if we missed something.

Thank you, I wasn't aware of this flag. We can try testing it, but I am certain the issue that we will encounter is that the driver uses DefaultTokenSource oauth token exchange for authentication, which is not supported with an alternate universe domain:

func isSelfSignedJWTFlow(data []byte, ds *DialSettings) (bool, error) {
// For non-GDU universe domains, token exchange is impossible and services
// must support self-signed JWTs with scopes.
if !ds.IsUniverseDomainGDU() {
return typeServiceAccount(data)
}

As the comment indicates, it is necessary to use self-signed JWTs instead of oauth in these environments. When testing with other components, such as cloud-provider-gcp we get an error message like:

ERROR failed to fetch Metadata: failed to load asset "Install Config": failed to create install config: failed to find project permissions: failed to get Iam permissions: Post "https://cloudresourcemanager.apis-berlin-build0.goog/v3/projects/eu0:openshift:testIamPermissions?alt=json&prettyPrint=false": oauth2: cannot fetch token: Post "https://oauth2.apis-berlin-build0.goog/token": dial tcp: lookup oauth2.apis-berlin-build0.goog on 169.254.169.254:53: no such host

@patrickdillon

Copy link
Copy Markdown
Contributor

@sunnylovestiramisu I tried setting the endpoint, and confirmed as expected that authentication can not work:

Name:                 gcp-pd-csi-driver-controller-66558b865f-khq9r
Namespace:            openshift-cluster-csi-drivers
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Service Account:      gcp-pd-csi-driver-controller-sa
Node:                 padillon-08041144-m8j7d-master-1/10.0.0.44
Start Time:           Tue, 04 Aug 2026 16:06:25 +0000
Labels:               app=gcp-pd-csi-driver-controller
                      pod-template-hash=66558b865f
                      topology.kubernetes.io/region=u-germany-northeast1
                      topology.kubernetes.io/zone=u-germany-northeast1-b
Annotations:          openshift.io/required-scc: hostnetwork-v2
                      openshift.io/scc: hostnetwork-v2
                      operator.openshift.io/dep-3003c66d1e7cfab0b092cf87127145aa0f888: xYGQpA==
                      operator.openshift.io/dep-9b2ad69e0e161cab32acb7d6e6b381665a89a: b3aTLg==
                      operator.openshift.io/dep-d227b2c1222d04406f604fc9bce845050ae07: Owk7Ew==
                      seccomp.security.alpha.kubernetes.io/pod: runtime/default
                      security.openshift.io/validated-scc-subject-type: serviceaccount
Status:               Running
SeccompProfile:       RuntimeDefault
IP:                   10.0.0.44
IPs:
  IP:           10.0.0.44
Controlled By:  ReplicaSet/gcp-pd-csi-driver-controller-66558b865f
Containers:
  csi-driver:
    Container ID:  cri-o://fe8fa43259f47e201a5d8bf57c437276f2902f094275897c6fcf8160c4bccc00
    Image:         quay.io/openshift-release-dev/ocp-v5.0-art-dev@sha256:a5bed62b7742e2dcffcd3705ed07c499b8c11322821cf422adb8aab1f1cb90b1
    Image ID:      quay.io/openshift-release-dev/ocp-v5.0-art-dev@sha256:a5bed62b7742e2dcffcd3705ed07c499b8c11322821cf422adb8aab1f1cb90b1
    Port:          10301/TCP (healthz)
    Host Port:     10301/TCP (healthz)
    Args:
      --endpoint=$(CSI_ENDPOINT)
      --logtostderr
      --v=2
      --enable-storage-pools=true
      --allow-hdha-provisioning=true
      --supports-dynamic-throughput-provisioning=hyperdisk-balanced,hyperdisk-throughput,hyperdisk-ml
      --supports-dynamic-iops-provisioning=hyperdisk-balanced,hyperdisk-extreme
      --extra-labels=kubernetes-io-cluster-padillon-08041144-m8j7d=owned
      --compute-endpoint=https://compute.apis-berlin-build0.goog
    State:       Running
      Started:   Tue, 04 Aug 2026 16:13:19 +0000
    Last State:  Terminated
      Reason:    Error
      Message:   h2: cannot fetch token: Post "https://oauth2.apis-berlin-build0.goog/token": dial tcp: lookup oauth2.apis-berlin-build0.goog on 169.254.169.254:53: no such host
E0804 16:11:21.405068       1 gce.go:418] error fetching initial token: oauth2: cannot fetch token: Post "https://oauth2.apis-berlin-build0.goog/token": dial tcp: lookup oauth2.apis-berlin-build0.goog on 169.254.169.254:53: no such host
E0804 16:11:26.405143       1 gce.go:418] error fetching initial token: oauth2: cannot fetch token: Post "https://oauth2.apis-berlin-build0.goog/token": dial tcp: lookup oauth2.apis-berlin-build0.goog on 169.254.169.254:53: no such host
E0804 16:11:26.411359       1 gce.go:418] error fetching initial token: oauth2: cannot fetch token: Post "https://oauth2.apis-berlin-build0.goog/token": dial tcp: lookup oauth2.apis-berlin-build0.goog on 169.254.169.254:53: no such host
E0804 16:11:26.411372       1 gce.go:345] Failed to get compute endpoint: timed out waiting for the condition
I0804 16:11:26.412080       1 gce.go:175] Compute endpoint for V1 version: https://compute.googleapis.com/compute/v1/
E0804 16:11:26.417895       1 gce.go:418] error fetching initial token: oauth2: cannot fetch token: Post "https://oauth2.apis-be

@sunnylovestiramisu

sunnylovestiramisu commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Oh I just realized that you are using tagging. Our internal TPC tests do not use that that is why it succeeds. If you remove ResourceTags usage, it should succeed.

In a sovereign/universe environment where compute and resource-manager hostnames differ from googleapis.com (or are region-prefixed differently), tagging calls will try to use compute.googleapis.com and cloudresourcemanager.googleapis.com and will fail even if compute calls succeed via --compute-endpoint.

This is a bug in TPC for sure. Have you tested this PR in your Google Cloud Dedicated environment?

@patrickdillon

Copy link
Copy Markdown
Contributor

Oh I just realized that you are using tagging. Our internal TPC tests do not use that that is why it succeeds. If you remove ResourceTags usage, it should succeed.

Interesting! I will try this out. Thank you, this comment is extremely helpful.

In a sovereign/universe environment where compute and resource-manager hostnames differ from googleapis.com (or are region-prefixed differently), tagging calls will try to use compute.googleapis.com and cloudresourcemanager.googleapis.com and will fail even if compute calls succeed via --compute-endpoint.

This is a bug in TPC for sure. Have you tested this PR in your Google Cloud Dedicated environment?

Yes, this code is included in the test results which you can see here. Everything worked except a few expected test failures (due to how our prow instance proxies to connect to the cluster).

@patrickdillon

patrickdillon commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

@sunnylovestiramisu So I tried again, making sure to unset tags, but we get the same result. The result makes intuitive sense because the driver uses DefaultTokenSource, which will utilize oauth token exchange but token exchange is impossible in non-default universe domains:

func isSelfSignedJWTFlow(data []byte, ds *DialSettings) (bool, error) {
// For non-GDU universe domains, token exchange is impossible and services
// must support self-signed JWTs with scopes.
if !ds.IsUniverseDomainGDU() {
return typeServiceAccount(data)
}

We can see from the failure in my testing ☝️ that the error is coming from newOauthClient, which is called when initializing the cloud service/provider--it fails before we get to tagging.

I do wonder if the staging sandbox you used for testing had a working oauth token endpoint, which would not be available in a real gcd environment.

@sunnylovestiramisu

Copy link
Copy Markdown
Contributor

Hmm sending this to the TPC team, it may mean that the test environment setup has flaws.

@sunnylovestiramisu

Copy link
Copy Markdown
Contributor

@patrickdillon Can you also share the error you see before setting the compute-endpoint flag?

Talked with the TPC team, and they agreed anything that would update the code to be universe aware is good (and usually harmless). So this PR is not blocked by anything.

It is odd that we see differences in internal PRP tests compared to TSP(Germany).

@patrickdillon

Copy link
Copy Markdown
Contributor

@sunnylovestiramisu No problem. The error is the same:

Containers:
  csi-driver:
    Container ID:  cri-o://d2aa0469296b784716ec4fb36404c65db24197cc91f685c90162ea2e8a8c472f
    Image:         quay.io/openshift-release-dev/ocp-v5.0-art-dev@sha256:256efb6c6831407518cb2ec8bb8ceec237b23fa4a5bb22c8ca59af08ad40c4c5
    Image ID:      quay.io/openshift-release-dev/ocp-v5.0-art-dev@sha256:256efb6c6831407518cb2ec8bb8ceec237b23fa4a5bb22c8ca59af08ad40c4c5
    Port:          10301/TCP (healthz)
    Host Port:     10301/TCP (healthz)
    Args:
      --endpoint=$(CSI_ENDPOINT)
      --logtostderr
      --v=2
      --enable-storage-pools=true
      --allow-hdha-provisioning=true
      --supports-dynamic-throughput-provisioning=hyperdisk-balanced,hyperdisk-throughput,hyperdisk-ml
      --supports-dynamic-iops-provisioning=hyperdisk-balanced,hyperdisk-extreme
      --extra-labels=kubernetes-io-cluster-padillon-08061535-bcw8z=owned
    State:       Running
      Started:   Thu, 06 Aug 2026 20:02:00 +0000
    Last State:  Terminated
      Reason:    Error
      Message:   h2: cannot fetch token: Post "https://oauth2.apis-berlin-build0.goog/token": dial tcp: lookup oauth2.apis-berlin-build0.goog on 169.254.169.254:53: no such host
E0806 20:01:35.516494       1 gce.go:418] error fetching initial token: oauth2: cannot fetch token: Post "https://oauth2.apis-berlin-build0.goog/token": dial tcp: lookup oauth2.apis-berlin-build0.goog on 169.254.169.254:53: no such host
E0806 20:01:40.518628       1 gce.go:418] error fetching initial token: oauth2: cannot fetch token: Post "https://oauth2.apis-berlin-build0.goog/token": dial tcp: lookup oauth2.apis-berlin-build0.goog on 169.254.169.254:53: no such host
E0806 20:01:40.525279       1 gce.go:418] error fetching initial token: oauth2: cannot fetch token: Post "https://oauth2.apis-berlin-build0.goog/token": dial tcp: lookup oauth2.apis-berlin-build0.goog on 169.254.169.254:53: no such host
E0806 20:01:40.525308       1 gce.go:345] Failed to get compute endpoint: timed out waiting for the condition
I0806 20:01:40.525981       1 gce.go:175] Compute endpoint for V1 version: https://compute.googleapis.com/compute/v1/
E0806 20:01:40.531171       1 gce.go:418] error fetching initial token: oauth2: cannot fetch token: Post "https://oauth2.apis-be

This test is from the most recent nightly of openshift 5.0.

It is odd that we see differences in internal PRP tests compared to TSP(Germany).

The difference must be that the internal environment has a valid oauth token endpoint... the actual environment does not have an oauth token endpoint.

@sunnylovestiramisu

sunnylovestiramisu commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

I reviewed the PR. The PR only makes endpoint/credential handling universe-domain-aware and preserves current behavior for the default universe (googleapis.com), so existing clusters should not be affected.

It is good for merging.

Question to you:

  1. Do you need it in a specific branch in PDCSI open source repo if you are doing self deployed PDCSI driver?
  2. Do you need it in a specific version in GKE in your TPC environment?

@kubernetes-prow kubernetes-prow Bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Aug 6, 2026
@jsafrane

jsafrane commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

For us it's enough if the code is merged into master and it's in the next driver release. Or at least the behavior is the same. We can cherry-pick it to our driver fork by ourselves and backport as far as we need.

@sunnylovestiramisu

Copy link
Copy Markdown
Contributor

/lgtm
/approve

@kubernetes-prow kubernetes-prow Bot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 7, 2026
@kubernetes-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rvagner78, sunnylovestiramisu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubernetes-prow kubernetes-prow Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 7, 2026
@kubernetes-prow

kubernetes-prow Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

@rvagner78: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-gcp-compute-persistent-disk-csi-driver-e2e-windows-2022 f5c8c3f link false /test pull-gcp-compute-persistent-disk-csi-driver-e2e-windows-2022
pull-gcp-compute-persistent-disk-csi-driver-e2e-windows-2019 f5c8c3f link false /test pull-gcp-compute-persistent-disk-csi-driver-e2e-windows-2019

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@sunnylovestiramisu

Copy link
Copy Markdown
Contributor

/retest-required

@kubernetes-prow
kubernetes-prow Bot merged commit 8675939 into kubernetes-sigs:master Aug 7, 2026
7 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants