Add universe domain support for sovereign cloud environments - #2373
Conversation
Adds custom universe domain support in order to allow operation in environments with non-default universe domains, such as Google Cloud Dedicated, GCP's sovereign cloud offering. The universe domain can be determined from the credential, and replaces the standard googleapis.com resource path.
|
Welcome @rvagner78! |
|
Hi @rvagner78. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/ok-to-test |
|
/test all |
|
Cross referencing the same problem in the cloud provider kubernetes/cloud-provider-gcp#1227 |
kubernetes/cloud-provider-gcp#1250 is the accepted solution |
|
We have tested the code in this PR downstream, in OpenShift, in a GCD (Berlin) environment. You can see the results here: Everything looks good. The 5 failing e2e tests are expected and will be skipped. |
|
Translate this to a term we know better: this is for support of Trusted Partner Cloud, but our driver should have already supported TPC. Is this some bugs you see? |
@sunnylovestiramisu Thanks for taking a look! Yes this proposes to add support for TPC (or Google Cloud Dedicated) by enabling support for alternate Universe Domains. I do not believe there is support in the driver for alternate Universe Domains/TPC for two primary reasons: First, the driver constructs API endpoints, and those are hard-coded to In a TPC-environment with an alternate universe domain, that endpoint will not be accessible and calls to those endpoints will fail. Second, the GCP SDK requires that callers opt-in and explicitly set the universe domain so that it matches the credentials. In a TPC environment, the credentials will include an alternate universe domain and therefore the client calls must also set that universe domain. Technically that universe domain could be set with the |
|
Can you try to configure the --compute-endpoint to your customized endpoint and see if the driver is working in TPC? But I am not sure about the cloudresourcemanager.googleapis.com one hmm. I checked that we did tested the TPC in TPC Staging internally at Google and it is working. I wonder if we missed something. |
Thank you, I wasn't aware of this flag. We can try testing it, but I am certain the issue that we will encounter is that the driver uses As the comment indicates, it is necessary to use self-signed JWTs instead of oauth in these environments. When testing with other components, such as cloud-provider-gcp we get an error message like:
|
|
@sunnylovestiramisu I tried setting the endpoint, and confirmed as expected that authentication can not work: |
|
Oh I just realized that you are using tagging. Our internal TPC tests do not use that that is why it succeeds. If you remove ResourceTags usage, it should succeed. In a sovereign/universe environment where compute and resource-manager hostnames differ from googleapis.com (or are region-prefixed differently), tagging calls will try to use compute.googleapis.com and cloudresourcemanager.googleapis.com and will fail even if compute calls succeed via --compute-endpoint. This is a bug in TPC for sure. Have you tested this PR in your Google Cloud Dedicated environment? |
Interesting! I will try this out. Thank you, this comment is extremely helpful.
Yes, this code is included in the test results which you can see here. Everything worked except a few expected test failures (due to how our prow instance proxies to connect to the cluster). |
|
@sunnylovestiramisu So I tried again, making sure to unset tags, but we get the same result. The result makes intuitive sense because the driver uses We can see from the failure in my testing ☝️ that the error is coming from I do wonder if the staging sandbox you used for testing had a working oauth token endpoint, which would not be available in a real gcd environment. |
|
Hmm sending this to the TPC team, it may mean that the test environment setup has flaws. |
|
@patrickdillon Can you also share the error you see before setting the compute-endpoint flag? Talked with the TPC team, and they agreed anything that would update the code to be universe aware is good (and usually harmless). So this PR is not blocked by anything. It is odd that we see differences in internal PRP tests compared to TSP(Germany). |
|
@sunnylovestiramisu No problem. The error is the same: This test is from the most recent nightly of openshift 5.0.
The difference must be that the internal environment has a valid oauth token endpoint... the actual environment does not have an oauth token endpoint. |
|
I reviewed the PR. The PR only makes endpoint/credential handling universe-domain-aware and preserves current behavior for the default universe (googleapis.com), so existing clusters should not be affected. It is good for merging. Question to you:
|
|
For us it's enough if the code is merged into master and it's in the next driver release. Or at least the behavior is the same. We can cherry-pick it to our driver fork by ourselves and backport as far as we need. |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: rvagner78, sunnylovestiramisu The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@rvagner78: The following tests failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/retest-required |
8675939
into
kubernetes-sigs:master
What type of PR is this?
What this PR does / why we need it:
The GCE PD CSI driver currently hardcodes
googleapis.comin all API endpoint construction — compute resource paths, Cloud Resource Manager tag endpoints, and credential handling. This prevents the driver from operating in Google Cloud environments that use a non-default universe domain, such as Google Cloud Dedicated (GCD), GCP's sovereign cloud offering.This PR is a proposal how to make the driver universe-domain-aware:
google.DefaultTokenSourcewithgoogle.FindDefaultCredentials, which exposes the universe domain from the credential//compute.googleapis.com/...format strings with acomputeParentPath()helper that interpolates the correct universe domaincloudresourcemanager.googleapis.comconstant with aresourceManagerHost()helper for tag value and tag binding clientsauthOpts()to centralize client option construction — usesWithAuthCredentialsJSONwhen credential JSON is available, falls back toWithHTTPClientfor alternate token sourcesOn standard GCP the behavior is unchanged — when the credential returns the default universe domain (
googleapis.com) or empty string, all endpoints resolve identically. No additional configuration is needed from the cluster operator; the universe domain is determined from the credential itself.Which issue(s) this PR fixes:
None
Special notes for your reviewer:
The universe domain is a DNS suffix that organizes Google Cloud API endpoints. Standard GCP uses
googleapis.com. Sovereign cloud deployments (e.g. Google Cloud Dedicated) use a different universe domain so that all API traffic stays within the sovereign boundary. See https://cloud.google.com/dedicated/docs/overview for details.Does this PR introduce a user-facing change?: