[core][kuberay] Support configurable Kubernetes API authentication in the autoscaler

[ankushbbbr]

Description

The KubeRay autoscaler's Kubernetes API client hardcodes in-cluster ServiceAccount authentication: load_k8s_secrets() reads a bearer token from /var/run/secrets/kubernetes.io/serviceaccount/token and verifies the API server with /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, and KubernetesHttpApiClient.get() / .patch() pass no cert= to requests.

Many organizations run customized Kubernetes distributions where API authentication differs from the out-of-the-box in-cluster mechanism — for example, requests must go through an authenticating proxy that expects a client TLS certificate rather than a ServiceAccount bearer token. Today the autoscaler cannot run in those environments, and it cannot be worked around by configuration: requests has no environment variable for client certificates, and REQUESTS_CA_BUNDLE only affects verify, which is inert here because Ray passes verify= explicitly.

This PR makes the authentication mechanism configurable via three optional environment variables, named after the existing KUBERAY_REQUEST_TIMEOUT_S / KUBERAY_CRD_VER convention in the same module:

Variable	Effect
RAY_KUBERAY_CLIENT_CERT_PATH	Client certificate passed to requests as part of cert=
RAY_KUBERAY_CLIENT_KEY_PATH	Client key passed to requests as part of cert=
RAY_KUBERAY_CA_CERT_PATH	CA certificate used for verify=, instead of the ServiceAccount CA

When none of these are set, behavior is unchanged. The client sends the same Authorization: Bearer <token> header, the same verify path, and cert=None — which is requests' own default for cert, so the outgoing request is identical to today's. test_client_get_defaults_unchanged and test_client_patch_defaults_unchanged assert the full kwargs dict passed to requests.get/requests.patch to lock this in.

Other properties, all deliberate:

• The ServiceAccount token becomes optional only when a client certificate is configured, since such environments often do not project one. In the default configuration a missing token file still raises exactly as before, so existing diagnostics don't change (test_load_k8s_secrets_token_required_without_client_cert).
• Setting only one of cert/key raises a ValueError naming both variables, rather than silently ignoring the configuration.
• url_from_resource is untouched. The API server host is already overridable through KUBERNETES_SERVICE_HOST / KUBERNETES_SERVICE_PORT_HTTPS, and it already accepts a host carrying the https:// scheme verbatim, so only authentication needed to change. That keeps this diff small.
• IKubernetesHttpApiClient is unchanged, and all existing construction sites keep working unchanged: node_provider.py, autoscaling_config.py, and autoscaler/v2/instance_manager/cloud_providers/kuberay/cloud_provider.py.
• The only signature changes are to two private symbols in a _private module: load_k8s_secrets() now returns a third element (cert), and _get_refreshed_headers_and_verify is renamed to _get_refreshed_credentials to match. Neither is referenced outside node_provider.py.

An alternative design would be to make the client fully pluggable, e.g. a config field or entry point naming a custom IKubernetesHttpApiClient implementation. That is more general but a much larger surface to support, and it isn't needed for this case. Happy to change direction if maintainers prefer that; discussed in the linked issue.

I'd like this to land in Ray 2.59.

Related issues

Closes #65826

Additional information

Example usage

# In the autoscaler container spec of a RayCluster
env:
  - name: KUBERNETES_SERVICE_HOST
    value: "https://k8s-api-proxy.example.internal"
  - name: RAY_KUBERAY_CLIENT_CERT_PATH
    value: /etc/ray-k8s-auth/tls.crt
  - name: RAY_KUBERAY_CLIENT_KEY_PATH
    value: /etc/ray-k8s-auth/tls.key
  - name: RAY_KUBERAY_CA_CERT_PATH
    value: /etc/ray-k8s-auth/ca.crt

Tests run

Ten new test functions (13 cases with parametrization) in python/ray/tests/kuberay/test_kuberay_node_provider.py, covering: unchanged defaults asserted at the requests kwargs level for both get and patch; cert= pass-through; CA override with and without a client cert; optional token with a client cert; token still required without one; cert/key required together; and the credential caching/refresh contract.

$ python -m pytest python/ray/tests/kuberay/test_kuberay_node_provider.py -q
34 passed in 0.49s

$ python -m pytest python/ray/tests/kuberay/test_autoscaling_config.py -q
55 passed in 0.26s

$ python -m pytest python/ray/autoscaler/v2/tests/test_node_provider.py -q -k KubeRayProviderIntegrationTest
25 passed, 7 deselected in 0.92s

$ python -m pytest python/ray/autoscaler/v2/tests/test_ippr_provider.py -q
31 passed in 0.23s

The last two cover the other KubernetesHttpApiClient / IKubernetesHttpApiClient consumers, to confirm nothing downstream broke.

Not tested locally: the client-certificate path against a real API server behind an authenticating proxy. The unit tests assert what is handed to requests; the requests side of cert= is library behavior.

Lint

Clean on both changed files: black --check, ci/lint/check-docstyle.sh, ci/lint/check_import_order.py, pyflakes, pycodestyle --max-line-length=88, and isort --check-only.

I could not run the full pre-commit suite in my environment: the pinned ruff==0.8.4 and pydoclint==0.8.3 were unavailable there, and the ruff binary I did install would not execute at all (killed on start). So ruff and pydoclint were not actually run locally — pyflakes / pycodestyle / isort above are stand-ins for the ruff and ruff --select I hooks, and nothing substitutes for pydoclint. Please treat CI as authoritative for those two hooks; happy to fix whatever they flag.

DCO

The commit is signed off (git commit -s).

Duplicate-work check

Per AGENTS.md, I checked for existing work before opening this:

gh pr list --repo ray-project/ray --state open --search "KubernetesHttpApiClient"
gh pr list --repo ray-project/ray --state open --search "kuberay autoscaler auth"
gh issue list --repo ray-project/ray --state all --search "load_k8s_secrets"
gh issue list --repo ray-project/ray --state all --search "kuberay autoscaler authentication"
gh issue list --repo ray-project/ray --state all --search "client certificate kubernetes api autoscaler"
gh issue list --repo ray-project/ray --state all --search "mTLS kuberay"

No open PR or issue covers configurable Kubernetes API authentication for the autoscaler. The nearest hits are unrelated (in-process TLS credential reloading for RayCluster components, RBAC questions).

AI assistance

AI assistance was used to write this change and its tests. I have reviewed every changed line, run the tests above locally, and can defend the design and the backwards-compatibility argument.

[gemini-code-assist]

Code Review

This pull request introduces support for optional client TLS credentials (client certificate, client key, and CA certificate) in the KubeRay node provider, allowing authentication with the Kubernetes API server via a client certificate instead of just the in-cluster ServiceAccount token. The changes update load_k8s_secrets and KubernetesHttpApiClient to handle these new credentials and include comprehensive unit tests. The review feedback suggests stripping trailing whitespace from the ServiceAccount token to avoid malformed headers and simplifying the _get_refreshed_credentials method by removing a redundant else block.

[gemini-code-assist]

Kubernetes ServiceAccount tokens read from disk can sometimes contain trailing newlines or whitespace, which can lead to malformed HTTP headers or authentication failures with strict API servers or proxies. It is safer to strip any leading/trailing whitespace from the token before adding it to the headers.

Suggested change

	if cert is None or os.path.exists(IN_CLUSTER_TOKEN_PATH):
	with open(IN_CLUSTER_TOKEN_PATH) as secret:
	token = secret.read()
	headers["Authorization"] = "Bearer " + token
	if cert is None or os.path.exists(IN_CLUSTER_TOKEN_PATH):
	with open(IN_CLUSTER_TOKEN_PATH) as secret:
	token = secret.read().strip()
	headers["Authorization"] = "Bearer " + token

[ankushbbbr]

I'd like to leave it out of this PR, because it is the one change here that would alter the default code path.

The core property this PR is trying to preserve is that with none of the new environment variables set, the outgoing request is identical to today's. secret.read() without .strip() is exactly what load_k8s_secrets() does on master, and test_client_get_defaults_unchanged / test_client_patch_defaults_unchanged assert the complete kwargs dict handed to requests to lock that down.

fwiw, I don't think the failure mode is reachable in practice today: Kubernetes projects the ServiceAccount token without a trailing newline.