Dynamically generate namespace and rbac from user attributes

[gberche-orange]

Thanks for maintaining this great project over the years. Here is an idea in the context of a k8s-centric Internal Develop Platform

In the pinniped tutorial at https://pinniped.dev/docs/tutorials/concierge-and-supervisor-demo/#configure-rbac-rules-for-the-developer-and-devops-users, the namespace and rbac granted to a user are statically created by an operator

For this tutorial, we will keep the Kubernetes RBAC configuration simple.

# Create a namespace in the first workload cluster.
kubectl create namespace "dev" \
  --kubeconfig workload1-admin.yaml

# Allow the developer to edit everything in the new namespace.
cat <<EOF | kubectl create --kubeconfig workload1-admin.yaml -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: developer-can-edit-dev-ns
  namespace: dev
subjects:
- kind: User
  name: walrus@example.com
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: edit
  apiGroup: rbac.authorization.k8s.io
EOF

# In the second workload cluster, allow the developer
# to view everything in all namespaces.
kubectl create clusterrolebinding developer-can-view \
  --clusterrole view \
  --user walrus@example.com \
  --kubeconfig workload2-admin.yaml

I'm exploring a scenario where namespaces and rbac are automatically created/mirrored from the upstream identity sources, using templating with upstream identity providers attributes as inputs (username, groups, email ...), with some vague similarities to pinniped transformation expressions https://github.com/vmware-tanzu/pinniped/blob/876f626e7d9c1cd655bed2fbe1c6a0a6498d399a/site/content/docs/howto/supervisor/configure-supervisor-federationdomain-idps.md?plain=1#L177-L184

Let's take the example of a user who has an identity on a gitlab IDP, is member of multiple groups, this templating system would map these groups to a set of namespaces and rbac resources, so that the user can then subscribe to cloud resources through K8S CRs within these namespaces.

Can you think of a way pinniped could help in this scenario ? Are you instead aware of other community projects that could be setup to fulfill this goal ?

[cfryanr]

Hi @gberche-orange, Pinniped will help your users auth and will help you extract their usernames and group names from your IDP (potentially transformed by CEL expressions too) and get those usernames and group names into your Kubernetes clusters. However, it does not offer any features to help you manage your RBAC policies for those usernames and group names. There are probably other projects to help you manage RBAC policies, but I don't personally have any specific recommendations for that.

[gberche-orange]

thanks @cfryanr for your response.

My team is considering using crossplane to observe user name, groups in the K8S api, and as a result dynamically create namespace and RBAC objects from observed data.

Pinniped will help your users auth and will help you extract their usernames and group names from your IDP (potentially transformed by CEL expressions too) and get those usernames and group names into your Kubernetes clusters.

Can usernames, group names be observed through the K8S api once a user has logged in thanks to pinniped ?

Looking at the API reference, I have so far identified that the WhoAmIRequest would include in its status the KubernetesUserInfo

However, I'm not yet clear on how to issue a WhoAmIRequest to the API server, as the associated tests seem to rather currently illustrate a user-generated REST api call.

https://github.com/vmware-tanzu/pinniped/blob/876f626e7d9c1cd655bed2fbe1c6a0a6498d399a/internal/registry/whoamirequest/rest_test.go#L136-L165

Is the pinniped cli automatically submitting this WhoAmIRequest API call upon user logging, and is the WhoAmIRequest observable as a cluster wide resource in the API server ?

Also, I'd hope that through CEL transformations I'd be able to propagate extra elements from the OIDC id token exchanged with the OIDC provider (say gitlab), into the WhoAmIRequest status extra fields. For instance, the id token issued by Gitlab, provides rich additional details that would be leveraged when generating the namespace and rbac objects.

[cfryanr]

Can usernames, group names be observed through the K8S api once a user has logged in thanks to pinniped ?

No, unfortunately k8s does not have an API for that, as far as I know. The usernames and group names are transient and are not stored anywhere. Usernames will show up in the Kubernetes API server's audit logs for all API calls, if you have those enabled.

However, I'm not yet clear on how to issue a WhoAmIRequest to the API server, as the associated tests seem to rather currently illustrate a user-generated REST api call.

End users can choose to create a WhoAmIRequest by using the Pinniped CLI or any other client. It is not part of the login flow, so it does not happen automatically. That API is equivalent to the new kubectl auth whoami feature that is now built into kubernetes. It only echos back the information about the currently logged in user.

Also, I'd hope that through CEL transformations I'd be able to propagate extra elements from the OIDC id token exchanged with the OIDC provider (say gitlab), into the WhoAmIRequest status extra fields.

That's not currently a feature of the CEL expressions. You can configure additional fields from the external identity provider's ID token that should get added to the Pinniped Supervisor's ID token, but those will not appear on the WhoAmIRequest's response.

[gberche-orange]

Thanks again @cfryanr for your detailed and prompt answers !

End users can choose to create a WhoAmIRequest by using the Pinniped CLI or any other client. It is not part of the login flow, so it does not happen automatically. That API is equivalent to the new kubectl auth whoami feature that is now built into kubernetes. It only echos back the information about the currently logged in user.

Thanks, for the records, history of whoami feature in kubernetes/enhancements#3326 and KEP-3325 README.md.

Since with pinniped WhoAmIRequest or k8s selfsubjectreviews endpoints are unrelated to common k8s resources, I wondered how they would be observed from the k8s api client.

I understand to pinniped WhoAmIRequest or k8s selfsubjectreviews endpoints can not be observed from k8s admission web hooks (such as kyverno) because admission webhooks seem to only be applied on resources and not on "virtual api resource types", see https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-rules

A heavy approach with control over the k8s api configuration could be to register an authorization webhook whose implementation record the SubjectAccessReview payload it receives, and record the user names and groups as resources in the k8s api. See https://kubernetes.io/docs/reference/access-authn-authz/authorization/#authz-config-example To limit the performance impact, this would require to limit this authorization webhook scope to a small number of path/resources. I'll next search for k8s authorization frameworks to see if one has implemented a similar pattern surfaced as an authorization webhook/.

No, unfortunately k8s does not have an API for that, as far as I know. The usernames and group names are transient and are not stored anywhere. Usernames will show up in the Kubernetes API server's audit logs for all API calls, if you have those enabled

I have also investigated use of K8S component metrics as an alternative way to observe user logins

The usernames from authenticated user requests are indeed observable according to https://kubernetes.io/docs/reference/instrumentation/metrics/

authenticated_user_requests
Counter of authenticated requests broken out by username.
Stability Level:ALPHA
Type: Counter
Labels:username

Which is recorded in K8s metrics filter at https://github.com/kubernetes/kubernetes/blob/fa03b93d25a5a22d4f91e4c44f66fc69a6f69a35/staging/src/k8s.io/apiserver/pkg/endpoints/filters/metrics.go#L124-L139

Unfortunately for supporting my use-case, the metric only include the username label and is missing groups.

I have yet to explore other K8s extension mechanisms (e.g custom request filters) for opportunities to intercept user names and groups.

[gberche-orange]

Another use-case I am exploring, is the ability during user enrollment to pinniped, to request for additional scope/claims beyond openid, and upon user consent, to leverage the return oauth access tokens to act on the user behalf to observe additional data beyond what is available in the id token.

I'm annotating the documented diagram at https://github.com/vmware-tanzu/pinniped/blob/8b7fec5049f79e790e27e009994e1069d93eba11/site/content/docs/img/pinniped-concierge-supervisor-sequence.svg to illustrate the flows that I would like to act upon, typically in the context of a gitlab IDP with Authorization code flow

• The authorize call would request additional scopes beyond openid
• the returned access token and refresh token would be observable within the k8s API as resource (likely a secret)
• ideally pinniped would periodically refresh the access token and update the k8s resource/secret

Our crossplane-based controller would then observe the secret produced by pinniped to request the gitlab api on behalf of the user, in accordance with the granted scope given during consent.

[cfryanr]

There is a feature where you can configure (on the OIDCIdentityProvider) the scopes to be requested during the authorization flow with the external identity provider. However, the tokens returned to the external identity provider to the Pinniped Supervisor are private to the Supervisor and are not meant to be used for other purposes, consistent with the design of OIDC itself. The Supervisor does store them in Secrets, although that is intended to be an internal implementation detail rather than an API. The Supervisor will only refresh those on-demand when needed as a result of the interactions of that end user, so if that end user is idle then no refreshes are happening for that user.

[gberche-orange]

Thanks again @cfryanr for your support. I'm starting a sample pinniped setup with an OIDC backend (cloudfoundry UAA). I'm properly observing the pinniped supervisor secrets

$ k get secrets -n  00-pinniped-supervisor | grep pinniped-storage
pinniped-storage-access-token-73impf7jv2mop37fjmrkaaffk5jrxl7qvbnoutl5fmii2huby2rq          storage.pinniped.dev/access-token                             2      72m
...
pinniped-storage-authcode-3jtzn2e7nhmlvbesax2qle3j73wsysvcb2vii5agy454w5dnodyq              storage.pinniped.dev/authcode                                 2      104m
...
pinniped-storage-refresh-token-2rzlynia3cev5i4paxnlnwgfyyvt3zqr6qagsuurquqxcc4f7b3a         storage.pinniped.dev/refresh-token                            2      103m

However, it it not yet clear to me why upstream access token is empty

k get secrets -n  00-pinniped-supervisor -l storage.pinniped.dev/type=access-token -o json | jq -r '.items[].data."pinniped-storage-data"'| base64 -d | jq | less
#>          "upstreamRefreshToken": "eyJqa...",
#>          "upstreamAccessToken": "",

# Same with refresh token secrets
k get secrets -n  00-pinniped-supervisor -l storage.pinniped.dev/type=refresh-token -o json | jq -r '.items[].data."pinniped-storage-data"'| base64 -d | jq | less

The upstreamRefreshToken as expected does not contain id token claims to infer user identity

k get secrets -n  00-pinniped-supervisor -l storage.pinniped.dev/type=authcode -o json | jq -r '.items[].data."pinniped-storage-data"'| base64 -d | jq -r .request.session.custom.oidc.upstreamRefreshToken | head -n 1 | cut -d'.' -f2 | base64 -d
#> {"sub":"81858649-4976-4b3d-8f84-9eaba79e3d7e","user_name":"gberche","amr":["ext","pwd"],"origin":"ldap","iss":"https://uaa.int-ops-...","client_id":"pinniped-client","aud":["openid","bosh","pinniped-client"],"zid":"uaa","grant_type":"authorization_code","user_id":"81858649-...","auth_time":1741796717,"granted_scopes":["openid","bosh.admin"],"exp":1741800402,"iat":1741796802,"jti":"a3671095fa8f495f9456fcd6a693aa32-r","rev_sig":"9aa7984f","cid":"pinniped-client"}base64: invalid input

I yet have to reproduce with other upstream OIDC provider (namely gitlab) to confirm behavior is the same, or whether this is specific to UAA.

For your experience, should the pinniped-storage secret contain the upstream access token ?

[cfryanr]

The Secret of type storage.pinniped.dev/type=refresh-token is used during token refreshes of end users.

The Pinniped Supervisor will only save into that Secret what it thinks it needs to support the refresh for that end user, not everything that was returned by the upstream OIDC provider. In this case, see this switch statement where it prefers to save the refresh token from the upstream provider over the access token from the upstream provider.

During the next refresh by this end user, it will use the upstream refresh token to perform a refresh grant with the upstream provider.