Dynamically add and remove secret from the trustee pod

[iroykaufman]

Currently, the flow for creating a LUKS key is:

1. When the operator detects new machine creation, it creates a new LUKS key.
2. The operator creates a Kubernetes secret to hold the LUKS key.
3. Patch the trustee deployment to add the secret as a confidential resource.

Step 3 will trigger the trustee deployment to restart with the new secret mounted. We want to avoid this. Other objects (policies, reference-values, etc) can be handled similarly.

I start implementing two different approaches to solving this issue:

1. Create a sidecar inside the trustee deployment that listens for new machine creation and manages the LUKS key as a Kubernetes secret. A draft implementation can be found here. Based on my tests, this implementation is functional.
2. Use the kbs API (kbs-client) from trustee to update the LUKS key and other trustee configuration. While the first two steps of the flow remain the same, the third step is replaced by a direct API call to the KBS. A draft implementation is available here. This version is currently incomplete and has not been fully tested.

I would like to get your feedback on what is a better approach

[alicefr]

1. Create a sidecar inside the trustee deployment that listens for new machine creation and manages the LUKS key as a Kubernetes secret. A draft implementation can be found here. Based on my tests, this implementation is functional.

I would simply watch for the secret in the namespace and act accordingly. We can also add labels to the machine secret in order to filter the secret and limit the reconciliation loop to that subset of secret.

3. Use the kbs API (kbs-client) from trustee to update the LUKS key and other trustee configuration. While the first two steps of the flow remain the same, the third step is replaced by a direct API call to the KBS. A draft implementation is available here. This version is currently incomplete and has not been fully tested.>

IIRC, the secrets managment API is experimental in trustee. Hence, please double check this with the trustee community.

Additionally, you also need to add each time you restart the pod the associated secrets that were already present.

I'm fine in using the KBS API if it is a stable API and the suggested way.