Skip to content

Add simple options for enabling Gremlin GPU access #147

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ZachNo wants to merge 8 commits into
base: master
Choose a base branch
from
Open

Add simple options for enabling Gremlin GPU access #147

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
70bb987
Add easy GPU driver setup
ZachNo Jul 20, 2026
0281768
Add runtimeclass creation
ZachNo Jul 20, 2026
c5e802d
Fix ICD not mounting
ZachNo Jul 21, 2026
a5526c8
Simplify ICD loader config
ZachNo Jul 21, 2026
ed9b656
Support Nvidia CDI config, clean up docs
ZachNo Jul 21, 2026
cd043d5
Remove orphaned variable
ZachNo Jul 22, 2026
9048fe0
Add README changes
ZachNo Jul 22, 2026
7be99b5
bump chart version
ZachNo Jul 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
177 changes: 177 additions & 0 deletions gremlin/templates/_helpers.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -244,6 +244,183 @@ When createSecret or existingSecret are configured
{{- end -}}
{{- end -}}

{{/*
gremlinGpuVendorPreset returns the GPU configuration for a given vendor. The context passed in
is the vendor string (gremlin.gpu.vendor). Supported values are "nvidia" and "amd"; any other value
(including "") yields an empty preset so the chart falls back to whatever the advanced options specify.

nvidia - relies on the NVIDIA container toolkit: run under the "nvidia" RuntimeClass and set
NVIDIA_VISIBLE_DEVICES / NVIDIA_DRIVER_CAPABILITIES so the runtime injects the driver
libraries and device nodes (no hostMounts required). Also projects the nvidia OpenCL
ICD registry file, since some runtimes inject libnvidia-opencl.so.1 but not the
/etc/OpenCL/vendors/nvidia.icd file the loader needs to discover it.
amd - relies on the ROCm/amdgpu driver on the host: mount the /dev/kfd and /dev/dri device
nodes plus the OpenCL ICD registry so the container can reach the GPUs directly.

The optional `openclIcd` key (filename + library) tells the chart to project an OpenCL ICD
registry file; vendors that already expose /etc/OpenCL/vendors (amd) omit it.
*/}}
{{- define "gremlinGpuVendorPreset" -}}
{{- $vendor := . -}}
{{- if eq $vendor "nvidia" -}}
runtimeClassName: nvidia
env:
- name: NVIDIA_VISIBLE_DEVICES
value: "all"
- name: NVIDIA_DRIVER_CAPABILITIES
value: "all"
hostMounts: []
openclIcd:
filename: nvidia.icd
library: libnvidia-opencl.so.1
{{- else if eq $vendor "amd" -}}
runtimeClassName: ""
env: []
hostMounts:
- name: kfd
hostPath: /dev/kfd
type: CharDevice
readOnly: false
- name: dri
hostPath: /dev/dri
type: Directory
readOnly: false
- name: opencl-vendors
hostPath: /etc/OpenCL/vendors
mountPath: /etc/OpenCL/vendors
type: DirectoryOrCreate
readOnly: true
{{- else -}}
runtimeClassName: ""
env: []
hostMounts: []
{{- end -}}
{{- end -}}

{{/*
gremlinGpuEffective returns the vendor preset (see gremlinGpuVendorPreset) with each advanced
option overriding the preset when set.
*/}}
{{- define "gremlinGpuEffective" -}}
{{- $gpu := .Values.gremlin.gpu -}}
{{- $preset := fromYaml (include "gremlinGpuVendorPreset" (default "" $gpu.vendor)) -}}
{{- $runtimeClassName := $preset.runtimeClassName -}}
{{- if $gpu.runtimeClassName -}}{{- $runtimeClassName = $gpu.runtimeClassName -}}{{- end -}}
{{- $env := $preset.env -}}
{{- if $gpu.env -}}{{- $env = $gpu.env -}}{{- end -}}
{{- $hostMounts := $preset.hostMounts -}}
{{- if $gpu.hostMounts -}}{{- $hostMounts = $gpu.hostMounts -}}{{- end -}}
{{- $effective := dict "runtimeClassName" $runtimeClassName "env" $env "hostMounts" $hostMounts -}}
{{- if $preset.openclIcd -}}{{- $_ := set $effective "openclIcd" $preset.openclIcd -}}{{- end -}}
{{- $effective | toYaml -}}
{{- end -}}

{{/*
gremlinGpuRuntimeClassName returns the effective RuntimeClass name for the Gremlin pods, or
nothing when GPU access is disabled or no RuntimeClass applies.
*/}}
{{- define "gremlinGpuRuntimeClassName" -}}
{{- if .Values.gremlin.gpu.enabled -}}
{{- $eff := fromYaml (include "gremlinGpuEffective" .) -}}
{{- $eff.runtimeClassName -}}
{{- end -}}
{{- end -}}

{{/*
gremlinGpuEnv returns the environment variables that enable GPU/OpenCL access.
When gremlin.gpu.enabled is true, the effective env entries (vendor preset or the gremlin.gpu.env
override) are emitted. For the NVIDIA container toolkit these include NVIDIA_VISIBLE_DEVICES and
NVIDIA_DRIVER_CAPABILITIES (which must include `compute` or `all` for OpenCL).
*/}}
{{- define "gremlinGpuEnv" -}}
{{- if .Values.gremlin.gpu.enabled -}}
{{- $eff := fromYaml (include "gremlinGpuEffective" .) -}}
{{- with $eff.env -}}
{{- toYaml . -}}
{{- end -}}
{{- end -}}
{{- end -}}

{{/*
gremlinGpuCdiAnnotation returns the CDI device pod annotation when gremlin.gpu.cdiDevice is set,
and nothing otherwise. This targets the annotation-based CDI injection path supported by
containerd >= 1.7 / CRI-O, which does not consume a schedulable GPU resource.
*/}}
{{- define "gremlinGpuCdiAnnotation" -}}
{{- if and .Values.gremlin.gpu.enabled .Values.gremlin.gpu.cdiDevice -}}
cdi.k8s.io/gremlin-gpu: {{ .Values.gremlin.gpu.cdiDevice | quote }}
{{- end -}}
{{- end -}}

{{/*
gremlinGpuOpenclIcdActive returns "true" when the chart should project an OpenCL ICD registry
file into the container, and nothing otherwise.

The ICD is projected automatically whenever gremlin.gpu.projectOpenclIcd is true and the selected
vendor preset defines one (nvidia does), UNLESS an effective hostMount already provides
/etc/OpenCL/vendors (e.g. the amd preset or a custom mount), in which case we defer to that mount
to avoid overlapping mount paths.
*/}}
{{- define "gremlinGpuOpenclIcdActive" -}}
{{- if and .Values.gremlin.gpu.enabled .Values.gremlin.gpu.projectOpenclIcd -}}
{{- $eff := fromYaml (include "gremlinGpuEffective" .) -}}
{{- if and $eff.openclIcd $eff.openclIcd.library -}}
{{- $dirMounted := false -}}
{{- range $eff.hostMounts }}
{{- if eq (default .hostPath .mountPath) "/etc/OpenCL/vendors" }}{{- $dirMounted = true -}}{{- end }}
{{- end -}}
{{- if not $dirMounted -}}true{{- end -}}
{{- end -}}
{{- end -}}
{{- end -}}

{{/*
gremlinGpuVolumeMounts returns the volumeMounts that expose host GPU/OpenCL drivers.
Each effective hostMounts entry (vendor preset or the gremlin.gpu.hostMounts override) becomes a
volumeMount. mountPath defaults to hostPath and readOnly defaults to true when not specified.
When gremlinGpuOpenclIcdActive, the projected OpenCL ICD file is mounted as well.
*/}}
{{- define "gremlinGpuVolumeMounts" -}}
{{- if .Values.gremlin.gpu.enabled -}}
{{- $eff := fromYaml (include "gremlinGpuEffective" .) -}}
{{- range $eff.hostMounts }}
- name: {{ .name }}
mountPath: {{ default .hostPath .mountPath }}
readOnly: {{ if hasKey . "readOnly" }}{{ .readOnly }}{{ else }}true{{ end }}
{{- end -}}
{{- if include "gremlinGpuOpenclIcdActive" . }}
- name: gremlin-opencl-icd
mountPath: /etc/OpenCL/vendors/{{ $eff.openclIcd.filename }}
subPath: {{ $eff.openclIcd.filename }}
readOnly: true
{{- end -}}
{{- end -}}
{{- end -}}

{{/*
gremlinGpuVolumes returns the hostPath volumes that expose host GPU/OpenCL drivers.
Each effective hostMounts entry (vendor preset or the gremlin.gpu.hostMounts override) becomes a
hostPath volume. An optional `type` (e.g. DirectoryOrCreate, CharDevice) is passed through when present.
*/}}
{{- define "gremlinGpuVolumes" -}}
{{- if .Values.gremlin.gpu.enabled -}}
{{- $eff := fromYaml (include "gremlinGpuEffective" .) -}}
{{- range $eff.hostMounts }}
- name: {{ .name }}
hostPath:
path: {{ .hostPath }}
{{- if .type }}
type: {{ .type }}
{{- end }}
{{- end -}}
{{- if include "gremlinGpuOpenclIcdActive" . }}
- name: gremlin-opencl-icd
configMap:
name: {{ include "gremlin.fullname" . }}-opencl-icd
{{- end -}}
{{- end -}}
{{- end -}}

{{/*
chaoTlsIdentityArgs returns the chao cli arguments needed to configure TLS client identity
When remoteSecret is configured
Expand Down
17 changes: 16 additions & 1 deletion gremlin/templates/daemonset.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ spec:
{{- if .Values.gremlin.podLabels }}
{{- toYaml .Values.gremlin.podLabels | nindent 8 }}
{{- end }}
{{- if or .Values.gremlin.apparmor .Values.gremlin.installApparmorProfile .Values.gremlin.podSecurity.seccomp.enabled .Values.gremlin.podSecurity.securityContextConstraints.create .Values.gremlin.podAnnotations }}
{{- if or .Values.gremlin.apparmor .Values.gremlin.installApparmorProfile .Values.gremlin.podSecurity.seccomp.enabled .Values.gremlin.podSecurity.securityContextConstraints.create .Values.gremlin.podAnnotations (include "gremlinGpuCdiAnnotation" .) }}
annotations:
{{- if .Values.gremlin.apparmor }}
container.apparmor.security.beta.kubernetes.io/{{ .Chart.Name }}: {{ .Values.gremlin.apparmor }}
Expand All @@ -44,12 +44,18 @@ spec:
{{- if .Values.gremlin.podSecurity.securityContextConstraints.create }}
openshift.io/required-scc: "gremlin"
{{- end }}
{{- if include "gremlinGpuCdiAnnotation" . }}
{{- include "gremlinGpuCdiAnnotation" . | nindent 8 }}
{{- end }}
{{- if .Values.gremlin.podAnnotations }}
{{- toYaml .Values.gremlin.podAnnotations | nindent 8 }}
{{- end }}
{{- end }}
spec:
serviceAccountName: gremlin
{{- if include "gremlinGpuRuntimeClassName" . }}
runtimeClassName: {{ include "gremlinGpuRuntimeClassName" . }}
{{- end }}
{{- if .Values.affinity }}
affinity: {{ toYaml .Values.affinity | trimSuffix "\n" | nindent 8 }}
{{- end }}
Expand Down Expand Up @@ -163,6 +169,9 @@ spec:
{{- if include "gremlinTlsIdentityEnv" . }}
{{- include "gremlinTlsIdentityEnv" . | nindent 10 }}
{{- end }}
{{- if include "gremlinGpuEnv" . }}
{{- include "gremlinGpuEnv" . | nindent 10 }}
{{- end }}
{{- with .Values.gremlin.extraEnv }}
{{- toYaml . | nindent 10 }}
{{- end }}
Expand Down Expand Up @@ -195,6 +204,9 @@ spec:
{{- if include "gremlinTlsIdentityVolumeMounts" . }}
{{- include "gremlinTlsIdentityVolumeMounts" . | nindent 10 }}
{{- end }}
{{- if include "gremlinGpuVolumeMounts" . }}
{{- include "gremlinGpuVolumeMounts" . | nindent 10 }}
{{- end }}
volumes:
- name: cgroup-root
hostPath:
Expand Down Expand Up @@ -235,6 +247,9 @@ spec:
{{- if include "gremlinTlsIdentityVolumes" . }}
{{- include "gremlinTlsIdentityVolumes" . | nindent 8 }}
{{- end }}
{{- if include "gremlinGpuVolumes" . }}
{{- include "gremlinGpuVolumes" . | nindent 8 }}
{{- end }}
{{- if .Values.gremlin.priorityClassName }}
priorityClassName: {{ .Values.gremlin.priorityClassName }}
{{- end }}
26 changes: 26 additions & 0 deletions gremlin/templates/opencl-icd-configmap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{{- /*
Projects an OpenCL ICD registry file into the Gremlin container.

Some NVIDIA container runtimes inject the OpenCL driver library (libnvidia-opencl.so.1) but do
NOT create the /etc/OpenCL/vendors/<vendor>.icd registry file that the ICD loader reads to
discover the driver. Without it, the OpenCL platforms cannot be listed.

The ICD filename and library come from the selected vendor preset (see gremlinGpuVendorPreset).
See gremlinGpuOpenclIcdActive for when this is rendered.
*/ -}}
{{- if include "gremlinGpuOpenclIcdActive" . }}
{{- $eff := fromYaml (include "gremlinGpuEffective" .) }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "gremlin.fullname" . }}-opencl-icd
namespace: {{ .Release.Namespace }}
labels:
helm.sh/chart: {{ include "gremlin.chart" . }}
app.kubernetes.io/name: {{ include "gremlin.fullname" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
data:
{{ $eff.openclIcd.filename }}: |
{{ $eff.openclIcd.library }}
{{- end }}
24 changes: 24 additions & 0 deletions gremlin/templates/runtimeclass.yaml
View file
Open in desktop

@philgebhardt philgebhardt Jul 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need unit tests for this document?

Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
{{- /*
Creates the RuntimeClass referenced by the GPU configuration when it does not already exist.

NOTE: `lookup` only sees the cluster during `helm install`/`upgrade`. Under `helm template` or a
GitOps renderer with no cluster access, `lookup` returns empty and the RuntimeClass is rendered.
Set gremlin.gpu.createRuntimeClass to false if you manage the RuntimeClass yourself.
*/ -}}
{{- if .Values.gremlin.gpu.enabled -}}
{{- $runtimeClassName := include "gremlinGpuRuntimeClassName" . -}}
{{- if and $runtimeClassName .Values.gremlin.gpu.createRuntimeClass -}}
{{- if not (lookup "node.k8s.io/v1" "RuntimeClass" "" $runtimeClassName) }}

@philgebhardt philgebhardt Jul 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, I think the use of lookup here is going to present an issue on subsequent upgrades, as the next time an upgrade lands (after the first), lookup will find something and omit this document. The net effect is behavior like helm/helm#10281, where the resource gets deleted after the first upgrade/install.

I think the problem of preventing collisions with an existing runtimeClass is solved by gremlin.gpu.createRuntimeClass. That is, if a user already has a runtime class, they should not be setting gremlin.gpu.createRuntimeClass=true.

If we want a better error message for someone that has the class already, we can use lookup to exit the chart installation with an error, e.g. "a RuntimeClass named X already exists; set gremlin.gpu.createRuntimeClass=false if you intend to reuse it"

@ZachNo ZachNo Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the normal use case will be that a customer already has the runtimeclass existing for their normal application, but unfortunately not having the runtime makes the gremlin pod unschedulable, so it's a hard call honestly.

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: {{ $runtimeClassName }}
labels:
app.kubernetes.io/name: {{ include "gremlin.name" . }}
helm.sh/chart: {{ include "gremlin.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
handler: {{ $runtimeClassName | quote }}
{{- end }}
{{- end }}
{{- end }}
Loading
Loading