Long-running containers: heap memory gradually increases and may lead to OOM

[ProGengi]

Description

In long-running containers, process-exporter's memory usage gradually increases over time, eventually causing OOM, even when the tracked process list is fixed.

We observed the following pattern in container memory metrics:

• Monitoring shows container_memory_working_set_bytes changing in stages, as illustrated in the figure below.

• Using pprof, heap memory is observed to continuously grow
• RSS and working set eventually reach the container limit (256MB), triggering OOM

Environment

• process-exporter version: 0.8.7
• Container runtime: K8s
• Memory limit: 256MB
• Command: /bin/process-exporter -config.path /etc/process-exporter.yml -procfs /host/proc -threads=false
• process-exporter.yml:
  process_names:
  - exe:
  - /sbin/auditd
  - /usr/bin/buildkitd
  - /opt/ceph_service/ceph-agent
  - /usr/local/bin/containerd
  - /usr/sbin/crond
  - /usr/bin/dbus-daemon
  - /sbin/agetty
  - /usr/sbin/gssproxy
  - /usr/sbin/irq_balancer
  - /usr/bin/kubelet
  - /usr/sbin/NetworkManager
  - /usr/sbin/ntpd
  - /usr/lib/polkit-1/polkitd
  - /sbin/rngd
  - /usr/bin/rpcbind
  - /usr/sbin/rsyslogd
  - /usr/sbin/sshd
  - /usr/lib/systemd/systemd-journald
  - /usr/lib/systemd/systemd-logind
  - /usr/lib/systemd/systemd-udevd
  - /usr/sbin/tuned