Skip to content

libct: do not hang on a half-dead "runc init" - #5431

Open
kolyshkin wants to merge 2 commits into
opencontainers:mainfrom
kolyshkin:fix-5087-init-hang
Open

libct: do not hang on a half-dead "runc init"#5431
kolyshkin wants to merge 2 commits into
opencontainers:mainfrom
kolyshkin:fix-5087-init-hang

Conversation

@kolyshkin

Copy link
Copy Markdown
Contributor

Fixes #5087.

The problem

SCMP_ACT_KILL is SECCOMP_RET_KILL_THREAD: the kernel does do_exit(SIGSYS) on the thread that made the syscall, and on that thread only. runc init locks its main goroutine to the initial OS thread, which is the thread group leader, so a syscall blocked this way kills the leader while the rest of the Go runtime threads live on.

What is left is a process that looks dead and is not:

  • the leader is a zombie, but release_task() for it is delayed until the last thread exits, so /proc/$pid/stat says Z;

  • exit_files() drops only the leader's reference to the shared files_struct, so the child ends of the sync socket and of the exec fifo stay open — the parent never sees an EOF;

  • exit_notify() defers do_notify_parent() until the thread group is empty, so there is no SIGCHLD, wait(2) does not return, and a pidfd never becomes readable — I checked the latter explicitly, with and without PIDFD_THREAD:

    leader state=Z
    flags=0x0   n=0 revents=0x0
    flags=0x80  n=0 revents=0x0   /* PIDFD_THREAD */
    

The parent therefore waits forever. Which wait it gets stuck in depends on where init dies; with write blocked and no --debug, init dies writing to the exec fifo (seccomp is applied just before that), and runc run hangs in waitForFifoReady. The reporter hit the same thing in ReadPacket on the sync socket.

This also means the pidfd fast path added for the fifo wait cannot detect this case at all — /proc/$pid/stat is the only signal that reports a zombie thread group leader.

The fix

Add waitForInitWrite(fd, pid): poll(2) with a 100 ms timeout, and on timeout check whether init went away via /proc. Use it for both places where the parent waits on init — reading the sync socket and waiting for the exec fifo — replacing the three fifo-specific wait helpers.

Two more things were needed to make the failure actually terminate:

  • once a dead init is detected, kill its leftover threads. They keep the container's stdio descriptors open, and whoever inherited those (runc run's own IO copying goroutines, or bats) would then wait for an EOF just as long.
  • drop the p.wait() in the setnsProcess.start error path — a half-dead child is never reaped, so that wait4(2) never returns. The deferred error handler terminates and reaps it anyway, and SIGKILL is group-wide.

Testing

Before, runc run with {"names": ["write"], "action": "SCMP_ACT_KILL"} hung indefinitely, leaving a zombie and a cgroup that could not be removed. After:

$ runc run test
ERRO[0000] runc run failed: wait for exec fifo: container init process died unexpectedly
real    0m0.136s

No leftover process, no leftover cgroup, runc list is empty. A container without the profile still runs in 28 ms, i.e. the poll timeout costs nothing on the healthy path (the first poll(2) returns immediately).

Two tests:

  • a unit test that builds the exact kernel state — a helper process that starts a second thread and then exit(2)s its leader from init(), the only place where Go guarantees the main thread — and checks that we notice. It needs no root, no busybox and no seccomp, and runs in 0.1 s.
  • an integration test in run.bats. It gives the container its own stdio rather than the one bats captures: should this regress, the leftover threads would keep bats' end open and the suite would hang instead of reporting a failure. Verified that it passes with the fix and fails (in 35 s, without hanging) without it.

https://claude.ai/code/session_01GtF433o8i3BrjqTZqT5xk7

@kolyshkin kolyshkin added this to the 1.6.0-rc.1 milestone Aug 29, 2026
@kolyshkin
kolyshkin marked this pull request as draft August 29, 2026 20:51
@kolyshkin
kolyshkin force-pushed the fix-5087-init-hang branch 2 times, most recently from 993c808 to 52ced03 Compare August 29, 2026 21:54
kolyshkin and others added 2 commits August 29, 2026 17:25
Seccomp's SCMP_ACT_KILL is SECCOMP_RET_KILL_THREAD: it kills the thread
that performed the syscall, not the whole thread group. As "runc init"
locks its main goroutine to the thread group leader, killing that thread
leaves a zombie leader whose remaining Go runtime threads are alive,
keeping the shared descriptor table -- and thus the child ends of the
sync socket and of the exec fifo -- open. The parent therefore never sees
an EOF, and since the thread group is not dead, neither wait(2) nor a
pidfd report anything either, so the parent waits forever.

Add waitForInitWrite, which polls the descriptor with a timeout and, when
nothing arrives, consults /proc/[pid]/stat -- the only thing that reports
a zombie thread group leader. Use it both for reading the sync socket and
for waiting on the exec fifo; the latter replaces the pidfd fast path,
which can not detect this case at all.

Kill and reap the leftover threads once a dead init is detected while
waiting for the exec fifo. They keep the container's stdio descriptors
open, so whoever inherited those would wait for an EOF just as long, and
they keep its cgroup populated, so destroying the container would fail.

Finally, do not wait(2) for the child in the setnsProcess.start error
path: a half-dead child is never reaped, and the deferred error handler
terminates and reaps it anyway.

Fixes opencontainers#5087

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
SCMP_ACT_KILL kills a single thread, so "runc init" is left as a zombie
thread group leader surrounded by live threads that keep its descriptors
open. Check that runc reports that instead of hanging.

The container gets its own stdio rather than the one bats captures: were
this to regress, the leftover threads would keep bats' end open, and the
suite would hang instead of reporting a failure.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@kolyshkin
kolyshkin marked this pull request as ready for review August 30, 2026 06:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Indefinite parent hang in ReadPacket upon abnormal child exit after procRun

1 participant