nerdctl build fails on buildkit '#syntax' directive in Fedora guest (mount /tmp/buildkit-metadata1419535027:/run/config/buildkit/metadata (via /proc/self/fd/6), flags: 0x1021: operation not permitted)

[vsiravar]

I am trying to build an image using nerdctl on Fedora guest and it stumbles on #buildkit syntax directive. The build however succeeds on Ubuntu guest.

Steps to reproduce.

[siravara@lima-fedora syntaxtests]$ cat Dockerfile
# syntax=docker/dockerfile:1
FROM public.ecr.aws/docker/library/golang:1.18 AS builder
WORKDIR /build
COPY . .

ARG TARGET_OS
ARG TARGET_ARCH
RUN go env -w GO111MODULE=off
RUN CGO_ENABLED=0 GOOS=${TARGET_OS} GOARCH=${TARGET_ARCH} go build -a -installsuffix cgo -ldflags '-extldflags "-static"' -o hello .

FROM scratch
EXPOSE 8080
COPY --from=builder /build/hello /app/
WORKDIR /app
ENTRYPOINT ["./hello"]

Debug log

[siravara@lima-fedora syntaxtests]$ nerdctl build --debug --no-cache  --build-arg TARGET_OS=linux --build-arg TARGET_ARCH=arm64 -t demo:0.1 .
DEBU[0000] stateDir: /run/user/504/containerd-rootless  
DEBU[0000] rootless parent main: executing "/usr/bin/nsenter" with [-r/ -w/Users/siravara/vishwas-tests/syntaxtests --preserve-credentials -m -n -U -t 1634 -F nerdctl build --debug --no-cache --build-arg TARGET_OS=linux --build-arg TARGET_ARCH=arm64 -t demo:0.1 .] 
DEBU[0000] Choosing the buildkit host "buildkit-default/buildkitd.sock", candidates=[buildkit-default/buildkitd.sock buildkit/buildkitd.sock] (in "/run/user/504") 
DEBU[0000] Chosen buildkit host "unix:///run/user/504/buildkit-default/buildkitd.sock" 
DEBU[0000] worker labels: map[org.mobyproject.buildkit.worker.containerd.namespace:default org.mobyproject.buildkit.worker.containerd.uuid:7eaadfd0-6d19-4331-bd3c-a462bdd97440 org.mobyproject.buildkit.worker.executor:containerd org.mobyproject.buildkit.worker.hostname:lima-fedora org.mobyproject.buildkit.worker.network:host org.mobyproject.buildkit.worker.snapshotter:overlayfs] 
DEBU[0000] running /usr/local/bin/buildctl [--addr=unix:///run/user/504/buildkit-default/buildkitd.sock build --progress=auto --frontend=dockerfile.v0 --local=context=. --output=type=image,unpack=true,name=docker.io/library/demo:0.1 --local=dockerfile=/Users/siravara/vishwas-tests/syntaxtests --opt=filename=Dockerfile --opt=build-arg:TARGET_OS=linux --opt=build-arg:TARGET_ARCH=arm64 --no-cache] 
[+] Building 1.2s (4/4) FINISHED                                                
 => [internal] load build definition from Dockerfile                       0.1s
 => => transferring dockerfile: 445B                                       0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => resolve image config for docker.io/docker/dockerfile:1                 0.9s
 => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:9ba7531bd8  0.0s
 => => resolve docker.io/docker/dockerfile:1@sha256:9ba7531bd80fb0a858632  0.0s
Dockerfile:1
--------------------
   1 | >>> # syntax=docker/dockerfile:1
   2 |     FROM public.ecr.aws/docker/library/golang:1.18 AS builder
   3 |     WORKDIR /build
--------------------
error: failed to solve: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/tmp/buildkit-metadata1419535027" to rootfs at "/run/config/buildkit/metadata": mount /tmp/buildkit-metadata1419535027:/run/config/buildkit/metadata (via /proc/self/fd/6), flags: 0x1021: operation not permitted: unknown

Wonder why the metadata is handled differently by buildkit on Fedora.

[AkihiroSuda]

Please report to https://github.com/moby/buildkit/issues

[vsiravar]

The interesting thing though is that it builds fine with docker on the same fedora guest with buildkit.

Logs

[siravara@lima-dfcd syntaxtests]$ docker build  --no-cache  --build-arg TARGET_OS=linux --build-arg TARGET_ARCH=arm64 -t demo:0.1 .
Sending build context to Docker daemon  11.26kB
Step 1/12 : FROM public.ecr.aws/docker/library/golang:1.18 AS builder
 ---> 8316a46deced
Step 2/12 : WORKDIR /build
 ---> Running in 53535ab8960c
Removing intermediate container 53535ab8960c
 ---> 30eeecddf6d4
Step 3/12 : COPY . .
 ---> 21bf95ea5c13
Step 4/12 : ARG TARGET_OS
 ---> Running in cf1847c44e26
Removing intermediate container cf1847c44e26
 ---> b09f98e8669d
Step 5/12 : ARG TARGET_ARCH
 ---> Running in f537dfba4635
Removing intermediate container f537dfba4635
 ---> a028de5ff684
Step 6/12 : RUN go env -w GO111MODULE=off
 ---> Running in f8814301a43e
Removing intermediate container f8814301a43e
 ---> 229d4b41436b
Step 7/12 : RUN CGO_ENABLED=0 GOOS=${TARGET_OS} GOARCH=${TARGET_ARCH} go build -a -installsuffix cgo -ldflags '-extldflags "-static"' -o hello .
 ---> Running in c0cc7a65e3ea
Removing intermediate container c0cc7a65e3ea
 ---> 4fb935229c27
Step 8/12 : FROM scratch
 ---> 
Step 9/12 : EXPOSE 8080
 ---> Running in 7d7bb1e4855c
Removing intermediate container 7d7bb1e4855c
 ---> dad651a29be9
Step 10/12 : COPY --from=builder /build/hello /app/
 ---> e87409ce2c13
Step 11/12 : WORKDIR /app
 ---> Running in 988d734e949f
Removing intermediate container 988d734e949f
 ---> 5895b057bd61
Step 12/12 : ENTRYPOINT ["./hello"]
 ---> Running in 7334f08efe35
Removing intermediate container 7334f08efe35
 ---> 38cca311a67c
Successfully built 38cca311a67c
Successfully tagged demo:0.1

I am using lima to launch the VM.

[AkihiroSuda]

with buildkit

Doesn't seem so

[vsiravar]

My bad, I misread. Thanks for getting back.

[joshspicer]

@vsiravar did you ever get help on this issue from elsewhere? Ever get to a solution?

[joshspicer]

I'm seeing something similar today with user namespace remapping enabled in the journal (journalctl -u docker)

Jan 13 00:40:10 f5e30636-1287-4fa2-82c3-c4fee7233074 dockerd[16476]: runc run failed: unable to start container process: error during container init: error mounting "/var/lib/docker/300000.300000/tmp/buildkit-metadata1667912507" to rootfs at "/run/config/buildkit/metadata": mount /proc/self/fd/8:/run/config/buildkit/metadata (via /proc/self/fd/9), flags: 0x1021: operation not permitted
Jan 13 00:40:11 f5e30636-1287-4fa2-82c3-c4fee7233074 dockerd[6106]: time="2024-01-13T00:40:11.153891700Z" level=warning msg="no trace recorder found, skipping"

[joshspicer]

Posted here moby/buildkit#4556