Why is prepping the build context so slow when Dockerfile uses COPY with glob syntax?

[tboddyspargo]

It's a little hard to provide a minimal repro for this because it depends on the build context having a lot of hierarchies.

Let's say you have a Dockerfile for a sub-project in a monorepo that has lots of directories and files. The Dockerfile assumes a build context of the whole monorepo (in order to access a couple of files in a sibling sub-project).

monoreop

- project-a/
  | - Dockerfile
  | - src/  (many files)
- project-b/ (many files)
- project-c/ (many files)
- project-d/ (many files)
- project-e/ (many files)
- project-f/ (many files)
- examples/
  | - data/
    | - test.csv
    | - other.csv

The following Dockerfile is a barebones example of this:

FROM scratch

COPY project-a/src /app/
COPY examples/data/*.csv /examples/

Build command from project-a directory

docker build -f Dockerfile ./..

All told, the files that will end up being copied out of the build context should number around a dozen, and given the examples/data/*.csv glob is fairly narrow, I would expect the load build context step to be very fast. However, in my case it is taking ~30s.

Shouldn't the algorithm for identifying the relevant files from the build context be able to efficiently notice that the only relevant directories are project-a/src and example/data? Then the wildcard should resolve very quickly since, *.csv within examples/data only needs to search through a couple of files. I would expect this to take milliseconds, all told, regardless of how big the build context is...

Am I misunderstanding the complexity of glob pattern matching? Is there an avoidable inefficiency in the implementation of prepping the build context?

[wellkilo]

COPY does not control which files are sent as the build context. The client must first walk the entire context directory and apply .dockerignore before the Dockerfile frontend can evaluate COPY. Therefore a narrow COPY glob does not let Buildx skip unrelated sibling trees during the “load build context” phase.

With docker build -f project-a/Dockerfile .., the context is the monorepo root. Every non-ignored path under it must at least be stat'ed/walked, and included files are hashed/transferred to BuildKit. The complexity is dominated by the context tree, not by matching examples/data/*.csv.

The primary fix is a root .dockerignore that excludes everything and re-includes the needed paths:

**
!project-a/
!project-a/src/
!project-a/src/**
!examples/
!examples/data/
!examples/data/*.csv
!project-a/Dockerfile

Be careful to re-include parent directories as shown; otherwise the matcher cannot descend to the desired files. If different Dockerfiles need different contexts, BuildKit also supports named additional contexts, which lets project-a be the main context and examples/data be mounted/copied separately.

Use --progress=plain and compare the context byte count before/after the .dockerignore change. If the byte count is already tiny but walking remains slow, filesystem metadata performance (network filesystem, antivirus, huge directory count) is the remaining cost.

So your glob is not intrinsically expensive; the assumption that COPY is used to prune the client-side context is the part that differs from Docker's build model.