CircleCI publishes dirty build for clean commit, hash mismatch vs local build

[mikevrpv]

Checked against current master:

47c1952...master

---

Context of the Build

1. What board are you trying to build?

EOL_x230-hotp-maximized

2. What repository:branch are you using to build from?

• Heads:Master
• Other (please specify)

Commit:

47c19525dcefe3c3cc2131b855ae252c7bda8ef7

3. What version of Heads/coreboot are you trying to build?

Heads:

v0.2.1-3117-g47c1952

4. In building the rom where did you get the blobs?

• No blobs required
• Provided by the company that installed Heads on the device
• Extracted from a backup rom taken from this device
• Extracted from another backup rom taken from another device
• Extracted from the online bios using the automated tools provided in Heads
• I don't know

N/A for this report. This is about reproducing the published CircleCI artifact using the same Heads commit and reproducible build environment.

5. If using the automated tools to get the blobs did you run the relevant scripts in the blobs directory

• Yes
• No

N/A

6. What operating system are you using

Fedora locally. The affected published build is from CircleCI.

Please describe the problem

Describe the bug

CircleCI job 53075 for commit 47c19525dcefe3c3cc2131b855ae252c7bda8ef7 publishes artifacts with the -dirty suffix.

A local build of the same commit from a clean working tree produces a different ROM hash.

I then rebuilt locally while forcing:

GIT_STATUS=dirty
HEADS_GIT_VERSION=v0.2.1-3117-g47c1952-dirty

This reproduced the CircleCI artifacts exactly.

CircleCI and my forced-dirty local build both produce:

final ROM:
b92aba28d3bbb77ce9797c71b13ebc96b208de2b3b367389d83b3176057657c4

tools.cpio:
7a34ae1fa9d513cc874e4274a6a80a5474c1c0a3175858173401fab37c400348

initrd.cpio.xz:
1b34551976c36a7e900978009b15783cd4f32abb5fd6ae67aa0203f1ab51ea28

The clean build of the same commit produces:

final ROM:
e94ad11d9ae53762d4330f406e8a935bcacf3aca016d6c621478e08ce4b08460

The kernel and the other hashed build components match. The first differing file is /etc/config inside tools.cpio, consistent with GIT_STATUS=dirty being embedded into the build.

To Reproduce

1. Check out commit 47c19525dcefe3c3cc2131b855ae252c7bda8ef7.
2. Confirm git status --short produces no output.
3. Build EOL_x230-hotp-maximized using docker_repro.sh.
4. Compare the result with CircleCI job 53075.
5. Rebuild with:

./docker_repro.sh make \
  BOARD=EOL_x230-hotp-maximized \
  GIT_STATUS=dirty \
  HEADS_GIT_VERSION=v0.2.1-3117-g47c1952-dirty

6. The resulting ROM, tools.cpio, initrd.cpio.xz, top ROM, and bottom ROM hashes match CircleCI exactly.

Expected behavior

CircleCI should build from a clean tracked working tree so the published artifact for a commit can be reproduced locally from the same clean commit.

It may also make sense for CI to fail before make if tracked files have been modified.

Screenshots

N/A

Additional context

CircleCI job:

https://app.circleci.com/pipelines/github/linuxboot/heads/1402/workflows/2565e7d3-e20e-492c-b4c0-5a1295a06e7a/jobs/53075/artifacts

Logging these immediately before make may show what is making the CI checkout dirty:

git status --short
git diff
git diff --cached
git describe --abbrev=7 --tags --dirty

[tlaurion]

Thanks @mikevrpv, this is a workspace cache reusal problem

blobs workspace cache was overwriting local git from cache with digest of only *.sh, where recent commit added READMEs under those directory. Old cache overwrote local git repo for blobs, causing dirty because git was unclean.

This was caused by #2165 adding READMEs in blobs, where an older CircleCI reused older blobs/* cache. Benign as you proved (content of rom good outside of /etc/config having dirty which causes non reproductibility which as you showed could be resulved by forcing a dirty build locally to match CircleCi's).

Will be fixed with #2183