install: refresh URL/local tarballs under --force and explicit re-add/update

[robobun]

What

bun install <url> --force did not refresh a URL tarball, and neither did explicitly re-adding the same URL (bun i <url>) even though Bun 1.3 refreshed it in that case. Reinstalling after the tarball at the URL changed kept serving the old code, and the only known workaround was clearing the cache and the lockfile by hand (bun pm cache rm, then delete node_modules and bun.lock).

Fixes #31864.

Repro

Serve a tarball at a fixed URL, install it, change the bytes served at that same URL, then reinstall with --force (or re-add the same URL).

# server serves v1 at http://localhost:PORT/pkg.tgz
$ bun i http://localhost:PORT/pkg.tgz
$ cat node_modules/my-url-pkg/index.js
module.exports = "VERSION_ONE";

# swap the served tarball to v2 (same URL), then:
$ bun i http://localhost:PORT/pkg.tgz --force
$ cat node_modules/my-url-pkg/index.js
module.exports = "VERSION_ONE";     # expected VERSION_TWO

Cause

URL and local tarballs are cached under a folder named from the URL/path hash (cached_tarball_folder_name), not from the tarball content. The cache gate that decides whether to download (PackageInstall::package_missing_from_cache, and the isolated installer's inline equivalent) only checks whether that folder exists, with no --force handling. When the bytes behind the same URL/path change, the folder name is identical, so --force reinstalls from the stale extraction and never re-downloads.

A naive re-download would then hit a second problem: ExtractTarball::run verifies the fresh bytes against the integrity pinned in the lockfile and fails with IntegrityCheckFailed, which is the error the reporter saw after clearing only the cache.

On Bun 1.3, bun i <url> (re-adding the URL, no --force) did refresh the tarball, but only as a side effect of the duplicate-package.json-entry bug fixed for #30499: each re-add appended a second, URL-keyed dependency whose name hash missed the lockfile dedup and forced a fresh download. Fixing #30499 removed the accidental refresh along with the duplicate entries. Users rely on re-add-to-refresh (see the issue thread), so this PR restores that behavior through an explicit mechanism instead.

npm/git/github resolutions are content-addressed by version/commit, so they are unaffected and keep hitting the cache.

Fix

For content-mutable tarballs only (RemoteTarball/LocalTarball, identified by Tag::is_tarball_cache_keyed_by_url):

• Treat the package as missing from the cache under --force, and also when the dependency was explicitly named on the command line (bun i <url>, bun update <name>; matched via UpdateRequest::matches against update_requests), so a download/read task is enqueued. Applies to both the hoisted gate (package_missing_from_cache) and the isolated installer's inline gate, on the initial install-phase pass only.
• The refresh is skipped when this run already fetched the tarball: either the preinstall state is already Done, or a tarball task for the same URL is already in task_queue (the resolve phase re-downloads it when bun update invalidates the resolution). Without these guards the install phase re-enqueues into the already-drained task and the package silently never installs.
• Drop the lockfile-pinned integrity when constructing the ExtractTarball (generate_network_task_for_tarball for remote, enqueue_tarball_for_reading for local), so the fresh bytes are accepted and the hash is recomputed from them.
• Persist the recomputed hash over the stale one after extraction, in the shared extract handler so both the hoisted and isolated installers stay consistent. This keeps a later cache-cleared install from rejecting the current content against the old pin.

Plain bun install (no package named, no --force) is unchanged and still installs from the cache, matching Bun 1.3.

Verification

The tarball --force refresh tests in test/cli/install/bun-install-tarball-integrity.test.ts serve a tarball over a local Bun.serve (port: 0, isolated BUN_INSTALL_CACHE_DIR), swap the bytes at the same URL/path, and assert the installed file is the new content, the tarball was actually re-requested, and the lockfile integrity was updated. Cases covered for both the hoisted and isolated linkers:

• bun install --force with a changed remote tarball
• bun install --force with a changed local tarball at the same path
• --force as the very first install with no lockfile (exercises the re-enqueue guard)
• bun i <url> re-add with no --force (the Bun 1.3 behavior from the issue thread), including asserting package.json does not grow a duplicate entry (bun install {tarball url} will add it multiple times to package.json #30499)
• bun update <name> of a URL tarball dep refreshing node_modules

All fail on the unfixed build. The rest of the tarball-integrity suite (including should fail integrity check when tarball URL content changes for the plain-install path) still passes, confirming normal integrity verification is unchanged. Verified against Bun v1.3.14 that plain bun install staying on the cached extraction matches 1.3, and that re-add/update refreshing matches 1.3. rust:check-all passes on all targets.

[coderabbitai]

Walkthrough

Implement --force re-download and integrity refresh for URL/path-keyed tarball caches. Add a resolution tag classifier, thread force-refresh through cache-miss detection, drop cached integrity to force recomputation during fetch, persist the newly computed integrity to the lockfile, and provide regression test coverage.

Changes

Force-refresh URL-keyed tarballs

Layer / File(s)	Summary
Resolution tag classifier for URL-keyed tarballs src/install/resolution.rs	Add Tag::is_tarball_cache_keyed_by_url() helper method to classify RemoteTarball and LocalTarball cache keys as URL/path-based, distinguishing them from content-addressed resolutions.
Cache-miss detection with force-refresh parameter src/install/PackageInstall.rs, src/install/PackageInstaller.rs	PackageInstall::package_missing_from_cache adds allow_force_refresh parameter and early-return logic to treat URL-keyed tarball caches as missing when force-install is enabled. PackageInstaller wires the parameter through its call site.
Isolated install missing-from-cache logic src/install/isolated_install.rs	Update install_isolated_packages to compute force_refresh_tarball and treat URL-keyed tarballs as missing when --force is enabled, forcing re-download/re-read. Otherwise preserve prior behavior: check PreinstallState::Done and cache existence.
Enqueue: choose integrity for local tarballs under --force src/install/PackageManager/PackageManagerEnqueue.rs	enqueue_tarball_for_reading selects Integrity::default() when forcing refresh for URL-keyed tarballs; otherwise uses the lockfile package meta integrity.
Integrity recomputation in network and extract tasks src/install/PackageManager/runTasks.rs	runTasks imports Integrity, selects between persisted and default integrity during network task generation, wires the selected integrity into ExtractTarball, recomputes integrity on extract, and persists the new integrity back to the lockfile with FORCE_SAVE_LOCKFILE.
Regression test for --force URL tarball refresh test/cli/install/bun-install-tarball-integrity.test.ts	Add regression tests verifying that bun install --force re-downloads tarballs when bytes at the same URL/path change. Tests build distinct tarballs with different version markers, verify re-fetch, installed content update, and lockfile integrity refresh for both remote and local tarballs.

Suggested reviewers

• RiskyMH
• dylan-conway
• alii

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The PR fully addresses issue #31864 by implementing --force tarball refresh: re-fetching URL/local tarballs, dropping stale integrity pins, and persisting recomputed hashes as required.
Out of Scope Changes check	✅ Passed	All changes are within scope: adding a new method Tag::is_tarball_cache_keyed_by_url(), modifying cache-miss checks to handle --force for URL tarballs, updating integrity handling, and adding comprehensive tests.
Description check	✅ Passed	The PR description is comprehensive and well-structured, covering the what, why, how, and verification of the fix for tarball refresh under --force.
Title check	✅ Passed	The title accurately summarizes the main change: adding tarball refresh support under --force flag for URL and local tarballs, which directly matches the primary objective of the PR.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

Warning

Review ran into problems

🔥 Problems

Stopped waiting for pipeline failures after 30000ms. One of your pipelines takes longer than our 30000ms fetch window to run, so review may not consider pipeline-failure results for inline comments if any failures occurred after the fetch window. Increase the timeout if you want to wait longer or run a @coderabbit review after the pipeline has finished.

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[robobun]

^{Updated 11:51 AM PT - Jun 10th, 2026}

❌ @robobun, your commit b410712 has 8 failures in Build #61771 (All Failures):

• test/js/sql/sql.test.ts - code 1 on 🍎 14 aarch64
• test/js/sql/sql-postgres-datetime-roundtrip.test.ts - code 1 on 🍎 14 aarch64
• test/js/sql/local-sql.test.ts - code 1 on 🍎 14 aarch64
• test/js/bun/test/parallel/test-docker-build-debian.ts - code 1 on 🍎 14 aarch64
• test/js/bun/test/parallel/test-docker-build-alpine.ts - code 1 on 🍎 14 aarch64
• test/js/bun/s3/s3.test.ts - timeout on 🍎 14 aarch64
• test/integration/mysql2/mysql2.test.ts - code 1 on 🍎 14 aarch64
• test/cli/init/init.test.ts - code 1 on 🍎 26 aarch64
• test/cli/init/init.test.ts - code 1 on 🍎 14 aarch64

---

🧪   To try this PR locally:

bunx bun-pr 31868

That installs a local version of the PR into your bun-31868 executable, so you can run:

bun-31868 --bun

[github-actions]

Found 1 issue this PR may fix:

1. bun install --force does not re-extract local .tgz dependencies after file is replaced #29372 - Local .tgz dependencies not re-extracted under --force due to the same path-based cache key root cause this PR addresses

If this is helpful, copy the block below into the PR description to auto-close this issue on merge.

Fixes #29372

🤖 Generated with Claude Code

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/install/PackageManager/runTasks.rs`:
- Around line 1163-1183: The integrity write is performed before the final
package mapping is finalized, so move the logic that persists
task.data_extract().integrity (and sets Enable::FORCE_SAVE_LOCKFILE) to after
process_extracted_tarball_package has been called and package_id has been
possibly retargeted; specifically, remove the block that writes to
manager.lockfile.packages.items_meta_mut()[package_id as usize].integrity and
instead, after process_extracted_tarball_package(...) confirms the final
package_id and new_integrity.tag.is_supported(), assign the integrity into
manager.lockfile.packages.items_meta_mut()[package_id as usize].integrity and
call manager.options.enable.set(Enable::FORCE_SAVE_LOCKFILE, true).

In `@test/regression/issue/31864.test.ts`:
- Around line 10-17: Remove the global 5-minute override by deleting the
setDefaultTimeout(1000 * 60 * 5) call in the test (do not replace it elsewhere);
rely on Bun’s default test timeout and keep the fixture small so the test fails
fast instead of stalling—ensure no other file-wide timeout overrides
(setDefaultTimeout) remain in this test file.
- Around line 61-138: Add a sibling test that covers the local-file path
variant: create two tarballs via buildTarball("VERSION_ONE") and
buildTarball("VERSION_TWO"), write v1.tgz to a local "./pkg.tgz" inside the
tempDir used for the test (instead of using Bun.serve/serveV2), run bun install
to populate node_modules, overwrite the same "./pkg.tgz" file with v2 bytes,
then run bun install --force (using the same spawnOpts and
BUN_INSTALL_CACHE_DIR) and assert the installed index file (installedIndex) now
contains "VERSION_TWO" and bun.lock contains v2.integrity (and not
v1.integrity); use the same helpers and assertions as the HTTP test so the
LocalTarball code path is exercised the same way.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 1df1589d-511f-4993-8816-49201f1d7fdb

📥 Commits

Reviewing files that changed from the base of the PR and between 91270aa and f472bcb.

📒 Files selected for processing (7)

• src/install/PackageInstall.rs
• src/install/PackageInstaller.rs
• src/install/PackageManager/PackageManagerEnqueue.rs
• src/install/PackageManager/runTasks.rs
• src/install/isolated_install.rs
• src/install/resolution.rs
• test/regression/issue/31864.test.ts

[claude]

Additional findings (outside current diff — PR may have been updated during review):

• 🟡 test/regression/issue/31864.test.ts:1-8 — Per CLAUDE.md and test/CLAUDE.md, test/regression/issue/${issueNumber}.test.ts is reserved for true regressions (worked in a previous release, then broke) — and as the PR's own Cause section explains, URL/local tarball refresh under --force never worked because the cache key is the URL/path hash, so this is a missing capability rather than a regression. Consider folding this test into one of the existing URL-tarball suites the PR already references (test/cli/install/bun-install.test.ts or bun-install-tarball-integrity.test.ts).

  Extended reasoning...

  What

  The new test is added at test/regression/issue/31864.test.ts, but the project's test-placement convention says that directory is reserved for true regressions, and this bug is not one.

  The convention

  Both guideline files state the rule explicitly and identically:

  • CLAUDE.md:66 — "Exception: test/regression/issue/${issueNumber}.test.ts is reserved for bugs that have a GitHub issue number and are true regressions (worked in a previous release, then broke). An issue number alone is not enough — if the behavior was never correct, it's not a regression and the test belongs in the existing file for that module."
  • test/CLAUDE.md:153 — "/test/regression/issue/${issueNumber}.test.ts is only for bugs that have a GitHub issue number and are true regressions (worked in a previous release, then broke). An issue number alone does not qualify — if it was never correct, put the test in the module's existing test file instead."

  So the gating question is not "does this have an issue number?" but "did this ever work in a shipped release and then break?"

  Why #31864 is not a regression — step by step

  1. The PR's Cause section says URL/local tarballs are cached under a folder named from cached_tarball_folder_name (the URL/path hash), not the content hash.
  2. The cache gate (PackageInstall::package_missing_from_cache and the isolated installer's inline equivalent) only checked whether that folder exists, with no --force handling. This is the design as it has always existed, not a recently introduced break.
  3. Therefore, when the bytes behind the same URL change, the cache key is identical and --force structurally could never have refreshed the tarball — the capability did not exist until this PR adds it (allow_force_refresh + is_tarball_cache_keyed_by_url).
  4. The PR also notes a second structural blocker: even a naive re-download would have failed IntegrityCheckFailed against the lockfile pin. There is no prior release in which this path worked.

  There is no "worked in version X, broke in version Y" claim anywhere in the issue or PR. This is a missing feature/capability being added for the first time, which is exactly the case the guideline calls out as not belonging in test/regression/issue/.

  Where it should go

  The PR's own Verification section already names the natural homes:

  • test/cli/install/bun-install.test.ts — already covers URL/local tarball installs.
  • test/cli/install/bun-install-tarball-integrity.test.ts — already covers the "tarball URL content changes" integrity path for the non---force case, which is the direct sibling of what this test exercises.

  Either is a better fit than a new standalone regression-issue file.

  Impact / fix

  No functional impact — purely test organization. Fix: move the describe.each(...) block (and its tar/server helpers, if not already available there) into one of the two existing suites above and delete test/regression/issue/31864.test.ts. The header comment can keep the https://github.com/oven-sh/bun/issues/31864 reference so the issue link is preserved.

• 🟡 test/regression/issue/31864.test.ts:61-62 — This test only exercises the RemoteTarball path (via Bun.serve), but the fix also touches the LocalTarball path — specifically the integrity-drop in enqueue_tarball_for_reading, which is only reached for file: tarball dependencies. Consider adding a sibling case that writes the .tgz to disk, depends on it via file:, swaps the bytes, and reinstalls with --force; right now deleting the enqueue_tarball_for_reading hunk would not break any test in this PR.

  Extended reasoning...

  What's missing

  The PR fixes --force refresh for both RemoteTarball and LocalTarball — is_tarball_cache_keyed_by_url() returns true for both, and the diff adds symmetric integrity-drop logic in two enqueue paths:

  • generate_network_task_for_tarball (runTasks.rs) — used by RemoteTarball
  • enqueue_tarball_for_reading (PackageManagerEnqueue.rs:258-267) — used by LocalTarball

  The new regression test, however, only drives the RemoteTarball path: it spins up a Bun.serve server and depends on http://127.0.0.1:${port}/my-url-pkg.tgz. There is no test that depends on a file:./pkg.tgz on disk, swaps the bytes, and asserts --force re-reads it.

  Why enqueue_tarball_for_reading is uncovered

  enqueue_tarball_for_reading is called exclusively from the ResolutionTag::LocalTarball arms — PackageInstaller.rs:1562 for the hoisted installer and isolated_install.rs:2542 for the isolated installer. The HTTP-served tarball in this test resolves as RemoteTarball and goes through generate_network_task_for_tarball instead, so it never touches the new code in enqueue_tarball_for_reading.

  Step-by-step proof

  1. Revert only the PackageManagerEnqueue.rs hunk (restore let integrity = this.lockfile.packages.items_meta()[package_id as usize].integrity;).
  2. Run test/regression/issue/31864.test.ts.
  3. The test installs http://127.0.0.1:PORT/my-url-pkg.tgz → Tag::RemoteTarball.
  4. On --force, package_missing_from_cache returns true (covered hunk), and the download is enqueued via generate_network_task_for_tarball (covered hunk), which drops the integrity.
  5. enqueue_tarball_for_reading is never called — its reverted body is dead for this test.
  6. Test passes despite the partial revert.

  So one of the four load-bearing clauses in this PR has no covering test.

  Project guidelines

  CLAUDE.md's "Tests reviewers reject" section (distilled from merged-PR review feedback) calls this out explicitly:

  Confirm deleting each load-bearing clause of your fix breaks at least one test.
  Cover the variant matrix, not just the repro. Every sibling entry point receiving the same fix.

  Both apply here: LocalTarball is the sibling entry point, and the enqueue_tarball_for_reading clause is deletable without test failure.

  Suggested fix

  Wrap the test body in a second describe.each(['remote', 'local']) (or add a sibling it) that, for the local variant:

  • Writes v1.tgz into the temp dir and sets dependencies: { "my-url-pkg": "file:./pkg.tgz" }.
  • Asserts VERSION_ONE after the first install.
  • Overwrites pkg.tgz with v2.tgz bytes (same path) and runs bun install --force.
  • Asserts VERSION_TWO is installed and bun.lock carries v2.integrity.

  This reuses the existing buildTarball helper and the hoisted/isolated matrix already in place.

  Severity

  This is a test-coverage gap, not a production correctness bug — the LocalTarball change is symmetric with the RemoteTarball change and looks correct by inspection. Filing as a nit, but it's the kind of gap the repo's own review checklist flags.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/cli/install/bun-install-tarball-integrity.test.ts`:
- Around line 713-750: Extract the duplicated tar helper functions octal,
tarHeader, and pad512 to file-level (top of the test file, after imports and
before setDefaultTimeout) so both test suites reuse the same implementations;
remove the duplicate definitions in both places and keep each test's
buildTarball variants local (the marker-based one in the
"bun-install-tarball-integrity.test.ts" suite and the generic Buffer-based one
in the other suite), and update any calls to reference the single top-level
helpers (no API changes to tarHeader/pad512/octal).

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 5662ae7c-46e9-46da-9a63-25ea50a3cea9

📥 Commits

Reviewing files that changed from the base of the PR and between f472bcb and 6de9881.

📒 Files selected for processing (7)

• src/install/PackageInstall.rs
• src/install/PackageInstaller.rs
• src/install/PackageManager/PackageManagerEnqueue.rs
• src/install/PackageManager/runTasks.rs
• src/install/isolated_install.rs
• src/install/resolution.rs
• test/cli/install/bun-install-tarball-integrity.test.ts

[robobun]

Status after the re-add/update conformance push (b410712): build 61771 is green on every lane except one.

The only failed job is darwin 26 aarch64 - test-bun (timed out), whose failing test is test/cli/init/init.test.ts "bun init in folder": the inline snapshot does not expect the CLAUDE.md file that bun init now generates. That is unrelated to this PR (no init changes here) and reproduces on other branches, e.g. build 61749.

One pre-existing test in the file this PR touches, should fail integrity check when tarball URL content changes, flaked once on Windows 2019 x64-baseline and passed on retry (CI classified it flaky). It runs plain bun install with no --force and no named packages, so the new code paths (gated on force_install() or a command-line update request) are unreachable from it.

The earlier repo-wide bunx node-24 break and the verify-baseline SDE flake are both gone on this build. I have already used my one automatic re-trigger; the remaining red is the unrelated init snapshot break.

[VillainsRule]

@robobun if you check, the behavior in bun 1.3 is actually that it checks with normal bun install even without force...please conform to this

[robobun]

@VillainsRule You are right, thanks for pushing on this. I tested Bun v1.3.14 against a local server that swaps the tarball bytes behind a fixed URL:

• plain bun install (lockfile + node_modules intact): does NOT re-check on 1.3 either (stays on the cached extraction, zero requests)
• bun i <url> (re-adding the URL, no --force): DOES refresh on 1.3
• bun update <name>: DOES refresh on 1.3

The 1.4 canary lost the re-add refresh as a side effect of fixing #30499 (each bun i <url> on 1.3 appended a duplicate URL-keyed entry to package.json, which forced a fresh download; fixing the duplicate entries removed the accidental refresh).

This PR now restores both behaviors explicitly: bun i <url> and bun update <name> re-fetch the tarball (no --force needed, and no duplicate package.json entries), on top of the existing --force refresh. Plain bun install still uses the cache, same as 1.3. All verified against 1.3.14 and covered by tests for both the hoisted and isolated linkers.