Skip to content

test(tool-output): stop expressing "unwritable path" as a magic absolute path - #4722

Open
DaoyuanLi2816 wants to merge 1 commit into
bytedance:mainfrom
DaoyuanLi2816:fix/externalize-test-portability
Open

test(tool-output): stop expressing "unwritable path" as a magic absolute path#4722
DaoyuanLi2816 wants to merge 1 commit into
bytedance:mainfrom
DaoyuanLi2816:fix/externalize-test-portability

Conversation

@DaoyuanLi2816

@DaoyuanLi2816 DaoyuanLi2816 commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Why

Two tests in test_tool_output_budget_middleware.py need an outputs_path that
os.makedirs refuses to create, so they can reach _externalize's
except OSError: return None branch and the truncation fallback behind it. Both
spell that condition as a literal path, /dev/null/cannot-mkdir-here.

That only holds where /dev/null is a character device. On Windows it is an
ordinary relative path, so os.makedirs succeeds and:

  • TestExternalize::test_returns_none_on_invalid_path and
    TestWrapToolCallFallback::test_fallback_when_disk_write_fails both fail;
  • the suite writes real files to C:\dev\null\cannot-mkdir-here\.tool-results\
    — outside any temporary directory, at the drive root. 66 stray files had
    accumulated there on my machine, one per run of each affected test.

The comment above the first test records that this is the second time the same
assumption has broken. /nonexistent/... came first and was silently created by
mkdir -p when CI ran as root in a container; /dev/null/... was the fix for
that. Both encode a guess about the environment rather than the condition
actually under test, so each one holds only until it meets an environment nobody
had in mind.

What changed

Tests only. The two tests now build their unwritable path from a regular file:
a TemporaryDirectory containing a plain file, with the outputs_path pointing
below that file.

Creating a directory under a file fails with an OSError subclass on every
platform, so the branch under test is reached deterministically, and the whole
path lives inside the test's own temporary directory, so nothing is written
outside it. Measured, rather than assumed:

file as parent /dev/null/...
Linux (WSL Ubuntu, non-root) NotADirectoryError, errno 20 NotADirectoryError, errno 20
Windows 11 FileNotFoundError, errno 2 succeeds — creates C:\dev\null\...

No production code changes, and no documented behaviour changes, so README.md
and AGENTS.md need no update here.

Surface area

  • Frontend UI — page / component / setting / interaction under frontend/
  • Backend API — endpoint / SSE event / request-response shape under backend/app
  • Agents / LangGraph — agent node, graph wiring, langgraph.json, or prompt change
  • Sandboxdocker/ or sandboxed execution
  • Skills — change under skills/
  • Dependencies — new/upgraded entry in backend/pyproject.toml or frontend/package.json
  • Default behavior change
  • Docs / tests / CI only — no runtime behavior change

Bug fix verification

  • Test paths: backend/tests/test_tool_output_budget_middleware.py::TestExternalize::test_returns_none_on_invalid_path
    and ::TestWrapToolCallFallback::test_fallback_when_disk_write_fails
  • Red on main, green on this branch — on Windows, which is where the old
    spelling breaks.
  • On Linux both spellings pass, before and after. That is the intended
    outcome, not a no-op: the point is not to turn the tests green on POSIX but
    to stop the condition under test from depending on the host to supply it.
    Checked rather than assumed — I ran main's version of both tests on Linux
    (2 passed) and this branch's version (2 passed), in a clean Ubuntu checkout.
  • The tests keep their teeth on both platforms. Deleting _externalize's
    except OSError: return None guard around os.makedirs makes both fail
    rather than pass — NotADirectoryError on Linux, FileNotFoundError on
    Windows — so they really do exercise that branch.
  • Also confirmed the stray-file count under C:\dev\null\... stops growing
    across runs on this branch (66 before, 66 after).

Validation

Linux (clean Ubuntu checkout of this branch, Python 3.12):

cd backend
make lint               # ruff check: All checks passed! / ruff format --check: 1092 files already formatted
make test               # 11065 passed, 73 skipped in 192.95s
make test-blocking-io   # 70 passed

Windows 11 (same branch):

cd backend
ruff check .                                         # All checks passed!
pytest tests/test_tool_output_budget_middleware.py   # 124 passed  (2 of these fail on main)

To be clear about scope: this is not a bid to support Windows

I ran the whole suite on Windows to see where this sits, and it reports
88 failed, 10939 passed. The rest are POSIX permission bits, symlink
privileges, path separators, hostPath mounts, pnpm/nginx shell assumptions —
a different and much larger question, and one I am not proposing to open.
Please read this PR as two tests that no longer depend on the host, not as
step 1 of a Windows-support effort.

The reason it is still worth taking on a Linux-only project: the two arguments
above hold there too. A test that writes outside its own TemporaryDirectory
is a hygiene bug wherever it runs, and the assumption that broke first —
/nonexistent/... being uncreatable — broke on Linux, in your CI container,
because the process was root. The current spelling narrowed that assumption
rather than removing it. Deriving the path from a real file removes it.

Happy to close this if you would rather keep the magic-path form.

…ute path

`test_returns_none_on_invalid_path` and `test_fallback_when_disk_write_fails`
both need an `outputs_path` that `os.makedirs` refuses to create, so they can
reach `_externalize`'s `except OSError: return None` branch. They spell that as
the literal path `/dev/null/cannot-mkdir-here`, which only works where
`/dev/null` is a character device.

On Windows it is an ordinary relative path, so `os.makedirs` succeeds, both
tests fail, and the suite writes real files to `C:\dev\null\cannot-mkdir-here\
.tool-results\` -- outside any temporary directory, at the drive root. Running
the backend suite a few times leaves dozens of stray files behind.

The comment above the first test records that this is the second time the
same assumption has broken: `/nonexistent/...` was silently created by `mkdir
-p` when CI ran as root in a container, and `/dev/null/...` was the fix. Both
encode a guess about the environment rather than the condition under test.

Use a regular file as the parent component instead. Creating a directory below
a file fails with an `OSError` subclass on every platform -- `NotADirectoryError`
(errno 20) on POSIX, `FileNotFoundError` (errno 2) on Windows -- so the branch
is reached deterministically, and the path lives inside the test's own
`TemporaryDirectory`, so nothing is written outside it.

Verified both spellings on Linux (WSL Ubuntu, non-root) and Windows; only the
file-as-parent form fails on both. The two tests still have teeth: dropping
`_externalize`'s `except OSError` guard makes both fail rather than pass.

Tests only -- no production code or documented behaviour changes.
Copilot AI lite review requested due to automatic review settings August 7, 2026 06:00
@github-actions github-actions Bot added risk:medium Medium risk: regular code changes size/S PR changes 20-100 lines labels Aug 7, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR makes the backend tool-output tests deterministic across platforms by replacing a POSIX-specific “unwritable path” assumption (/dev/null/...) with a guaranteed-invalid outputs_path derived from a temporary directory containing a regular file.

Changes:

  • Add _unwritable_outputs_path() context manager to generate an outputs_path that reliably makes os.makedirs fail on all platforms.
  • Update TestExternalize::test_returns_none_on_invalid_path to use the new helper instead of a hard-coded /dev/null/... path.
  • Update TestWrapToolCallFallback::test_fallback_when_disk_write_fails to use the new helper, preventing stray file creation on Windows.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:medium Medium risk: regular code changes size/S PR changes 20-100 lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants