Skip to content

test(eql): raise the Postgres test container's memory limit to 4096 MB - #1730

Draft
auxesis wants to merge 1 commit into
mainfrom
test/eql-postgres-memory
Draft

auxesis wants to merge 1 commit into
mainfrom
test/eql-postgres-memory

Conversation

@auxesis

@auxesis auxesis commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

Summary

The EQL tests run against a Postgres database in a Docker container. EQL is the SQL that stores and queries encrypted values. That container may use at most 2,048 MB of memory. This pull request raises the limit to 4,096 MB.

When Postgres reaches the limit, the kernel kills one Postgres process. Postgres then ends every connection and restarts, so every test running at that moment fails at once. This happened in a Bench EQL run and in a Test EQL run this week.

Changes

  • packages/eql/tests/docker-compose.yml: the memory limit goes from 2048mb to 4096mb. All five Postgres services inherit it: Postgres 14, 15, 16, and 17, and the default service.
  • The same file: a comment says why the limit is 4,096 MB, and the lock-table comment now names the new limit.

Evidence

Verification

  • I parsed the file with YAML merge keys turned on, as Docker Compose does. All five services resolve to 4096mb.
  • pnpm run test:scripts: all 83 test files passed. Two of them read this file: eql-matrix-triggers.test.mjs and eql-workflow-filters.test.mjs.
  • Docker is not installed on the machine I used, so I did not start the container locally.
  • This pull request's Test EQL runs every database job, because packages/eql/tests/** is in its relevance filter.
  • Bench EQL does not run on pull requests, so this pull request does not run it.
  • Because the failure is rare, a passing run here does not prove the fix. The evidence will be fewer connection-loss failures over the coming weeks.

Who this affects

  • CI: six jobs use this file: bench in bench-eql.yml, and build-archive, test, validate, splinter, and e2e in test-eql.yml. Each runs on a 16 vCPU machine: 64 GB of memory on Blacksmith, and 32 GiB or 64 GiB on RunsOn. One container runs per job, so 4,096 MB fits.
  • Developers: anyone who runs mise run postgres:up in packages/eql gets the new limit. Docker does not reserve memory for a limit. The limit only caps what Postgres can use.

Related

Review notes

  • local/docker-compose.yml has the same 2,048 MB limit. tests-bench.yml uses it on a 4 vCPU machine. I left it unchanged, because no failure there points to it.
  • The container's CPU limit stays at 2, through CPU_LIMIT.
  • No changeset: this file is not part of any published package.

A Bench EQL run on #1419's branch failed when 15 tests lost their
database connection in the same second. Each failed with "expected to
read 5 bytes, got 0 bytes at EOF". RunsOn reported that the kernel
killed one process for running out of memory. The machine's own memory
use peaked at 12.8 GB of 61.6 GiB, so the kill came from a smaller
limit. This container's 2048 MB limit is the only one in that job.

When the kernel kills a Postgres process, Postgres ends every other
connection and restarts. Every test running at that moment then fails.

The same symptom appeared in a Test EQL shard on Blacksmith, in #1416:
16 tests lost their connection in the same second. Blacksmith does not
report killed processes, so the cause there is not confirmed.

Every CI job that uses this file runs on a 16 vCPU machine with at
least 32 GB of memory, so 4096 MB fits. On a developer machine, Docker
reserves nothing up front. The limit only caps what Postgres can use.
@changeset-bot

changeset-bot Bot commented Nov 20, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 2765a8b

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@coderabbitai

coderabbitai Bot commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant