Repository navigation
Conversation
A Bench EQL run on #1419's branch failed when 15 tests lost their database connection in the same second. Each failed with "expected to read 5 bytes, got 0 bytes at EOF". RunsOn reported that the kernel killed one process for running out of memory. The machine's own memory use peaked at 12.8 GB of 61.6 GiB, so the kill came from a smaller limit. This container's 2048 MB limit is the only one in that job. When the kernel kills a Postgres process, Postgres ends every other connection and restarts. Every test running at that moment then fails. The same symptom appeared in a Test EQL shard on Blacksmith, in #1416: 16 tests lost their connection in the same second. Blacksmith does not report killed processes, so the cause there is not confirmed. Every CI job that uses this file runs on a 16 vCPU machine with at least 32 GB of memory, so 4096 MB fits. On a developer machine, Docker reserves nothing up front. The limit only caps what Postgres can use.
|
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The EQL tests run against a Postgres database in a Docker container. EQL is the SQL that stores and queries encrypted values. That container may use at most 2,048 MB of memory. This pull request raises the limit to 4,096 MB.
When Postgres reaches the limit, the kernel kills one Postgres process. Postgres then ends every connection and restarts, so every test running at that moment fails at once. This happened in a Bench EQL run and in a Test EQL run this week.
Changes
packages/eql/tests/docker-compose.yml: the memory limit goes from2048mbto4096mb. All five Postgres services inherit it: Postgres 14, 15, 16, and 17, and the default service.Evidence
mainpassed on each of the last 7 nights.Verification
4096mb.pnpm run test:scripts: all 83 test files passed. Two of them read this file:eql-matrix-triggers.test.mjsandeql-workflow-filters.test.mjs.packages/eql/tests/**is in its relevance filter.Who this affects
benchinbench-eql.yml, andbuild-archive,test,validate,splinter, ande2eintest-eql.yml. Each runs on a 16 vCPU machine: 64 GB of memory on Blacksmith, and 32 GiB or 64 GiB on RunsOn. One container runs per job, so 4,096 MB fits.mise run postgres:upinpackages/eqlgets the new limit. Docker does not reserve memory for a limit. The limit only caps what Postgres can use.Related
Review notes
local/docker-compose.ymlhas the same 2,048 MB limit.tests-bench.ymluses it on a 4 vCPU machine. I left it unchanged, because no failure there points to it.CPU_LIMIT.