Repository navigation
sandbox-cpu: grade provider CPUs — SHA-256 throughput, single + all-core - #522
Closed
HeyGarrison wants to merge 2 commits into
Closed
HeyGarrison wants to merge 2 commits into
HeyGarrison wants to merge 2 commits into
Conversation
New benchmark measuring raw CPU inside a fresh sandbox at the standardized 8 vCPU / 16 GiB profile: one dd|sha256sum pipe per phase — single-core speed, then nproc parallel pipes for whole-box throughput. No network, disk, or installs on the clock. Composite score = weighted ops/s (multi-core headline, single-core so weak cores can't hide) x success rate. The per-provider sizing map moves to sizes.ts, shared with dax; the script reports observed vcpus and CPU model so under-provisioning shows in results.
Contributor
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
One phases arm per market size (small/medium/large/xlarge, matching benchmarks-platform PLATFORM_SIZES) with scoring.groupBy='phase', so every provider gets a composite per tier. Providers taking raw cpu/memory params scale to all four tiers; template/preset-locked providers grade only the tiers they can provision — a missing tier fails as size_not_supported rather than substituting a wrong-size box. The xlarge layer is exported unchanged as STANDARD_RESOURCE_OPTIONS for dax.
Contributor
|
Closing — the benchmark moved to the private suite ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
sandbox-cpu, a dedicated CPU benchmark that grades each sandbox provider's delivered compute — per market size tier — so composite scores can drive CPU provider selection in the market. Unlike dax (which mixes clone/install/typecheck), this is a pure CPU test: no network, disk, or package installs touch the clock.Workload (
benchmarks/scripts/cpu-benchmark.sh, uploaded via heredoc like dax's):single:dd if=/dev/zero bs=1M count=4096 | sha256sum— per-core speedmulti:nproccopies of that pipe in parallel (2 GiB each) — whole-box throughputBENCH_META(cpu model/arch/kernel/logical_cpus/memory),BENCH_PHASE,BENCH_BYTES,BENCH_DONE/BENCH_ERROR; fails rather than substituting a different tool whensha256sum/ddare missingPer-size grading: the config declares one
phasesarm per market size —small/medium/large/xlarge, matchingPLATFORM_SIZESin benchmarks-platform (lib/sandboxes/sizes.ts) — 3 iterations each.scoring.groupBy: 'phase'produces a composite per size per provider, so a buyer sees "this provider's large is a 90."benchmarks/sandbox/sizes.tsis rewritten asPROVIDER_SIZE_OPTIONS(provider → tier → options): providers taking raw cpu/memory params scale to all four tiers; template/preset-locked providers (e2b, superserve, novita, codesandbox, arker, declaw, sail, gravixlayer, createos, northflank, upstash) only grade the tiers they can provision — a missing tier fails assize_not_supportedinstead of substituting a wrong-size box.STANDARD_RESOURCE_OPTIONSis now the derived xlarge layer, byte-identical to the old map, so dax is unaffected (verified).Scoring:
multiCoreMBps60% (floor 500 / ceiling 20000) +singleCoreMBps40% (floor 50 / ceiling 3000), per tier × success rate. Create/destroy are timed but unscored steps.Files:
cpu.bench.ts,scripts/cpu-benchmark.sh,sizes.tsrework,cpu.mdtech doc, CI workflow (weekly Friday cron +provider/iterations/dry_rundispatch, ~35-provider matrix, same vault pattern as dax),bench:sandbox:cpuscript, and a one-line dax refactor to import from sizes.ts.Verified:
tsc --noEmitclean; real e2b dry run — small/medium/large fail assize_not_supported(e2b is template-locked), xlarge succeeds: 213.8 MB/s single, 1915.5 MB/s all-core, vcpus=8,phase: "xlarge"tagged on the record.Link to Devin session: https://app.devin.ai/sessions/08621fc28d124168920f539e8f9eb771
Open in Devin Desktop: https://app.devin.ai/desktop/session/08621fc28d124168920f539e8f9eb771?variant=devin
Requested by: @HeyGarrison