Repository navigation
Register ec2-firecrab as a direct-mode sandbox provider - #542
Merged
Merged
Conversation
Co-Authored-By: Noah Kiser <noah@computesdk.com>
Contributor
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
…phost Co-Authored-By: Noah Kiser <noah@computesdk.com>
Co-Authored-By: Noah Kiser <noah@computesdk.com>
Co-Authored-By: Noah Kiser <noah@computesdk.com>
Co-Authored-By: Noah Kiser <noah@computesdk.com>
The provider stays in the matrix so a manual dispatch can select it, but SHOULD_RUN now requires workflow_dispatch + provider='ec2-firecrab' for its leg — it never runs on push/schedule, and a blank-provider 'all' dispatch skips it too (its host pool is spun up per manual run). Also adds EC2_FIRECRAB_* to the vault secret regex so a dispatched leg actually receives its env. Co-Authored-By: Noah Kiser <noah@computesdk.com>
benchmarks-daily builds its scheduled matrix by iterating this providers list; the flag keeps dispatch-only providers out of those legs. Co-Authored-By: Noah Kiser <noah@computesdk.com>
Co-Authored-By: Noah Kiser <noah@computesdk.com>
kisernl
marked this pull request as ready for review
October 23, 2026 19:27
Contributor License AgreementAll contributors are covered by a CLA. |
![superagent-security[bot]](https://avatars.githubusercontent.com/in/3287076?s=60&v=4){kind=small}
The shared regex exported the host SSH private key into every provider leg's env while every provider package runs in-process. Matches the sandbox-capabilities.yml pattern of per-provider vault scoping. Also adds the host_count pool steps (pool creds step is already gated to the firecrab leg via FIRECRAB_POOL). Co-Authored-By: Noah Kiser <noah@computesdk.com>
Co-Authored-By: Noah Kiser <noah@computesdk.com>
Contributor
Sandbox Benchmark ResultsBurst
View full run · SVGs available as build artifacts |
Sandbox Dax Benchmark Results
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Registers
ec2-firecrabas a traditional direct-mode sandbox provider so the sandbox TTI bench can run against Firecracker microVMs on our firecrab-on-EC2 hosts — manual dispatch only, never on push/schedule or an "all providers" dispatch.benchmarks/sandbox/providers.ts— new direct-mode entry aftere2b:createComputebuildsec2Firecrab({sshTargets, sshIdentityPem, microNetwork, template, cpus:1, memoryMib:256, diskGb:2, egressPolicy:'internet'})from env — sized for thenode -vTTI probe. One SSH target list + key is the whole credential — the adapter self-opens an SSH local-forward per host for the loopback API and derives the guest-exec jumphost from the same credentials, so no tunnel step is needed in the workflow.microNetworktakes a network name (lab), resolved to its UUID via the API.destroyTimeoutMs: 90_000sincestopis ~30s graceful ACPI.manualOnly: truekeeps it out of benchmarks-daily's derived matrix (computesdk/benchmarks-daily#45)..github/workflows/sandbox-tti-benchmarks.yml—ec2-firecrabis active in the matrix but its leg is gated inSHOULD_RUN: it runs only onworkflow_dispatchwithprovider: ec2-firecrab. Push, Friday cron, and blank-provider "all" dispatches all skip it. Newhost_countinput:0(default) uses the vault's standingEC2_FIRECRAB_SSH_TARGETS;>0launches a fresh pool in-job — loads AWS creds +FIRECRAB_*launch config from vault, checks outcomputesdk/ec2-firecrab, runshost/pool-up.sh N(targets flow to the bench viaPOOL_SSH_TARGETS), andpool-down.shterminates underif: always().EC2_FIRECRAB_*is loaded only in the firecrab leg — the host SSH key never reaches other providers' env.benchmarks/scripts/provider-vars.json— vault declaration for the suite:EC2_FIRECRAB_*plus the pool vars (AWS_*,FIRECRAB_*,FIRECRAB_REPO_TOKEN).package.json—@computesdk/ec2-firecrab: ^1.0.0(published via computesdk#1004 → #1005).Vault vars needed
Static (always):
EC2_FIRECRAB_SSH_KEY(PEM for the hosts'ubuntuuser),EC2_FIRECRAB_MICRO_NETWORK(lab),EC2_FIRECRAB_TEMPLATE(catthehacker-ubuntu-act-latest),EC2_FIRECRAB_SSH_TARGETS(a standing host likeubuntu@x.y.z.w— used whenhost_count=0).Pool runs additionally need:
AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY,AWS_REGION,FIRECRAB_AMI_ID,FIRECRAB_KEY_NAME,FIRECRAB_SG_ID,FIRECRAB_SUBNET_ID,FIRECRAB_REPO_TOKEN(checkout for the private ec2-firecrab repo).To run it
Standing host: vault
EC2_FIRECRAB_SSH_TARGETS=ubuntu@<ip>→ Actions → Sandbox TTI →provider: ec2-firecrab,host_count: 0.Ephemeral pool: dispatch with
provider: ec2-firecrab,host_count: N— everything else is automatic.Verified live on the spike host (burst-2 smoke through the real
tti.bench.ts): 2/2 concurrent sandboxes, TTI ~36.9s each.Link to Devin session: https://app.devin.ai/sessions/6a5fd0700bcf437c8f189d45b35e4cd4
Open in Devin Desktop: https://app.devin.ai/desktop/session/6a5fd0700bcf437c8f189d45b35e4cd4?variant=devin
Requested by: @kisernl