Repository navigation
eval: §6.3 instant-RQE rerun on CloudLab (planning times from one node type) - #221
Merged
Merged
Conversation
The 8 classic and all tables' 95% runs, rerun on clnode109/155/167/178 (the multi-grouping runs' nodes, each running its share serially), so every planning time comes from one node type. Every cost, latency, frontier point and SLA result matches #220's desktop run exactly; only planning times change (CloudLab is 2-2.7x slower). Planning-time figure and table updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- scripts/plot_sec63_paper.py: the paper's cost bars (replacing Table 5), frontier (without the no-roll-up line) and roll-up ablation, from the committed results, into autosketch-vs-asap-hydra/paper/. - scripts/export_case_studies.py and results/case-studies-6.3.1/: the performance-monitoring and anomaly-detection workloads, every method's plan (chosen kept), and a README of what each method deploys. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…stacked 3.4 in column version), larger fonts Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r bar (ASAP's cost, each baseline's multiple), no overlaps Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…is titles; ASAP's cost on its bar, baseline multiples inside theirs, darker bar colors Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… LiteOlly, LiteOllyPerQuery, AutoSketch baseline; W-RangeHeavy/W-RollupHeavy/W-Mixed (the three-panel version kept as sec63_cost_bars_wide) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…(sec63_cost_bars_fargate) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
milindsrivastava1997
force-pushed
the
eval/s63-cloudlab-planning
branch
from
October 26, 2026 18:21
03772df to
1a546bb
Compare
milindsrivastava1997
changed the base branch from
eval/s63-rerun-instant-scrape
to
main
October 26, 2026 18:22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #220, as asked in Slack: the 8 changed tables' 95% runs, rerun on CloudLab.
Sanity check of #220: costs don't depend on the machine.
What changes: only planning times. CloudLab is 2–2.7× slower than the desktop.
Now every planning time in
autosketch-vs-asap-hydracomes from the same node type as the multi-grouping rows. The planning-time figure and table,summary_synthetic.mdand the README provenance are updated. AutoSketch's benchmark times are reused (no new probes).Paper figures and case studies (c165995)
scripts/plot_sec63_paper.pywrites the §6.3 figures toautosketch-vs-asap-hydra/paper/(PDF + PNG), reading the committed results:sec63_cost_barsreplaces Table 5: each method's absolute cost at AutoSketch's latency, with each baseline's multiple of ASAP's;sec63_frontier, without the no-roll-up line;sec63_rollups.scripts/export_case_studies.pyandresults/case-studies-6.3.1/hold the §6.3.1 workloads (performance monitoring, anomaly detection): tables, every method's plan with the per-RQE choices kept, and a README of what each method deploys.🤖 Generated with Claude Code