Skip to content

Batch support for Solid Queue - #2

Open
jpcamara wants to merge 12 commits into
mainfrom
batches
Open

Batch support for Solid Queue#2
jpcamara wants to merge 12 commits into
mainfrom
batches

Conversation

@jpcamara

@jpcamara jpcamara commented Jul 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Batch support for Solid Queue in Mission Control, built fresh against current main and solid_queue's batch-review-fixes branch (rails/solid_queue#142 plus the review-fix series).

What's here

Batches follow the recurring-tasks architecture end to end:

  • Adapter capability: supports_batches? (false by default) with batches / find_batch returning attribute hashes; SolidQueueExt::Batches implements them straight off SolidQueue::Batch — symbol statuses, total/completed/failed/pending counters, progress_percentage, metadata, timestamps. Index lists the most recent 100.
  • Job filtering: JobsRelation gains batch_id, filtered through the job association on the executions path and directly on the finished path, so batch.jobs works across statuses.
  • UI: a capability-gated Batches tab; index with status tags, progress and failed counts; a show page with progress bar, linked counters and the batch's jobs. The job list is status-aware via the standard per-status partials — failed batches default to their failed jobs with full error details and retry/discard actions, finished batches to finished, running to pending, switchable via jobs_status (same convention as the jobs index).
  • Wiring: server concern, ResourceNotFound → batches index redirect, routes, dummy schema + BatchCallbackJob for exercising callbacks.

How it was tested

  • 6 controller tests (list, details + job list, progress with failures, failed-jobs view with error details, jobs_status switching, not-found redirect)
  • Exercised with real workers: supervisor + dispatcher against the dummy app; batches completed, failed and fired callbacks through the genuine claim/perform flow. A worker process died mid-run (sqlite fork-safety artifact of runner-launch) and the batch correctly recorded the claimed jobs as failures — crash-failover accounting verified incidentally.
  • Verified in a real browser (index, running/failed/completed batch pages, retry/discard actions rendered).
  • This exercise surfaced two solid_queue-side mixed-adapter leaks (EmptyJob and callback jobs enqueuing via the app-default adapter), fixed on batch-review-fixes (2459b29) and re-verified end to end from this branch.

Known/out of scope

  • Workers controller tests fail with solid_queue 1.4 independently of this change (process registration changed since the 1.0.1 pin) — needs a separate fix.
  • Batch index pagination (currently recent-100), and richer filtering, left for follow-up.

🤖 Generated with Claude Code

jpcamara and others added 4 commits July 23, 2026 09:32
@jpcamara @claude
Add batch support for Solid Queue
7fb3730 
Batches surface in the UI the way recurring tasks do: an adapter
capability (supports_batches?) with batches/find_batch returning
attribute hashes, a Server concern wrapping them in
MissionControl::Jobs::Batch models, an index of recent batches and a
show page with status, progress, counters, metadata and the batch's
jobs, gated by a navigation tab that only appears when the adapter
supports batches.

Jobs can be filtered by batch: JobsRelation gains a batch_id property,
filtered through the job association on the executions path and
directly on the finished-jobs path.

Built against solid_queue's batch-poc branch (via path: ../solid_queue
for development): symbol statuses, total/completed/failed/pending
counters and progress_percentage come straight from SolidQueue::Batch.

Note: the workers controller tests fail with solid_queue 1.4
independently of this change (process registration changed since the
1.0.1 pin); to be addressed separately.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P39PSB74nP3a8B2F7Z5xvz
@jpcamara @claude
Disable HTTP basic auth in the dummy development environment
793e32c 
No credentials are configured for the dummy app, so every dev-server
request 401s otherwise.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P39PSB74nP3a8B2F7Z5xvz
@jpcamara @claude
Show a batch's failed jobs with the standard failed-jobs treatment
91bc4ca 
The batch page's job list is now status-aware, reusing the per-status
job partials: failed batches default to their failed jobs with error
details and retry/discard actions, finished batches to their finished
jobs, running batches to pending. The progress counters link to each
list via the jobs_status param, following the jobs index convention.

Exercised end to end with real workers and a dispatcher against
solid_queue's batch-review-fixes branch: batches completed, failed and
fired callbacks through the genuine claim/perform flow, including
correct failure accounting when a worker process died mid-batch. Adds
BatchCallbackJob to the dummy app for exercising callbacks.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P39PSB74nP3a8B2F7Z5xvz
@jpcamara @claude
Bundle solid_queue from the batch-review-fixes branch
3fbb0b1 
Git-sourced so the branch bundles anywhere; switch to a path source when
co-developing against a local solid_queue checkout.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P39PSB74nP3a8B2F7Z5xvz
@cursor

cursor Bot commented Jul 25, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

jpcamara and others added 5 commits July 29, 2026 07:53
@jpcamara @claude
Re-pin solid_queue to the rebased batch-review-fixes
f62478f 
Now based on batch-poc with rails/solid_queue main merged forward,
picking up the Rails-main stopping? compatibility fix among 19 upstream
commits.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P39PSB74nP3a8B2F7Z5xvz
@jpcamara @claude
Pin minitest below 6 for railties 7.2 compatibility
0f9d95b 
bundle update pulled minitest 6, whose Runnable#run arity breaks
railties 7.2's line filtering before any test executes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P39PSB74nP3a8B2F7Z5xvz
@jpcamara @claude
Fix batch review findings and paginate the batches index
0c98dcf 
supports_batches? now verifies the Solid Queue batch models, API, and
schema instead of always advertising support. Batch pages expose
pending, in-progress, blocked, and scheduled jobs separately so
unfinished work is visible, with an unfinished batch defaulting to its
first non-empty status. Pagination preserves the selected jobs_status,
find_by_id honors the batch filter, and the batches index loads each
batch's counters in a single statement snapshot instead of five
queries per batch.

The batches index itself is now paginated like the jobs index, backed
by a BatchesRelation the Page understands (the adapter takes offset
and limit and exposes batches_count), and shows an empty-state notice
instead of a bare table when there are no batches.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jpcamara @claude
Filter the batches index by finished and unfinished
7757064 
Tabs on the batches index narrow the list to finished or unfinished
batches, threaded through the adapter as a status argument on batches
and batches_count so the database does the filtering, and preserved
across pagination. A filtered page with no matches gets its own empty
notice so the tabs remain reachable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jpcamara @claude
Add a failed tab to the batches index filter
1106699 
Failed batches have also finished, so they appear under both tabs; the
failed tab answers the operator question directly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jpcamara jpcamara mentioned this pull request Jul 30, 2026
Open
jpcamara and others added 2 commits July 31, 2026 10:13
@jpcamara @cursoragent
Make the batches index usable at multi-million-row scale.
bb3585b 
Cap batches_count like job counts, skip live job-count subqueries for
finished/failed listings, short-circuit those subqueries for finished
rows on mixed listings, and default the index to unfinished batches.

Co-authored-by: Cursor <cursoragent@cursor.com>
@jpcamara @claude
Align the batches feature with project conventions
9103efd 
Sibling-file comparison pass: constants before the private section, helper_method
beside its helpers, relation locals as private methods, a modifier_for_batch_status
helper mirroring modifier_for_status, plain markup instead of content_tag, batch
cleanup consolidated into delete_solid_queue_data, and the adapter batches section
moved after workers to match the feature's ordering everywhere else.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P39PSB74nP3a8B2F7Z5xvz
@jpcamara @claude
Match the batches relation and status validation to their siblings
318839d 
Rework BatchesRelation to the workers_relation shape (clone_with,
ALL_BATCHES_LIMIT, count clamped into the pagination window with the
internal-limit Float::INFINITY cap preserved), validate jobs_status via
supported_job_statuses like the discards controller, drop the unused
dummy batch_callback_job, and restore the dummy development config to
match upstream.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P39PSB74nP3a8B2F7Z5xvz
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jpcamara