Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions docs/agent-guides/STATE-PATTERNS.md
Original file line number Diff line number Diff line change
Expand Up @@ -510,3 +510,27 @@ Each AI tab within a session:
| `agentError` | `AgentError?` | Per-tab error state |

**Model/effort resolution chain** (used at user-facing spawn time in `useInputProcessing` and `agentStore.processQueuedItem`): `tab.customModel ?? session.customModel ?? agentConfig.model`. The MainPanel model/effort pill writes to the active tab via `tabStore.setTabModel`/`setTabEffort` - only the Edit Agent modal mutates `session.customModel`/`customEffort`. Programmatic spawns (Auto Run batch, synopsis, Cue, group chat, fork/merge) intentionally read the session value only.

## Auto-Resume On Limit

When an agent pauses on a provider limit (a rate / token / credit limit - `isLimitError(err)` in `src/shared/types.ts`, true for `rate_limited` and `token_exhaustion`), Maestro can auto-resume it once the window reopens. The coordinator is a renderer singleton, `useAutoResumeCoordinator` (`src/renderer/hooks/agent/useAutoResumeCoordinator.ts`), mounted once in `App.tsx` beside the other agent listeners.

**Settings** (General tab → "Auto-Resume on Limit"; metadata in `settingsMetadata.ts`, defaults in `main/stores/defaults.ts`):

| Setting | Default | Meaning |
| ------------------------------ | ------- | ------------------------------------------------------------ |
| `autoResumeOnLimit` | `true` | Master toggle. Off = no timer, nothing scheduled. |
| `autoResumeCheckIntervalHours` | `2` | How often the coordinator probes every limit-paused session. |
| `autoResumeGiveUpDays` | `7` | Time-based give-up window measured from the first pause. |

**Lifecycle** - limit-pause → probe → resume:

1. **Pause** (Phase 2): `useAgentErrorListener` sets `agentError` + `agentErrorPaused: true` + `state: 'error'`, seeds `resumeAttemptCount`, and best-effort stamps `limitResetAt` (when the provider window reopens, via `agents:getLimitResetAt`).
2. **Probe** (on the interval; a kickoff tick fires ~10s after mount so a restart probes fast). The coordinator selects sessions matching `isLimitPausedSession` and past their `limitResetAt`, then calls `probeAvailability`:
- **Claude** (local): reads a freshly re-sampled usage snapshot and returns available only when both the session and weekly windows are below `LIMIT_THRESHOLD`. Missing/unauthenticated snapshot → stay paused.
- **All other providers** (and **SSH-backed Claude**): no trustworthy usage signal, so availability is **unknown** and the coordinator falls back to **resume-as-probe** - the resume attempt itself is the probe; if it re-hits the limit, Phase 2 re-pauses it and the next interval retries. (The usage sampler `maestro-p --status` runs locally only and does not honor `sessionSshRemoteConfig`, so it can't describe a remote account - see `claude-usage-sampler.ts`.)
3. **Resume**: dispatched by run kind. Spec-/goal-driven Auto Runs resolve the shared `errorResolution` promise both runners await (`resumeAfterError`); a standard query clears the error so the persisted `executionQueue` drains (re-firing any captured in-flight direct send). A green "Resumed" toast fires.

**Restart behavior**: a limit pause is the ONE error state that survives an app restart. `prepareSessionForPersistence` (`useDebouncedPersistence.ts`) and `restoreSession` (`useSessionRestoration.ts`) preserve `agentError`/`agentErrorPaused`/`agentErrorTabId`/`state: 'error'` only for limit pauses (every other error stays stripped). On a cold start the coordinator re-finds the session with no extra wiring and resumes the **agent conversation** (the agent re-spawns with its native `--resume <agentSessionId>` and the persisted queue drains). The in-memory **Auto Run / goal-run orchestration loop is NOT reconstructed** (`batchStore` is in-memory by design), so even a formerly Auto-Run session resumes via the standard queue-drain path - the agent continues from its own transcript, the loop controller does not resume.

**Give-up (time-based)**: the coordinator keeps probing on the normal interval the entire window - there is NO attempt-count cap (`resumeAttemptCount` is telemetry only). Only once `limitPausedAt + autoResumeGiveUpDays` elapses does it stop retrying that session, leave it paused, and fire ONE distinct orange "Auto-resume stopped" toast. The window anchor survives a resume-then-re-hit (so "N days of repeated limits" actually elapses), and resets on any successful resume or manual clear (`clearAgentError`) so a later limit starts fresh. Any manual recovery action drops the session from consideration (its selector no longer matches).
12 changes: 12 additions & 0 deletions docs/autorun-playbooks.md
Original file line number Diff line number Diff line change
Expand Up @@ -280,6 +280,18 @@ Click the **Stop** button at any time. The runner will:
- Preserve all completed work
- Allow you to resume later by clicking Run again

## Auto-Resume on Limit

If an agent pauses mid-run because it hit a provider limit (a rate, token, or credit limit), Maestro can pick the run back up on its own once the window reopens - so you can queue a batch of work, walk away, and come back to it finished. Enable it in **Settings → General → Auto-Resume on Limit**. Three settings drive it:

- **Auto-Resume on Limit** (on by default) - the master toggle.
- **Check interval** (default 2 hours) - how often Maestro re-checks each paused agent.
- **Give up after** (default 7 days) - if an agent is still stuck this long after the first pause, Maestro stops retrying it, leaves it paused, and posts a one-time notice so you can resume manually.

How it decides to resume: for Claude it reads your actual plan usage and only resumes when credits are genuinely available again; for every other provider (and Claude on an SSH remote) it simply retries on the interval - if the limit is still in force the agent re-pauses and the next check tries again. Probing is cheap, so it keeps trying the whole window.

This survives a full app restart. If you reboot while an agent is limit-paused, Maestro restores the pause and resumes the **agent's conversation** (it continues from its own transcript) and drains any work you had queued. One caveat: the Auto Run / Goal-Driven **loop controller** does not survive a restart - the agent session and its queued messages resume, but the orchestration loop that was stepping through your document does not pick back up automatically. Manually resolving the error, or manually resuming or stopping the agent, always takes precedence and cancels auto-resume for that agent.

## Halt Marker (Agent Early Exit)

Sometimes the agent itself discovers that the rest of the playbook cannot meaningfully proceed - a missing dependency, a broken precondition, an ambiguous spec it cannot resolve, or a destructive change it refuses to make. In that case the agent can abort the entire run by writing a halt marker into the current document:
Expand Down
66 changes: 66 additions & 0 deletions src/__tests__/main/agents/limitResetEstimator.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
import { describe, it, expect, vi, beforeEach } from 'vitest';

// Mock the usage store so the estimator can be exercised without electron-store.
const { mockGetSnapshot } = vi.hoisted(() => ({ mockGetSnapshot: vi.fn() }));
vi.mock('../../../main/stores/claudeUsageStore', () => ({
getSnapshot: (...args: unknown[]) => mockGetSnapshot(...args),
resolveConfigDirKey: (env: NodeJS.ProcessEnv) => env.CLAUDE_CONFIG_DIR ?? '/home/u/.claude',
}));

import { getLimitResetAt } from '../../../main/agents/limitResetEstimator';

/** Build a minimal usage snapshot with the two reset windows the estimator reads. */
function snap(sessionResetsAt: string, weekResetsAt: string) {
return {
sampledAt: new Date().toISOString(),
configDirKey: '/home/u/.claude',
session: { percent: 100, resetsAt: sessionResetsAt },
weekAllModels: { percent: 50, resetsAt: weekResetsAt },
weekSonnetOnly: { percent: 0, resetsAt: weekResetsAt },
};
}

describe('getLimitResetAt', () => {
beforeEach(() => {
mockGetSnapshot.mockReset();
});

it('returns the nearest FUTURE reset for Claude', () => {
const now = Date.now();
const soon = new Date(now + 60_000).toISOString();
const later = new Date(now + 3_600_000).toISOString();
mockGetSnapshot.mockReturnValue(snap(soon, later));

expect(getLimitResetAt('claude-code')).toBe(new Date(soon).getTime());
});

it('skips a past reset and returns the future one', () => {
const now = Date.now();
const past = new Date(now - 60_000).toISOString();
const future = new Date(now + 120_000).toISOString();
mockGetSnapshot.mockReturnValue(snap(past, future));

expect(getLimitResetAt('claude-code')).toBe(new Date(future).getTime());
});

it('returns undefined when every reset window is already in the past (stale/expired)', () => {
const now = Date.now();
const past1 = new Date(now - 120_000).toISOString();
const past2 = new Date(now - 60_000).toISOString();
mockGetSnapshot.mockReturnValue(snap(past1, past2));

expect(getLimitResetAt('claude-code')).toBeUndefined();
});

it('returns undefined when no snapshot is cached', () => {
mockGetSnapshot.mockReturnValue(null);

expect(getLimitResetAt('claude-code')).toBeUndefined();
});

it('returns undefined for non-Claude providers without touching the store', () => {
expect(getLimitResetAt('codex')).toBeUndefined();
expect(getLimitResetAt('opencode')).toBeUndefined();
expect(mockGetSnapshot).not.toHaveBeenCalled();
});
});
1 change: 1 addition & 0 deletions src/__tests__/main/ipc/handlers/agents.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,7 @@ describe('agents IPC handlers', () => {
'agents:getRemoteMaestroPAvailable',
'agents:getClaudeUsageSnapshots',
'agents:getClaudeUsageAccountKeys',
'agents:getLimitResetAt',
'claude:usage:refresh-all',
'agents:getCodexUsageSnapshots',
'agents:getCodexUsageAccountKeys',
Expand Down
12 changes: 12 additions & 0 deletions src/__tests__/main/stores/defaults.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,18 @@ describe('stores/defaults', () => {
it('should have null installationId by default', () => {
expect(SETTINGS_DEFAULTS.installationId).toBeNull();
});

it('should enable autoResumeOnLimit by default', () => {
expect(SETTINGS_DEFAULTS.autoResumeOnLimit).toBe(true);
});

it('should default autoResumeCheckIntervalHours to 2', () => {
expect(SETTINGS_DEFAULTS.autoResumeCheckIntervalHours).toBe(2);
});

it('should default autoResumeGiveUpDays to 7', () => {
expect(SETTINGS_DEFAULTS.autoResumeGiveUpDays).toBe(7);
});
});

describe('SESSIONS_DEFAULTS', () => {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,10 @@ describe('searchableSettings', () => {
// General tab
['Auto Run Inactivity Timeout', 'general-autorun-inactivity-timeout'],
['refactor', 'general-autorun-inactivity-timeout'],
['resume paused', 'general-auto-resume'],
['rate limit', 'general-auto-resume'],
['quota', 'general-auto-resume'],
['give up', 'general-auto-resume-interval'],
['forced parallel execution', 'general-input-behavior'],
['shift+enter', 'general-input-behavior'],
['prompt composer', 'general-input-behavior'],
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import { describe, it, expect, vi, beforeEach } from 'vitest';
import { renderHook } from '@testing-library/react';
import { renderHook, waitFor } from '@testing-library/react';
import { useAgentErrorListener } from '../../../../../renderer/hooks/agent/internal/useAgentErrorListener';
import { useSessionStore } from '../../../../../renderer/stores/sessionStore';
import { useModalStore } from '../../../../../renderer/stores/modalStore';
Expand Down Expand Up @@ -191,6 +191,44 @@ describe('useAgentErrorListener', () => {
expect(deps.activeHiddenToolRef.current.has('sess-1:tab-1')).toBe(false);
});

it('seeds auto-resume metadata and stashes the last prompt on a limit error', async () => {
const getLimitResetAt = vi.fn().mockResolvedValue(1700000123456);
(window as any).maestro.agents = { getLimitResetAt };

const userLog = {
id: 'log-user',
timestamp: 100,
source: 'user' as const,
text: 'do the rate-limited thing',
};
const tab = createMockAITab({ id: 'tab-1', logs: [userLog] });
const session = createMockSession({ id: 'sess-1', aiTabs: [tab], activeTabId: 'tab-1' });
useSessionStore.setState({ sessions: [session] } as any);

renderHook(() => useAgentErrorListener(makeDeps()));
handler!('sess-1-ai-tab-1', { ...baseError, type: 'rate_limited', message: 'usage limit' });

// Synchronous pause: error stamped, paused, retry counter seeded to 0.
const paused = useSessionStore.getState().sessions[0];
expect(paused.state).toBe('error');
expect(paused.agentErrorPaused).toBe(true);
expect(paused.agentError?.type).toBe('rate_limited');
expect(paused.agentError?.resumeAttemptCount).toBe(0);

// The captured prompt rides along on the error log so Phase 3 can re-fire it.
const errorLog = paused.aiTabs[0].logs.find((l) => l.source === 'error');
expect(errorLog?.recoveryAction).toEqual({
lastUserPrompt: 'do the rate-limited thing',
tabId: 'tab-1',
});

// Best-effort reset estimate is patched in asynchronously.
expect(getLimitResetAt).toHaveBeenCalledWith('claude-code');
await waitFor(() => {
expect(useSessionStore.getState().sessions[0].agentError?.limitResetAt).toBe(1700000123456);
});
});

it('skips synopsis-process errors', () => {
const tab = createMockAITab({ id: 'tab-1' });
const session = createMockSession({ id: 'sess-1', aiTabs: [tab] });
Expand Down
Loading