fix(antigravity): resume pending executions via empty-input continue (#190)#330
Open
joycel-github wants to merge 427 commits into
Open
fix(antigravity): resume pending executions via empty-input continue (#190)#330joycel-github wants to merge 427 commits into
joycel-github wants to merge 427 commits into
Conversation
…util (#242) Refactor: Move newControllerFromConfig to shared package internal/cliutil
This PR optimizes the building of execution traces in cmd/ax/trace.go by iterating over the already ordered execIDs list and looking up events in a map. This avoids a redundant sorting step and reduces complexity from O(N^2) to O(N). A unit test was also added to verify order preservation.
This PR adds a new conversation_id field in AgentMessage. Also add the conversationID to the signature of Agent.Connect and Executor.Exec so that we can: 1. propagate conversationID from controller to sub-agent, like remote agent, a2a agent 2. propagate conversationID to substrate worker The conversationID is unused by anything yet. It will be picked up by a2a agents in follow up PRs, and possibly substrate agent as well.
bytes allow arbitary configuration encoding format, e.g. JSON. Rename AgentStart.config to AgentStart.agent_config for consistency.
We allow any encoding for sub agent configuration after google-gemini/ax#248. This change is removing the proto encoding requirement for the Gemini and the test agents.
AX is primarily a server project. The Serve section is accidentally burried undernath other sections.
- ConversationId is added in the reply agent message of the remote agent example. Without conversation id the message gets rejected. (The ate agent and k8s sandbox agent have it added in the previous PR).
Allows us to more easily track and update experimental features.
Add content types for different media types. These new types will be used by the A2A bridge. Mirror the exact content types from Interactions with a slight difference: - Added two MimeType in DocumentContent for A2A bridge: `TYPE_JSON`, `TYPE_PYTHON`. The current enum in Interactions has a small set of values. We may update this in the future to make them consistent.
This PR moves the internal_only boolean from `AgentOutputs` to `Message`. 1. Internal-only messages are persisted in event log as `Message`. The `internal_only` bit is not persisted. 2. I'd like to store some execution metadata as an internal event, e.g. conversation id, task id, etc. 3. When resume, these internal metadata is read by AX controller, but I want to filter them out of the history that we feed into planner agents or custom agents since these messages are of no meaning to them. Thus, moving it to Message level for log filtering.
# Refactor Antigravity Planner to use WebSocket and AgentMessage
This PR refactors the Antigravity Planner to use the standard
`AgentMessage` protocol for communication between the Go controller and
the Python planner server. It also changes the protocol from HTTP/JSON
to Websocket to support tool call. AX agents are invoked as Tools.
Python side CL cl/904297174 (need to improve the code quality)
Next up:
* clean up the logic for HITL.
* get cl/906138865 into google3 and further github once antigravity SDK
is released
## E2E Test Result: Antigravity Planner with Tool Call
### Commands Run
* Start remote agent: `go run ./examples/remote_agent`
* Start Python planner server:
`/google/src/cloud/lhuan/jetski-ax/google3/blaze-bin/experimental/users/lhuan/jetski_planner_2`
* Start AX server: `./bin/ax serve --config ax.yaml`
* Run client: `./bin/ax exec --server localhost:8496 --input "Use the
'secret_code' agent to get the secret code
### Logs and Outputs
#### Python Server Log
```
[AX Server] Loaded subagents: [{'id': 'secret_code', 'description': 'Retrieves secret codes and performs sensitive text transformations that only this agent can do.'}]
[AX Server] Generated tool for subagent: secret_code
Starting connection to harness and creating conversation...
Sending prompt to harness: Use the 'secret_code' agent to get the secret code
Waiting for steps...
[AXToolRunner] Intercepted tool call: secret_code
[AXToolRunner] Raw response from Go: {"outputs":{"messages":[{"role":"assistant", "content":{"text":{"text":"secret code 111 from ax"}}}]}}
[AXToolRunner] Received result from AX: secret code 111 from ax
Final Response: The secret code for user 'sundar' is `secret code 111 from ax`.
```
#### AX Server Log
```
2026/04/28 07:27:15 [AX] Connecting to Antigravity WebSocket for execution b8549e17-483a-4016-a06b-f81cd1a579dc
2026/04/28 07:27:22 [AX] Handling tool call: secret_code
2026/04/28 07:27:22 [AX] Captured 1 tool outputs for secret_code
2026/04/28 07:27:24 [AX] Conversation completed.
```
#### Client Output
```
⏺ Use the 'secret_code' agent to get the secret code
The secret code for user 'sundar' is `secret code 111 from ax`.
```
TAG=agy
CONV=4d4e96b3-966e-488c-8de2-31a1c6bb1978
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Also add a TODO about the current bug and why we are not yet fixing it.
Gemini planner is overwhelmingly choosing these skills. New comers to the project will fork the repo and observe this behavior. Also most people will bring skills through the global directories or SKILLS_DIR.
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
- **Add unit and integration tests for gemini_planner** - **Handle missing skills directory gracefully in NewSkillsTool**
…te caller (#261) Moved antigravity_planner.go to internal/experimental/planner and updated the caller in cliutil.go to use the new location. Also broke an import cycle by removing unused Registry reference.
## Summary
This branch reorganizes the codebase by moving Gemini-related test files
to their appropriate package and resolves several circular dependencies
that arose from the file relocation. It also ensures that the project
builds, lints, and passes all tests (including integration tests).
## Motivation
* **Cohesion**: Test files were located in `internal/controller` but
were testing functionality belonging to `internal/gemini`. Moving them
improves project structure.
* **Decoupling**: The `gemini` package was tightly coupled to
`controller` via concrete types, causing import cycles when files were
moved.
## Changes
### File Relocations
* Moved the following files from `internal/controller` to
`internal/gemini`:
* `gemini_planner_integration_test.go`
* `gemini_planner_test.go`
* `gemini_test.go`
### Refactoring & Cycle Breaking
* **`internal/agent`**:
* Moved `AgentInfo` struct from `internal/controller` to
`internal/agent/agent.go` to allow sharing without circular
dependencies.
* **`internal/gemini`**:
* Defined `AgentRegistry` interface in `gemini_planner.go` to decouple
the planner from `controller.Registry`.
* Updated package names in moved test files to `gemini`.
* Converted `gemini_planner_integration_test.go` to an external test
(`package gemini_test`) to resolve a test-time import cycle.
* **`internal/controller`**:
* Updated `controller.go` and `registry.go` to use `agent.AgentInfo` and
import `gemini` symbols correctly.
* **`internal/cliutil`**:
* Updated `cliutil.go` to reference `gemini.NewGeminiPlannerAgent`
instead of the old `controller` path.
## Verification Results
* **Build**: `go build ./...` completed successfully.
* **Lint**: `go vet ./...` reported no issues.
* **Tests**: All unit tests passed.
* **Integration Tests**: `TestIntegrationGeminiPlanner` passed
successfully using the provided `GEMINI_API_KEY` and the `integration`
build tag.
This PR introduces a standalone `DockerAgent` example and fixes an infinite loop bug in the planner. It also addresses several code review comments for robustness and cleanups.
The 17 tests in python/antigravity/harness_server_test.py have never run in CI. .github/workflows/go.yml only runs 'go test ./...' and the Makefile 'test' target likewise only runs Go. That let PR #244 land with 2 stale assertions unnoticed (fixed in #291). Adds: - .github/workflows/python.yml -- dedicated Python workflow (kept separate from the Go-only go.yml so each can evolve independently). - Makefile 'test-python' target for local parity. Test deps are inlined in the workflow rather than added as a separate requirements-test.txt -- the diff is 2 packages ('pytest>=7.0', 'pytest-timeout>=2.0'), not worth its own file. Version floors match the upstream google-antigravity SDK's dev extras so this doesn't drift as pytest evolves. --timeout=30 --timeout-method=thread guards against hung gRPC servers so a stuck test can't eat the whole workflow budget.
* antigravity: parse and validate request harness_config Parse HarnessStart.harness_config (JSON-in-bytes) and overlay it onto the server's default LocalAgentConfig before each turn. - Blocks a request from overriding AX-owned persistence fields (save_dir, conversation_id) via _AX_MANAGED_CONFIG_FIELDS; these are injected last so a request can't redirect trajectory storage. - Reconstructs the config (not model_copy) so the SDK re-validates the overlaid values and surfaces its own error; invalid config fails the turn with INVALID_ARGUMENT instead of crashing. - save_dir derives from the injected state_dir (#292) as state_dir / conversation_id. * antigravity: address harness_config review comments - Inline _parse_harness_config into _build_config_for so the parsed dict stays a local intermediate; the method's only boundary type is the validated LocalAgentConfig (no dict[str, object] across a helper boundary). Addresses the "introduce a type for the config" comment. - Make per-conv save_dir test assertions portable: compare against str(tmp_path / conversation_id) instead of hardcoding "/conv-1", and drop the now-vestigial Path.home monkeypatch (save_dir derives from the injected state_dir since #292). Addresses the Windows path separator comment. _AX_MANAGED_CONFIG_FIELDS keeps guarding both conversation_id and save_dir on purpose: harness_config is a separate client-controlled surface from the request's conversation_id, so blocking it prevents a request from redirecting another conversation's trajectory storage.
LocalAgentConfig uses pydantic's default extra="ignore", so an
unrecognized field in a request's harness_config -- e.g. a typo like
"system_instruction" for "system_instructions" -- was silently dropped
and the caller believed the override took effect when it did not.
Add _reject_disallowed_fields(overrides), which factors two key checks
into one helper and raises HarnessConfigError (mapped to
INVALID_ARGUMENT) naming the offending field(s):
- fields that come from outside harness_config (_NON_HARNESS_CONFIG_
FIELDS: conversation_id from the runtime request, save_dir from the
server's state_dir), and
- unknown top-level fields.
Validation is best-effort and top-level only: nested-key and value/type
validation is delegated to the SDK's own LocalAgentConfig validation at
construction time.
state_dir is an AGY SDK implementation detail (where trajectory / resume-cursor storage lives), not something AX users should configure. Remove the yaml surface for both built-in harnesses and derive the path internally from config.AXAssetsDir(). - config: drop StateDir from AntigravityHarnessConfig and AntigravityInteractionsHarnessConfig. - antigravity: add DefaultStateDir() -> ~/.ax/antigravity/conversations, mirroring antigravityinteractions.DefaultStateDir(). cliutil now passes this into antigravity.New instead of the yaml value. - cliutil: interactions harness always uses DefaultStateDir(); drop the yaml read + fallback. The internal APIs (antigravity.New's stateDir arg, the required AntigravityInteractionsConfig.StateDir field, and the Python sidecar's --state-dir default) are unchanged -- only the user-facing ax.yaml knob is removed. Addresses rakyll's follow-up on #292.
…300) Materialize agentskills.io skills from the Gemini Enterprise Skill Registry (Vertex AI v1beta1) into on-disk folders before the harness starts, so the built-in harnesses can use registry-hosted skills on the local `ax exec` / `ax serve` path. - internal/skills/geminienterprise: harness-agnostic package that reads config.SkillsConfig, drives the registry client (ListSkills / GetSkill / GetSkillRevision / skills:retrieve), safe-unzips payloads to <target_dir>/<skill-id>/, and reports what it wrote. First-wins on duplicate ids with a warning; fail-safe (a registry error never blocks harness startup). - config: top-level `skills.registries[]` (harness-agnostic -- each actor runs a single harness that consumes the materialized folder). Per-registry selection (skills / query / all), required target_dir, and a validated "exactly one selection mode" rule. Also wires the interactions harness's system_instruction from ax.yaml. - cliutil: materializes skills once, up front, at controller construction; for the interactions harness (no SKILLS_DIR concept) it appends a discovery pointer to the system instruction. Scope: local path only; the substrate/pod path does not yet read ax.yaml. Verified end-to-end against a live registry (by-id and by-query).
The exec example used `--harness coding`, but no "coding" harness exists; runHarness only accepts "antigravity" and "antigravity-interactions" and errors otherwise, so the command as written fails. Point the example at the real "antigravity" harness.
* Honor view_file line range and cap result size
execViewFile read the whole file and ignored the agent's StartLine/EndLine,
so viewing a large file returned its entire content as the tool result --
a ~900KB blob for a 3k-line CSV stalled the following turn.
Implement the Antigravity view_file contract (1-indexed inclusive line range
with slice-notation windowing) plus defensive caps:
- StartLine/EndLine window (neither=first N lines; start-only=next N forward;
end-only=previous N backward; both=precise range, capped to N).
- viewFileMaxLines / viewFileMaxBytes caps so a large file can never blob.
- ContentOffset honored as the read position within the windowed content.
The result payload is just {"content": ...}; the view_file result schema is
defined server-side, so no extra fields are added.
Adds intArg/intArgOK helpers and tests for windowing, byte cap, offset read,
and range honoring.
* view_file: UTF-8-safe byte cap + pagination metadata
Address review feedback on the byte cap:
- applyByteWindow backs the cut off to the last complete UTF-8 rune so a
multi-byte character straddling viewFileMaxBytes is never split (raw byte
slicing could produce invalid UTF-8, corrupting the last char and JSON
serialization).
- execViewFile returns the metadata the server needs to distinguish a
complete read from a paginated/byte-truncated one: content, start_line/
end_line (0-indexed inclusive served range), content_offset,
line_range_bytes (total bytes of the line range pre-byte-cap), and
num_lines/num_bytes. The server detects truncation via
content_offset+len(content) < line_range_bytes and prompts the model to
resume from that offset.
A paired server change reads these fields instead of hardcoding
start_line=0/end_line=last.
Adds tests for UTF-8 boundary capping, pagination metadata, and resume.
* docs(readme): refresh Antigravity setup, auth, and CLI usage Re-land the README refresh that was approved as #310 but never reached main (it merged into #309's branch, which was then squash-merged, orphaning this content). - Document the built-in Antigravity harness: local execution needs python3 and pip on PATH; AX auto-starts the Python sidecar and installs the pinned SDK dependencies on first use. - Promote Authentication to its own top-level section (Google AI Studio and Vertex AI / ADC). - Refresh Quick Start / Usage examples, document the --harness-config and --harness-config-json flags, and add a per-request configuration example. * docs(readme): use gemini-3.5-flash in per-request config example Update the per-request harness config example from gemini-2.5-pro to a current model, matching the model naming used in examples/skills.
…strate (#322) runAntigravityHarness (the `ax harness antigravity` path used by the substrate ActorTemplate) forked the Python sidecar with only --host/--port. The sidecar then fell back to its own default state dir, so on a substrate actor per-conversation save_dir landed on a path AX does not control, which breaks SDK-native resume across actor Run/Restore and snapshotting. Derive the trajectory dir via antigravity.DefaultStateDir() and pass it as --state-dir, mirroring the local path in antigravity.New and the interactions harness in runAntigravityInteractionsHarness (both own the path internally rather than taking it from ax.yaml/ActorTemplate). Fixes #321
google-antigravity 0.1.7's _connect_websocket retries both "localhost" and "127.0.0.1" when dialing localharness (local_connection.py:993), so the sidecar no longer needs to patch /etc/hosts for substrate/gVisor environments where localhost may not resolve. Ref #325
harness_server uses the caller-supplied conversation_id directly as a per-conversation storage directory name (save_dir = state_dir / conversation_id) before the Antigravity harness ever sees it, so a value like "../escape" could traverse outside state_dir. Reject an empty id or one containing a path separator or a "." / ".." component at the boundary in _run_turn, returning an INVALID_ARGUMENT end frame. This is intentionally only a path-safety guard, not the id-format contract. The Antigravity harness owns that: it validates the forwarded cascade_id (length and character set) per cl/948016352. Keeping the format rule in one place avoids the two layers drifting.
- CONTRIBUTING.md: drop 'make run-remote' (no such Makefile target) - .gitignore: remove orphaned entries (local/remote_agent, sandbox-router, tasklog, axepp, google-cloud-sdk, test-sandbox-config.yaml) - Makefile: add missing deps, clean-logs to .PHONY
joycel-github
force-pushed
the
resume-pending-execution
branch
3 times, most recently
from
July 17, 2026 23:55
004aa1e to
88f689d
Compare
…190) `ax exec --resume` on a STATE_PENDING conversation (the harness died mid-turn) failed with "no input messages queued for execution turn". The controller resumes by re-running the pending turn with no new input, but both the Go adapter and the Python sidecar rejected an empty payload. Empirically, the Antigravity SDK persists the interrupted turn's input in its save_dir DB and an empty chat("") continues that turn from the persisted state. So resume just needs the empty payload to flow through: - Go adapter (antigravity.go): drop the empty-queue guard in Run(). An empty queue only occurs on resume (the controller returns early for new turns with no input), so the empty start frame is forwarded. - Python sidecar (harness_server.py): treat empty HarnessStart.messages as a resume signal and call chat("") to continue the persisted conversation instead of returning STATE_FAILED. Verified end-to-end: `ax exec --resume` on a pending conversation now continues the interrupted turn to STATE_COMPLETED. Test: add test_run_turn_empty_messages_resumes_with_empty_prompt, which asserts empty messages resume with chat("") and complete rather than fail.
joycel-github
force-pushed
the
resume-pending-execution
branch
from
July 17, 2026 23:57
88f689d to
f73c69f
Compare
joycel-github
marked this pull request as ready for review
July 18, 2026 00:00
wjjclaud
approved these changes
Jul 18, 2026
rakyll
added a commit
that referenced
this pull request
Jul 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #190.
ax exec --resumeon aSTATE_PENDINGconversation (the harness died mid-turn) failed withno input messages queued for execution turn. The controller resumes by re-running the pending turn with no new input, but both the Go adapter and the Python sidecar rejected an empty payload.This change lets the empty resume flow through so the harness continues the interrupted turn from its own persisted conversation state:
internal/harness/antigravity/antigravity.go): drop the empty-queue guard inRun().python/antigravity/harness_server.py): drop the empty input guard callchat("")to continue the persisted conversation, instead of returningSTATE_FAILED.Why empty-input continue (not input replay)
Verified empirically against the Antigravity SDK : the SDK persists an interrupted turn's input in its
save_dirDB, and an emptychat("")continues that turn from persisted state. So AX does not need to reconstruct/replay inputs.This follows the general principle: AX passes through and lets the harness/SDK
own conversation state.
Scope
Only hndles resume/continue after an interrupted turn (the #190 case).
Any other resumption use cases including true mid-step crash recovery (continuing a half-finished turn without redoing work) still requires SDK checkpoint/reattach and is out of scope here and will be handled more systematically with next milestone
Manual E2E test case
TLDR Reproduces #190 end-to-end: interrupt a real turn mid-stream (kill the live
axharnesssidecar), thenax exec --resume. Before this change the resumefailed with
no input messages queued for execution turn; after it, theinterrupted turn is continued to
STATE_COMPLETED.More details below or http://shortn/_MeaAYOZkFY
Setup
1. Start a turn, then kill the harness mid-stream
The client stream fails with an EOF, and the event log is left
PENDING:2. Resume
./bin/ax exec --conversation demo190 --harness antigravity --resumeBefore this change:
After this change — the harness continues the interrupted turn (recalls the
secret and resumes the essay), and the conversation reaches
STATE_COMPLETED: