An LLM-powered agent harness for the ARC-AGI-3 interactive reasoning benchmark.
USAI stands for Understand, Simulate, Act, Improve -- uniting "us" with "AI" (pun intended).
The primary features of this harness are:
- Specialized agents that refine their work iteratively
- A python simulator that the agents write and execute in order to plan efficient batches of actions.
This agent completed ft09 (version 9ab2447a) in 76 actions, surpassing all previously published results and approaching the theoretical minimum of 75 actions (see below for recording, conversation log, and comparison table).
| Player | Actions |
|---|---|
| a hypothetical perfect player | 75 |
| USAI Agent (this project) | 76 |
| RGB-Agent | 78 |
| Arcgentica | 344 |
| Humans (2nd best human out of those tested by ARC) | 91 |
The agents were powered by Claude Opus 4.6 with adaptive thinking and high thinking effort. The game was completed at a cost of about $53.17 in API credits (excluding tax).
-
Two specialized agents work together to complete the game:
- Hypothesis-Refinement (HR) Agent -- analyzes frames, identifies objects, writes/updates a Python simulator, and maintains a full hypothesis about how the game works.
- Plan-Refinement (PR) Agent -- devises action sequences, tests them with the simulator, and submits plans for execution.
-
The naming of the agents is based on the fact that both agents use refinement loops. Each agent is invoked repeatedly until it is satisfied with the work of its previous invocation. For example, when the HR agent submits a hypothesis, a new instance of the HR agent is invoked (with a clean context window) to review and potentially refine the hypothesis. When the hypothesis is acceptable, the agent adopts it.
-
I was surprised to find that the agents would usually adopt the first hypothesis or plan that was submitted, rather than iterating several times. However, the iteration loops did trigger a few times, likely because the fresh context window between invocations allowed the agents to catch and correct their previous mistakes. Also, the agents were able to iterate their solutions after each new batch of actions was submitted, which likely contributed to their high performance.
The agents wrote, maintained, and used a simulator.py file (shared across all levels, validated after each new action, and re-validated after every modification to the simulator's code). Constant validation of the simulator was critical for grounding the AI agents in a verifiable source of truth (the actual frame pixels produced after each action). The simulator approach is not perfect and may break down with certain types of games, but it shows the power of grounding AI agents.
In order to allow the simulator to capture the dynamics of the actual games, it is important to consider whether the games are stateless or stateful. For the most part, the preview games exhibited stateless behavior, but games can sometimes be stateful (e.g. valid_action_count matters for the behavior of the health bar on ft09).
The harness focused on initial and final frames for each action. Transition frames (the ones occurring between the initial and final frames) were treated as reference only.
-
Scaffolding matters. This project demonstrates that harness/scaffolding approaches can meaningfully improve the raw capabilities of LLMs. The RGB-Agent project advocates for simple scaffolding with more freedom for the LLM. I also noticed the benefits of harness simplicity during my testing. It's very easy for an AI system to regress when adding more complexity. However, there appear to be additional performance improvements possible if the extra complexity is carefully considered before implementing.
-
Future LLMs may not require heavy scaffolding, but effective scaffolding could help them to initiate self-improvement loops earlier. Scaffolded AI systems could achieve self-improvement through:
- Trying new architectures, seeing what works well on smaller scales, self-applying those architectures, and validating for measurable improvements (similar to Karpathy's autoresearch). However, this can require expensive and time-consuming iterations.
- Analyzing and improving its own outputs, then retraining on the better outputs. This approach is also challenging due to subjectivity in rating response quality and accuracy, as well as the unreliable nature of current LLMs.
I believe option 2 is the most promising approach for the short-term, and option 1 is the most promising approach for the long-term. Either way, both approaches will likely benefit from well-designed scaffolding.
This project was coded almost entirely by AI (mostly using Opus 4.6 with thinking enabled). The code is very buggy and roughly implemented and does not perfectly match my original specifications. I intentionally chose not to prioritize a clean codebase since this was a research experiment. There is significant room for improvement, especially within the instruction files that are presented in context every time a new agent is invoked.
- Even when using Opus 4.6 (arguably the best LLM currently available), the agents still have very clear weaknesses. For example, their vision is very imprecise, their initial hypotheses are overly complex and sometimes nonsensical, and they often try to avoid following instructions unless forced to do so. These weaknesses will surely improve quickly in frontier models as AI research advances.
- I did not finish testing the other 2 preview games (ls20 and vc33) due to the high cost of running these games, but based on partial testing, I would expect similar results to those achieved by the RGB-Agent harness.
- I encourage others to iterate on this project and test other ARC-AGI-3 games, especially the new ones in the official benchmark (released on 2026-03-25).
- Feel free to borrow ideas from this project when building your own agents.
- I focused on implementing this project with Anthropic's API in mind, but the code should be easily adaptable to handle APIs from other LLM providers.
- Anthropic for developing an amazingly capable LLM and a well-documented API.
- The ARC-AGI-3 organizers for all of their hard work in creating a unique and challenging benchmark.
- The ARC-AGI-3 community for their willingness to share ideas and collaborate.
- Son Pham for providing feedback, guidance, and encouragement.
- Eugenia Stanisauskis Weiss for putting up with my incessant ramblings about AI.
- Python 3.12+
- Docker (for the agent sandbox)
- An ARC-AGI-3 API key (free)
- An Anthropic API key
# 1. Change directory to the project folder
cd USAI-agent
# 2. Install dependencies (using uv)
uv sync
# ...or install them using pip
pip install -e .
# 3. Copy the environment variables file
cp .env.example .env
# 4. Edit the .env file by adding your API keys
# (in text editor)
# 5. Build the Docker Sandbox Image
docker build -t usai-sandbox:latest src/usai/sandbox/docker/
# 6. Launch the web dashboard
uv run usai --dashboard-only
# 7. Open http://localhost:5000 in your browser, adjust the settings, and start a game.uv run pytest tests/ -v