Skip to content

Add ClawBench (live-website everyday-task agent benchmark) #73

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
xdotli merged 1 commit into from
Aug 20, 2026
Merged

Add ClawBench (live-website everyday-task agent benchmark) #73

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -370,6 +370,7 @@ Most "awesome" lists are link dumps. This one is **annotated and verified**: eve
- **[Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge](https://arxiv.org/abs/2506.21506)** — Gou et al., OSU NLP Group (NeurIPS 2025 D&B) — <https://arxiv.org/abs/2506.21506> · *benchmark* — 130 long-horizon live-web agentic-search tasks; novel Agent-as-a-Judge rubric-tree grader for time-varying, citation-backed answers — a serious answer to the Deep Research evaluation gap. 🆕
- **[Online-Mind2Web (An Illusion of Progress? Assessing the Current State of Web Agents)](https://arxiv.org/abs/2504.01382)** — Xue et al., OSU NLP Group — <https://arxiv.org/abs/2504.01382> · *benchmark* — 300 realistic tasks on 136 live websites with an LLM-as-a-Judge auto-grader (~85% human agreement); exposes overstated web-agent progress vs simple baselines. 🆕
- **[REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites](https://github.com/agi-inc/REAL)** — AGI Inc (agi-inc/REAL), powers realevals.xyz — <https://github.com/agi-inc/REAL> · *benchmark* — 112 tasks on deterministic Next.js replicas of Amazon/Uber/LinkedIn etc.; reproducible LLM evaluator plus state validators — fixes the flakiness of live-site web benchmarks. 🆕
- **[ClawBench: Can AI Agents Complete Everyday Online Tasks?](https://arxiv.org/abs/2604.08523)** — TIGER-AI-Lab — <https://arxiv.org/abs/2604.08523> · *benchmark* — 153 everyday online tasks (purchases, bookings, job applications, email) across 144 live production websites in 15 categories; a Chrome-extension + CDP submission-interception layer blocks only the final write request so agents run end-to-end on real sites safely (no sandbox); best model Claude Sonnet 4.6 reaches 33.3%. (code: github.com/TIGER-AI-Lab/ClawBench · site: claw-bench.com) 🆕
- **[WebGames: Challenging General-Purpose Web-Browsing AI Agents](https://arxiv.org/abs/2502.18356)** — Thomas et al., Convergence AI — <https://arxiv.org/abs/2502.18356> · *benchmark* — 50+ client-side challenges isolating specific browser interaction skills with verifiable pass/fail; best agent 41% vs 96% human, a sharp diagnostic gap. 🆕
- **[Berkeley Function Calling Leaderboard (BFCL) V4](https://gorilla.cs.berkeley.edu/leaderboard.html)** — Patil et al., UC Berkeley (Gorilla / ICML 2025) — <https://gorilla.cs.berkeley.edu/leaderboard.html> · *leaderboard* — Executable + AST-based grading of tool/function calling; V4 adds multi-turn agentic, web-search and memory tasks — the de facto tool-calling leaderboard. 🆕
- **[GTA: A Benchmark for General Tool Agents](https://arxiv.org/abs/2407.08713)** — Wang et al., Shanghai AI Laboratory (NeurIPS 2024 D&B) — <https://arxiv.org/abs/2407.08713> · *benchmark* — 229 human-written real-world queries with implicit multimodal tool use; executable evaluation platform across perception/operation/logic/creativity tools (GTA-2 follow-up in 2026). 🆕
Expand Down
Loading