From 50a7bfa91c7b9c8a64fbdf381422b0fc504239be Mon Sep 17 00:00:00 2001 From: reacher-z Date: Fri, 7 Aug 2026 03:36:27 -0700 Subject: [PATCH] Add ClawBench (live-website everyday-task agent benchmark) --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 4ea2ad2..7ba9f5c 100644 --- a/README.md +++ b/README.md @@ -370,6 +370,7 @@ Most "awesome" lists are link dumps. This one is **annotated and verified**: eve - **[Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge](https://arxiv.org/abs/2506.21506)** — Gou et al., OSU NLP Group (NeurIPS 2025 D&B) — · *benchmark* — 130 long-horizon live-web agentic-search tasks; novel Agent-as-a-Judge rubric-tree grader for time-varying, citation-backed answers — a serious answer to the Deep Research evaluation gap. 🆕 - **[Online-Mind2Web (An Illusion of Progress? Assessing the Current State of Web Agents)](https://arxiv.org/abs/2504.01382)** — Xue et al., OSU NLP Group — · *benchmark* — 300 realistic tasks on 136 live websites with an LLM-as-a-Judge auto-grader (~85% human agreement); exposes overstated web-agent progress vs simple baselines. 🆕 - **[REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites](https://github.com/agi-inc/REAL)** — AGI Inc (agi-inc/REAL), powers realevals.xyz — · *benchmark* — 112 tasks on deterministic Next.js replicas of Amazon/Uber/LinkedIn etc.; reproducible LLM evaluator plus state validators — fixes the flakiness of live-site web benchmarks. 🆕 +- **[ClawBench: Can AI Agents Complete Everyday Online Tasks?](https://arxiv.org/abs/2604.08523)** — TIGER-AI-Lab — · *benchmark* — 153 everyday online tasks (purchases, bookings, job applications, email) across 144 live production websites in 15 categories; a Chrome-extension + CDP submission-interception layer blocks only the final write request so agents run end-to-end on real sites safely (no sandbox); best model Claude Sonnet 4.6 reaches 33.3%. (code: github.com/TIGER-AI-Lab/ClawBench · site: claw-bench.com) 🆕 - **[WebGames: Challenging General-Purpose Web-Browsing AI Agents](https://arxiv.org/abs/2502.18356)** — Thomas et al., Convergence AI — · *benchmark* — 50+ client-side challenges isolating specific browser interaction skills with verifiable pass/fail; best agent 41% vs 96% human, a sharp diagnostic gap. 🆕 - **[Berkeley Function Calling Leaderboard (BFCL) V4](https://gorilla.cs.berkeley.edu/leaderboard.html)** — Patil et al., UC Berkeley (Gorilla / ICML 2025) — · *leaderboard* — Executable + AST-based grading of tool/function calling; V4 adds multi-turn agentic, web-search and memory tasks — the de facto tool-calling leaderboard. 🆕 - **[GTA: A Benchmark for General Tool Agents](https://arxiv.org/abs/2407.08713)** — Wang et al., Shanghai AI Laboratory (NeurIPS 2024 D&B) — · *benchmark* — 229 human-written real-world queries with implicit multimodal tool use; executable evaluation platform across perception/operation/logic/creativity tools (GTA-2 follow-up in 2026). 🆕