Website: harp-bench.vercel.app · Paper: arXiv:2611.11123 (PDF)
HARP-Bench is a simulation benchmark for robot manipulation around active
humans, built on Genesis. Animated human avatars change what the robot should
do, which objects are available, when it can act, and how it must move.
The repository (Python packages, paths, and dataset IDs) keeps its historical
genesis-hr-bench / genesis_hr_bench names.
Ships 49 canonical manipulation tasks plus 32 compatibility aliases (the
pre-HARP task names), with six robot embodiments. The 37 benchmark tasks
evaluated in the paper are the rows of
eval_sets/eval_v3_sampled; the
remaining registered tasks are no-human base tasks and factory work-cell
scenarios. The release runtime is pinned to genesis-world==1.2.0 with the
NYX GPU renderer.
Start with installation, quick start, and the documentation index. The code map explains module ownership.
The 37 tasks are task-condition instances over four human-present
conditions. Task names follow the paper: a Cueing task uses the paper task
name, and every other task is <paper task>_<condition>. The condition value
(in parentheses) is what TaskVariant, eval-set manifest category fields,
and summaries store.
- Cueing (
cueing): the robot infers its objective from human activity, e.g. a human carrying, offering, or using an object. - Contribution (
contribution, suffix_contribution): the human completes part of the task, and the robot finishes the remaining subtasks. - Interruption (
interruption, suffix_interruption): the human temporarily changes execution conditions, e.g. by inspecting or holding the object the robot needs, and the robot must wait or adapt and then resume. - Co-presence (
copresence, suffix_copresence): the robot completes its original goal while a human does unrelated work nearby.
That is 7 Cueing, 10 Contribution, 10 Interruption, and 10 Co-presence tasks.
envs.tasks.paper_label(task_name) returns (condition, paper task name) for
any of them. The pre-HARP names (*_assist, *_interrupt, *_neutral,
categorize_cooperative, deliver_to_human_easy, stamp_documents) still resolve to the same
classes, so old commands, manifests, checkpoints and datasets keep working;
the task framework has the old-to-new
table, also exported as envs.tasks.LEGACY_TASK_NAMES.
| Paper task | Task name | Description |
|---|---|---|
| Deliver to Human | deliver_to_human |
Pick up the requested object from the table and deliver it to the human. |
| Pour Water | pour_water |
Pour water from the mug into the cup. |
| Oil Bottle Recovery | oil_bottle_recovery |
Recover the knocked-over oil bottle from the cooking area and place it upright. |
| Open Microwave | open_microwave |
Open the microwave door for the human carrying a plate. |
| Feed Documents | feed_documents |
Push each document into the stamp zone so the human can stamp it. |
| Take from Human (Easy) | take_from_human_easy |
Take the requested compact object from the person and place it on the table. |
| Take from Human (Safety) | take_from_human_safety |
Take the kitchen knife offered by the chopping human and place it in the knife box. |
| Paper task | Contribution (contribution) |
Interruption (interruption) |
Co-presence (copresence) |
|---|---|---|---|
| Categorize Objects | categorize_objects_contribution: Sort the objects into matching baskets. |
categorize_objects_interruption: Sort objects into matching baskets while yielding when the human inspects one. |
categorize_objects_copresence: Sort objects while the human works nearby. |
| Put Object in Cabinet | put_object_in_cabinet_contribution: Place the object in a human-opened cabinet drawer. |
put_object_in_cabinet_interruption: Place the object after the human inspects it. |
put_object_in_cabinet_copresence: Put the object in the cabinet while the human works nearby. |
| Stack Three Bowls | stack_three_bowls_contribution: Stack one bowl while the human stacks another. |
stack_three_bowls_interruption: Stack bowls while waiting if the human inspects one. |
stack_three_bowls_copresence: Stack bowls while the human works nearby. |
| Place Bread in Basket | place_bread_in_basket_contribution: Place one bread while the human places the other. |
place_bread_in_basket_interruption: Place breads while waiting if the human inspects one. |
place_bread_in_basket_copresence: Place breads while the human works nearby. |
| Dump Bin | dump_bin_contribution: Empty the table cubes while the human tosses another cube into the bin. |
dump_bin_interruption: Place each cube into the trash can, waiting if the human inspects one. |
dump_bin_copresence: Put table objects into the bin while the human works nearby. |
| Place Burger and Fries | place_burger_and_fries_contribution: Place meal items while the human adds one item. |
place_burger_and_fries_interruption: Place meal items while waiting if the human inspects one. |
place_burger_and_fries_copresence: Place meal items while the human works nearby. |
| Place Pair of Shoes | place_pair_of_shoes_contribution: Put one shoe in the shoebox while the human puts the other. |
place_pair_of_shoes_interruption: Put shoes in the shoebox while waiting if the human inspects one. |
place_pair_of_shoes_copresence: Place shoes in the shoebox while the human works nearby. |
| Place Food in Skillet | place_food_in_skillet_contribution: Place the skillet on the stove and put foods in it while the human helps place one food. |
place_food_in_skillet_interruption: Place the skillet and foods while waiting if the human inspects one. |
place_food_in_skillet_copresence: Place food in the skillet while the human works nearby. |
| Rank Blocks by Color | rank_blocks_by_color_contribution: Place red, green, and blue blocks in left-to-right order while the human helps place one. |
rank_blocks_by_color_interruption: Order RGB blocks while waiting if the human inspects one. |
rank_blocks_by_color_copresence: Order RGB blocks while the human works nearby. |
| Rank Blocks by Size | rank_blocks_by_size_contribution: Place blocks from largest to smallest while the human helps place one. |
rank_blocks_by_size_interruption: Order blocks by size while waiting if the human inspects one. |
rank_blocks_by_size_copresence: Order blocks by size while the human works nearby. |
Each family also has a no-human base task, which keeps its historical name
(put_object_cabinet, stack_bowls_three, blocks_ranking_rgb, dump_bin,
...; the Categorize Objects tasks collapse to categorize under
--no-human). The factory work-cell scenarios (factory_inspect_pack, factory_kit_packing,
factory_line_feeding) are registered extras outside the 37-task benchmark;
see the task inventory.
Supported robots are Franka Panda, Piper, Stretch, ARX X5, UR5+WSG, and xArm7. Scripted experts use mplib planning and IK; policy evaluation has a separate high-rate control path. See the task framework.
The commands in this README run on one Linux machine with an NVIDIA GPU. To use a SLURM cluster instead, see running on a SLURM cluster.
- Python 3.10 and an NVIDIA GPU with a CUDA driver for the benchmark
itself. Every
eval_v3_sampledrow renders with the NYX renderer, which runs on the Genesis GPU backend (GENESIS_BACKEND=gpu) and draws through Vulkan, so it needs a graphics-capable card. RTX 2080 Ti, RTX 8000, RTX A4000, A16, A40, L4 and L40S have been verified. The compute-only data centre parts (A100, H100, H200) have no Vulkan device and cannot run NYX. Pre-Turing cards (GTX 1080 Ti, Titan X, M40) are too old for the Genesis GPU backend. See GPU requirements. - CPU-only fallback. With
GENESIS_BACKEND=cpuand--renderer rasterizer, scripted experts, data-collection debugging and the test suite run without a GPU. A row that asks for NYX falls back to the rasterizer on the CPU backend, andscripts/eval_expert_set.pyprints a notice when that happens. CPU rasterizer results are not the reference numbers, and the VLA baselines need a GPU. - Memory: at least 32 GB of RAM; some scene builds need 64 GB.
# 1. Genesis runtime env (genesis-world 1.2.0 + NYX, all from PyPI).
python3.10 -m venv .venv-genesis-latest
.venv-genesis-latest/bin/python -m pip install -U pip wheel
.venv-genesis-latest/bin/python -m pip install -r requirements.txt
# Optional: the test runner and correctness linter.
.venv-genesis-latest/bin/python -m pip install -r requirements-dev.txtrequirements.txt is the installation entry point. It includes
requirements-genesis-latest.txt, the narrow simulator/planner/renderer lock,
and adds the task and offline-data helpers. Keep the simulator lock pinned:
changing it requires a new robot, task, planner, and renderer compatibility
sweep.
source scripts/env.sh activates .venv-genesis-latest, exports MAWM_PY
(the simulator interpreter), and sets PYTHONPATH and GENESIS_BACKEND=gpu.
If your interpreters live elsewhere, set them in scripts/env.local.sh: copy
scripts/env.local.sh.example. With an
external MAWM_PY, run "$MAWM_PY" wherever this README says python.
The VLA baselines run their policy servers in separate LeRobot environments
(LEROBOT_PY, PI0_PY); see VLA fine-tuning
for how to create them.
Everything below is public and ungated, so no Hugging Face login or token is needed.
-
Asset pack (required). The Hugging Face dataset
zhouqh/harp-bench-assetsholds objects, avatars, motions, robot descriptions and HDRIs. Download it to<repo>/assets, or download it elsewhere and symlink it there:pip install -U huggingface_hub # provides the `hf` command hf download zhouqh/harp-bench-assets --repo-type dataset --local-dir assetsThe pack is self-contained and ships a
manifest.jsonwith a SHA-256 for every file. Its dataset card lists per-asset provenance and licensing. -
Released baseline checkpoints (for VLA evaluation). These are the Hugging Face model repos
zimplex/harp-vla-train37-{act,diffusion,pi0,pi05,smolvla}. Runscripts/vla_train37/fetch_ckpts.pywith the LeRobot environment's interpreter, which already hashuggingface_hub:source scripts/env.sh "$LEROBOT_PY" scripts/vla_train37/fetch_ckpts.py # all five "$LEROBOT_PY" scripts/vla_train37/fetch_ckpts.py --models act,pi0 # a subset
Each snapshot lands in
checkpoints/train37/<model>/and is checked against itsSHA256SUMS. The servers loadcheckpoints/train37_eval/<model>/, a view of symlinks plus JSON configs whose training-machine paths point at public Hugging Face model ids.--root DIRputs both trees somewhere other thancheckpoints/. -
LeRobot source (for training or serving the baselines).
git submodule update --init --recursive # baseline/lerobot
The LeRobot training datasets are not published as a download. To train, record your own demos with the scripted experts; see portable data collection.
source scripts/env.sh
# Collect one scripted demo (writes data/pour_water/ and a video).
python scripts/collect.py --task pour_water \
--config config/latest_genesis/pour_water_no_avatar.yml --episodes 1
# Run the privileged scripted expert with NYX on one eval-set row
# (row 1176 is pour_water; rows are zero-based, --end-index is exclusive).
python scripts/eval_expert_set.py --eval-set eval_sets/eval_v3_sampled \
--policy expert_full_state --start-index 1176 --end-index 1177Without a usable GPU, add --renderer rasterizer and prefix the commands
with GENESIS_BACKEND=cpu (debugging only; see Requirements).
Tasks are declared in envs/tasks/registry.py (TASK_DESCRIPTORS), which
projects the compatibility maps exported from envs.tasks (TASK_MAP for
canonical tasks, TASK_ALIASES for compatibility names).
scripts/eval_expert_set.py runs the expert_full_state (privileged) and
expert (blind to the human) policies on a fixed manifest. The release
evaluation set is eval_sets/eval_v3_sampled: 3,403 expert-solvable seeds
over the 37 benchmark tasks (see its README).
source scripts/env.sh
python scripts/eval_expert_set.py --eval-set eval_sets/eval_v3_sampled \
--policy expert_full_state-
Renderer. Each row's
renderer(NYX foreval_v3_sampled) is used unless you pass--renderer {nyx,rasterizer}, which overrides every row. IfGENESIS_BACKEND=cpuwhile rows ask for NYX, the script prints a one-line notice that those rows fall back to the rasterizer. -
One scene per process. NYX can build only one scene per Python process, so when more than one row renders with NYX each row runs in a fresh subprocess (
--subprocess-per-episode auto, the default). -
Output. Results go to
runs/expert_eval_set/<eval-set name>/<policy>/(results.jsonl,results.csv,summary.json), or to--output-dir. The console summary reports success per paper condition (Cueing, Contribution, Interruption, Co-presence);summary.jsonkeys keep the manifestcategoryvalues. -
Sharding. The full set takes days on one GPU.
--start-indexand--end-indexselect a zero-based row range (end exclusive), and each shard writes to its own.../shard_<start>_<end>/directory, so shards can run in parallel on several GPUs or machines:for start in 0 500 1000 1500 2000 2500 3000; do python scripts/eval_expert_set.py --eval-set eval_sets/eval_v3_sampled \ --policy expert_full_state --start-index $start --end-index $((start + 500)) done
Other eval sets live under eval_sets/; eval_v0_smoke has one seed per task.
scripts/generate_eval_set.py builds new manifests and
scripts/validate_eval_set_init.py checks that every row resets.
-
One checkpoint, one task.
scripts/run_vla_eval.shstarts the LeRobot policy server for a checkpoint, waits for it, runsscripts/eval_vla.py, and stops the server:source scripts/env.sh ACT_SERVER_ENV=lerobot scripts/run_vla_eval.sh --model act \ --checkpoint checkpoints/train37_eval/act --task pour_water --episodes 1ACT_SERVER_ENV=lerobotserves the released ACT checkpoint fromLEROBOT_PY; without it ACT uses the legacyPI0_PYenvironment. -
Released checkpoints on an eval set.
scripts/vla_train37/shards a manifest into per-model batch files, runs each batch with NYX rendering, and aggregates success and safety tables:source scripts/env.sh scripts/vla_train37/launch_full.sh eval_sets/eval_v3_sampled/manifest.jsonl <run_name> --local python scripts/vla_train37/summarize.py data/vla_train37_eval/<run_name>
See VLA inference and control for running a subset of rows, per-model defaults and GPU memory needs.
scripts/vla_data/collect_vla.py records one expert episode per seed as
images, proprioception and actions (H5). scripts/collect_vla.sh wraps
scripts/collect.py --record-stride for the per-seed pickle layout under
data/<task>/vla_raw/. Convert recordings with
scripts/build_genesis_hr_bench_rlds.py (RLDS) and
scripts/convert_genesis_hr_bench_to_lerobot.py (LeRobot). See
portable data collection.
scripts/vla_data/nyx_campaign.py is an optional SLURM campaign runner for
large collections (cluster guide).
The released baselines are LeRobot policies: ACT, LeRobot Diffusion,
pi0, pi0.5, and SmolVLA. They are trained with scripts/finetune.sh
(which dispatches to finetune_lerobot.sh for ACT/Diffusion and
finetune_lerobot_v2.sh for pi0/pi0.5/SmolVLA) and served by
baseline/servers/lerobot_server.py;
the benchmark talks to the server through scripts/vla_client.py. Apply
patches/lerobot-genesis-hr-bench.patch to the baseline/lerobot submodule as
described in VLA fine-tuning.
Core versions (pinned in requirements-genesis-latest.txt):
- Python 3.10
torch==2.4.1+cu121genesis-world==1.2.0gs-nyx==0.1.3/gs-nyx-plugin==0.1.4numpy==1.26.4formplib==0.2.1compatibilityh5py==3.16.0for VLA data recording
Genesis provides two camera paths here: NYX (default; GPU sensor renderer
that needs GENESIS_BACKEND=gpu on a Vulkan-capable GPU, and falls back to
the rasterizer when it cannot run) and the rasterizer (fast, also runs on
CPU). Switch with --renderer; see
NYX rendering.
Run the correctness linter and the complete regression suite (CPU only, a few minutes):
source scripts/env.sh
GENESIS_BACKEND=cpu bash scripts/check_quality.shIt exits 0 when Ruff and every test pass. Two known open failures in
tests/test_take_handover_stance.py are marked xfail(strict=True) (see
release checks).
Native integration check and the reset/step smoke for every registered task (the second takes hours):
source scripts/env.sh
GENESIS_BACKEND=cpu python scripts/check_latest_genesis_integration.py \
--python "$MAWM_PY" --robots franka --tasks blocks_ranking_rgb --steps 1
GENESIS_BACKEND=cpu python scripts/probe_latest_genesis_robot_tasks.py \
--all-registered --settle-steps 1 --timeout 300A cold first kernel/JIT compile can exceed the default timeouts; retry with
larger --robot-timeout / --task-timeout. See
release checks and the
pinned runtime.
| Topic | Doc |
|---|---|
| Documentation index | docs/README.md |
| Code layout and module ownership | docs/code_layout.md |
| Collect VLA training data (RLDS / LeRobot) | docs/collect-data-portable.md |
| Fine-tune the LeRobot baselines | docs/vla-finetune-pipeline.md |
| Serve and evaluate policies | docs/vla-inference-control-pipeline.md |
| Task registry and interaction architecture | docs/task_framework.md |
| GPU requirements for NYX | docs/general/gpu_resources.md |
| Running jobs, outputs and task rules | docs/general/running_jobs.md |
| Optional: running on a SLURM cluster | docs/cluster.md |
| Contributing | CONTRIBUTING.md |