Evaluation¶
Evaluate a checkpoint now → Run the benchmark
The end-to-end recipe: download the benchmark, serve your checkpoint, run a family across ID + OOD, and read the results in the paper's metrics. This page covers the architecture and the checkers behind it.
ManiGuard evaluates a VLA policy on frozen benchmark scenes and scores it on two independent axes — success and safety. A capable-but-reckless policy that finishes the task while knocking a glass off the table is not a pass; the benchmark reports safe-success vs. unsafe-success vs. failure.
ManiGuard's contribution to evaluation is the benchmark + the checkers, not any one model's serving stack:
- the eval client (
maniguard.eval.benchmark) — loads a frozen scene, steps whatever policy is connected, and drives the two checkers; - the success checker (
eval/goal_checker.py) and the LTL safety monitor (utils/safety_monitor.py) — the model-agnostic core; - a small, well-defined policy-integration contract so any VLA can plug in.
How a given checkpoint is served / loaded is the model family's own concern (openpi, GR00T, SmolVLA each have their own serving docs) — see Plugging in a policy.
policy (any framework/GPU/Python) eval client (OmniGibson + Isaac Sim)
┌────────────────────────────┐ obs ┌──────────────────────────────────┐
│ a policy exposed to the │ ─────► │ maniguard.eval.benchmark │
│ eval contract │ │ load snapshot → step policy │
│ (websocket, or in-process) │ ◄───── │ → success check + LTL safety │
└────────────────────────────┘ action │ → results.jsonl │
└──────────────────────────────────┘
The policy is decoupled from the simulator because OmniGibson can't reliably reload
scenes in one long-lived process, and because the policy and simulator often need
different Python/CUDA stacks. For per-scene isolation,
scripts/run_benchmark_all_scenes.sh spawns one eval-client process per scene and
aggregates the results.
New simulation runs default to five 640×640 DLAA source views; each policy receives its configured view subset and applies its own processor. High fidelity is the only supported rendering route for new runs. Video previews and full source recording are independent options; see Visual profiles and source recording.
The eval loop (eval/benchmark.py)¶
- Resolve the source — local dir or HF dataset repo via
maniguard.data.scene.hf_benchmark.resolve_benchmark_root. - Discover scenes —
eval/scene_discovery.py:discover_sceneswalks the benchmark root forscene_ep1.json+diagnostics.jsonlpairs and resolves each scene's target, prompt, cameras, goal conditions, and LTL safety spec. - Connect the policy — through the integration contract below (or a
--random-policysmoke stub). - Per scene: build the OmniGibson env from the frozen snapshot, reload the
controller after
env.reset(), warm up, then step the policy in chunks ofexecute_horizon— checking success every step and running the LTL safety monitor every step. - Record — write
results.jsonl(one line/scene, success + safety outcome),summary.json,eval_config.json, and an optional per-scene MP4. Each run lands in its ownoutput_dir/<run_name>subfolder (autoYYYYmmdd_HHMMSS, or..._<TAG>with--tag), so runs never overwrite.
Plugging in a policy¶
The eval client and the policy meet at one model-agnostic contract:
- Client → policy: an observation — one external overview image + a wrist image,
the robot state, and the language prompt (exact keys/shapes driven by
EvalConfigstate_mode/external_cam, to match the checkpoint's training config). - Policy → client: an action chunk (
(horizon, action_dim)), consumedexecute_horizonsteps at a time.
The default transport is an openpi-client-compatible websocket (msgpack-numpy): any server implementing that interface connects with no glue. But VLA families do not share one serving convention — openpi serves over websocket, GR00T ships its own inference server, LeRobot/SmolVLA policies typically run in-process — so a non-openpi model plugs in through a small model-specific adapter that bridges its serving to this contract. The benchmark fixes the interface once; each model family we evaluate gets its own thin adapter.
In code the boundary is already explicit — connect_policy returns
(policy, client_type) and query_policy dispatches per type — so adding a family
is a new adapter, not a change to the contract or the checkers.
| Policy family | Adapter | Notes |
|---|---|---|
| π0 / π0.5 (openpi) | maniguard.serve.openpi_native |
wraps openpi's own loader; serves any registered train config |
| GR00T N1.6 | maniguard.serve.gr00t_native |
wraps GR00T's inference policy; also serves the real-robot observation schema (--real) |
| SmolVLA | maniguard.serve.smolvla_native |
wraps the LeRobot policy in-process |
| anything else | your own thin adapter | bridge its serving to the websocket contract — no change to the checkers |
Each adapter runs inside its model family's own env (openpi venv, GR00T uv env, LeRobot env, ...) — the eval client never imports the model. Launch commands: Run the benchmark §2.
Configuring a run (EvalConfig)¶
One YAML per experiment, with CLI overrides (eval/eval_config.py:config_from_cli).
Key knobs:
| Group | Fields |
|---|---|
| Source | benchmark_root, benchmark_revision, scene_filter, scenes, max_scenes |
| Policy link | host, port, use_openpi_client, random_policy |
| Obs / action | state_mode, external_cam (left / right — which third-person overview feeds the fixed 1-overview + wrist layout), action_dim, execute_horizon, gripper_binarize |
| Controller | controller_preset, override_controller_config, ik_eef_to_joint, joint_pos_kp |
| Sim | action_frequency, rendering_frequency, physics_frequency, headless, longfinger |
| Eval | max_steps, camera_resolution, save_video, output_dir, run_name / tag |
Match training
state_mode, external_cam, action_dim, and especially the controller must
match how the SFT data was produced, or the realized motion diverges from
training. See Controller, data, action & eval.
Success checking (eval/goal_checker.py)¶
GoalChecker— evaluates the scene'sgoal_conditions, either a flat list (implicit AND) or a compoundand/or/nottree, over OmniGibson predicates (inside,ontop,touching,grasping,open,closed,covered).GoalRegionChecker— spatial success: the target is held and intersects the goal-region sphere (viamaniguard.utils.goal_region).
Safety checking (utils/safety_monitor.py)¶
This is what sets ManiGuard apart from a plain success benchmark. Each task ships an
LTL safety spec (diagnostics.ltl_safety: a Spot-compatible formula over atomic
propositions such as dropped / upright / touching / premature-lift). The eval
runs a TaskLTLMonitor every step over the rollout; a single violation marks the
episode unsafe. Success and safety are reported together (safe-success /
unsafe-success / failure), so a policy is credited only for finishing the task
without tipping, dropping, spilling, or lifting-before-closing. The same monitor
and active-object resolution (build_active_objects_for_ltl) are shared with datagen
and bench-finalize — one source of truth for what "safe" means.
Safety is contact-gated: a rollout that never touches an object is not safety-evaluated (a do-nothing policy is vacuously safe, not credited as safe behaviour), and violations count only from first contact. The report pairs this with a target-engagement ladder that separates inert from clumsy failures — see Engagement & contact-gated safety.
Snapshot validation (eval/snapshot_validator.py)¶
Offline checks inspect target/support references, goals and manifest metadata for
the validator's supported family schemas. Its optional runtime mode reconstructs
a BehaviorTask and requires the matching problem0.bddl and activity definitions;
a frozen benchmark with only inline ltl_safety is not sufficient for that mode.
Policy evaluation reads the frozen scene and inline specification directly through
maniguard.eval.benchmark.
The validator also supports batch --root checks. With a video output directory
and --video-output-exact-dir, batch videos are separated by each task's path
relative to the input root.
Preparing benchmark scenes (data/scene/)¶
| Tool | Purpose |
|---|---|
hf_benchmark.py |
resolve a benchmark source (local path or HF repo id) to a local dir |
benchmark_repair.py |
rebuild diagnostics.jsonl to the current scene-registry format |
rewrite_scene_robot.py |
copy a scene dir and swap its robot to match the training distribution |
trim_scene_to_room.py |
trim a snapshot to the target room only (faster load, fewer distractors) |
Perturbation sets (data/perturbation_scaling.py)¶
Generates self-contained single-level perturbation task sets from frozen base
tasks — object / position / semantic / env kinds — for measuring policy
robustness. Perturbations are materialized at load by
maniguard.envs.perturbation_runtime.
Running it¶
The end-to-end recipe — benchmark download, per-family serving, the family
runner (scripts/eval_family.sh, ID + 4 OOD levels), and reading the results in
the paper's metrics — is Run the benchmark. For a
one-off smoke test the client also runs directly:
# serve a policy (see the adapter table above), then:
python -m maniguard.eval.benchmark \
--benchmark-root IDEAS-Lab-Northwestern/ManiGuard-Bench --config <eval_yaml> --tag smoke
Code map¶
| Concern | Code |
|---|---|
| Eval loop, obs packing, policy dispatch | maniguard/eval/benchmark.py |
| Run config (YAML + CLI) | maniguard/eval/eval_config.py |
| Scene enumeration | maniguard/eval/scene_discovery.py |
| Success predicates / regions | maniguard/eval/goal_checker.py |
| LTL safety monitor | maniguard/utils/safety_monitor.py |
| Snapshot QA | maniguard/eval/snapshot_validator.py |
| Policy servers (one per family) | maniguard/serve/{openpi,gr00t,smolvla}_native.py |
| Family runner (ID / OOD buckets) | scripts/eval_family.sh |
| Metric summary (paper metrics) | tools/eval_summary.py |
| Benchmark prep / perturbation | maniguard/data/scene/, maniguard/data/perturbation_scaling.py |