Skip to content

Data flow: offline pools → selection → placement

Every task-generation pipeline runs the same four-stage flow, with pipeline-specific code only at the per-stage edges. Understanding these stages makes it easier to:

  • Add a new pipeline (you mostly fill in select_objects and place_objects).
  • Diagnose "why was this object/scene picked?" or "why isn't this container in the pool?".
  • Regenerate the offline pools after asset changes or graspability re-runs.

Stage 1 — offline pool generation

One-time object filtering (raycasts + simulator), produces JSON files committed to the repo so runtime selection is a pure dict lookup. Mostly the objects that are going to be interacted with need to be graspable which is filtered by the graspability csv. Here are some examples with the existing pipelines:

maniguard/task_generation/utils/stack_pipeline/

File Generated by Purpose
scan_top_full.json (gitignored, ~122 MB) scan_top_surface.py Top-down 24×24 raycast over each graspable object's world-XY AABB, normal-z filtered for up-facing geometry. Per cell: hit z + face normal. Source of truth for top-surface geometry.
derived_top_features.json derive_top_features.py Per object: largest axis-aligned square at z_max (flat plateau side) and at z_min (cavity floor side) via dynamic programming, plus z-range of facing hits.
stack_self_full.json test_stack_self_stability.py 3-copy self-stack stability with shake perturbation in an empty Scene; per-axis 1.05× bbox tolerance.
stack_same_pool.json build_stack_same_pool.py Self-stable models intersected with docs/graspability_classified.csv (status=graspable) and complaints.json (no unresolved).
stack_flat_compatibility.json build_stack_flat_compat.py max(item.bbox_xy) ≤ target.z_max_side: fit matrix for stack-on-flat-target.
stack_recep_compatibility.json build_stack_recep_compat.py max(item.bbox_xy) ≤ target.z_min_side: fit matrix for stack-into-receptacle.

maniguard/task_generation/utils/food_transfer_pipeline/

File Generated by Purpose
container_openings.json derive_container_openings.py Per graspable wide-opening container: cavity centroid (relative to AABB center), opening-floor depth, largest inscribed-square side, opening area. Pure-Python from scan_top_full.json — no OG required, ~1 sec for ~270 containers.
transfer_compatibility.json build_transfer_compatibility.py Per container: list of foods that satisfy max(food.bbox_dims) ≤ container.opening_square_side_m, restricted to graspable foods + containers.

maniguard/task_generation/utils/lid_transport_pipeline/

File Generated by Purpose
lid_cap_container_pairs.json build_lid_cap_container_pairs.py Every (lid | cap, container) attachment pair with per-side status + per-pair verdict (kept / kept_via_relax / dropped:<reason> / manually_added). The full inventory; admission gating happens at runtime.
lid_transport_food_compat.json build_lid_transport_food_compat.py Admitted pairs joined with transfer_compatibility.json's food list per container, restructured to {food_cat: [model, …]} so sampling can do uniform-by-category-then-model and avoid the "hardback has 245 entries" bias.

Manual overrides

docs/graspability_classified.csv is the canonical readiness gate. Rows can be flipped manually (e.g. an asset marked no_grasp in the survey but geometrically valid for transport) by tagging the note column with ;manual_override. The override survives a future regeneration of the CSV because the note tag is the marker.

When CSV rows are flipped, regenerate the downstream pools:

conda activate behavior

# scan_top_full.json: extend the raycast scan for newly-graspable models
python -m maniguard.task_generation.utils.stack_pipeline.scan_top_surface \
  --output scan_top_full.json --resume

# Then in dependency order:
python -m maniguard.task_generation.utils.food_transfer_pipeline.derive_container_openings
python -m maniguard.task_generation.utils.food_transfer_pipeline.build_transfer_compatibility
python -m maniguard.task_generation.utils.lid_transport_pipeline.build_lid_transport_food_compat

Stage 2 — object selection (runtime)

Each pipeline package has a select.py that pulls from its JSON pool. Selection is uniform-by-category, then uniform-by-model — the category-first restructure prevents categories with many models (e.g. hardback ×245) from drowning out single-model categories.

Pipeline Selector Pool
Stack stack_pipeline/select.select_stack_objects(mode, rng, ...) stack_same_pool.json / stack_flat_compatibility.json / stack_recep_compatibility.json per --stack-mode
Transfer food_transfer_pipeline/... (called inline in transfer_scene_pipeline.build_transfer_objects) transfer_compatibility.json
Lid transport (food) lid_transport_pipeline/select.select_pair_for_food(rng, ...) lid_transport_food_compat.json
Lid transport (liquid) lid_transport_pipeline/select.select_pair_for_liquid(rng, ...) lid_cap_container_pairs.json (kept verdicts only)

CLI overrides (--target-model, --item-model, --food-model, --source-model, --dest-model, …) bypass the random sampler by pinning a specific model id; the selector still validates the pin against the pool.

Stage 3 — scene + surface selection (runtime)

pipeline_common.pick_scene_from_placeable picks a (scene, support_object, region_of_object) triple that fits the pipeline's required_area_m2 (computed from the selected objects' footprints via estimate_object_set_footprint).

The picker reads pre-computed surface profiles from placeable_surfaces_v1.json per scene/object. Each profile is a 24×24 raycast at the placeable plane that yields connected-component regions (with area, x/y bounds, top-z) — so a 2-region desk like desk/puapey (split by a centerline divider) exposes both halves independently.

CLI pins:

Flag Effect
--scene-model Restrict to a single scene.
--surface-category Require the support's BEHAVIOR category (e.g. countertop, breakfast_table).
--surface-model Pin the support's asset model id. Picker still chooses the region within the model based on required_area_m2.

Output of the picker becomes args._picked_surface = {"scene_model", "category", "model", "instance_name", "room", region geometry, top_plane_z_local, ...}.

Stage 4 — placement (runtime)

BasePipeline._setup_session orchestrates placement in this order:

  1. env.reset() — load the picked scene from its snapshot. Partial-room load (load_room_instances=[...]) is used by default for speed, except when GPU dynamics is on (PhysX articulation pool sized to the partial scene crashes when new task objects are spawned).
  2. Build task-object configs upfront for ALL episodes via build_task_object_cfgs (into cfg["objects"], spawned at env load), with episode-labelled inst_ids like bowl_ep1_1, and role-prefixed object names like target_bowl_ep1_1 / fragile_cup_ep1_2 so downstream code can filter task objects by role); ep > 0 are parked at z = −100. Spec-level abilities are merged INTO the BEHAVIOR taxonomy abilities (not replacing).
  3. Find the support object by (category, model) from the picked surface; pin to world.
  4. clear_support_area removes non-structural scene props overlapping the picked region + a margin. Excludes the support, spawned task objects, and env.robots (the FrankaMounted at world origin would otherwise be deleted in scenes where the surface is near origin).
  5. identify_objects + place_objects (pipeline-specific). The shared placement primitives are:
    • _upright_half_height(obj) — orientation-independent Z extent from native_bbox * scale (avoids aabb's shrinkage when the object is tilted).
    • Gap-aware X offset for paired objects (container + lid/cap) so wide pairs don't interpenetrate.
    • food_transfer_pipeline.lookup.container_drop_xy(obj) — applies the offline-derived AABB-relative cavity-opening offset (from container_openings.json) to the live AABB center, so food drops above the actual opening (e.g. jug spout offset +7.9 cm Y) rather than the AABB center.
    • place_food_on_source(env, food, source) — drop just above source rim, settle 60 sim steps so gravity actually descends the food into the cavity.
  6. place_franka_edge_aligned picks a table edge for the robot mount (collision-checked against scene + spawned objects), then clear_robot_base_region clears any preset props in the keepout zone (also excludes env.robots).
  7. set_position_orientation moves the FrankaMounted to the chosen edge pose at floor_z; gripper at home pose.

Per-episode work between rollouts is just teleporting the next episode's objects from their park position onto the surface — no scene mutations during the run, sidestepping OG's registry-staleness on objects added while the sim is playing.

See also