Object-centric retargeting is a way of transferring a human manipulation demonstration to a robot in which the object’s 6-DoF trajectory is treated as the goal, and each robot is left free to find its own grasp and its own joint motion to reproduce that trajectory. It is the opposite of copying the hand. Nothing about the human’s arm — wrist path, finger joints, reach envelope — is asserted as the thing that has to transfer. The path the object took is.
At DXTR the contract is written down and enforced: the object’s 6-DoF trajectory is the reward; how a robot reproduces it — which grasp, which base pose — is the robot’s own business; and the object is never rescaled.
Why does copying the human hand fail?
A human hand has five fingers, and a wrist and a reach that a robot does not share. Replay those joint angles on a two-finger gripper or a short-reach arm and it grasps air, over-extends, or gets silently rescaled until the demonstration no longer means what it did.
The rescaling is the part worth dwelling on, because it is the failure that hides. If the pipeline shrinks the whole world by a quarter so a small arm can reach, a 20 cm drawer pull becomes a 5 cm drawer pull. The drawer does not open. Inverse kinematics reports success the whole way, because IK was only ever asked whether the end-effector could hit its targets — and it could, because the targets moved. The failure is not noise in the tracking. It is that the wrong thing was recorded as the goal.
Two families of prior work record the human’s motion rather than the object’s. Teleoperation retargeting maps a tracked human hand onto a robot hand-arm system in real time — DexPilot and AnyTeleop are the reference points. Egocentric imitation learns from video of a person doing the task, either head-mounted (EgoMimic, EgoDex) or through a gripper-shaped handheld device (UMI, DexCap). What both record is a hand pose or a device pose, so a mapping step has to produce commands for whichever body is being driven. Object-centric retargeting records a different quantity — the object’s pose — and so has a different mapping step to do. Which of those is the better trade is not something this page measures.
What does the object-centric contract actually say?
What a task requires is that the object move along a certain path: the mug lifts, the bottle pours, the box lands on the shelf. That trajectory is body-agnostic. It is the same whether a Franka, a UR5e or a humanoid produces it. Store the object motion as the goal, with a tolerance, and each robot is free to satisfy it its own way.
Three consequences follow immediately, and they are the reason to bother:
- Gripper width becomes correct. It comes from the object’s geometry at the grasp point, not from measuring a human pinch and hoping it maps.
- Different bodies legitimately differ. A parallel gripper, a humanoid hand and a short-reach arm each choose a different grasp on the same object, and all three are right, because the object still went where it was supposed to go.
- The goal is checkable. “Did the object follow its path?” is a question a simulator can answer. “Did the robot look like the human?” is not.
How does a robot choose its own grasp?
Per robot, in the object’s canonical frame:
- Synthesise candidates. Sample antipodal grasps on the reconstructed object mesh. Each candidate is a gripper pose relative to the object plus the width it requires. Filter by that robot’s maximum gripper width, by force-closure quality, and by surface-normal alignment. Where the human held the object is used as a bias, not a constraint — the robot may grasp elsewhere if elsewhere works better for it.
- Select by sweep feasibility. This is the load-bearing step. For each surviving candidate, compose the grasp with the object’s recorded pose at every frame of the contact interval and ask whether the resulting end-effector path stays reachable for the whole interval. Score candidates by how much of the required object motion the robot can actually follow, and keep the best. Different embodiments score different grasps highest — which is precisely the point.
- Build end-effector targets. During contact, the target is the object’s pose composed with the chosen grasp. Before contact onset, reach from rest; after release, retreat. The gripper closes to the width the object’s geometry calls for.
- Solve IK for those targets, with collision constraints against the reconstructed scene, so an arm that would sweep through the counter is not quietly allowed to.
Two bodies in the fleet do not reach the end of that list. Apptronik Apollo has no actuated gripper, so there is nothing to filter by width and nothing to close: it refuses with no_actuated_gripper rather than pretending. The Unitree G1’s three-finger Dex3 does close, but its closure is not yet scorable, so the G1’s outcomes rest on arm motion. Both are stated that way on the playground rather than counted as grasps.
What is the tolerance tube, and what does “pass, fail or refused” mean?
The tolerance tube is the envelope around the recorded object trajectory inside which the object still counts as having done the task. It turns the contract into a measurement. Every retargeted episode is replayed in simulation and scored against it before it ships.
If the object stays in the tube, the episode passes. If it does not, the episode does not ship a prettier number — it ships an infeasibility certificate naming the sub-interval that failed and what blocked it. This is checked in simulation at the kinematic tier: pass, fail or refused, each with its reason. On the demo grid, the Unitree G1 is the honest case: parts of that path sit outside its reach, so it holds where it cannot go and the certificate says so.
A certificate replaces something worse. The alternative to a certificate is not a better episode; it is a silently rescaled one that looks clean in a video and teaches the wrong thing.
Two kinds of non-pass, and they are not equally good. A refusal is decided before the robot moves: the pre-flight check finds no grasp, no reach or no clearance, and the body ships a rest pose with the named reason. A failure in replay is a trajectory that shipped as an attempt and then missed the tube — the system moved without knowing, which is the worse of the two and is never called a refusal here. Today most non-passing outcomes on the playground are failures in replay; the counts are published split rather than pooled.
What happens when no grasp works?
There is an explicit ladder, tried in order, and the provenance records which rung each frame used:
- A fixed-base solution that follows the whole path.
- Failing that, base motion — let a mobile base reposition.
- Failing that, a regrasp: release, reposition, grasp again.
- Failing that, an infeasibility certificate for the uncovered sub-interval.
What is never on the ladder is scaling the object. That rung was removed on purpose.
Where does the hand still matter?
Demoted, not deleted. The hand supplies contact timing — pinch closure marks grasp onset and release. It rescues the object’s pose during occlusion, when the object is rigid to the palm. It biases grasp synthesis toward where a person actually held the thing. And it covers what the object contract structurally cannot see: a press, a wipe, a reach to a switch — everything that happens while nothing is moving.
That is why the shipped policy is a hybrid: the object contract where an object moves, wrist imitation where none does, with each frame labelled by which rule it obeyed and checked accordingly. The three policies — hand only, object only, object plus hand — are switchable side by side on the home page, driven from the same capture.
What ships out of it?
Eight robot embodiments are retargeted today — Franka Panda, UR5e, PR2, MyCobot 280, Unitree G1, Apptronik Apollo, Fanuc LR Mate 200iD and SO-101, seven of them in the public playground fleet — and the results export as LeRobot datasets per robot (the format the pi0, ACT and Diffusion Policy stacks read) and MCAP for the raw multimodal record. Every render on this site is simulation.