Robotics data collection for manipulation: teleoperation vs egocentric video vs phone capture

Robotics data collection for manipulation is the practice of gathering demonstrations of physical tasks — picking, placing, pouring, opening, transporting — in a form a learned policy can train on. There are four working approaches in the field today, and they differ less in what they record than in what they assume. The assumption is always about the body: whose arm produced the motion, and whether that matters downstream.

What are the four approaches?

Teleoperation. A person drives the target robot directly — leader-follower arms, VR controllers, a 3D mouse — and the robot’s own joint states are what gets recorded. What comes out is, by construction, a trajectory that robot executed.

Egocentric video with hand tracking. Record a person doing the task from a head-mounted or fixed camera, track the wrist and fingers, and map that hand pose onto a robot’s end effector and hand. DexPilot and AnyTeleop do this for live teleoperation; EgoMimic and EgoDex use egocentric video as training data.

Purpose-built handheld rigs. A gripper-shaped device a person carries, instrumented so that what is recorded is an end-effector pose and a jaw width rather than a hand pose. UMI and DexCap are the reference points; the device has to be built and shipped to each collector.

Phone capture with object-centric retargeting. Record the person with a phone, but recover the object’s path rather than the hand’s, and let each robot re-solve its own grasp and joint motion to reproduce it. This is what DXTR does.

What does each one record, and what does it produce?

Stated as inputs and outputs, with the published work named as a reference point rather than characterised. We have not run these systems, so this table says what each approach records and emits — nothing about how well any of them works.

The last row of that table is the one this site is built on, and the row above it is why: what a method records is what it can transfer. Everything below is DXTR’s own reasoning about its own choice, and is not a measurement of anyone else’s system.

What does teleoperation actually cost you?

Not money in the first instance — coupling. A teleoperated episode is an episode of that robot. Change the arm, the gripper, the mount height, and the dataset does not follow; you collect again. That is fine when there is one robot and it is the product. It becomes the dominant cost the moment there is a fleet, because the collection budget multiplies by the number of embodiments rather than the number of tasks.

The second cost is throughput shape. Every episode consumes an operator’s undivided attention and a robot’s uptime simultaneously, so the two scarcest resources are consumed in lockstep and neither can be parallelised past the number of robots you own.

What you buy for that is real: contact that actually happened, forces that were actually exerted, and feasibility you never have to prove. No video-based method gets those for free.

Why does copying the hand not solve it?

Recording video instead of robot motion takes the robot out of the collection loop, and puts the human body in as the quantity being transferred. A five-finger grasp has no counterpart on a parallel gripper, and a wrist path recorded from a human arm is not always reachable by a robot arm, so something has to absorb the difference.

One available move is to scale the world until the motion fits, and it is the move DXTR’s contract forbids — because it does not look like a failure. A 20 cm drawer pull scaled to a quarter is a 5 cm drawer pull. The drawer stays shut. IK reports success throughout, because the targets moved with the world. The episode passes review and teaches the wrong thing. That is a property of the rescaling, not a claim about any particular published system.

What does phone capture change?

It changes what gets recorded as the goal. DXTR’s Eye app records RGB video at 1920×1440, per-frame LiDAR depth and ARKit camera pose while a person does the task eyes-free — audio coaching and hand gestures run each take, so the collector never touches the screen mid-demonstration. From that, the pipeline recovers the object’s 6-DoF path through the task, and each robot solves its own grasp and joint motion to move the object along that same path. The object is never rescaled to make a body fit.

Then it is checked. Every retargeted episode is replayed in simulation against the object’s tolerance tube. It passes, or it ships an infeasibility certificate naming what blocked it — checked in simulation at the kinematic tier: pass, fail or refused, each with its reason. That is the substitute for the feasibility guarantee teleoperation gets by construction, and it is deliberately an explicit, inspectable one.

The honest limits are the ones above: contact forces are inferred rather than measured, quality depends on object pose recovery and metric reconstruction, and tasks where nothing moves — a press, a wipe, a reach to a switch — fall back to wrist imitation for those frames, labelled as such.

Which should a team choose?

• One robot, contact-rich tasks, and it is the product. Teleoperate. Nothing else gives you real forces.
• A fleet of different embodiments. Object-centric capture, because the collection cost stops multiplying by the number of bodies.
• Broad task coverage on a budget. Phone capture — the collectors already own the hardware.
• A dataset you have to trust without re-running it. Whichever method you pick, insist on an explicit pass/refuse record per episode. An episode that cannot say why it is good is not evidence.

What DXTR runs today

The full loop — push a task, accept it, capture eyes-free, upload, score, review, approve, credit — runs end to end in production on a real phone. Eight robot embodiments are retargeted: Franka Panda, UR5e, PR2, MyCobot 280, Unitree G1, Apptronik Apollo, Fanuc LR Mate 200iD and SO-101 — seven of them in the public playground fleet — each keeping its arm clear of the reconstructed scene and each picking its own grasp where the body has a gripper whose closure we score (Apollo has none and refuses on that ground; the G1’s Dex3 closure is not yet scorable). Data ships as LeRobot datasets per robot and MCAP for the raw multimodal record. Articulated objects and bimanual and mobile-base support are the next tiers on the same certificate machinery. Every render on this site is simulation.