A person holds two handheld grippers, one per hand, and does the task themselves. No robot in the loop — the gripper records what a robot would need to reproduce the motion.
Two deliveries. The first is short pick-and-place, zips, lids and bins on a bare table. The second is long-form cleaning in real rooms: spray, wipe, flip, re-wipe, put back — up to three minutes per take, with the operator's own segment-by-segment labels.
Each gripper has a camera-based gel pad on both fingers. The gel image is recorded at 120 fps; the vendor derives deformation, shear and depth fields and a six-axis force from it at 60 Hz.
Both grippers are tracked at about 1 kHz by their own stereo-inertial units and placed in one world frame off an AprilGrid. The operator wears a stereo headset whose head pose runs at 200 Hz.
The thirteen episodes below are drawn from a 622-episode delivery recorded over five days on two identical rigs — one on a white table (pens and pencil cases), the other mostly on a black table (boxes, bins and a backpack), with ten bin takes on the white table as well. Every episode passed the vendor's quality check, both gripper trajectories aligned to the AprilGrid, and none is missing a file. Six tasks, eight objects; counts below are exact, every episode was opened.
White table, 08-07. One to three pens lie on the table; the right gripper drops each into a white holder. The left gripper is carried open. 24–44 s per take.
08-11. The left gripper pinches the case body, the right takes the slider and pulls the zip open. 6–19 s per take.
08-11 / 12. The same two-hand zip on a stiffer navy case with a smaller slider. 7–14 s per take.
Black table, second rig, 08-10. A hinged blue box sits open; the left gripper steadies the base while the right flips the lid shut. 4–6 s per take.
08-10. The same close on a lighter clear box whose lid snaps — the right pads peak higher than on the blue box. 4–8 s per take.
08-11, second rig. A small backpack with a double-slider zip; each gripper takes one slider and they pull apart in opposite directions. 7–12 s per take.
08-05 / 10 / 11, second rig. A white bin lies on its side; the right gripper grips the rim, lifts it and sets it upright. Ten takes are on the white table, the rest on the black one. 7–18 s per take.
08-10, second rig. A light-blue bin starts upside down; the right gripper grips it, lifts and turns it over. 6–7 s per take.
Long-form cleaning work, recorded in offices, at a window, in front of filing cabinets and a shelving unit. One towel, one spray bottle, one target object: spray it, wipe it, turn it over, wipe again, put it back where it came from. Takes run 47 to 264 seconds against 4 to 46 in the first delivery, and every take arrives with the operator's own segment-by-segment labels — start and end times, which hand, what was done, in plain language. Ten of the eleven takes below are one per object family.
Mugs and a thermos flask. Wipe, spray, wipe again, set down on a magazine. 143–264 s.
Open plastic boxes, a storage bin, trays. Inner and outer wall, one stood on its edge to reach the base. 104–204 s.
Cola, mineral water, a capped plastic bottle. Alcohol spray over the body, then wiped dry. 62–82 s.
Tins and a glass preserve jar, two of them lifted off a shelving unit and put back. 69–98 s.
A crisp packet — it deforms under the towel, so the pads never settle. 81 s.
A rectangular carton off the shelf, wiped on the desk, returned. 103 s.
Notebooks, two comic books wiped on both faces and stacked square, a pencil, a folder from a cabinet. 47–112 s.
An air-conditioner remote, front then back, then into its box. 173 s.
A round glass ashtray — pens taken out of it first. 114 s.
A desk top, a window sill and its glass, the face of a glass cabinet, a shelf board. One take swaps the towel and the sprayer between hands mid-job. 79–142 s.
| take | task window | ego offset | L travel | R travel | L peak | R peak | contacts L · R |
|---|