Teaching robots to feel
Robots learned to see from the internet. There is no internet for touch — so I started building it. A $3K glove, a sensorless robot hand, and the first measured proof: force in the data controls force in the robot.
Robots learn to see from video. But video has no sense of touch — a camera cannot tell how hard a hand is gripping. So I built a glove that records what fingers feel, and taught a robot hand to hold a flimsy plastic cup without crushing it. Three policies, trained on the same demonstrations: one feels force, one doesn't, one only sees. The one that feels holds the cup with half the force. The one that only sees can't grasp it at all.
The result
Same demonstrations. Same success. Half the grip force.
50 autonomous attempts, pre-registered, none excluded:
- Both policies grasped the cup 25 of 25.
- Without force: squeezes at 2.55 N.
- With force: holds at 1.20 N — the way the human did.
- 53% less grip force (p < 0.0001).
And a policy that only sees — no sense of force or of its own fingers? 0 of 15. It repeats the average motion blindly and misses the cup.
Force is measured from the hand's motor currents — the same instrument for every policy, and the headline statistic is a rank test, so it doesn't depend on calibration. The vision-only round ran in a separate matched session with control runs grasping cleanly in the same light.
How this compares
| Work | Team | Robot needs touch sensors? | Measured gain from touch |
|---|---|---|---|
| OSMO — Meta FAIR (Dec 2025) | 10 authors | Yes — the robot wears the same glove | Wiping: 55.8% → 71.7% |
| 3D-ViTac — Columbia | university lab | Yes — sensor pads on the gripper | +20–45 points across four tasks |
| This work (Aug 2026) | 1 person, $3K | No — zero sensors on the robot | Half the grip force; vision-only collapses (0/15) |
Different tasks, so results aren't directly comparable — the point is the setting: prior work transfers touch to robots that carry touch sensors. This transfer crosses hardware: from human fingertips to a robot with none.
Why this matters
The frontier labs say it themselves: robot learning has hit the limits of vision-only data. NVIDIA's DreamZero paper names tactile sensing as the missing modality; NVIDIA's Cosmos world-model family has no touch channel at all. And 100,000+ hours of egocentric video were just released for free — without a single Newton of force in them.
Touch cannot be scraped. It has to be measured — with hardware, on real hands, doing real work. That is what this project builds: the capture rig, the calibrated data, and the proof that it transfers to robots — including robots with no touch sensors of their own. A person wearing the glove just does the task; no robot, no teleoperation, no lab. Collection scales like video. The skills transfer like software.
The glove
One microcontroller, one clock, three senses:
- 5 flex sensors — finger curl
- 5 fingertip force sensors (FSRs)
- 1 wrist IMU — orientation
Everything off-the-shelf and cheap — a rig anyone could rebuild.
The robot
An Aero Hand (5 fingers, 7 motors) on a static SO-101 arm. The glove can drive it live for testing; a calibrated mapping translates finger poses, and torque limits keep the grip gentle.
The task
Grasp a flimsy plastic cup, lift it, don't crush it. The arm rides a fixed path; the policy controls only the fingers. Chosen on purpose: a camera can't see grip force, and the failure modes are obvious — too hard crushes, too soft drops.
The data
73 sessions, ≈38,000 synchronized frames — me, wearing the glove, doing the task. No robot, no teleoperation during collection. Each frame: the camera image, finger pose, fingertip force, and the next hand pose as the action.
Honesty notes: thumb and index FSRs are calibrated to Newtons against a digital scale (the other fingers carry little signal in this pinch grip), and the thumb channel is imputed from the index due to a hardware flaw — one reliable independent force channel plus a symmetric estimate, not two.
How it's tested
Three identical policies (ACT, trained with LeRobot — same data, same steps, same seed). Only the input differs: camera + hand state + force · camera + hand state · camera only.
The robot hand has no tactile sensors. At deployment, force is estimated from its own motor currents — touch learned from human fingertips, running on sensorless hardware.
Everything was pre-registered before the first scored run, no runs excluded — and earlier rounds that failed their gates are reported as failed.
What's next
- From one task to a catalog of touch-rich skills — and from one robot hand to any robot hand.
- A short technical writeup with calibrated force axes.
- A downloadable dataset sample in LeRobot format.
- A better glove: denser force sensing, shift-length recording in real workplaces.
Working on manipulation, robot learning, or tactile sensing? I'd genuinely like to compare notes — email me or say hello on LinkedIn.