What if robots could learn dexterous skills by watching videos on the internet?
Teaching robots to manipulate objects is challenging because collecting training data is expensive, time-consuming, and difficult to scale across different tasks and environments. Humans, on the other hand, demonstrate countless everyday skills in videos uploaded online. The challenge is finding the right demonstrations that can help a robot learn how to perform a specific action.
A new system called RoboTok tackles this problem by turning internet videos into a searchable resource for robot learning. Given a reference video of a person performing a manipulation task, RoboTok searches for other videos showing similar hand movements and interactions with objects. Rather than relying mainly on visual similarities, it analyzes the motion trajectories of the hands in three-dimensional space, using a representation centered on the person performing the action.
This approach allows the system to recognize similar manipulation behaviors even when videos differ in camera angles, backgrounds, visual appearances, or when parts of the person’s body are obscured. Because the motion representation is compact, RoboTok can also index and search large collections of online videos efficiently.
The researchers evaluated the system using video-retrieval benchmarks and real-world robot experiments. Their results showed that RoboTok retrieved more relevant demonstrations than existing approaches and helped robots perform manipulation tasks more successfully. In real-world tests, RoboTok-guided robot policies achieved an average improvement of 42.2 percentage points in task success rates compared with the strongest baseline for each task.
The broader implication is significant: the internet could become a vast, continuously growing source of training data for robots. Instead of relying exclusively on costly, specially collected robot demonstrations, researchers could extract useful manipulation examples from videos of humans performing everyday tasks.
RoboTok does not mean robots can automatically master every skill simply by watching videos. They still need suitable learning methods and robot-specific training. But by making human demonstrations easier to find and reuse, this approach offers a promising path toward more scalable robot learning and more dexterous machines.
#ArtificialIntelligence #robotics #ImitationLearning #DexterousManipulation.
Can robots learn dexterous skills by watching internet videos?
What if the millions of videos showing humans manipulating objects could become a training ground for more capable robots?
Join us for a discussion on “RoboTok: A Scalable Data Engine for Internet Demonstration Video Retrieval and Dexterous Manipulation Learning,” featuring Howard Qian of Rice University.
RoboTok introduces a scalable approach to finding relevant human manipulation demonstrations from internet videos to help robots learn physical skills. By analyzing 3D hand-motion trajectories, the system can identify similar actions even when camera angles, backgrounds, and visual conditions differ.
In real-world experiments, RoboTok-guided robot policies achieved an average improvement of 42.2 percentage points in task success over the strongest baseline for each task.
Could the vast amount of human activity captured on video help overcome one of robotics’ biggest challenges: the cost and complexity of collecting training data?
Come explore the research, its implications for dexterous manipulation, and the future of robot learning!
📅 Date: October 15, 2026 🕓 Time: 4:00 PM PDT 🎙️ Speaker: Howard Qian, Rice University.
🎤 Hosted by: Cecile G. Tamura of the Quantum Photonics Club, with Kevin Filson and Lihua Tan.
🔗 Join us on Clubhouse:
[ https://www.clubhouse.com/invite/ZzRHWZYE7JRYa5jYKn6n37OJdK4NS5pA4qK:
W/ howard qian of rice university.