What if robots could learn dexterous skills by watching videos on the internet?
Teaching robots to manipulate objects is challenging because collecting training data is expensive, time-consuming, and difficult to scale across different tasks and environments. Humans, on the other hand, demonstrate countless everyday skills in videos uploaded online. The challenge is finding the right demonstrations that can help a robot learn how to perform a specific action.
A new system called RoboTok tackles this problem by turning internet videos into a searchable resource for robot learning. Given a reference video of a person performing a manipulation task, RoboTok searches for other videos showing similar hand movements and interactions with objects. Rather than relying mainly on visual similarities, it analyzes the motion trajectories of the hands in three-dimensional space, using a representation centered on the person performing the action.
This approach allows the system to recognize similar manipulation behaviors even when videos differ in camera angles, backgrounds, visual appearances, or when parts of the person’s body are obscured. Because the motion representation is compact, RoboTok can also index and search large collections of online videos efficiently.
The researchers evaluated the system using video-retrieval benchmarks and real-world robot experiments. Their results showed that RoboTok retrieved more relevant demonstrations than existing approaches and helped robots perform manipulation tasks more successfully. In real-world tests, RoboTok-guided robot policies achieved an average improvement of 42.2 percentage points in task success rates compared with the strongest baseline for each task.
The broader implication is significant: the internet could become a vast, continuously growing source of training data for robots. Instead of relying exclusively on costly, specially collected robot demonstrations, researchers could extract useful manipulation examples from videos of humans performing everyday tasks.
RoboTok does not mean robots can automatically master every skill simply by watching videos. They still need suitable learning methods and robot-specific training. But by making human demonstrations easier to find and reuse, this approach offers a promising path toward more scalable robot learning and more dexterous machines.
#ArtificialIntelligence #robotics #ImitationLearning #DexterousManipulation.
