Imagine working at a warehouse or office sometime in the near future, and you’re asked to help a new trainee learn the basics of their job. The catch: It’s a robot. To teach them, you might want to play a game of “show and tell”—that is, physically showing how to do something a few different ways, while also explaining what you’re doing.
Let’s say you asked the robot to place some coffee on your desk without disturbing you during a Zoom call. You’ll prefer that the robot doesn’t get close to you and the laptop so that it doesn’t interrupt your meeting. To enable this behavior, the robot should be trained with data that clearly demonstrates the full task. Computer scientists have attempted to explain manipulation tasks to robots by recording lots of physical demonstrations or writing extensive directions. But if you don’t have both, the machine is likely to misunderstand what it needs to do.
It’s laborious for humans to do all that showing and telling, so researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have automated the process of teaching a robot, while clarifying instructions automatically and using nearly five times less demonstration data.
This is a fascinating development in the intersection of LLMs and robotics. The ability to parse vague instructions into actionable steps has so many applications beyond what the paper covers. I have been exploring how AI-generated video content can help explain complex robotics concepts to broader audiences. Platforms like vidglory.com make it easy to create visual demonstrations of these robotic learning processes. Would love to see follow-up research on how these systems handle truly ambiguous multi-step instructions in real-world scenarios.