Sep 20, 2022
Just say the magic word: using language to program robots
Posted by Shubham Ghosh Roy in category: robotics/AI
Language is the most intuitive way for us to express how we feel and what we want. However, despite recent advancements in artificial intelligence, it is still very hard to control a robot using natural language instructions. Free-form commands such as “Robot, please go a little slower when you pass close to my TV” or “Stay far away from the swimming pool!” are hard to parse into actionable robot behaviors, and most human-robot interfaces today still rely on complex strategies such directly programming cost functions which define the desired behavior.
With our latest work, we attempt to change this reality through the introduction of “LaTTe: Language Trajectory Transformer”. LaTTe is a deep machine learning model that lets us send language commands to robots in an intuitive way with ease. When given an input sentence by the user, the model fuses it with camera images of objects that the robot observes in its surroundings, and outputs the desired robot behavior.
Continue reading “Just say the magic word: using language to program robots” »