If “to err is human,” you could say robots are superhuman. They’re more error-prone than we are, even when tasked with things like home and factory chores, which we might consider easy. One reason for these mistakes is that robots struggle to extract relevant details from the past and act on them in real time, even with vision-language models (VLMs), which interpret visual inputs such as videos and images along with natural language.

Recent approaches use AI agents, which are semiautonomous programs based on VLMs, that, in this case, decide when robots should interpret specific details from a long history of camera frames. However, this can lead to errors, high compute costs and poor reactivity.

A new approach called “Workspace Models” from researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) and Carnegie Mellon University’s Machine Learning Department (MLD) reformulates the agentic pipeline used to compress history for robots.