Can we learn robot manipulation for everyday tasks, only by watching videos of humans doing arbitrary tasks in different unstructured settings? Unlike widely adopted strategies of learning task-specific behaviors or direct imitation of a human video, we develop a a framework for extracting agent-agnostic action representations from human videos, and then map it to the agent’s embodiment during deployment. Our framework is based on predicting plausible human hand trajectories given an initial image of a scene. After training this prediction model on a diverse set of human videos from the internet, we deploy the trained model zero-shot for physical robot manipulation tasks, after appropriate transformations to the robot’s embodiment. This simple strategy lets us solve coarse manipulation tasks like opening and closing drawers, pushing, and tool use, without access to any in-domain robot manipulation trajectories. Our real-world deployment results establish a strong baseline for action prediction information that can be acquired from diverse arbitrary videos of human activities, and be useful for zero-shot robotic manipulation in unseen scenes.

Examples the team gives include choosing an object to use as a hammer when there’s no hammer available (the robot chooses a rock) and picking the best drink for a tired person (the robot chooses an energy drink).

“RT-2 shows improved generalization capabilities and semantic and visual understanding beyond the robotic data it was exposed to,” the researchers wrote in a Google blog post. “This includes interpreting new commands and responding to user commands by performing rudimentary reasoning, such as reasoning about object categories or high-level descriptions.”

The dream of general-purpose robots that can help humans with whatever may come up—whether in a home, a commercial setting, or an industrial setting—won’t be achievable until robots can learn on the go. What seems like the most basic instinct to us is, for robots, a complex combination of understanding context, being able to reason through it, and taking actions to solve problems that weren’t anticipated to pop up. Programming them to react appropriately to a variety of unplanned scenarios is impossible, so they need to be able to generalize and learn from experience, just like humans do.

Digital sociology explores the interplay between digital technology, social behaviors, and societal structures. As an emerging discipline, it delves into how online interactions, social media platforms, and digital tools shape and reflect cultural dynamics, personal identities, and group norms. Digital sociologists analyze trends in cyber communities, the implications of digital data collection, and the broader impact of the internet on social evolution.

In this age of rapid technological advancement — where we find ourselves in the midst of a Digital Revolution and most recently, in an AI Revolution, the fast-spinning changes in society require analysis and understanding. Digital sociology provides critical insights into the multifaceted relationship between technology and the fabric of society.