{"id":152121,"date":"2022-12-07T21:27:17","date_gmt":"2022-12-08T03:27:17","guid":{"rendered":"https:\/\/lifeboat.com\/blog\/2022\/12\/talking-to-robots-in-real-time"},"modified":"2022-12-07T21:27:17","modified_gmt":"2022-12-08T03:27:17","slug":"talking-to-robots-in-real-time","status":"publish","type":"post","link":"https:\/\/lifeboat.com\/blog\/2022\/12\/talking-to-robots-in-real-time","title":{"rendered":"Talking to Robots in Real Time"},"content":{"rendered":"<p style=\"padding-right: 20px\"><a class=\"aligncenter blog-photo\" href=\"https:\/\/lifeboat.com\/blog.images\/talking-to-robots-in-real-time.gif\"><\/a><\/p>\n<p>A grand vision in robot learning, going back to the <a href=\"https:\/\/hci.stanford.edu\/~winograd\/shrdlu\/AITR-235.pdf\">SHRDLU experiments<\/a> in the late 1960s, is that of helpful robots that inhabit human spaces and follow a wide variety of natural language commands. Over the last few years, there have been significant advances in the application of machine learning (ML) for <a href=\"https:\/\/arxiv.org\/abs\/1906.03926\">instruction following<\/a>, both <a href=\"https:\/\/language-play.github.io\/\">in<\/a> <a href=\"https:\/\/arxiv.org\/abs\/2112.03763\">simulation<\/a> and in real world systems. Recent <a href=\"https:\/\/sites.research.google\/palm-saycan\">Palm-SayCan<\/a> work has produced robots that leverage language models to plan long-horizon behaviors and reason about abstract goals. <a href=\"https:\/\/ai.googleblog.com\/2022\/11\/robots-that-write-their-own-code.html\">Code as Policies<\/a> has shown that code-generating language models combined with pre-trained perception systems can produce language conditioned policies for zero shot robot manipulation. Despite this progress, an important missing property of current \u201clanguage in, actions out\u201d robot learning systems is <em>real time<\/em> interaction with humans.<\/p>\n<p>Ideally, robots of the future would react in real time to any relevant task a user could describe in natural language. Particularly in open human environments, it may be important for end users to customize robot behavior as it is happening, offering quick corrections (\u201cstop, move your arm up a bit\u201d) or specifying constraints (\u201cnudge that <em>slowly<\/em> to the right\u201d). Furthermore, real-time language could make it easier for people and robots to collaborate on complex, long-horizon tasks, with people iteratively and interactively guiding robot manipulation with occasional language feedback.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A grand vision in robot learning, going back to the SHRDLU experiments in the late 1960s, is that of helpful robots that inhabit human spaces and follow a wide variety of natural language commands. Over the last few years, there have been significant advances in the application of machine learning (ML) for instruction following, both [\u2026]<\/p>\n","protected":false},"author":556,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20,6],"tags":[],"class_list":["post-152121","post","type-post","status-publish","format-standard","hentry","category-futurism","category-robotics-ai"],"_links":{"self":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/posts\/152121","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/users\/556"}],"replies":[{"embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/comments?post=152121"}],"version-history":[{"count":0,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/posts\/152121\/revisions"}],"wp:attachment":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/media?parent=152121"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/categories?post=152121"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/tags?post=152121"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}