Imagine a surgeon being able to “step inside” a digital version of a patient’s body — studying organs, tissues, and complex structures, rehearsing procedures, and evaluating possible approaches before performing the actual operation.

The integration of robotic surgical assistants (RSAs) in operating rooms offers substantial advantages for both surgeons and patient outcomes. Currently operated through teleoperation by trained surgeons at a console, these surgical robot platforms provide augmented dexterity that has the potential to streamline surgical workflows and alleviate surgeon workloads. Exploring visual behavior cloning for next-generation surgical assistants could further enhance the capabilities and efficiency of robotic-assisted surgeries.

This post introduces two template frameworks for robotic surgical assistance: Surgical First Interactive Autonomy Assistant (SuFIA) and Surgical First Interactive Autonomy Assistant – Behavior Cloning (SuFIA-BC). SuFIA uses natural language guidance and large language models (LLMs) for high-level planning and control of surgical robots, while SuFIA-BC enhances the dexterity and precision of robotic surgical assistants through behavior cloning (BC) techniques. These frameworks explore the recent advances in both LLMs and BC techniques and tune them to excel to the unique challenges of surgical scenes.

This research aims to accelerate the development of surgical robotic assistants, with the eventual goal of alleviating surgeon fatigue, enhancing patient safety, and democratizing access to high-quality healthcare. SuFIA and SuFIA-BC advance this field by demonstrating their capabilities across various surgical subtasks in simulated and physical settings. Moreover, the photorealistic assets introduced in this work enable the broader research community to explore surgical robotics—a field that has traditionally faced significant barriers to entry due to limited data accessibility, the high costs of expert demonstrations, and the expensive hardware required.