Many of today’s self-driving cars “see” the road through two senses at once. Cameras capture color, texture and the outline of a pedestrian, while light detection and ranging (LiDAR) fires laser pulses to measure exactly how far away that pedestrian stands. Together, they provide complementary information and outperform either sensor alone.

This pairing has a weakness, however. Trained mostly on clear daytime footage, the fusion model learns to lean on whichever sensor performs best there and may fail in challenging scenarios. Rain can scatter LiDAR beams, for instance, while a night drive leaves cameras battling darkness and glare.

To address this, assistant professor Zhao Na at Singapore University of Technology and Design (SUTD) led two studies that built different models. In “CCF: Complementary collaborative fusion for domain generalized multi-modal 3D object detection,” the research team rebalances how a detector weighs its sensors so it holds up in new surroundings. In “PanDA: Unsupervised domain adaptation for multimodal 3D panoptic segmentation in autonomous driving,” they harness unlabeled data from a new environment to help the model adapt once it arrives.