By Raji Natarajan,
Special to Rice News
Picture a busy street during rush hour with vehicles zipping past and pedestrians rushing by. While humans can effortlessly perceive and interpret such dynamic scenes in real time, current 3D imaging technologies often struggle to create accurate representations of rapidly changing environments. A new study published in Nature Communications by researchers at Rice University and the University of Arizona introduces an innovative approach that could help machines see the world more clearly.
Ashok Veeraraghavan, chair of the Department of Electrical and Computer Engineering in Rice’s George R. Brown School of Engineering and Computing, and Aniket Dashpute, a graduate student in his lab, in collaboration with associate professor Florian Willomitzer and his team at the University of Arizona’s Wyant College of Optical Sciences, developed a two-step computational imaging method that uses a laser and a high-speed camera to capture complex scenes in three dimensions with exceptional speed and accuracy.
The key innovation lies in the system’s ability to transform ordinary matte surfaces into virtual screens. By using surrounding walls, furniture, clothing and other nonreflective surfaces as part of the imaging process, the technique enables accurate 3D reconstruction of scenes containing both matte and reflective objects, a long-standing challenge in computer vision.
The advance could improve machine vision for applications ranging from industrial inspection and facial recognition to human sensing and autonomous vehicles.
Most 3D imaging systems rely on structured light, which projects patterns onto a scene and measures how those patterns deform across object surfaces to create depth maps. While widely used, these systems can struggle with motion, challenging lighting conditions and scenes containing both matte and reflective materials. In mixed-reflectance environments, light bouncing between surfaces can distort measurements and reduce image quality.
“To address these challenges, we leveraged a well-established technique in computer vision to devise a novel solution that turns all matte surfaces in a scene into virtual screens by projecting light onto these surfaces,” said Veeraraghavan, a professor of electrical and computer engineering and computer science and a member of the Ken Kennedy Institute and the Houston Methodist-Rice Digital Health Institute. “Deflectometry is typically used to measure the shape of shiny surfaces by observing how projected light patterns distort upon reflection. While precise, it traditionally requires large, carefully positioned screens, making it expensive and inflexible.”
“We used a laser to scan and create precise 3D maps of matte surfaces such as walls, clothing and furniture,” said Dashpute, the first author on the study. “When laser points reflect off a shiny object, the system effectively repurposes the surrounding matte surfaces into virtual display screens.”
The researchers then used a neuromorphic event camera, which records changes in light intensity rather than capturing full image frames, to reconstruct high-speed 3D videos.
“The event camera can handle vastly different light levels, from very dim to extremely bright,” said Jiazhang Wang, a postdoctoral research associate at the Wyant College of Optical Sciences and the paper’s second author. “This allows us to measure all object surfaces in a scene with equally high accuracy and speed, regardless of variations in their reflectivity.”
While the technology has so far been demonstrated in a tabletop laboratory setting, the researchers say the approach can scale to a wide range of applications.
“Scalability is an important requirement for 3D imaging,” Willomitzer said. “Whether measuring tiny reflective blood vessels during surgery or digitizing entire rooms and buildings, this approach offers the flexibility needed to adapt to very different environments and scales.”
Other authors on the study include James Taylor, a doctoral student at the Wyant College of Optical Sciences, and Oliver Cossairt, adjunct associate professor of electrical and computer engineering at Northwestern University. The research was supported by grants from the National Science Foundation and the Defense Advanced Research Projects Agency.
Adapted from a text by Daniel Stolte at University of Arizona.