New tech developed by U of A researchers turns rooms into virtual screens for 3D imaging

University of Arizona researchers have developed technology that transforms everyday surroundings into virtual screens, helping machines see three-dimensional scenes more accurately than current systems.

The breakthrough addresses a major challenge in 3D imaging where most sensors struggle to capture both shiny and matte surfaces simultaneously, according to the university. Real-world environments contain a mix of reflective materials like mirrors and glass alongside dull surfaces like walls and fabric.

Florian Willomitzer, an associate professor at the U of A Wyant College of Optical Sciences, leads the research team. The work was published in Nature Communications.

“One of our goals is to enable computers and machines to see in 3D better than any human, which is crucial for a multitude of technological challenges, such as reliable navigation of self-driving cars, accurate guidance during robotic surgery or improved sensing capabilities in industrial inspection and biomedical imaging,” Willomitzer said.

The technology uses a laser scanner to capture everything in a room, then separates shiny surfaces from matte ones using algorithms. The matte surfaces essentially become a giant virtual screen that helps measure the reflective objects.

“This effectively allows us to repurpose everything inside that room to into a giant display – essentially everything around you becomes a virtual screen,” Willomitzer said.

The system uses a neuromorphic event camera that captures only important parts of measurements at high speeds. This allows the technology to handle vastly different light levels and measure all surfaces accurately despite variations in reflectivity.

The approach has been demonstrated in laboratory settings but can scale to different applications, from measuring small blood vessels during surgery to digitizing entire buildings. The work was supported by the National Science Foundation and the Defense Advanced Research Projects Agency.