Plato-NeRF is a computer vision system that combines LiDAR measurements with machine learning to reconstruct a 3D scene.Researchers at the Massachusetts Institute of Technology have developed a computer vision technique to create highly accurate three-dimensional models of objects hidden from view. The researchers used a single camera position to capture shadows and machine learning to determine what was behind the obscured portions of a scene.
“Our key idea was taking these two things that have been done in different disciplines before and pulling them together — multibounce lidar and machine learning,” said Tzofi Klinghoffer, a graduate at MIT who was involved in the research. “It turns out that when you bring these two together, you find many new opportunities to explore and get the best of both worlds. Reconstructing a scene using a single-camera viewpoint is challenging.
While some light pulses bounce off a point and return directly to the sensor, others bounce off to other objects before returning to the sensor. PlatoNeRF works with these secondary bounces to view objects hidden from direct view. The system then uses secondary rays of light to determine which points lie in the shadow and use it to determine the geometry of hidden objects. metamorworks/iStockThe LiDAR and NeRF can construct an entire 3D scene by sequentially illuminating 16 points and capturing their images.