A team of researchers from the University of Maryland has made significant progress in turning some of the advanced enhancement tools depicted in crime investigation shows like CSI into a reality. Their latest breakthrough involves reconstructing interactive 3D models of what a person was looking at in a short video clip using the reflections captured in their eyes.
While the eyes are often referred to as the window to the soul, this research suggests that they can serve as mirrors that provide valuable data for recreating visual information. Leveraging the concept of neural radiance field (NeRF) technology, which enables the creation of 3D models from a limited set of 2D images, the scientists focused on the cornea of the eye, which remains relatively consistent across healthy adults. By analyzing the imagery extracted from eye reflections, they successfully generated 3D representations of simple scenes.
Image: “Eye Reflections” by Martin Cathrae
Some limitations and challenges
It is important to note that this innovative approach is not intended for crime-solving purposes. The process comes with its own set of challenges. Unlike traditional methods that require high-quality source material, this technique extracts reflections from a small, low-resolution portion of each frame. The complex textures of the iris and variations in its color among individuals further complicate the post-processing stage.
Another obstacle is the limitation imposed by the source footage, as the 2D images used for reconstruction originate from the same location with minimal variations caused by eye movements. Unlike capturing a comprehensive 3D model with a smartphone and moving around the subject, this approach relies on fixed positioning.
Consequently, the resulting 3D models exhibit low resolution and lack intricate details. While objects like plush toys can still be identified in basic scenes with deliberate lighting and high-resolution source imagery, the application of this method to footage captured under less controlled conditions proves challenging.
For example, when attempting to reconstruct a clip from Miley Cyrus’s “Wrecking Ball” music video sourced from YouTube, the resulting 3D model becomes indiscernible. The vague representation likely corresponds to a hole in a white shroud seen through the camera lens to achieve the specific shot’s visual effect.
Although this research is captivating, it will take time before it can be practically applied in real-world scenarios. The current limitations in resolution and detail prevent its immediate use as a reliable tool. Nonetheless, the advancements made by the University of Maryland team open up intriguing possibilities for further exploration in the field of 3D reconstruction and visualization.
Filed in . Read more about Science.