Project Starline Ultra-Real Video Chat Is Google’s Coolest Project

There are a few constants in the universe, like the speed of light, but you would not expect the “average laptop webcam quality” to be so constant that it has barely evolved for years.

2020 made it painfully evident that video communication needs a massive revamp, and PC OEMs like Lenovo have slowly improved that.

But Google has taken the whole thing to a few levels above everything we know about teleconferencing. Project Starline (official page) is a photorealistic video-conferencing system that relies on 3D capture instead of video capture.

As such, the subjects appear to have volume and be physically present right in front of you because your eyes and brain can perceive small details such as parallax, a critical ingredient that distinguishes a nice, flat, 2D image and a real-world-like 3D perception.

Starline works by capturing each person as a 3D model with a camera array and transmitting all the data to the other side, to be rendered 30 times or more per second! Note: all the images below come from Google.

This young woman and the baby (image below) are rendered in real-time 3D. Think about that for a moment…

One of the challenges here is the low latency it takes to make the conversation seem more life-like.

Any communications lag would crush the realism of the experience, but fortunately, Google owns a worldwide uber-fast fiber-optic network and can leverage it to its advantage. That’s also one strength of its Google Cloud service.

This technology uses a 3D light field screen to display images with a true “3D feel” on the receiving end. The best light field displays are a topic of intense research and investment.

The basic idea is that the display constantly shows multiple views rendered at various angles, and depending on your position relative to the screen, you see the most appropriate view. A higher number of views would yield a more accurate 3D image and cost a lot more to render.

In the present demo, the system can scan enough space to have a person-to-person conversation, but not a group. However, the general principle should extend to more expansive spaces and it becomes a matter of computing power in the real-to-3D conversion process and rendering.

In the previous animation, the demo also shows a person holding a baby, but the area to 3D-scan is the same as a one-person chat.

The demo is set up in a booth-like environment, probably to avoid challenging light situations, and I do not think that Google is trying to interfere with the lighting at the moment (in theory, it would be possible to remove+replace the original lighting).

However, this is very promising, and I can see how such a system could evolve to smaller form-factors and perhaps into TVs someday.

Starlight might be something that makes it into the wealthiest corporate environments in the coming years, but I expect this to remain “sci-fi” for regular consumers. That said, it is the coolest demo I’ve seen from Google in many years.

Hubert Nguyen
Categories: General
Tags: Google, google i/o, Social Hit