����volg

Read how AIML’s leading research is bridging the 3D / 2D domain gap.

Humans move effortlessly around our rich and detailed three-dimensional world without much second thought. But, like most mammals, our eyes actually sense the world two-dimensionally — it’s our brains that take those 2D images and interpret them into a 3D understanding of reality.

Even without the stereo visual input from our two eyes, we’re experts at looking at a flat 2D image and instantly ‘lifting’ it back to its 3D origins; we do it every time we watch TV or look at photos on our phone.

But computers and robots have a much harder time doing this ‘lifting’. It’s a problem that AI researchers are hard at work to fix.

Making computers able to understand 3D space from only 2D input is considered such an important capability—with diverse applications ranging from mobile phones to driverless vehicles—that Professor Simon Lucey, director of the Australian Institute for Machine Learning (AIML), has received a $435,000 grant from the Australian ����volg Council to build a geometric reasoning system that can exhibit human-like performance.

AIML researchers are among the world’s leaders in computer vision, a field of AI that enables computers to obtain meaningful information from digital images and video footage.

Building computer vision systems that can understand the real world typically requires vast troves of labeled training data using something called supervised machine learning. That means millions of images, each labeled ‘dog’, ‘strawberry’ or ‘President Obama’; or thousands of hours of driving footage where coloured boxes are drawn to mark each pedestrian, stop sign and traffic light. If you’ve ever had a website ask you to ‘click all the squares with bicycles’ to prove you’re really human, you’ve helped train a supervised machine learning model.

AI researchers are using the vast collections of labeled 2D training data, and working out how to apply it so AI systems can develop a 3D geometric understanding similar to what humans can do.

“How can I take 2D supervision that humans can easily provide,” asks Professor Lucey, “and, using some elegant math, allow it to act as 3D supervision for modern AI systems?”

One application of this kind of computer vision is something called 3D motion capture, where earlier advances brought us Gollum in The Lord of the Rings movies. It’s still a popular technique and one that’s widely used in film visual effects, video game production and even medicine and sports science. But even today it uses a number of expensive and finely calibrated cameras, and sometimes still requires people to wear special reflective dots on their body and perform in front of a greenscreen, and that’s a problem.

“People want the data they’re collecting to be realistic…they don’t want a white background. They don’t want a green screen. They would love to be out in the field, or in areas that are highly unconstrained. And the sheer cost of this limits the application of technology at the moment,” says Professor Lucey. “You can only apply it to problems where companies are willing to invest millions to build these things.”