Artificial intelligence from Facebook turns 2D images into 3D

Artificial intelligence from Facebook turns 2D images into 3D

Facebook is investing in artificial intelligence that can create three-dimensional models from images of objects. This should help with VR and AR.


When we see a Cup in front, we think about the back end — whether it’s in front of us or in the pictures. We understand that our world has three spatial dimensions.


This ability helps us to identify objects, even when they move. This, in turn, is the basis for orientation and navigation in our environment.


Therefore, artificial intelligence which considers the objects in the pictures as three-dimensional, is an important step towards more accurate object recognition, robotic navigation, or the best applications of augmented reality.


New AI technique detects 3D to 2D


Facebook shows a new AI, which, like the human brain, can infer 3D form from 2D images. “Grid R-CNN” identificeret the object in the image and generates buy a simple 3D model.


First, the AI creates a rough model of voxels (Voxel — formed of words: volume (eng. volumetric) and pixel (eng. pixel) — element of volume image containing value of an element of a raster in three-dimensional space), which then converts it to a more accurate model of the polygon at the second stage. This also works for more complex objects such as a chair leg or furniture, hiding each other.


The AI was trained using matching pairs of images and polygons from a dataset ShapeNet. It includes items such as furnishings and electronic devices.



Facebooks KI erstellt erst ein Voxel-Modell aus dem zweidimensionalen Foto und verfeinert es dann zu eiem Polygon-Modell. Bild: Facebook


3D face recognition without training data


But the training datasets for the 3D models, such as ShapeNet, it is difficult to build. Not every real object has a 3D digital model. This is reflected in the volume data sets: ShapeNet covers a little less than 3000 categories. The ImageNet dataset, which is commonly used for recognition of 2D-objects, contains 21 800 categories.


From Facebook may be the answer to the problem of learning the Canonical network 3D poses (briefly: C3DPO): AI can output so-called 3D key points for different objects from the images. These points together form a kind of three-dimensional skeleton of the object. The researchers successfully tested the AI on 14 categories of objects, including birds, people and cars.


AI Facebook


3D reconstruction from 2D images is still not possible, says Facebook. The technology even allows you to calculate 3D models of large objects for which hardware 3D, the picture is not so simple, such as buildings or airplanes.


Facebook wants to continue to study, because the digital world is changing towards 3D photographs and experience AR and VR. According to the blog post, progress in AI is absolutely necessary for a better understanding of and interaction with 2D content.



Go to our cases Get a free quote