Researchers from Google, Columbia University and Synthesis AI, a data generation platform for computer vision, together developed ClearGrasp, a machine learning algorithm, that is capable of estimating accurate 3D data of transparent objects from RGB-D images (combination of a RGB image and its corresponding depth images). ClearGrasp works with inputs from any standard RGB-D camera, using deep learning, to accurately reconstruct the depth of transparent objects and generalise to completely new objects. 

Optical sensors such as cameras and lidar are a fundamental part of modern robotics platforms. However, transparent objects like glass containers tend to confuse them since algorithms analysing data from those sensors assume all surfaces reflect light evenly in all directions and from all angles. By contrast, transparent objects both refract and reflect light. This makes depth data invalid.

Previous methods required prior knowledge of the transparent objects (like 3D models) and often combined with maps of background lighting and camera positions. ClearGrasp can benefit robotic manipulation by integrating it into Google’s pick and place robot’s control system. This has significant improvements in the grasping success rate of transparent plastic objects, notes the research team.

ClearGrasp comprises three machine learning algorithms in total. A network to estimate surface normals, one for occlusion boundaries and one that masks transparent objects. This mask removes all pixels belonging to transparent objects. Thus the correct depths can be filled in, and an optimisation module can extend the surface’s depth using predicted surface normals to guide the reconstruction’s shape.  

Training sophisticated AI models requires large data sets. The researchers created their own data sets containing more than 50,000 photorealistic renders with corresponding depth, edges, surface normals, and more. Each image shows up to five transparent objects, either on a flat ground plane or inside a tote with various backgrounds and lighting. And a separate set of 286 real-world images with corresponding ground truth depth serves as a test set.

In experiments, the researchers trained the models on their custom data set, as well as real indoor scenes from the open-source Matterport3D and ScanNet corpora. ClearGrasp managed to reconstruct depth for transparent objects with much higher fidelity than the baseline methods, and that its output depth could be directly used as input to manipulation algorithms that use images. When using a robot parallel-jaw gripper arm, the gripping success rate of transparent objects improved from 12 to 74 per cent, and from 64 to 86 per cent with suction.