ASGrasp: Generalizable Transparent Object Reconstruction and 6-DoF Grasp Detection from RGB-D Active Stereo Camera

ICRA 2024

Jun Shi1*    Yong A1    Yixiang Jin1    Dingzhe Li1    Haoyu Niu1    Zhezhu Jin1    He Wang2,3,4†   

1Samsung R&D Institute China-Beijing    2CFCS, Peking University    3Gallbot    4Beijing Academy of Artificial Intelligence (BAAI)   

corresponding author  


We propose ASGrasp, a 6-DoF Grasp detection network that uses an RGB-D Active Stereo camera, to tackle the problem of grasping transparent and specular objects.


In this paper, we tackle the problem of grasping transparent and specular objects. This issue holds importance, yet it remains unsolved within the field of robotics due to failure of recover their accurate geometry by depth cameras. For the first time, we propose ASGrasp, a 6-DoF grasp detection network that uses an RGB-D active stereo camera. ASGrasp utilizes a two-layer learning-based stereo network for the purpose of transparent object reconstruction, enabling material-agnostic object grasping in cluttered environments. In contrast to existing RGB-D based grasp detection methods, which heavily depend on depth restoration networks and the quality of depth maps generated by depth cameras, our system distinguishes itself by its ability to directly utilize raw IR and RGB images for transparent object geometry reconstruction. We create an extensive synthetic dataset through domain randomization, which is based on GraspNet-1Billion. Our experiments demonstrate that ASGrasp can achieve over 90% success rate for generalizable transparent object grasping in both simulation and the real via seamless sim-to-real transfer. Our method significantly outperforms SOTA networks and even surpasses the performance upper bound set by perfect visible point cloud inputs.


Full pipeline


The architecture of our proposed approach involves the extraction of features \(F_c\), \({F_{ir}^{l}}\), and \({F_{ir}^{r}}\) from the RGB image \({I_c}\), the left IR image \({I_{ir}^{l}}\), and the right IR image \({I_{ir}^{r}}\). \({F_{ir}^{l}}\) and \({F_{ir}^{r}}\) are used to construct a correlation pyramid. This correlation pyramid, along with \({F_c}\), is fed into a GRU network for the prediction of a complete point cloud, which includes the first-layer depth (visible point cloud) and the second-layer depth (invisible point cloud). Subsequently, GSNet generates grasp poses based on the complete point cloud.


We visualize a qualitative comparison of the predictions for both layers. The simulated raw point cloud contains missing and wrong points. Our method performs much better in both the first and second layers, yielding a high-quality complete point cloud reconstruction.


We tested our method in real-world experiments, showing sim2real results. The test cases demonstrate significant potential in our methods for grasping transparent and specular objects.





  title={ASGrasp: Generalizable Transparent Object Reconstruction and 6-DoF Grasp Detection from RGB-D Active Stereo Camera},
  author={Jun Shi, Yong A, Yixiang Jin, Dingzhe Li, Haoyu Niu, Zhezhu Jin, He Wang},
  journal={arXiv preprint arXiv:2405.05648},


If you have any questions, please feel free to contact us:

  • Jun Shi: jun7.shiPrevent spamming@Prevent
  • He Wang: hewangPrevent spamming@Prevent