We propose ASGrasp, a 6-DoF Grasp detection network that uses an RGB-D Active Stereo camera, to tackle the problem of grasping transparent and specular objects.
1Samsung R&D Institute China-Beijing
2CFCS, Peking University
3Gallbot
4Beijing Academy of Artificial Intelligence (BAAI)
† corresponding author
We propose ASGrasp, a 6-DoF Grasp detection network that uses an RGB-D Active Stereo camera, to tackle the problem of grasping transparent and specular objects.
In this paper, we tackle the problem of grasping transparent and specular objects. This issue holds importance, yet it remains unsolved within the field of robotics due to failure of recover their accurate geometry by depth cameras. For the first time, we propose ASGrasp, a 6-DoF grasp detection network that uses an RGB-D active stereo camera. ASGrasp utilizes a two-layer learning-based stereo network for the purpose of transparent object reconstruction, enabling material-agnostic object grasping in cluttered environments. In contrast to existing RGB-D based grasp detection methods, which heavily depend on depth restoration networks and the quality of depth maps generated by depth cameras, our system distinguishes itself by its ability to directly utilize raw IR and RGB images for transparent object geometry reconstruction. We create an extensive synthetic dataset through domain randomization, which is based on GraspNet-1Billion. Our experiments demonstrate that ASGrasp can achieve over 90% success rate for generalizable transparent object grasping in both simulation and the real via seamless sim-to-real transfer. Our method significantly outperforms SOTA networks and even surpasses the performance upper bound set by perfect visible point cloud inputs.
The architecture of our proposed approach involves the extraction of features \(F_c\), \({F_{ir}^{l}}\), and \({F_{ir}^{r}}\) from the RGB image \({I_c}\), the left IR image \({I_{ir}^{l}}\), and the right IR image \({I_{ir}^{r}}\). \({F_{ir}^{l}}\) and \({F_{ir}^{r}}\) are used to construct a correlation pyramid. This correlation pyramid, along with \({F_c}\), is fed into a GRU network for the prediction of a complete point cloud, which includes the first-layer depth (visible point cloud) and the second-layer depth (invisible point cloud). Subsequently, GSNet generates grasp poses based on the complete point cloud.
@article{shi2024gsnet,
title={ASGrasp: Generalizable Transparent Object Reconstruction and 6-DoF Grasp Detection from RGB-D Active Stereo Camera},
author={Jun Shi, Yong A, Yixiang Jin, Dingzhe Li, Haoyu Niu, Zhezhu Jin, He Wang},
journal={arXiv preprint arXiv:2405.05648},
year={2024}
}
If you have any questions, please feel free to contact us: