GAMMA: Graspability-Aware Mobile MAnipulation
Policy Learning based on Online Grasping Pose Fusion

Jiazhao Zhang1,2*         Nandiraju Gireesh3*         Jilong Wang2         Xiaomeng Fang2
Chaoyi Xu3         Weiguang Chen2         Liu Dai4         He Wang1,2
1Peking University           2BAAI           3Galbot           4Tongji University ICRA 2024
Paper Code Video
teaser

Abstract

We propose a graspability-aware mobile manipulation approach powered by an online grasping pose fusion framework that enables a temporally consistent grasping observation. Specifically, the predicted grasping poses are online organized to eliminate the redundant, outlier grasping poses, which can be encoded as a grasping pose observation state for reinforcement learning. Moreover, on-the-fly fusing the grasping poses enables a direct assessment of graspability, encompassing both the quantity and quality of grasping poses. This assessment can subsequently serve as an observe-to-grasp reward, motivating the agent to prioritize actions that yield detailed observations while approaching the target object for grasping. Through extensive experiments conducted on the Habitat and Isaac Gym simulators, we find that our method attains a good balance between observation and manipulation, yielding high performance under various grasping metrics. Furthermore, we discover that the incorporation of temporal information from grasping poses aids in mitigating the sim-to-real gap, leading to robust performance in challenging real-world experiments.

Method

method

Our method processes the gripper depth map to region-of-interest point cloud, which then be sent to GSNet for predicting grasping poses. The predicted grasping poses will then be integrated into the so far fused grasping poses. The fused grasping poses will be encoded, along with visual information and state information for learning graspability policy.

Results

Visualization of the quality of the fused grasping poses during mobile manipulation. The grasping are color-coded based on their graspability feature (to red the better).


Real-world Grasping


   

BibTeX

@article{zhang2023gamma,
      title={GAMMA: Graspability-Aware Mobile MAnipulation Policy Learning based on Online Grasping Pose Fusion},
      author={Zhang, Jiazhao and Gireesh, Nandiraju and Wang, Jilong and Fang, Xiaomeng and Xu, Chaoyi and Chen, Weiguang and Dai, Liu and Wang, He},
      journal={arXiv preprint arXiv:2309.15459},
      year={2023}
    }