NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation

Jiazhao Zhang1,2,*       Kunyu Wang2,*       Rongtao Xu2,3,*       Gengze Zhou4       Yicong Hong5 Xiaomeng Fang2      Qi Wu4      Zhizheng Zhang6,†      He Wang1,2,†
1 Peking University       2Beijing Academy of Artificial Intelligence       3CASIA 4University of Adelaide       5Australian National University       6GalBot *Indicates Equal Contribution,  †Indicates Equal Advising.
Paper

示例图片

Highlights

  • NaVid is the first video-based Vision-Language Model (VLM) for the task of vision-and-language navigation (VLN).
  • NaVid navigates in a human-like manner, requiring solely an on-the-fly video stream from a monocular camera as input, without the need for maps, odometers, or depth inputs.
  • NaVid incorporates 510K VLN video sequences from simulation environments and 763K real-world caption samples to achieve cross-scene generalization.
  • NaVid achieves state-of-the-art (SOTA) performance in both simulated and real-world environments, and exhibits strong generalizability on unseen scenarios.

Sim-to-Real Demos: Simple Instruction VLN

In these demos, the agent navigates following relatively simple instructions, such as walking to a single landmark. NaVid demonstrates the ability to accurately distinguish differences in similar instructions and accordingly complete precise navigation behaviors.

Sim-to-Real Demos: Complex Instruction VLN

In these demos, the agent navigates according to complex instructions composed of multiple simple instructions in sequence. NaVid can accurately execute them in the correct order.

Abstract

Vision-and-Language Navigation (VLN) stands as a key research problem of Embodied AI, aiming at enabling agents to navigate in unseen environments following linguistic instructions. In this field, generalization is a long-standing challenge, either to out-of-distribution scenes or from Sim to Real. In this paper, we propose NaVid, a video-based large vision language model (VLM), to mitigate such a generalization gap. NaVid makes the first endeavour to showcase the capability of VLMs to achieve state-of-the-art level navigation performance without any maps, odometer and depth inputs. Following human instruction, NaVid only requires an on-the-fly video stream from a monocular RGB camera equipped on the robot to output the next-step action. Our formulation mimics how humans navigate and naturally gets rid of the problems introduced by odometer noises, and the Sim2Real gaps from map or depth inputs. Moreover, our video-based approach can effectively encode the historical observations of robots as spatio-temporal contexts for decision making and instruction following. We train NaVid with 510k navigation samples collected from VLN-CE trajectories, including action planning and instruction-reasoning samples, along with 763k large-scale web data. Extensive experiments show that NaVid achieves SOTA performance in simulation environments and the real world, demonstrating superior cross-dataset and Sim2Real transfer. We thus believe our proposed VLM approach plans the next step for not only the navigation agents but also this research field. We will release the code and data to benefit the community.

Method Overview

NaVid

The overview of NaVid. The inputs of NaVid consist of the RGB frames from the online video observation {x0, · · · , xt} along with the human instruction I. For each frame, we use an observation encoder to extract the visual information with the instruction to obtain observation tokens, including, instruction-queried tokens (orange blocks) and instruction-agnostic tokens (blue blocks). At the current step t, the history frames and current frame xt are encoded as observation tokens, with 4 and 64 instruction-agnostic tokens for history frames and current frames, respectively. Besides, our method obtains language tokens by a text encoder. Finally, split by the special tokens [HIS], [OBS], and [NAV], we concatenate the observation tokens and language tokens and send the tokens to the Vicuna-7B then obtain the next-step action.

Data Collection

NaVid

We co-train NaVid using real-world caption data (763k) and simulated VLN data (510k). The simulated VLN data consists of 500k action planning samples and 10k instruction reasoning samples.

Results of Navigation Video Captioning

Given an egocentric RGB video, describe the trajectory using NaVid.

NaVid: Walk forward and turn left. Wait by the first door on the left.
NaVid: Walk forward and turn left. Wait near the first doorway.
NaVid: Turn around and walk to the other side of the room. Wait by the chair and table.
NaVid: Exit the bathroom. Turn left and enter the bedroom. Wait near the bed.
NaVid: Walk through the doorway and turn left. Walk past the pool and wait by the first chair.
NaVid: Turn around and go into the hallway. Go into the room with the stairs and turn right. Go into the room with the kitchen and wait there.

Cross-scene Generalization Results on R2R

(R2R training split -> R2R validation unseen split)

Cross-scene Generalization Results from R2R to RxR

(R2R training split -> RxR validation unseen split )

BibTeX

@article{zhang2024navid,
        title={NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation},
        author={Zhang, Jiazhao and Wang, Kunyu and Xu, Rongtao and Zhou, Gengze and Hong, Yicong and Fang, Xiaomeng and Wu, Qi and Zhang, Zhizheng and Wang, He},
        journal={arXiv preprint arXiv:2402.15852},
        year={2024}
      }