Where2comm: Communication-Efficient Collaborative Perception via Spatial Confidence Maps

Accepted by NeurIPS 2022 Spotlight

Yue Hu1, Shaoheng Fang1, Zixing Lei1, Yiqi Zhong2, Siheng Chen1
1Cooperative Medianet Innovation Center, Shanghai Jiao Tong University
2Department of Engineering Science, University of Southern California
Paper arXiv Code Youtube Bilibili

Video




Collaborative perception could contribute to safety-critical scenarios, where the white car and the red car may collide due to occlusion. This collision could be avoided when the blue car can share a message about the red car's position. Such a message is spatially sparse, yet perceptually critical. Considering the precious communication bandwidth, each agent needs to speak to the point!

Abstract

Multi-agent collaborative perception could significantly upgrade the perception performance by enabling agents to share complementary information with each other through communication. It inevitably results in a fundamental trade-off between perception performance and communication bandwidth. To tackle this bottleneck issue, we propose a spatial confidence map, which reflects the spatial heterogeneity of perceptual information. It empowers agents to only share spatially sparse, yet perceptually critical information, contributing to where to communicate.

Contribution

  • We propose a novel fine-grained spatial-aware communication strategy, where each agent can decide where to communicate and pack messages only related to the most perceptually critical spatial areas. This strategy not only enables more precise support for other agents, but also more targeted request from other agents in multi-round communication.
  • We propose Where2comm, a novel collaborative perception framework based on the spatial-aware communication strategy. With the guidance of the proposed spatial confidence map, Where2comm leverages novel message packing and communication graph learning to achieve lower communication bandwidth, and adopts confidence-aware multi-head attention to reach better perception performance.
  • We conduct extensive experiments to validate Where2comm achieves state-of-the-art performance-bandwidth trade-off on multiple challenging real/simulated datasets across views and modalities.

    • Method



    Framework of Where2comm, a multi-round, multi-modality, multi-agent collaborative perception framework based on a spatial-confidence-aware communication strategy. Where2comm includes an observation encoder, a spatial confidence generator, the spatial confidence-aware communication module, the spatial confidence-aware message fusion module and a detection decoder. Among five modules, the proposed spatial confidence generator generates the spatial confidence map. Based on this spatial confidence map, the proposed spatial confidence-aware communication generates compact messages and sparse communication graphs to save communication bandwidth; and the proposed spatial confidence-aware message fusion module leverages informative spatial confidence priors to achieve better aggregation.

    Qualitative  Results


    Where2comm qualitatively outperforms When2com and DiscoNet in DAIR-V2X dataset. Green and Red boxes denote ground-truth and detection, respectively. Yellow and blue denote the point clouds collected from vehicle and infrastructure, respectively.



    BibTeX

    @article{Hu22Where2comm,
      author    = {Yue Hu, Shaoheng Fang, Zixing Lei, Yiqi Zhong, Siheng Chen},
      title     = {Where2comm: Communication-Efficient Collaborative Perception via Spatial Confidence Maps},
      booktitle={Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS)}
      month     = {November},  
      year      = {2022}
  }