Video

Abstract

Collaborative perception has recently shown great potential to improve perception capabilities over single-agent perception. Existing collaborative perception methods usually consider an ideal communication environment. However, in practice, the communication system inevitably suffers from latency issues, causing potential performance degradation and high risks in safety-critical applications, such as autonomous driving. To mitigate the effect caused by the inevitable latency, from a machine learning perspective, we present the first latency-aware collaborative perception system, which actively adapts asynchronous perceptual features from multiple agents to the same time stamp, promoting the robustness and effectiveness of collaboration. To achieve such a featurelevel synchronization, we propose a novel latency compensation module, called SyncNet, which leverages feature-attention symbiotic estimation and time modulation techniques. Experiments results show that the proposed latency aware collaborative perception system with SyncNet can outperforms the state-of-the-art collaborative perception method by 15.6% in the communication latency scenario and keep collaborative perception being superior to single agent perception under severe latency.

Contribution

To the best of our knowledge, We formulate the communication latency challenge in collaborative perception for the first time and propose a novel latency-aware collaborative perception system, which promotes robust multi-agents perception by mitigating the effect of inevitable communication latency.

We propose a novel latency compensation module, termed SyncNet, to achieve feature-level synchronization. It achieves symbiotic estimation of two types of critical collaboration information, including intermediate features and collaboration attention, mutually enhancing each other.

We conduct comprehensive experiments to show that our proposed SyncNet achieves huge performance improvement in latency scenarios compared with the previous method and keeps collaborative perception being superior to single agent perception under severe latency.

Method

Overview of the proposed latency-aware collaborative perception system: The key module is a latency compensation module. To realize this, we propose SyncNet, which leverages historical collaboration information to synchronize the asynchronized information from multiple agents caused by the latency issue.

Qualitative Results

Collaborative 3D detection. Red: Detected, green: Ground truth. Collaboration without considering latency could be even worse than no collaboration.The figure shows that the detection results of DiscoNet without latency, DiscoNet with latency, DiscoNet+VE and DiscoNet + SyncNet. Comparing (a) with (b), we see that the correctly detected vehicles in the purple box in (a) are missed or incorrectly detected in (b) due to the latency. (c) shows that the vanilla estimation partially compensates latency error in the blue box but fails to achieve accurate estimation in the orange box, while our SyncNet could precisely recover the true position of both vehicles, shown in purple box of (d). Plot (d) shows that SyncNet achieves the best compensation and precisely recovers the true position of vehicles.

BibTeX

@inproceedings{lei2022latency,
  title={Latency-aware collaborative perception},
  author={Lei, Zixing and Ren, Shunli and Hu, Yue and Zhang, Wenjun and Chen, Siheng},
  booktitle={European Conference on Computer Vision},
  pages={316--332},
  year={2022},
  organization={Springer}
  }

Latency-aware collaborative perception

Accepted to ECCV 2022