2024 · 18 citations · 50 references
ComputingAutonomous SystemsComputer ScienceComputer VisionMachine LearningSensor FusionDeep LearningRepresentation LearningRobotic VisionScene UnderstandingSemantic SegmentationVideo AnalysisDual-encoder ModelsEncoder-decoder ArchitecturesSemantic RetrievalSpatial Context FusionSpatio-temporal RepresentationSpiking Network ArchitecturesSpiking Neural Network ModelsVisual Signal Processing
Event cameras detect changes in per-pixel intensity to generate asynchronous ‘event streams’. They offer great potential for accurate semantic map retrieval in real-time autonomous systems owing to their much higher temporal resolution and high dynamic range (HDR) compared to conventional cameras. However, existing implementations for event-based segmentation suffer from sub-optimal performance since these temporally dense events only measure the varying component of a visual signal, limiting their ability to encode dense spatial context compared to frames. To address this issue, we propose a hybrid end-to-end learning framework HALSIE, utilizing three key concepts to reduce inference cost by up to 20× versus prior art while retaining similar performance: First, a simple and efficient cross-domain learning scheme to extract complementary spatio-temporal embeddings from both frames and events. Second, a specially designed dual-encoder scheme with Spiking Neural Network (SNN) and Artificial Neural Network (ANN) branches to minimize latency while retaining cross-domain feature aggregation. Third, a multi-scale cue mixer to model rich representations of the fused embeddings. These qualities of HALSIE allow for a very lightweight architecture achieving state-of-the-art segmentation performance on DDD-17, MVSEC, and DSEC-Semantic datasets with up to 33× higher parameter efficiency and favorable inference cost (17.9mJ per cycle). Our ablation study also brings new insights into effective design choices that can prove beneficial for research across other vision tasks.
50
Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos et al. · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2017 · 22.1K citations
Xception: Deep Learning with Depthwise Separable Convolutions
François Chollet · 2017 · 19K citations
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi et al. · 2017 · 15.5K citations
Advances in Neural Information Processing Systems 28
Peter Flach, Meelis Kull · 2015 · 14.7K citations
Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
Xingjian Shi, Zhourong Chen, Hao Wang et al. · arXiv (Cornell University) · 2015 · 6.6K citations · Full text