Complex & Intelligent Systems · 2024 · 14 citations · 36 references
ComputingComputer ScienceComputer VisionFeature EngineeringVisual AttentionImage Feature EngineeringScene UnderstandingSemantic SegmentationAttentionCosine SimilarityFeature ConcatenationFew-shot SegmentationFew-shot Semantic SegmentationTransformer ModelsTransformer TheoryTransformer-based Segmentation
Abstract Few-shot Semantic Segmentation (FSS) was proposed to segment unseen classes in a query image, referring to only a few annotated examples named support images. One of the characteristics of FSS is spatial inconsistency between query and support targets, e.g., texture or appearance. This greatly challenges the generalization ability of methods for FSS, which requires to effectively exploit the dependency of the query image and the support examples. Most existing methods abstracted support features into prototype vectors and implemented the interaction with query features using cosine similarity or feature concatenation. However, this simple interaction may not capture spatial details in query features. To address this limitation, some methods utilized pixel-level support information by computing pixel-level correlations between paired query and support features implemented with the attention mechanism of Transformer. Nevertheless, these approaches suffer from heavy computation due to dot-product attention between all pixels of support and query features. In this paper, we propose a novel framework, termed ProtoFormer, built upon the Transformer architecture, to fully capture spatial details in query features. ProtoFormer treats the abstracted prototype of the target class in support features as the Query and the query features as Key and Value embeddings, which are input to the Transformer decoder. This approach enables better capture of spatial details and focuses on the semantic features of the target class in the query image. The output of the Transformer-based module can be interpreted as semantic-aware dynamic kernels that filter the segmentation mask from the enriched query features. Extensive experiments conducted on PASCAL- $$5^{i}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:msup> <mml:mn>5</mml:mn> <mml:mi>i</mml:mi> </mml:msup> </mml:math> and COCO- $$20^{i}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:msup> <mml:mn>20</mml:mn> <mml:mi>i</mml:mi> </mml:msup> </mml:math> datasets demonstrate that ProtoFormer significantly outperforms the state-of-the-art methods in FSS.
36
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
ImageNet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher et al. · 2009 IEEE Conference on Computer Vision and Pattern Recognition · 2009 · 61.8K citations
Fully convolutional networks for semantic segmentation
Jonathan Long, Evan Shelhamer, Trevor Darrell · 2015 · 36.9K citations
Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos et al. · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2017 · 22.1K citations