Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval · 2022 · 111 citations · 12 references
ComputingComputer ScienceInformation RetrievalMachine LearningNatural Language ProcessingDeep LearningDocument RepresentationLanguage ModelingLanguage ModelsRepresentation LearningRetrieval ModelsDense RetrievalHard Negative MiningInverted IndexesLexical MatchingNegative SamplingNeural Information RetrievalSparse RetrievalZero-shot Retrieval
Neural retrievers based on dense representations combined with Approximate Nearest Neighbors search have recently received a lot of attention, owing their success to distillation and/or better sampling of examples for training -- while still relying on the same backbone architecture. In the meantime, sparse representation learning fueled by traditional inverted indexing techniques has seen a growing interest, inheriting from desirable IR priors such as explicit lexical matching. While some architectural variants have been proposed, a lesser effort has been put in the training of such models. In this work, we build on SPLADE -- a sparse expansion-based retriever -- and show to which extent it is able to benefit from the same training improvements as dense models, by studying the effect of distillation, hard-negative mining as well as the Pre-trained Language Model initialization. We furthermore study the link between effectiveness and efficiency, on in-domain and zero-shot settings, leading to state-of-the-art results in both scenarios for sufficiently expressive models.
12
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Nils Reimers, Iryna Gurevych · 2019 · 10.9K citations · Full text
Omar Khattab, Matei Zaharia · 2020 · 1.1K citations
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction
Keshav Santhanam, Omar Khattab, Jon Saad-Falcon et al. · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · 2022 · 277 citations · Full text