2024 · 17 citations · 52 references
ComputingComputer ScienceEducational TechnologyComputer VisionLanguage AcquisitionNatural Language ProcessingDeep LearningLanguage ModelingMultimodal AnalysisMultimodal CommunicationMultimodal InteractionMultimodal LiteracyBiasesContext-based LearningIn-context LearningLarge Language ModelsMajority VotingMajority Voting SystemsMultimodal Language LearningMultimodal Machine Learning
Large Language Models have demonstrated remarkable performance across various tasks, exhibiting the capacity to swiftly acquire new skills, such as through In-Context Learning (ICL) with minimal demonstration examples. In this work, we present a comprehensive framework for investigating Multimodal ICL (M-ICL) in the context of Large Multimodal Models. We consider the best open-source multimodal models (e.g., IDEFICS, OpenFlamingo) and a wide range of multimodal tasks. Our study unveils several noteworthy findings: (1) M-ICL primarily relies on text-driven mechanisms, showing little to no influence from the image modality. (2) When used with advanced-ICL strategy (like RICES), M-ICL is not better than a simple strategy based on majority voting over context examples. Moreover, we identify several biases and limitations of M-ICL that warrant consideration prior to deployment. Code available at gitlab.com/folbaeni/multimodal-icl
52
ImageNet Large Scale Visual Recognition Challenge
Olga Russakovsky, Jia Deng, Hao Su et al. · International Journal of Computer Vision · 2015 · 40.2K citations
Advances in neural information processing systems 7
Computers & Mathematics with Applications · 1996 · 14.4K citations