2024 · 13 citations · 26 references
ComputingFood ScienceComputer VisionFood ProcessingGenerative ArtMultimedia RetrievalMultimodal InteractionCross-modal EmbeddingCross-modal Image RetrievalCross-modal Representation LearningCross-modal RetrievalMultimodal Image RetrievalMultimodal RetrievalMultimodal Retrieval ModelsRecipe Representation
Vision-language pre-trained models have exhibited significant advancements in various multimodal and unimodal tasks in recent years, including cross-modal recipe retrieval. However, a persistent challenge in multimodal frameworks is the lack of alignment between the encoders of different modalities. Although previous works addressed image and recipe embedding alignment, the alignment of individual recipe components has been overlooked. To address this gap, we present Fine-grained Alignment for Recipe eMbeddings (FARM), a cross-modal retrieval approach that aligns the encodings of recipe components, including titles, ingredients, and instructions, within a shared representation space alongside corresponding image embeddings. Moreover, we introduce a hyperbolic loss function to effectively capture the similarity information inherent in recipe classes. FARM improves Recall@1 by 1.4% for image-to-recipe and 1.0% for recipe-to-image retrieval. Additionally, FARM achieves up to 6.1% and 15.1% performance improvement in image-to-recipe retrieval tasks, when just one and two components of the recipe are available, respectively. Comprehensive qualitative analysis of retrieved images for various recipes showcases the semantic capabilities of our trained models. Code is available at https://github.com/PLAN-Lab/FARM.
26
Laurens van der Maaten, Geoffrey E. Hinton · Journal of Machine Learning Research · 2008 · 35.7K citations
Learning Cross-Modal Embeddings for Cooking Recipes and Food Images
Amaia Salvador, Nicholas Hynes, Yusuf Aytar et al. · 2017 · 408 citations · Full text
Inverse Cooking: Recipe Generation From Food Images
Amaia Salvador, Michal Drozdzal, Xavier Giró-i-Nieto et al. · 2019 · 171 citations