2024 · 17 citations · 39 references
ComputingComputer ScienceComputer VisionNatural Language ProcessingPsycholinguisticsDeep LearningLanguage ModelingMultimodal CompositionMultimodal CorporaMultimodal InteractionText GenerationFew-shot LearningLarge Language ModelsMultimodal FusionMultimodal Machine LearningMultimodal Representation LearningSequence-to-sequence GenerationSynthetic Image GenerationText-to-image Generation
We present CoDi-2, a Multimodal Large Language Model (MLLM) for learning in-context interleaved multimodal representations. By aligning modalities with languagefor both encoding and generation, CoDi-2 empowers Large Language Models (LLMs) to understand modality-interleaved instructions and in-context examples and autoregressively generate grounded and coherent multimodal outputs in an any-to-any input-output modality paradigm. To train CoDi-2, we build a large-scale generation dataset encompassing in-context multimodal instructions across text, vision, and audio. CoDi-2 demonstrates a wide range of zero-shot and few-shot capabilities for tasks like editing, exemplar learning, composition, reasoning, etc. CoDi-2 surpasses previous domain-specific models on tasks such as subject-driven image generation, vision transformation, and audio editing and showcases a significant advancement for integrating diverse multimodal tasks with sequential generation.
39
LLaMA: Open and Efficient Foundation Language Models
Hugo Touvron, Thibaut Lavril, Gautier Izacard et al. · arXiv (Cornell University) · 2023 · 3.9K citations · Full text
Adding Conditional Control to Text-to-Image Diffusion Models
Lvmin Zhang, Anyi Rao, Maneesh Agrawala · 2023 · 3.6K citations
Audio Set: An ontology and human-labeled dataset for audio events
Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman et al. · 2017 · 3K citations
Llama 2: Open Foundation and Fine-Tuned Chat Models
Hugo Touvron, Louis Martin, Kevin H. Stone et al. · arXiv (Cornell University) · 2023 · 2.6K citations · Full text