IEEE Transactions on Visualization and Computer Graphics · 2023 · 114 citations · 54 references
ComputingComputer ScienceVisualizationInformation RetrievalInteractive VisualizationNatural Language ProcessingInteractive Data ExplorationInteractive Generative SystemsMultimodal InteractionVisual Data ExplorationCross-modal EmbeddingExpert InterviewsExplorationNatural Language InterfacesPrompt-based GenerationText-to-image GenerationUser Study Evaluation
Generative text-to-image models have gained great popularity among the public for their powerful capability to generate high-quality images based on natural language prompts. However, developing effective prompts for desired images can be challenging due to the complexity and ambiguity of natural language. This research proposes PromptMagician, a visual analysis system that helps users explore the image results and refine the input prompts. The backbone of our system is a prompt recommendation model that takes user prompts as input, retrieves similar prompt-image pairs from DiffusionDB, and identifies special (important and relevant) prompt keywords. To facilitate interactive prompt refinement, PromptMagician introduces a multi-level visualization for the cross-modal embedding of the retrieved images and recommended keywords, and supports users in specifying multiple criteria for personalized exploration. Two usage scenarios, a user study, and expert interviews demonstrate the effectiveness and usability of our system, suggesting it facilitates prompt engineering and improves the creativity support of the generative text-to-image model.
54
Laurens van der Maaten, Geoffrey E. Hinton · Journal of Machine Learning Research · 2008 · 35.7K citations
Training language models to follow instructions with human feedback
Long Ouyang, Jeff Wu, Xu Jiang et al. · arXiv (Cornell University) · 2022 · 4.3K citations · Full text
Adding Conditional Control to Text-to-Image Diffusion Models
Lvmin Zhang, Anyi Rao, Maneesh Agrawala · 2023 · 3.6K citations