2024 · 11 citations · 20 references
ComputingComputer ScienceLinguisticsNatural Language ProcessingPsycholinguisticsVisual CommunicationLanguage ModelingMultimodal CorporaMultimodal InteractionOptical Character RecognitionSpeech RecognitionText GenerationAutomatic Multimodal AnnotationBenchmark Dataset EvaluationBenchmark DatasetsLinguistic BoundariesManga IllustrationReading OrderReading Order Detection
In the past few decades, Japanese comics, commonly referred to as Manga, have transcended both cultural and linguistic boundaries to become a true worldwide sensation. Yet, the inherent reliance on visual cues and illustration within manga renders it largely inaccessible to individuals with visual impairments. In this work, we seek to address this substantial barrier, with the aim of ensuring that manga can be appreciated and actively engaged by everyone. Specifically, we tackle the problem of diarisation i.e. generating a transcription of who said what and when, in a fully automatic way. To this end, we make the following contributions: (1) we present a unified model, Magi, that is able to (a) detect panels, text boxes and character boxes, (b) cluster characters by identity (without knowing the number of clusters apriori), and (c) associate dialogues to their speakers; (2) we propose a novel approach that is able to sort the detected text boxes in their reading order and generate a dialogue transcript; (3) we annotate an evaluation benchmark for this task using publicly available [English] manga pages. The code, evaluation datasets and the pretrained model can be found at: https://github.com/ragavsachdeva/magi.
20
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
Decoupled Weight Decay Regularization
Ilya Loshchilov, Frank Hutter · arXiv (Cornell University) · 2017 · 9.1K citations · Full text
Deformable DETR: Deformable Transformers for End-to-End Object Detection
Xizhou Zhu, Weijie Su, Lewei Lu et al. · arXiv (Cornell University) · 2020 · 1.9K citations · Full text
DINOv2: Learning Robust Visual Features without Supervision
Maxime Oquab, Timothée Darcet, Théo Moutakanni et al. · arXiv (Cornell University) · 2023 · 1K citations · Full text
The VIA Annotation Software for Images, Audio and Video
Abhishek Dutta, Andrew Zisserman · 2019 · 973 citations · Full text