IEEE/ACM Transactions on Computational Biology and Bioinformatics · 2020 · 42 citations · 12 references
Life SciencesBioinformaticsComputer ScienceComputational LinguisticsGenomicsNatural Language ProcessingDeep LearningGenome AnnotationGenomic AnnotationSequence AnnotationSequence LabelingTranscription Start SitesConvolutional Neural NetworksTranscription Start Site AnnotationTranscription Start Site MappingTranscriptional Start SitesTransformer ModelsWhole-genome Sequence Analysis
In genomics, a wide range of machine learning methodologies have been investigated to annotate biological sequences for positions of interest such as transcription start sites, translation initiation sites, methylation sites, splice sites and promoter start sites. In recent years, this area has been dominated by convolutional neural networks, which typically outperform previously-designed methods as a result of automated scanning for influential sequence motifs. However, those architectures do not allow for the efficient processing of the full genomic sequence. As an improvement, we introduce transformer architectures for whole genome sequence labeling tasks. We show that these architectures, recently introduced for natural language processing, are better suited for processing and annotating long DNA sequences. We apply existing networks and introduce an optimized method for the calculation of attention from input nucleotides. To demonstrate this, we evaluate our architecture on several sequence labeling tasks, and find it to achieve state-of-the-art performances when comparing it to specialized models for the annotation of transcription start sites, translation initiation sites and 4mC methylation in E. coli.
12
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · 2023 · 76.3K citations · Full text
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context
Zihang Dai, Zhilin Yang, Yiming Yang et al. · 2019 · 3.1K citations · Full text