2019 · 13 citations · 16 references
ComputingComputer ScienceComputational LinguisticsNatural Language ProcessingSyntaxMachine TranslationNatural Language GenerationNatural Language UnderstandingPart-of-speech TaggingText GenerationText SummarizationConvolutional Neural NetworksLanguage GenerationPositional EncodingSequence GenerationSequence-to-sequence GenerationSequence-to-sequence TaggingSummarizationTransformer Models
The Transformer based neural networks have been showing significant advantages on most evaluations of various natural language processing and other sequence-tosequence tasks due to its inherent architecture based superiorities. Although the main architecture of the Transformer has been continuously being explored, little attention was paid to the positional encoding module. In this paper, we enhance the sinusoidal positional encoding algorithm by maximizing the variances between encoded consecutive positions to obtain additional promotion. Furthermore, we propose an augmented Transformer architecture encoded with additional linguistic knowledge, such as the Part-of-Speech (POS) tagging, to boost the performance on some natural language generation tasks, e.g., the automatic translation and summarization tasks. Experiments show that the proposed architecture attains constantly superior results compared to the vanilla Transformer.
16
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
Sepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 98.1K citations
Kishore Papineni, Salim Roukos, Todd J. Ward et al. · 2001 · 21.5K citations · Full text
Distributed Representations of Words and Phrases and their Compositionality
Tomáš Mikolov, Ilya Sutskever, Kai Chen et al. · arXiv (Cornell University) · 2013 · 18.1K citations · Full text