IEEE Transactions on Computational Social Systems · 2022 · 14 citations · 33 references
ComputingComputer ScienceComputational LinguisticsNatural Language ProcessingPsycholinguisticsContext DependenceConversational InterfacesConversational User InterfacesDialogue SystemsLanguage ModelingText GenerationBenchmark DatasetsContextContextual DependenciesDialog SystemsDialogue Response GenerationMulti-turn Conversation ModelingTransformer Models
Transformer structure has shown promising results in multiturn dialog generation. The self-attention mechanism can learn global dependencies but ignores local information, limiting the model’s ability to model context information. In this article, we propose an information-enhanced hierarchical self-attention network (IEHSA). In the word-level encoder, words in successive windows are automatically encoded as local information, and words with dependent words are automatically encoded as syntactic information, both of which are used to enhance word information and then feed into the self-attention mechanism. In the utterance-level encoder, adjacent utterance representations are automatically encoded as dialog structure information, and the self-attention mechanism is used to update the utterance representations. The context and masked response representation are then updated using the self-attention mechanism in the decoder. Finally, the correlation between context and reply is calculated and used in further decoding. We compared IEHSA with the current popular hierarchical model on several datasets, and the experiments show that the proposed method has substantial improvements in both metric-based and human evaluations.
33
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · 2023 · 76.3K citations · Full text
Dropout: a simple way to prevent neural networks from overfitting
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky et al. · 2014 · 34.3K citations
Sequence to Sequence Learning with Neural Networks
Ilya Sutskever, Oriol Vinyals, Quoc V. Le · arXiv (Cornell University) · 2014 · 13.4K citations · Full text
Computing, Computer Science, French +14
Recurrent neural network based language model
Tomáš Mikolov, Martin Karafiát, Lukáš Burget et al. · 2010 · 5.4K citations