2017 · 29 citations · 17 references
ComputingComputer ScienceComputational LinguisticsMachine LearningNatural Language ProcessingDeep LearningLanguage ModelingLanguage ModelsMachine TranslationWord EmbeddingsIncremental LearningLanguage Model PretrainingNegative SamplingNeural Language ModelsSamplingSkip-gram Model
This paper explores an incremental training strategy for the skip-gram model with negative sampling (SGNS) from both empirical and theoretical perspectives. Existing methods of neural word embeddings, including SGNS, are multi-pass algorithms and thus cannot perform incremental model update. To address this problem, we present a simple incremental extension of SGNS and provide a thorough theoretical analysis to demonstrate its validity. Empirical experiments demonstrated the correctness of the theoretical analysis as well as the practical usefulness of the incremental algorithm.
17
Distributed Representations of Words and Phrases and their Compositionality
Tomáš Mikolov, Ilya Sutskever, Kai Chen et al. · arXiv (Cornell University) · 2013 · 18.1K citations · Full text
Software Framework for Topic Modelling with Large Corpora
Petr Sojka · 2010 · 3.8K citations · Full text