KTH Publication Database DiVA (KTH Royal Institute of Technology) · 2021 · 86 citations · 24 references
Open access
ComputingComputer ScienceComputational LinguisticsMachine LearningNatural Language ProcessingDeep LearningLanguage ModelingLanguage ModelsNatural Language UnderstandingRepresentation LearningSelf-supervised LearningBiasesContrastive LearningLanguage Model PretrainingSelf-supervised PretrainingSemantic SimilaritySemantic Textual SimilarityTransformer Language ModelsTransformer Models
Extracting semantically useful natural language sentence representations frompre-trained deep neural networks such as Transformers remains a challenge. Wefirst demonstrate that pre-training objectives impose a significant task bias ontothe final layers of models, with a layer-wise survey of the Semantic Textual Similarity (STS) correlations for multiple common Transformer language models. Wethen propose a new self-supervised method called Contrastive Tension (CT) tocounter such biases. CT frames the training objective as a noise-contrastive taskbetween the final layer representations of two independent models, in turn makingthe final layer representations suitable for feature extraction. Results from multiple common unsupervised and supervised STS tasks indicate that CT outperformsprevious State Of The Art (SOTA), and when combining CT with supervised datawe improve upon previous SOTA results with large margins.
24
Learning representations by back-propagating errors
David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams · Nature · 1986 · 30.9K citations
Distributed Representations of Sentences and Documents
Quoc V. Le, Tomáš Mikolov · arXiv (Cornell University) · 2014 · 5.1K citations · Full text
How transferable are features in deep neural networks?
Jason Yosinski, Jeff Clune, Yoshua Bengio et al. · arXiv (Cornell University) · 2014 · 3.5K citations · Full text