2021 · 20 citations · 32 references
ComputingComputer ScienceComputational LinguisticsMachine LearningNatural Language ProcessingLanguage ModelingRepresentation LearningContextual EmbeddingsContextualized EmbeddingsEmbedding LearningLinguistic Structure
Understanding how linguistic structure is encoded in contextualized embedding could help explain their impressive performance across NLP. Existing approaches for probing them usually call for training classifiers and use the accuracy, mutual information, or complexity as a proxy for the representation's goodness. In this work, we argue that doing so can be unreliable because different representations may need different classifiers. We develop a heuristic, DIRECTPROBE, that directly studies the geometry of a representation by building upon the notion of a version space for a task. Experiments with several linguistic tasks and contextualized embeddings show that, even without training classifiers, DIRECTPROBE can shine light into how an embedding space represents labels, and also anticipate classifier performance for the representation.
32
Adam: A Method for Stochastic Optimization
Diederik P. Kingma, Jimmy Ba · UvA-DARE (University of Amsterdam) · 2014 · 84.8K citations · Full text
Scikit-learn: Machine Learning in Python
Fabián Pedregosa, Gaël Varoquaux, Alexandre Gramfort et al. · arXiv (Cornell University) · 2012 · 63.7K citations · Full text
Chih-Chung Chang, Chih‐Jen Lin · ACM Transactions on Intelligent Systems and Technology · 2011 · 41.3K citations
Libsvm : A library for support vector machines
Vittorio Ferrari · Medical Entomology and Zoology · 2008 · 10.1K citations