Concepedia

Pretraining

Pretraining is the academic concept of a machine learning procedure in which a model is initially trained on a source task or dataset to learn generalizable features, distinguishing it from subsequent fine-tuning by focusing on broad representation learning prior to task-specific adaptation.

1.2K

Publications

135.2K

Citations

4.9K

Authors

1K

Institutions

Publications per year

2017–2026

908

Parents

Publications

1.2K

HISTORIAE, History of Socio-Cultural Transformation as Linguistic Data Science. A Humanities Use Case

Yinhan Liu, Myle Ott, Jingfei Du et al. · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · 2019 · 17.3K citations · Full text

, , +13

Understanding the difficulty of training deep feedforward neural networks

Xavier Glorot, Yoshua Bengio · 2010 · 12.7K citations

, , +13

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

Victor Sanh, Lysandre Debut, Julien Chaumond et al. · arXiv (Cornell University) · 2019 · 4.6K citations · Full text

, , +15

HuggingFace's Transformers: State-of-the-art Natural Language Processing

Thomas Wolf, Lysandre Debut, Victor Sanh et al. · arXiv (Cornell University) · 2019 · 3.1K citations · Full text

, , +12

Why Does Unsupervised Pre-training Help Deep Learning?

Dumitru Erhan, Aaron Courville, Yoshua Bengio et al. · 2010 · 2.1K citations

, , +6

1–5 of 1.2K

Authors

4.9K

  1. ZL
    Zhiyuan Liu

    Tsinghua University, China

    9 publications · 2.3K citations

    9

    h-index

  2. NA
    Noah A. Smith

    Carnegie Mellon University, United States

    8 publications · 390 citations

    8

    h-index

  3. JL
    Junyang Lin

    Peking University, China

    8 publications · 507 citations

    8

    h-index

  4. LT
    Lorenzo Torresani

    Dartmouth College, United States

    7 publications · 1K citations

    7

    h-index

  5. YB
    Yoshua Bengio

    Université de Montréal, Canada

    7 publications · 15.7K citations

    7

    h-index

1–5 of 4.9K

Institutions

1K

  1. Google (United States)

    Mountain View, United States

    33 h-index

    138 publications

    16K citations

  2. University of California, Berkeley

    Berkeley, United States

    21 h-index

    92 publications

    12K citations

  3. Stanford University

    Stanford, United States

    20 h-index

    85 publications

    10.6K citations

  4. Tsinghua University

    Beijing, China

    20 h-index

    155 publications

    43K citations

  5. Meta (Israel)

    Tel Aviv, Israel

    20 h-index

    74 publications

    13.7K citations

  6. Zhejiang University

    Hangzhou, China

    19 h-index

    66 publications

    3.7K citations

  7. Chinese Academy of Sciences

    Beijing, China

    18 h-index

    62 publications

    3.6K citations

  8. University of Cambridge

    Cambridge, United Kingdom

    17 h-index

    45 publications

    2.9K citations

1–8 of 1K

Venues

  1. arXiv (Cornell University)

    Repository · 241 publications · 27.9K citations

    58

    h-index

  2. 15

    h-index

  3. Proceedings of the AAAI Conference on Artificial Intelligence

    Conference · 18 publications · 1.2K citations

    15

    h-index

  4. PubMed

    Repository · 15 publications · 730 citations

    14

    h-index

  5. 13

    h-index

1–5