Concepedia

Pretraining Objectives

Pretraining Objectives is a formal specification of the learning goal or task used to guide the initial training phase of a language model on large-scale unlabeled text data, distinguishing it from downstream fine-tuning objectives by focusing on general representation learning through self-supervised signals such as next-word prediction or masked token reconstruction.

286

Publications

26.7K

Citations

1.3K

Authors

273

Institutions

Publications per year

2017–2026

250

Variants

Pre-training Objectives

Parents

Publications

286

Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

Robert Tinn, Hao Cheng, Michael Lucas et al. · ACM Transactions on Computing for Healthcare · 2021 · 2K citations · Full text

, , +12

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Zhilin Yang · arXiv (Cornell University) · 2019 · 1.9K citations · Full text

, , +16

Cross-lingual Language Model Pretraining

Guillaume Lample, Alexis Conneau · arXiv (Cornell University) · 2019 · 1.6K citations · Full text

, , +12

Pre-trained models for natural language processing: A survey

Xipeng Qiu, Tianxiang Sun, Yige Xu et al. · Science China Technological Sciences · 2020 · 1.5K citations · Full text

, , +13

Conditional Prompt Learning for Vision-Language Models

Kaiyang Zhou, Jingkang Yang, Chen Change Loy et al. · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · 2022 · 1.5K citations

, , +14

1–5 of 286

Authors

1.3K

  1. QV
    Quoc V. Le

    Google (United States), United States

    7 publications · 484 citations

    7

    h-index

  2. XQ
    Xipeng Qiu

    Fudan University, China

    6 publications · 2.6K citations

    6

    h-index

  3. XR
    Xiang Ren

    University of Southern California, United States

    5 publications · 114 citations

    5

    h-index

  4. SR
    Samuel R. Bowman

    New York University, United States

    5 publications · 351 citations

    5

    h-index

  5. LZ
    Luke Zettlemoyer

    University of Washington, United States

    4 publications · 197 citations

    4

    h-index

1–5 of 1.3K

Institutions

273

  1. Google (United States)

    Mountain View, United States

    16 h-index

    66 publications

    10.4K citations

  2. Tsinghua University

    Beijing, China

    11 h-index

    55 publications

    13.1K citations

  3. Meta (Israel)

    Tel Aviv, Israel

    8 h-index

    27 publications

    2.5K citations

  4. Carnegie Mellon University

    Pittsburgh, United States

    7 h-index

    17 publications

    2.9K citations

  5. Stanford University

    Stanford, United States

    7 h-index

    26 publications

    1.5K citations

  6. University of Science and Technology of China

    Hefei, China

    6 h-index

    16 publications

    1.2K citations

  7. Microsoft Research (United Kingdom)

    Cambridge, United Kingdom

    6 h-index

    17 publications

    2K citations

  8. Fudan University

    Shanghai, China

    6 h-index

    19 publications

    10.8K citations

1–8 of 273

Venues

  1. arXiv (Cornell University)

    Repository · 80 publications · 8.6K citations

    34

    h-index

  2. Proceedings of the AAAI Conference on Artificial Intelligence

    Conference · 6 publications · 171 citations

    6

    h-index

  3. IEEE Access

    5 publications · 251 citations

    5

    h-index

  4. 5

    h-index

  5. 4

    h-index

1–5