bioRxiv (Cold Spring Harbor Laboratory) · 2023 · 46 citations · 17 references
Life SciencesBioinformaticsCell BiologyComputational BiologyGenomicsTranscriptomicsCell Type AnnotationSingle-cell Rna Sequencing AnalysisSingle-cell TranscriptomicsAblation StudiesFew-shot LearningFine-tuningFoundationSingle-cell Rna Sequencing
Abstract Large-scale foundation models, which are pre-trained on massive, unlabeled datasets and subsequently fine-tuned on specific tasks, have recently achieved unparalleled success on a wide array of applications, including in healthcare and biology. In this paper, we explore two foundation models recently developed for single-cell RNA sequencing data, scBERT and scGPT. Focusing on the fine-tuning task of cell type annotation, we explore the relative performance of pre-trained models compared to a simple baseline, L1-regularized logistic regression, including in the few-shot setting. We perform ablation studies to understand whether pretraining improves model performance and to better understand the difficulty of the pre-training task in scBERT. Finally, using scBERT as an example, we demonstrate the potential sensitivity of fine-tuning to hyperparameter settings and parameter initializations. Taken together, our results highlight the importance of rigorously testing foundation models against well established baselines, establishing challenging fine-tuning tasks on which to benchmark foundation models, and performing deep introspection into the embeddings learned by the model in order to more effectively harness these models to transform single-cell data analysis. Code is available at https://github.com/clinicalml/sc-foundation-eval .
17
Proceedings of the 24th international conference on Machine learning
2007 · 11.7K citations
Massively parallel digital transcriptional profiling of single cells
Grace Zheng, Jessica M. Terry, Phillip Belgrader et al. · Nature Communications · 2017 · 7.8K citations · Full text
Language Models are Few-Shot Learners
T. B. Brown, Benjamin Mann, Nick Ryder et al. · arXiv (Cornell University) · 2020 · 3K citations · Full text