bioRxiv (Cold Spring Harbor Laboratory) · 2023 · 175 citations · 68 references
Life SciencesCell BiologyGeneticsComputational BiologyMulti-omics IntegrationSingle-cell Systems BiologyCell Type AnnotationSingle-cell Multi-omics IntegrationBatch Data IntegrationCell AnnotationFoundation Model PretrainingIntegration Of Single-cell DatasetsSingle-cell Rna Sequencing Cell Type Annotation
Generative pre‑trained models have achieved remarkable success in NLP and computer vision, and large‑scale diverse datasets with transformers are emerging as a promising approach for foundation models. This study investigates whether foundation models can be applied to cellular biology and genetics by treating cells as analogous to words in language. We built scGPT, a generative pre‑trained transformer trained on over 33 million single‑cell profiles, and fine‑tuned it via transfer learning for tasks such as cell‑type annotation, multi‑batch and multi‑omic integration, perturbation prediction, and gene‑network inference. scGPT successfully extracts critical biological insights about genes and cells, and its codebase is publicly available at https://github.com/bowang-lab/scGPT.
Abstract Generative pre-trained models have achieved remarkable success in various domains such as natural language processing and computer vision. Specifically, the combination of large-scale diverse datasets and pre-trained transformers has emerged as a promising approach for developing foundation models. Drawing parallels between linguistic constructs and cellular biology — where texts comprise words, similarly, cells are defined by genes — our study probes the applicability of foundation models to advance cellular biology and genetics research. Utilizing the burgeoning single-cell sequencing data, we have pioneered the construction of a foundation model for single-cell biology, scGPT, which is based on generative pre-trained transformer across a repository of over 33 million cells. Our findings illustrate that scGPT, a generative pre-trained transformer, effectively distills critical biological insights concerning genes and cells. Through the further adaptation of transfer learning, scGPT can be optimized to achieve superior performance across diverse downstream applications. This includes tasks such as cell-type annotation, multi-batch integration, multi-omic integration, genetic perturbation prediction, and gene network inference. The scGPT codebase is publicly available at https://github.com/bowang-lab/scGPT .
68
Highly accurate protein structure prediction with AlphaFold
John Jumper, Alexander Pritzel, Tim Green et al. · Nature · 2021 · 45.2K citations · Full text
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov et al. · arXiv (Cornell University) · 2020 · 21.6K citations · Full text