iProX in 2021: connecting proteomics data sharing with big data

TLDR

Proteomics research has generated vast experimental datasets, and emerging big‑data platforms now enable efficient handling of these volumes. This paper outlines the key updates to the iProX proteome resource since its 2019 Nucleic Acids Research publication. iProX employs a hyper‑converged, scalable architecture with a Hadoop cluster for storage and a distributed Elastic‑Search engine for sub‑second queries, supplemented by USI integration, RESTful APIs, and a high‑efficiency reanalysis pipeline. By August 2021, iProX hosted 1,526 datasets totaling 92.42 TB, and the upgraded platform supports petabyte‑scale storage, billions of spectra, and second‑level latency suitable for the rapidly expanding proteomics field.

Abstract

The rapid development of proteomics studies has resulted in large volumes of experimental data. The emergence of big data platform provides the opportunity to handle these large amounts of data. The integrated proteome resource, iProX (https://www.iprox.cn), which was initiated in 2017, has been greatly improved with an up-to-date big data platform implemented in 2021. Here, we describe the main iProX developments since its first publication in Nucleic Acids Research in 2019. First, a hyper-converged architecture with high scalability supports the submission process. A hadoop cluster can store large amounts of proteomics datasets, and a distributed, RESTful-styled Elastic Search engine can query millions of records within one second. Also, several new features, including the Universal Spectrum Identifier (USI) mechanism proposed by ProteomeXchange, RESTful Web Service API, and a high-efficiency reanalysis pipeline, have been added to iProX for better open data sharing. By the end of August 2021, 1526 datasets had been submitted to iProX, reaching a total data volume of 92.42TB. With the implementation of the big data platform, iProX can support PB-level data storage, hundreds of billions of spectra records, and second-level latency service capabilities that meet the requirements of the fast growing field of proteomics.