Requirements for Science Data Bases and SciDB

TLDR

We have been gathering requirements from users across astronomy, particle physics, fusion, remote sensing, oceanography, and biology to inform a new science database system. The paper aims to specify a common set of requirements for SciDB and outline its preliminary design. SciDB development is being funded through a partnership of companies—eBay, LSST, Microsoft, SLAC, and Vertica—while the authors report the identified requirements and sketch the system design. We found that complex business analytics share most of the same requirements as big science, and identified LSST and eBay as initial lighthouse customers for the system.

Abstract

For the past year, we have been assembling requirements from a collection of scientific data base users from astronomy, particle physics, fusion, remote sensing, oceanography, and biology. The intent has been to specify a common set of requirements for a new science data base system, which we call SciDB. In addition, we have discovered that very complex business analytics share most of the same requirements as “big science”. We have also constructed a partnership of companies to fund the development of SciDB, including eBay, the Large Synoptic Survey Telescope (LSST), Microsoft, the Stanford Linear Accelerator Center (SLAC) and Vertica. Lastly, we have identified two “lighthouse customers” (LSST and eBay) who will run the initial system, once it is constructed. In this paper, we report on the requirements we have identified and briefly sketch some of the SciDB design.