By Kuldeep Chawla, Arica Chhay
Data is the lifeblood of science. The ever increasing volume of data has, in many ways, fundamentally changed the scientist’s approach to conducting research.
At Berkeley Lab, the next-generation instruments that are being deployed will generate many petabytes of data per year each. The success of these big-data science projects demands a high-performance data pipeline, of which a performant, resilient, scalable, and cost-effective storage platform is a crucial part.
To meet the storage needs of Lab researchers, the IT Division developed the Science Project Storage Service (SPSS) as part of the ScienceIT initiative, focusing on providing services to support the Lab’s scientific mission. SPSS was first launched in 2019 as a centrally managed storage solution and met with reasonable success. The ongoing growth of research and computational technology coupled with direct engagement with scientists has now given rise to SPSS v2, a new, improved, more evolved version of the initial offering. The revamped storage service enables ScienceIT to better meet the insatiable demand of Lab researchers to store, process, and share increasingly large volumes of data.
Features
Scientific Platform Storage Service (SPSS v2) is a scalable, cost-effective storage service available to Berkeley Lab researchers across all disciplinary spaces. SPSS v2 is based on the Ceph storage system, an open-source platform which implements object storage on a distributed computer cluster without a single point of failure; it is scalable to the exabyte level, freely available, and fault-tolerant. Object storage is widely preferred as the technology for storing and sharing vast amounts of data over Wide Area Networks. Originally designed as an HPC filesystem, Ceph has gained broad adoption in the enterprise space as the storage backend for production OpenStack installations.
SPSS v2 offers Lab researchers a file storage platform to support data workflows, without the cost of dedicated hardware or the administrative burden of running and maintaining their own IT infrastructure.
Purchasing a building block with hardware maintenance upfront is permitted, where the addition of such hardware serves to increase the performance envelope of the entire system. Beside local storage, avenues for data lifecycle management are available, including cloud storage platform offloading.
As a result of its design, the system is both self-healing and self-managing, thus minimizing maintenance time and other costs. Various levels of fault tolerance are supported (with tradeoff between reliability and capacity), ranging from replication to erasure-coding, so a PI can choose a tolerance level that is tailored to their needs. Lastly, Ceph utilizes the common Linux stack. Unlike the first incarnation of SPSS which utilized a proprietary operating system from Dell, the adoption of commodity Linux allows for the customization of services and protocols with fewer limitations.
Benefits
SPSS v2 is well-equipped to address requests for large volume storage, a condo storage model to allow customer buy-in, object storage, and the use of commodity hardware and open-source software to minimize cost and avoid vendor lock-in. The platform consists of a redundant 4 server nodes (which serve as Ceph Monitor Servers, Metadata Servers, and Object Gateways). Initial rollout of v2 includes 12 Object Storage Daemons (OSD) storage nodes with a storage capacity of 5.6 PB, which are the scalable building blocks of the Ceph cluster, with the ability to add more OSD nodes on the fly.
- Object Storage: Ceph object storage supports two APIs, Amazon S3 and OpenStack Swift. It is ideal for establishing a data lake on premise, for storing and sharing unstructured data. Its security is not tied with Lab accounts, therefore permitting users of facilities, such as the Foundry and ALS, to easily share and move their data. The Ceph object storage will be offered as a lower-cost but higher-performance alternative to Amazon S3 or Google Cloud Storage.
- Block Storage: Ceph block storage is ideal for providing disks for Scientific Virtual Machines (SVM), as well as persistent volumes for the Kubernetes/JupyterHub cluster that is planned for the near future. When finally deployed, the Kubernetes cluster will be another critical component of the high-performance data pipeline at the lab.
- Compatible File System: CephFS is a fully POSIX-compliant file system, initially developed as a Lustre® replacement. CephFS will serve as a higher-performance alternative to NFS or as a lower-cost alternative to Lustre® for LRC clusters.
- Second Tier Copy: Users can purchase SPSS to store or keep a second copy of data from their local server or any instrument.
- No Ingress or Egress Cost: Public cloud object storage options typically have ingress and egress cost attached to move the data in and out whereas there are no additional costs attached to moving data in SPSS object storage.
- Connectivity with Lawrencium: SPSS v2 has connectivity with Lawrencium HPC cluster. Users can move the data from the SPSS v2 environment to the Cluster Environment for computation and then move it back to the system for long-term storage or publish the results on a Web Server hosted on SVM.
- No COPEX: SPSS v2 is a pay-per-user which can be scaled up or down per request. Users only pay for what they are consuming and will not have to invest in any upfront cost in buying and standing up a storage system.
Costs
The price of the SPSS v2 Raw capacity tier is $7/TB/month for NFS or Object Store buckets, based on configured capacity. To encourage uptake of the service, a free three-month trial of SPSS is available to all Lab researchers.
For comparison, prices for storage on AWS, GCP, and National Laboratories ($/TB/month) are listed below:
| Cost Comparison | Cost per TB |
|---|---|
| Berkeley Lab SPSS v2 Storage | $7/Mo. |
| UC Irvine Storage | $5/Mo. |
| UCLA CASS Storage | $11.41/Mo. |
| Stanford File Storage | $38/Mo. |
| UCB Utility Storage | $40/Mo. |
| Amazon AWS S3 Standard | $23/Mo. |
| Google GCP Standard | $20/Mo. |
Support
Complete the SPSS Request Form to get started with the capacity tier of SPSS v2.
Contact scienceit@lbl.gov for any related questions.
Notes:
- The SPSS service is free of cost for the first three months for Lab researchers.
- Use of SPSS requires consent to the terms of the Service Level Agreement.
in the bottom right or navigate to