Science IT and JGI Partner to create Dori

Over the past 3 years, the Berkeley Lab’s Science IT and the Joint Genome Institute (JGI) have  built a transformative partnership, driving customized IT solutions forward with the new Dori computer cluster. JGI, the leading research institute in environmental genomics, previously relied solely on the National Energy Research Scientific Computing Center (NERSC) for its high-performance computing (HPC) needs. However, as its computational demands grew and diversified, JGI looked for additional help from Berkeley Lab’s Science IT department. Together, they launched a specialized 100-node direct-liquid cooled computer cluster named Dori, designed to handle JGI’s daily research workloads, while still utilizing NERSC for larger-scale tasks. This partnership represents a step forward in building accessible, tailored resources to meet complex scientific needs at Berkeley Lab.

The Vision Behind Dori

For Dori, the goal was to create a cluster that would operate like an extension of a researcher’s laptop but with the power to handle intensive genomic data workloads. Unlike supercomputers, which often require its users to do an extensive setup and have expertise, Dori is designed to be accessible and easy to use, making it perfect for JGI’s needs, which include hundreds of users at the Berkeley Lab and beyond.

“JGI is not a typical research group. As a user facility, it supports thousands of external users, many of whom need reliable, accessible computing resources. This makes dependability and integration with JGI’s existing systems, including authentication and data transfer, critical,” explained Kjiersten Fagnan, JGI’s Chief Informatics Office, highlighting the uniqueness of JGI’s requirements. 

Building a Customize Computer Cluster for JGI

Science IT approached Dori’s development with the specific needs of JGI’s data-intensive research in mind. Genomic research, which often involves millions of small files, presents unique challenges. Unlike typical scientific computations, which may handle fewer, larger files, JGI’s workloads involve vast quantities of data in smaller formats which can wreak havoc on most supercomputer storage systems. This unique requirement demanded a computer cluster customized from the ground up to handle millions of small data files effectively and efficiently.

According to Karen Fernsler, Systems Engineer IV, in Science IT, “The Dori cluster had to be customized not just in hardware but also in its setup, including an authentication configuration that integrated seamlessly with JGI’s existing system.” Her work focused on creating an environment where users could log in easily, access files quickly, and manage jobs without the hassles typical of larger supercomputing environments. The success of Dori relied on close, collaborative planning with JGI to ensure the system would meet JGI’s dynamic needs.

An innovative idea put into practice was the JGI development  and adoption of the JGI Analysis Workflow Service (JAWS) platform which itself in turn leverages  the ScienceIT developed “Singularity” container software. Support for container-based workflows was crucial in enabling the JGI to have “Mobility of Compute” so that they could easily move to the new Dori cluster as well as other computing resources. “This portability is becoming increasingly important as scientists are moving to utilize a portfolio of computing resources instead of relying on just one,” says Gary Jung, ScienceIT Department Head.

A Partnership in Action

The development of Dori highlighted a level of partnership that extended beyond the typical customer-provider relationship. 

“This was a true partnership. Every aspect of Dori’s development, from procurement to networking and facilities, required collaboration and adaptability,” Georg Rath, Computer Systems Supervisor, JGI who led the Dori project, “Both teams worked hand-in-hand, holding bi-weekly advisory group meetings to discuss the health and status of the system, address any issues, and continuously refine the cluster to better serve JGI’s goals.”

Another of the innovative features of Dori is its use of Slack as a primary communication tool for users. Rather than a ticketing system, Dori users can troubleshoot and collaborate in real-time with JGI Systems Infrastructure and ScienceIT staff  within Slack channels. This open communication style fosters a sense of community among Dori users, who can search for past questions, learn from each other, and collaborate on solutions. “It’s an unusual approach for a National user facility but has proven to be invaluable for problem-solving and community building,” says Fernsler. With 188 active members and over 130 specific sub-channels, Slack allows users to crowdsource solutions and share best practices seamlessly.

Meeting Changing Needs with Flexible Solutions

Science IT and JGI also built adaptability into the Dori project. Unlike many high performance computing (HPC) systems, which are built to run at a single high performance, Dori is a “dynamic system” that evolves as JGI’s research needs change. A key factor in the Dori project’s success is its ongoing support structure. The advisory group, which includes representatives from Science IT and JGI, meets regularly to discuss system health, potential upgrades, and how to adapt Dori to future research challenges.

Security and accessibility have also been prioritized to ensure that Dori meets the needs of a diverse user base. Science IT worked with JGI to streamline access so that any JGI team member can log into Dori without an elaborate approval process. The cluster is as close to plug-and-play as possible for researchers, which makes the transition from local desktop computing to Dori seamless.

The Impact of Dori on JGI’s Research

With Dori, JGI researchers can quickly access a cluster tailored to bioinformatics workflows without competing with the high-demand schedules of national supercomputing facilities like NERSC. This ease of access has allowed JGI to focus on its mission of advancing genomics research without the delays or limitations that often come with using shared computing resources.

The Dori project exemplifies the kind of collaboration that makes ambitious research projects possible. As Science IT and JGI continue to work together, they are setting a powerful example of how research institutions can adapt and modernize computing resources to meet ever-evolving scientific needs.