How High-Performance Computing Enables a Multi-Decade Journey to the Next Generation Gamma-Ray Detector

Story written by Erica Lee

The Hulk may only exist in comic books and on-screen, but the machine that exposed fictional physicist Bruce Banner to gamma radiation and turned him into a green giant was based on a real-life Berkeley Lab device. Director Ang Lee replicated the Lab’s Gammasphere for his 2003 Hulk film with one key difference — the actual device detected gamma-rays rather than emitted them.

Twenty years later, Lab scientists have built the next-generation device that uses high-performance computing to detect gamma-rays with even more sensitivity. When staff scientist Mario Cromaz arrived at the Lab in the late 90’s, the Nuclear Science Division’s Nuclear Structure Group was first prototyping the Gammasphere’s far more precise successor, eventually called GRETINA (Gamma-Ray Energy Tracking In-beam Nuclear Array). With GRETINA proven successful, the Nuclear Structure Group has now followed with GRETA, an advanced device set to start experimental operations at Michigan State University’s Facility for Rare Isotope Beams (FRIB) in 2026.

GRETA (Gamma-Ray Energy Tracking Array) measures the high-energy electromagnetic radiation released when atomic nuclei de-excite following a nuclear reaction. “We basically measure the properties of the gamma-rays to try to infer the structure of nuclei,” said Cromaz, who is responsible for GRETA’s computing systems.

In a fluorescent ceiling light, an electric current excites the atoms in the bulb, and those atoms then emit photons that are seen as visible light. Scientists use particle accelerators to excite nuclei in a similar way. After stripping electrons from atomic nuclei such as silicon or carbon, the nuclei are fired down a beamline to collide with nuclei in a target. The nuclei interact, become excited and de-excite by emitting gamma-rays and other particles. By measuring the energy of these gamma-rays, nuclear physicists can infer the forces operating in the nucleus.

“When I was a student, we used to think if anybody could ever build this, it would be the best thing ever,” Cromaz recalled. “It’s a huge deal to be able to track, but it also requires computing that really became available 10 or 15 years ago.”

The computational challenge

GRETA’s sphere, where the radioactive nuclei collisions occur, is tiled with soda can-sized germanium crystals. The gamma-rays from the de-exciting nucleus interact with the germanium crystals producing a set of electrical signals Scientists can then determine where the gamma-ray interacts within the crystal by comparing the measured signals to  linear combinations of simulated signals they have pre-calculated with simulations. Knowing where the gamma-rays interact is key in precisely determining the gamma-rays energy but this process is computationally expensive and demands computing resources that can perform half a million such calculations per second continuously during experiments that can last 10 days.

Running this algorithm at this rate requires high-performance computing. To accomplish this, Cromaz worked with Gary Jung, ScienceIT Department Head and his staff, notably Tin Ho along with others in the IT Division, to build a specialized AMD-based Linux cluster equipped with Nvidia GPUs to handle the signal decomposition along with a high performance storage subsystem to handle the data analysis. Eli Dart and Eric Pouyoul from ESNet designed and built a high speed forward buffer infrastructure to capture all the packets from the detector and forward them to the cluster. By the way, Gary’s group was also similarly involved with GRETINA and Gammasphere dating back to the 1990s.

One of GRETA’s distinguishing features is that it streams the processed data in real time. “What happens in the array is basically what you see on the screen a few seconds later,” said Cromaz. “There’s a lot of tuning that happens as the experiments are running, and you can’t really wait to analyze your data later.”

Probing new physics insights

Both devices have a peculiar attribute: They are designed to be movable. Since its assembly at Berkeley Lab, GRETINA has spent time at nuclear science laboratories at Michigan State University and  Argonne National Laboratory in Illinois. The different laboratories have different accelerators and capabilities which allow for a variety of experiments. Since beginning operating in 2012, GRETINA has been used in over 100 experiments and clearly demonstrated the science reach and impact of a gamma-ray tracking array to answer pressing questions about the structure of both subatomic and visible matter and set the stage for the full GRETA array.

GRETA will have an unmatched combination of high efficiency and high resolution to study exotic nuclei. Exotic nuclei refer to those with a large asymmetry between the number of neutrons and the number of protons. The more exotic they are, the more they tend to be unstable. 

The power of GRETA at FRIB will allow scientists to carry-out a broad science program spanning the next several decades to understand how nuclei change when the ratio of neutrons and protons change and how these changes affect their decays and reactions. This is important for example to determine how elements are created in the stars. 

GRETA will provide the nuclear structure data to help understand the abundances of elements in the Universe and where they were created, such as supernovae explosions. An Hubble Space Telescope image of a supernova remnant 160,000 light-years away from our Milky Way galaxy. (Credit: NASA/ESA/HEIC and The Hubble Heritage Team)

“We measure indirectly in the lab, and from that we can infer what happens in the stellar environment,” said Augusto Macchiavelli, a senior staff scientist in Berkeley Lab’s Nuclear Structure Group.

Increasing importance of computing

The team plans for GRETA to operate into the 2030s, so a significant challenge is how to scale along with the rapid pace of computational advancements.

“It’s a really tough call because you have to choose technologies that aren’t going to go obsolete. But you can’t choose technologies that are really old or not very forward-looking, because then you cheat yourself out of all the potential,” said Cromaz. “We’re in this transitional place between science and engineering. So it’s very important to have groups you can talk to like ScienceIT. You can come to them from the science end, they can come from the engineering end, and then you can come to some conclusion about how things should work.”