The challenge: dealing with data deluge

The ODISSEE project is addressing some of the most significant data challenges in physics and astronomy. With the future commissioning of the High-Luminosity Large Hadron Collider (HL-LHC) and the Square Kilometre Array Observatory (SKAO), the community of physicists will have to deal with vast amounts of data, that current technical technologies can’t process.

To cope with this challenge, the ODISSEE project aims to revolutionise the way we process, analyse and store data. We are developing new technologies using AI to process and filter only the relevant data into the data stream on the fly. This approach will enable scientists to build more complex, yet reliable, physical models, whether at micro or astronomical scale.

At the same time, the project also focuses on redesigning hardware and software solutions that cover the entire data stream continuum, from generation to analysis. The primary objective is to create tools that are energy efficient, adaptable and flexible for the future. This will involve the development of a reconfigurable network of diverse processing elements driven by AI.

The initial process starts with a collection of distributed science sensors, such as antennas or particle detectors, which gather raw data. These sensors are complemented by monitoring and environmental sensors to ensure accurate calibration and facility maintenance.

This second stage involves collecting, filtering, and converging raw data streams in real-time. This step is crucial for handling the immense data throughput without any loss in quality. It occurs locally, near the science sensors.

The third phase focuses on processing and reducing the incoming data streams into science data products. This is achieved using state-of-the-art High-Performance Computing (HPC) facilities, which must meet demanding specifications for data ingestion, computation, and storage.