Coping with data deluge

The ODISSEE project is addressing some of the most significant data challenges in physics and astronomy. With the future commissioning of the High-Luminosity Large Hadron Collider (HL-LHC) and the Square Kilometre Array Observatory (SKAO), the community of physicists will have to deal with vast amounts of data, reaching up to 200 terabits per second, that current technical technologies can’t process.

To cope, the ODISSEE project aims to revolutionise the way we process, analyse and store data. We are developing new technologies using AI to process and filter only the relevant data into the data stream on the fly. This approach will enable scientists to build more complex, yet reliable, physical models, whether on a micro or astronomical scale.

At the same time, the project also focuses on redesigning hardware and software solutions that cover the entire data stream continuum, from generation to analysis. The primary objective is to create tools that are energy efficient, adaptable and flexible for the future. This will involve the development of a reconfigurable network of diverse processing elements driven by AI.

The initial process starts with a network of distributed science sensors, such as antennas or particle detectors, which gather raw data. These sensors are complemented by monitoring and environmental sensors to ensure accurate calibration and facility maintenance.

The third phase focuses on processing and reducing the incoming data streams into science data products. This is achieved using state-of-the-art High-Performance Computing (HPC) facilities, which must meet demanding specifications for data ingestion, computation, and storage.