Selbstadaptiver dCache

Michael Köhler-Bußmeier, HAW Hamburg

Tigran Mkrtchyan, DESY

Kilian Schwarz, DESY

Jan Sudeikat, HAW Hamburg

Christian Voss, DESY

Experiments at Deutsches Elektronen-Synchrotron (DESY) produce large-scale data which need to be stored and distributed globally for examination and analysis. The dCache framework allows to set-up and control distributed data storages and is conjointly developed at DESY, the Nordic Data Grid Facility (NeIC, NDGF), and the Fermi National Accelerator Laboratory. The dCache system is a fundamental building block of the WLCG storage infrastructure, supporting all major experiments in the fields of High Energy and Astro-Physics. The majority of the large data centres for the LHC computing uses dCache and overall 60% of the LHC data is stored on dCache systems. Hence dCache is an essential component for all results published by the LHC collaborations, e.g. the discovery of the Higgs Boson, but also the Flavour-Anomalies that are the best indicator for New Physics. DESY alone operates about 100PiB of disk storage using dCache for High Energy Physics, accelerator research and operations as well as research with photons. Through dCache most of the data produced by the latter two are archived on tape as well. Also, the data taken during the Corona research in 2020/2021 are stored and archived on dCache. Thus, managing the flow of petabyte-scale data is a challenging and pivotal task.

While dCache allows operating networks of storage nodes, which is of central importance, it still requires manual administration and configuration. The development of self-adaptive systems is an active research field, which aims at engineering computational systems which can adapt themselves at run-time. Besides providing frameworks for implementing these systems it is particularly challenging to enable run-time reasoning about appropriate adjustments in changing execution contexts. A prime application example is the management of computational resources, e.g. data centers.

This project aimed at initiating sustainable collaborations between DESY and HAW in order to study how reliable adaptive systems can be engineered for automating the administration of dCache-based storage systems. Within the project a testbed shall be created as a starting point and basis for the ongoing development of autonomous self-adaptive algorithms.

dCache Pool Access Heatmap: Die Heatmap von dCache-Pool-Zugriffen eines Nutzers oder einer Nutzerin im Verlauf einer Woche. Farbabstufungen veranschaulichen die Auslastung der Pools; von Grün nach Rot steigt die Auslastung an. [Copyright: Deutsches Elektronen-Sychrotron (DESY), Scientific Computing Group]