XDC-demo-CTA (EOSC-hub week, Prague)

1. eXtreme DataCloud is co-funded by the Horizon2020
Framework Program – Grant Agreement 777367
Copyright © Members of the XDC Collaboration, 2017-2020
Data Management for extreme scale computing
CTA USE CASE
Review
Gillardo Frédéric
gillardo@lapp.in2p3.fr
11/04/2019 1

2. CTA Data Workflow
Reconst
Calib
DL0/Bulk Archive
x
Scientist
Interface
DL3 / High
level data
Canary Island ,
Online
Process
Chile ,
Online
Process

3. eXtreme DataCloud 3
Use Case Goals
• Objectives:
Archive PBs of data using an ingest method.
Index files using metadata contained in the header files
Query files using metadata parameters
• XDC Services Requirements:
• OneData:
• Onedata
• Metadata management
• QoS (policy definition)
3
Ingest
DL0
Preprocessing
Metadata
database
Query
(Metadata)
List of filenames
files
Data
management
Archive
Producer Consumer
Retrieve
(fileName)

4. User Stories
As an “Archive Manager”, I can configure the system to extract metadata during the
preprocessing operation.
Related with :
As an “Archive Manager”, I can define rules based on metadata to archive files on tape only or on disc & on tape, at ingest time.
As an “Archive Manager”, I can define rules based on metadata to duplicate archive files on tape on low latency storage to be soon quickly retrieved.
As an “Archive Manager”, I can define rules based on metadata to create replicas on a specific data center, at ingest time.
As an “Archive Manager”, I can define rules based on metadata to prevent deletion of files
As an “Archive Manager”, I can delete files based on metadata values
As a “Archive User”, I can make a query based on metadata parameters to get a list of
logical files names
4

5. OneData installation at LAPP
and CCIN2P3
POSIX
40TB
LAPP-
PROVIDER
XDC-DEMO1
S3
4TB
CC-PROVIDER-02
POSIX
6TB
5

6. Use Case Architecture
eXtreme DataCloud 6
Query by
Metadata Restquery
Query by LFMFile
System
preprocessing
CTA HDF5
Extraction
Metada
ingestFile
Generator

7. Conclusion
What has been demoed:
• Metadata index creation using REST API
• Deploy the « preprocessing’s service » using docker compose
• Ingest a HDF5 files
• Query File ID using HDF5 header
• Retrieve file using CDMI interface and File ID
eXtreme DataCloud 7