CERN has provided open access to a vast amount of data. Links to these datasets can be affected over time. In this slides, I present how I checked that there were no broken links to the CMS Experiment datasets.

1. Daniel Bustamante | Supervisor: Clemens Lange | July 08, 2019
Checking the
CMS datasets
Non-Member State
Summer Student Programme

2. CERN have made
PUBLIC data collected
by the CMS detector
BACKGROUND
Checking the CMS datasets 02/16

7. BACKGROUND
Checking the CMS datasets 03/16
First of all, you need to know that ROOT is the
framework used to work with
the collected data

8. BACKGROUND
Checking the CMS datasets 04/16
The primary data of CMS is in
AOD (Analysis Object Data)
files
They store raw data that
contain all the information
needed for analysis
What does that mean?
So, how can I read them?
ROOT is required to read the
files and understand the
reconstructed data
© CERN, 2014–2019

11. CERN Open Data Portal contains
datasets of real data recorded with
the CMS detector

13. Inside the datasets there is
information of different events and
physics object collections

15. The information is stored in the aforementioned
AOD files, which are conveniently listed in index
files

18. PROBLEM
Checking the CMS datasets 09/16
That's the problem!
With such a large list of files,
deletion, corruption or loss of data could occur
How do we make sure that is not happening now?

19. PROPOSED SOLUTION
Checking the CMS datasets 10/16
GFAL (Grid File Access Library) version 2
provides useful command line tools…
gfal-ls is equivalent to the system ls command
Support protocol (root://)
-l option allows long listing format (including size)

20. PROPOSED SOLUTION
Checking the CMS datasets 11/16
On the other hand…
JSON (JavaScript Object Notation) version of the index
files provides more organized and detailed information
about the ROOT files of each dataset
… just do the replacement .txt → .json

22. There we can find the size that we expect the ROOT
file to have in case there have been no
manipulations

23. A SMALL EXAMPLE
Checking the CMS datasets 13/16
gfal-ls will tell us if the file is still available…
It exists (expected result)
gfal-ls error: 2 (No such file or directory) - Failed to
stat file (No such file or directory)
root://eospublic.cern.ch//eos/opendata/cms/Run2010B/Ele
ctron/PATtuples/Electron_PAT_data_500files_1.root
It has not been found (Houston, we have a problem)
$ gfal-ls root://eospublic.cern.ch//eos/opendata/cms/Run2010B/
Electron/PATtuples/Electron_PAT_data_500files_1.root
Possibleresults

24. A SMALL EXAMPLE
Checking the CMS datasets 14/16
If it exists, we must check the size using gfal-ls -l,
comparing it with the JSON file
Result
$ gfal-ls -l root://eospublic.cern.ch//eos/opendata/cms/
Run2010B/Electron/PATtuples/Electron_PAT_data_500files_1.root
-r-------- 1 1399 125433 12117591860 Sep 4 2014
root://eospublic.cern.ch//eos/opendata/cms/Run2010B/Electron
/PATtuples/Electron_PAT_data_500files_1.root
Do you remember the JSON file?

26. RESULTS
Checking the CMS datasets 16/16
files with
broken link
index files with
reading problems