HEPData is a repository for data from high energy physics (HEP) experiments dating back to the 1950s. It provides a standardized way for scientists to submit the underlying data from their published papers and analysis results. This includes tables, plots, scripts and files used in the analysis to enable reproducibility. HEPData offers features like simplified submission processes, versioning, DOIs, and tools to access and search data in various environments and formats to help both data providers and consumers.

1. Eamonn Maguire, CERN
hepdata.net
HEPData Workshop

2. What is it?
HEP Scattering
experiments going
back to the 1950s
Each group of scientists will
analyse particular signals by
processing large numbers
of collisions.
The resulting analysis
will be published as
a paper.
But where does the
processed data go?
RE P P --> X
SQRT(S) IN GEV SIG IN MB
7000 95.35 ± 0.38 (stat) ± 1.25 (sys,experimental) ± 0.37
(sys,extrapolation)

3. What is it?
HEPDATA
Physics paper
Table description
RE P P --> X
SQRT(S) IN GEV SIG IN MB
7000 95.35 ± 0.38 (stat) ± 1.25 (sys,experimental) ± 0.37
(sys,extrapolation)
RE P P --> X
SQRT(S) IN GEV SIG IN MB
7000 95.35 ± 0.38 (stat) ± 1.25 (sys,experimental) ± 0.37
(sys,extrapolation)
Table 1
Table 2 (F1)
HEPData is the go to place for physicists to get access to the data
underlying plots and tables in a publication.
It also links to the scripts and ROOT files for instance used in the
analysis (for reproducibility).

4. What is it?

5. Data Providers
1. A Simplified Submission Process
2. A standard entry data format
3. Full review management system
4. Versioning
5. DOI minting
6. Sandbox
Data Consumers
1. Publication Driven Search
2. Data Driven Search
3. Semantic Publishing
4. Data Conversion
5. Access in Analysis Environments
What is new for you?
Whether you’re a data provider, or consumer,
the new HEPData has many functionalities

6. Data Providers
1. A Simplified Submission Process
2. A standard entry data format
3. Full review management system
4. Versioning
5. DOI minting
6. Sandbox

7. Data Providers
1. A Simplified Submission Process
2. A standard entry data format
3. Full review management system
4. Versioning
5. DOI minting
6. Sandbox

8. Submission Process

9. Data Providers
1. A Simplified Submission Process
2. A standard entry data format
3. Full review management system
4. Versioning
5. DOI minting
6. Sandbox

10. HEPData submission archive
YAML
Data Record
YAML
Data Record
ROOT PYTHON C++ ROOT
submission.yaml
data records
external data files & links
links the submission together by detailing the
data files to be loaded, their name and descrip-
tion, and their assocated analysis files and code.
YAML (or JSON) representation of the underlying
data files including value errors in a verbose format.
analysis files, code, links to code repositories, etc.

11. HEPdata submission archive
RE P P --> X
SQRT(S) IN GEV SIG IN MB
7000 95.35 ± 0.38 (stat) ± 1.25 (sys,experimental) ± 0.37
(sys,extrapolation)
HEPDATA
Table 1
{JSON}
Tables and plots
Processes YAML file, inserts records in to database
and links publication record with data and files.
Web Server
Table description
Plots rendered automatically
using a custom library built upon D3.js
Tables rendered from
JSON
DownloadScripts

12. Data Providers
1. A Simplified Submission Process
2. A standard entry data format
3. Full review management system
4. Versioning
5. DOI minting
6. Sandbox

13. Comprehensive Review System

14. Comprehensive Review System

15. Dashboard for Submission Management

16. Dashboard for Submission Management

17. Interactive Plotting Library

18. Data Providers
1. A Simplified Submission Process
2. A standard entry data format
3. Full review management system
4. Versioning
5. DOI minting
6. Sandbox

19. Versioning

20. Data Providers
1. A Simplified Submission Process
2. A standard entry data format
3. Full review management system
4. Versioning
5. DOI minting
6. Sandbox

21. DOIs
All HEPData records get
DOIs.
Each data table gets a
versioned DOI.
The whole HEPData
record is also given a DOI
to encompass the whole
collection.

22. Data Providers
1. A Simplified Submission Process
2. A standard entry data format
3. Full review management system
4. Versioning
5. DOI minting
6. Sandbox

24. Sandbox

25. Sandbox

26. Data Consumers
Get access to the data in many environments
1. Publication Driven Search
2. Data Driven Search
3. Semantic Publishing
4. Data Conversion
5. Access in Analysis Environments

27. Data Consumers
Get access to the data in many environments
1. Publication Driven Search
2. Data Driven Search
3. Semantic Publishing
4. Data Conversion
5. Access in Analysis Environments

29. The System - Demo
hepdata.net

30. Data Consumers
Get access to the data in many environments
1. Publication Driven Search
2. Data Driven Search
3. Semantic Publishing
4. Data Conversion
5. Access in Analysis Environments

31. HEPData
Coming soon!
Masters Project of Juan Luis Boya Garcia,
University of Salamanca, Spain.

32. You’ll be able to ask HEPData’s API for all data
matching some number of variables.
Retrieve all relevant data, then run your analysis
HEPData
Programmatically or through our interface
No need to manually go through every publication.
We do the hard work for you.

33. HEPData Search HEPData Submit Data Sign inSearch Data Query
Independent Variable
PT (GeV)
OR
Phrase
elastic scattering
Delete
10
20
30
Download as
5
15
25
Found 3,400 matching data points in 36 publications Download all asView publications
P (GeV) vs Dim 2 P (GeV) vs Dim 3
View publications Download asView publications
cmenergies
7000 GeV
3500 GeV
4.5 GeV
12
6
3.3 GeV
4
3
23 more
reactions
E+ E- --> HADRONS
P P --> X
P P --> P P
12
6
PBAR P --> X
4
3
45 more
observables
SIG
DSIG/DT
SLOPE
12
6
DSIG/DOMEGA
4
3
34 more
Data Query Engine
phrases
Exclusive
Inclusive
Single Differential
12
6
Strange production
4
3
34 more
AND
Independent Variable
COS(THETA)
Delete
AND
Phrase
cross section
DeleteDelete

34. Beta mode within the next month
The prototype
HEPData

36. A new type of search
Data driven instead of publication driven
Full data provenance captured
Every data point is recorded with its parent publication’s inspire
id and table number
HEPData
Fast data rendering
With WebGL, now capable of rendering 100s of 1000s of data
points in the browser.
API
Query HEPData directly from mathematica, ROOT, etc. and
process our results as you need

37. We’re interested in more use cases….
Please get in touch with us to get early access!
HEPData

38. Data Consumers
Get access to the data in many environments
1. Publication Driven Search
2. Data Driven Search
3. Semantic Publishing
4. Data Conversion
5. Access in Analysis Environments

39. Semantic Publishing
Every article is tagged with schema.org vocabulary.
Makes it possible for Google and other search engines
to understand your content.
https://hepdata.net/record/ins1397180
Google’s view
Google’s View
https://hepdata.net/search

40. Data Consumers
Get access to the data in many environments
1. Publication Driven Search
2. Data Driven Search
3. Semantic Publishing
4. Data Conversion
5. Access in Analysis Environments

41. Getting Data in, getting data out…
Converter
Convert from YAML to ROOT, YODA, CSV
Install via PIP, use as a web service, and contribute to
more conversions!
Interoperability with other tools

42. Conversion to many formats

43. Data Consumers
Get access to the data in many environments
1. Publication Driven Search
2. Data Driven Search
3. Semantic Publishing
4. Data Conversion
5. Access in Analysis Environments

44. The same can be said for use in ROOT or any analysis platform with
a file parser :)
e.g. use data directly in Mathematica
No need to leave the software
environment you like.

45. Example File here

46. Sustainability
HEPData is now hosted at CERN and deployed by the RCS-SIS-OA team.

47. Load Balancer
hepdata-lb1
Web nodes
hepdata-web1 hepdata-web2
Tasks
hepdata-task1hepdata-task2
Elastic Search Cluster
Client
hepdata-search-client
Master
hepdata-search-master
CERN DB on Demand
Data
hepdata-search-data2
Data
hepdata-search-data1
Task 2Task 1
Converter Node
hepdata-converter
Converter
Cache
Cache
hepdata-cache1
Data hosted on the new EOS file system.
Hosted on OpenStack, managed with Puppet

48. Everything on Github! http://www.github.com/hepdata

49. Acknowledgements
Eamonn Maguire
Jan Stypka
Salvatore Mele
HEPData @ CERN
Graeme Watt
Michael Whalley
Frank Kraus
HEPData @ Durham
Lukas Heinrich
HEPData @ NYU
Kyle Cramner
Alumni
Laura Rueda-Garcia
Michal Szoziak Summer Student
and all the Invenio team, and the inspire team including Javier Martin Montull,
Jan Age Lavik, and Samuel Kaplun for their support!
Juan Luis Boya Garcia
HEPData @ Salamanca
Thanks to Luca Maschetti and the EOS team for their superb assistance!

50. Any questions?