Presented at JAX London 2013 The Large Hadron Collider experiments manage tens of petabytes of data spread across hundreds of data centres. Managing and processing this volume required significant infrastructure and novel software systems, involving years of R&D and significant commissioning to prepare for the LHC First Data. The evolution of this global computing infrastructure, and the specialisations made by the experiments, have lessons relevant for many commercial "big data" users.

1. Big Data from the LHC
Commissioning
!
Practical Lessons from Big Science
Simon/@drsm79

2. Hello!

6. Time at places I’ve worked
Bristol University
Cloudant

7. Python
Perl
Bash
C++
Java
Javascript
Fortran
100
75
50
25
0
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013

8. The formula
G*E

9. The formula
Fixed
G* E
Fixed
Usually fixed

10. The formula
Grant * Effectiveness

11. The life of LHC data
1. Detected by experiment
2. “Online” filtering (hardware and software)
3. Transferred to CERN main campus, archived & reconstructed
4. Transferred to T1 sites, archived, reconstructed & skimmed
5. Transferred to T2 sites, reconstructed, skimmed, filtered & analysed
6. Written into locally analysable files, put on laptops
7. Turned into a plot in a paper

12. The life of LHC data
1. Detected by experiment
2. “Online” filtering (hardware and software)
3. Transferred to CERN main campus, archived & reconstructed
4. Transferred to T1 sites, archived, reconstructed & skimmed
5. Transferred to T2 sites, reconstructed, skimmed, filtered & analysed
6. Written into locally analysable files, put on laptops
7. Turned into a plot in a paper

13. Dig big
tu nne ls

14. Chain u p se rie s o f
“ato m smashe rs”

15. Pu t se nsitive cam eras in
aw kw ard places

16. Re co rd e ve nts

17. Process data on
high end
machines
http://www.chilton-computing.org.uk

18. The life of LHC data
1. Detected by experiment
2. “Online” filtering (hardware and software)
3. Transferred to CERN main campus, archived & reconstructed
4. Transferred to T1 sites, archived, reconstructed & skimmed
5. Transferred to T2 sites, reconstructed, skimmed, filtered & analysed
6. Written into locally analysable files, put on laptops
7. Turned into a plot in a paper

19. CMS online data flow
We have a big digital camera

20. It takes photos of this
courtesy of James Jackson

21. which come out like this
courtesy of James Jackson

22. CMS online data flow
We have a big digital camera
Which goes into lots of
computers (the HLT)

23. CMS online data flow
We have a big digital camera
Which goes into lots of
computers (the HLT)
Which goes into lots of
disk (the Storage Manager)

24. CMS data flow
Write to digital
We have a big HLT at camera
~200GB/s
Which goes into lots of
Write to Storage
computers ~2GB/s
(the HLT)
Manager at
Which goes into lots of
Write to T0 at ~2GB/s
disk (the Storage Manager)

25. The life of LHC data
1. Detected by experiment
2. “Online” filtering (hardware and software)
3. Transferred to CERN main campus, archived & reconstructed
4. Transferred to T1 sites, archived, reconstructed & skimmed
5. Transferred to T2 sites, reconstructed, skimmed, filtered & analysed
6. Written into locally analysable files, put on laptops
7. Turned into a plot in a paper

26. 10PB of data/year

27. The life of LHC data
1. Detected by experiment
2. “Online” filtering (hardware and software)
3. Transferred to CERN main campus, archived & reconstructed
4. Transferred to T1 sites, archived, reconstructed & skimmed
5. Transferred to T2 sites, reconstructed, skimmed, filtered &
analysed
6. Written into locally analysable files, put on laptops
7. Turned into a plot in a paper

28. 1PB/week

29. Why transfer so much
data?

30. To process all the data
taken in one year on
one computer would
take ~64,000 years

32. The life of LHC data
1. Detected by experiment
2. “Online” filtering (hardware and software)
3. Transferred to CERN main campus, archived & reconstructed
4. Transferred to T1 sites, archived, reconstructed & skimmed
5. Transferred to T2 sites, reconstructed, skimmed, filtered & analysed
6. Written into locally analysable files, put on laptops
7. Turned into a plot in a paper

33. Analysis
• Each analysis is ~unique
• Query language is C++
• Runs on distributed system and local resources
• Series of “cut” selections to identify interesting
events
• Data in the final plot may be substantially
reduced from the original dataset

34. Workflow ladder
Number of users
Large datasets (>100 TB)
Complex computation
Large datasets (>100 TB)
Simple computation
Shared datasets (>500 GB)
Complex computation
Shared datasets (10-500 GB)
Complex computation
Shared datasets (10-100 GB)
Simple computation
Shared datasets (0.1-10 GB)
Simple computation
Private datasets (0.1-10 GB)
Simple computation
}
}
}
Use Grid compute and storage
exclusively
Work on departmental resources,
store resulting datasets to Grid storage
Work on laptop/desktop machine,
store resulting datasets to Grid storage

35. The life of LHC
simulated data
1. Simulated by experimentalists at T0/T1/T2 sites
2. Transferred to T1 sites, archived possibly reconstructed &
skimmed
3. Transferred to T2 sites, reconstructed, skimmed, filtered &
analysed
4. Written into locally analysable files, put on laptops
5. Turned into a plot in a paper

36. Most events get cut

37. !
“We are going to die, and that makes us the
lucky ones. Most people are never going to
die because they are never going to be born.”
!
- Richard Dawkins

39. Adoption & Use

40. Setup
• Maybe a bit different to other people
• Many sites (>100) with >100’s TB storage,
10000’s worker nodes
• Global system
• Why not at one site?
• politics, power budget, cost

41. The grid

42. We Have a “Big Data”
Problem

43. We Have a Big “Data
Problem”

44. Do what you do best,
out source the rest

45. What's interesting is
that big data isn't
interesting any more

46. NIH

47. Define and refine
workflows

48. Our situation
•
Expert users, who are not
interested in infrastructure
• Will work around things they
perceive as unnecessary
limitations

49. Disruptive users

50. How to engage
disruptive users?

51. Open access

52. 1PB/week

53. Open access

54. Our situation
• Limited resources for integration/
testbed style activities
• Strange organisation

55. Data temperature

56. There is no such thing
as now

57. Keep things as local as
possible

58. Defining monitoring is
difficult

59. Small files are bad,
m'kay

60. Compartmentalise
metadata

61. Recognise, embrace and
communicate failures

62. People are harder than
computers

63. People are important

64. The formula
64

65. Consequences
• Automate all the things
• Learn to love a configuration management
system
• Make sure everyone in the team knows
how to interact with it
• Simple human solutions go a long way

66. Build good abstractions

67. Encourage
collaboration

68. Workflow ladder
Number of users
Large datasets (100 TB)
Complex computation
Large datasets (100 TB)
Simple computation
Shared datasets (500 GB)
Complex computation
Shared datasets (10-500 GB)
Complex computation
Shared datasets (10-100 GB)
Simple computation
Shared datasets (0.1-10 GB)
Simple computation
Private datasets (0.1-10 GB)
Simple computation
}
}
}
Use Grid compute and storage
exclusively
Work on departmental resources,
store resulting datasets to Grid storage
Work on laptop/desktop machine,
store resulting datasets to Grid storage

69. Summary