Downloaded 12 times
More Related Content
Similar to Big data from the LHC commissioning: practical lessons from big science - Simon Metson (Cloudant)
Big data from the LHC commissioning: practical lessons from big science - Simon Metson (Cloudant)
- 1. Big Data fromthe LHC Commissioning ! Practical Lessons from Big Science Simon/@drsm79
- 2.
- 6. Time at placesI’ve worked Bristol University Cloudant
- 7.
- 8.
- 9.
- 10. The formula Grant *Effectiveness
- 11. The life ofLHC data 1. Detected by experiment 2. “Online” filtering (hardware and software) 3. Transferred to CERN main campus, archived & reconstructed 4. Transferred to T1 sites, archived, reconstructed & skimmed 5. Transferred to T2 sites, reconstructed, skimmed, filtered & analysed 6. Written into locally analysable files, put on laptops 7. Turned into a plot in a paper
- 12. The life ofLHC data 1. Detected by experiment 2. “Online” filtering (hardware and software) 3. Transferred to CERN main campus, archived & reconstructed 4. Transferred to T1 sites, archived, reconstructed & skimmed 5. Transferred to T2 sites, reconstructed, skimmed, filtered & analysed 6. Written into locally analysable files, put on laptops 7. Turned into a plot in a paper
- 13.
- 14. Chain u pse rie s o f “ato m smashe rs”
- 15. Pu t sensitive cam eras in aw kw ard places
- 16. Re co rde ve nts
- 17. Process data on highend machines http://www.chilton-computing.org.uk
- 18. The life ofLHC data 1. Detected by experiment 2. “Online” filtering (hardware and software) 3. Transferred to CERN main campus, archived & reconstructed 4. Transferred to T1 sites, archived, reconstructed & skimmed 5. Transferred to T2 sites, reconstructed, skimmed, filtered & analysed 6. Written into locally analysable files, put on laptops 7. Turned into a plot in a paper
- 19. CMS online dataflow We have a big digital camera
- 20. It takes photosof this courtesy of James Jackson
- 21. which come outlike this courtesy of James Jackson
- 22. CMS online dataflow We have a big digital camera Which goes into lots of computers (the HLT)
- 23. CMS online dataflow We have a big digital camera Which goes into lots of computers (the HLT) Which goes into lots of disk (the Storage Manager)
- 24. CMS data flow Writeto digital We have a big HLT at camera ~200GB/s Which goes into lots of Write to Storage computers ~2GB/s (the HLT) Manager at Which goes into lots of Write to T0 at ~2GB/s disk (the Storage Manager)
- 25. The life ofLHC data 1. Detected by experiment 2. “Online” filtering (hardware and software) 3. Transferred to CERN main campus, archived & reconstructed 4. Transferred to T1 sites, archived, reconstructed & skimmed 5. Transferred to T2 sites, reconstructed, skimmed, filtered & analysed 6. Written into locally analysable files, put on laptops 7. Turned into a plot in a paper
- 26.
- 27. The life ofLHC data 1. Detected by experiment 2. “Online” filtering (hardware and software) 3. Transferred to CERN main campus, archived & reconstructed 4. Transferred to T1 sites, archived, reconstructed & skimmed 5. Transferred to T2 sites, reconstructed, skimmed, filtered & analysed 6. Written into locally analysable files, put on laptops 7. Turned into a plot in a paper
- 28.
- 29. Why transfer somuch data?
- 30. To process allthe data taken in one year on one computer would take ~64,000 years
- 32. The life ofLHC data 1. Detected by experiment 2. “Online” filtering (hardware and software) 3. Transferred to CERN main campus, archived & reconstructed 4. Transferred to T1 sites, archived, reconstructed & skimmed 5. Transferred to T2 sites, reconstructed, skimmed, filtered & analysed 6. Written into locally analysable files, put on laptops 7. Turned into a plot in a paper
- 33. Analysis • Each analysisis ~unique • Query language is C++ • Runs on distributed system and local resources • Series of “cut” selections to identify interesting events • Data in the final plot may be substantially reduced from the original dataset
- 34. Workflow ladder Number ofusers Large datasets (>100 TB) Complex computation Large datasets (>100 TB) Simple computation Shared datasets (>500 GB) Complex computation Shared datasets (10-500 GB) Complex computation Shared datasets (10-100 GB) Simple computation Shared datasets (0.1-10 GB) Simple computation Private datasets (0.1-10 GB) Simple computation } } } Use Grid compute and storage exclusively Work on departmental resources, store resulting datasets to Grid storage Work on laptop/desktop machine, store resulting datasets to Grid storage
- 35. The life ofLHC simulated data 1. Simulated by experimentalists at T0/T1/T2 sites 2. Transferred to T1 sites, archived possibly reconstructed & skimmed 3. Transferred to T2 sites, reconstructed, skimmed, filtered & analysed 4. Written into locally analysable files, put on laptops 5. Turned into a plot in a paper
- 36.
- 37. ! “We are goingto die, and that makes us the lucky ones. Most people are never going to die because they are never going to be born.” ! - Richard Dawkins
- 39.
- 40. Setup • Maybe abit different to other people • Many sites (>100) with >100’s TB storage, 10000’s worker nodes • Global system • Why not at one site? • politics, power budget, cost
- 41.
- 42. We Have a“Big Data” Problem
- 43. We Have aBig “Data Problem”
- 44. Do what youdo best, out source the rest
- 45. What's interesting is thatbig data isn't interesting any more
- 46.
- 47.
- 48. Our situation • Expert users,who are not interested in infrastructure • Will work around things they perceive as unnecessary limitations
- 49.
- 50. How to engage disruptiveusers?
- 51.
- 52.
- 53.
- 54. Our situation • Limitedresources for integration/ testbed style activities • Strange organisation
- 55.
- 56. There is nosuch thing as now
- 57. Keep things aslocal as possible
- 58.
- 59. Small files arebad, m'kay
- 60.
- 61.
- 62. People are harderthan computers
- 63.
- 64.
- 65. Consequences • Automate allthe things • Learn to love a configuration management system • Make sure everyone in the team knows how to interact with it • Simple human solutions go a long way
- 66.
- 67.
- 68. Workflow ladder Number ofusers Large datasets (100 TB) Complex computation Large datasets (100 TB) Simple computation Shared datasets (500 GB) Complex computation Shared datasets (10-500 GB) Complex computation Shared datasets (10-100 GB) Simple computation Shared datasets (0.1-10 GB) Simple computation Private datasets (0.1-10 GB) Simple computation } } } Use Grid compute and storage exclusively Work on departmental resources, store resulting datasets to Grid storage Work on laptop/desktop machine, store resulting datasets to Grid storage
- 69.