CERN’s Next Generation Data Analysis Platform with Apache Spark with Enric Tejedor
The CERN experiments and their particle accelerator, the Large Hadron Collider (LHC), will soon have collected a total of one exabyte of data. Moreover, the next upgrade of the accelerator, the high-luminosity LHC, will dramatically increase the rate of particle collisions, thus boosting the potential for discoveries but also generating unprecedented data challenges. In order to process and analyse all those data, CERN is investigating complementary ways to the traditional approaches, which mainly rely on Grid and batch jobs for data reconstruction, calibration and skimming combined with a phase of local analysis of reduced data. The new techniques should allow for interactive analysis on much bigger datasets by transparently exploiting dynamically pluggable resources. In that sense, Spark is being used at CERN to process large physics datasets in a distributed fashion. The most widely used tool for high-energy physics analysis, ROOT, implements a layer on top of Spark in order to distribute computations across a cluster of machines. This makes it possible for physics analysis written in either C++ or Python to be parallelised on Spark clusters, while reading the input data from CERN’s mass storage system: EOS. On the other hand, another important use case of Spark at CERN has recently emerged. The LHC logging service, which collects data from the accelerator to get information on how to improve the performance of the machine, is currently migrating its architecture to leverage Spark for its analytics workflows. This talk will discuss the unique challenges of the aforementioned use cases and how SWAN, the CERN service for interactive web-based analysis, now supports them thanks to a new feature: the possibility for users to dynamically plug Spark clusters into their sessions in order to offload computations to those resources.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to CERN’s Next Generation Data Analysis Platform with Apache Spark with Enric Tejedor
Similar to CERN’s Next Generation Data Analysis Platform with Apache Spark with Enric Tejedor (20)
More from Databricks
More from Databricks (20)
Recently uploaded
Recently uploaded (20)
CERN’s Next Generation Data Analysis Platform with Apache Spark with Enric Tejedor
- 1. Enric Tejedor, Prasanth Kothuri, CERN CERN´s Platform for Data Analysis with Spark #SAISExp1
- 2. Outline 1. What we do at CERN 2. Interactive data analysis with Spark 3. Use cases 1. Physics data 2. Controls data 2#SAISExp1
- 3. ##SAISExp1 Founded in 1954 Mission: fundamental research in Physics 3
- 4. ##SAISExp1 Alice The Large Hadron Collider (LHC) The largest scientific experiment to date 1 PB / second of raw physics data 4 TOTEM
- 5. ##SAISExp1 Discovery of the Higgs boson ATLAS & CMS 2012 5
- 6. ##SAISExp1 Invention of the WWW Sir Tim Berners-Lee 1989 http://info.cern.ch 6
- 7. ##SAISExp1 7 >15,000 scientists >100 nationalities
- 8. Physics Data Processing and Analysis at CERN #SAISExp1
- 9. LHC Data Pipeline at CERN 9#SAISExp1 Data collection Data analysis Offline processing Event selection, stat. treatment… Reconstruction Processing, skimming Event filtering Reco Results Raw data Analysis formats CMS ALICE LHCb ATLAS
- 10. ROOT • The data analysis framework for high-energy physics (HEP) • Data processing, statistical analysis, visualisation, I/O and storage • ~1 EB stored in ROOT format – Binary, compressed – Columnar 10#SAISExp1 https://root.cern
- 11. ROOT Dataset 11#SAISExp1 px py pz e Row: collision event ROOT dataset stored in one or multiple files Column: physics entity Can be nested structures (tree-like)
- 12. ##SAISExp1 The LHC Computing Grid Started in 2002 Provide processing power and data access to physicists ~170 centres in 42 countries Running 24/7/365 12
- 13. Distributed Computing in HEP • Physicists use Grid and batch resources to process LHC data in parallel 13#SAISExp1 Initial Dataset Merge Merge job Data processing Job Range 1 Job Range 2 Job Range N … Final Result … Partial results
- 14. HL-LHC: Even More Data! • Coming upgrade: High-Luminosity LHC • 30x more data collected • Big challenge for software and computing 14#SAISExp1 LHC HL-LHC CMS estimated disk space required by tier Source:HSFcommunitywhitepaper
- 15. LHC Data Pipeline at CERN 15#SAISExp1 Data collection Data analysis Offline processing Event selection, stat. treatment… Reconstruction Processing, skimming Event filtering Reco Results Raw data Analysis formats CMS ALICE LHCb ATLAS
- 16. Final Steps of Data Analysis 16#SAISExp1 Data collection Data analysis Offline processing Event selection, stat. treatment… Reconstruction Processing, skimming Event filtering Reco Results Raw data Analysis formats CMS ALICE LHCb ATLAS
- 18. The SWAN Service • SWAN: Service for Web-based Analysis • Interactive computing platform for scientists – Based on Jupyter notebooks • Analysis with only a web browser • Easy sharing of results • Integrated with CERN resources – Storage, software and computing 18#SAISExp1 https://swan.web.cern.ch
- 20. SWAN Architecture 20#SAISExp1 Web Portal User Session Scheduler CERN Resources EOS/CERNBox (Data/User files) CVMFS (Software) User 1 User 2 User n... CERN Credentials
- 23. Integration with Spark 23 Spark Cluster Spark Master Spark Executor Task User Notebook Task Task Spark Driver Offload computations to pluggable resources #SAISExp1
- 24. Spark Connector 24 Configure Spark and connect to cluster with a click #SAISExp1
- 25. Spark Monitor • Bridge the gap between interactive computing and distributed data processing • Automatically appears when a Spark job is submitted from a cell • Progress bars, task timeline, resource utilisation 25#SAISExp1 Code here!
- 27. The HEP DataFrame • RDataFrame – Implemented in C++, interfaced also to Python – Tailored for ROOT and HEP 27 df = RDataFrame(dataset) df2 = df.Filter('x > 0') .Define('r2', 'x*x + y*y') h = df2.Histo1D('r2') g = df2.Graph('x', 'y') data x,y Filter x>0 Define r²=x²+y² Histo1D r² Graph x,y #SAISExp1
- 28. Distributed RDataFrame • Exploratory work to parallelise RDataFrame computations with multiple backends 28 d = RDataFrame(dataset) f = d.Define(...) .Define(...) .Filter(...) h1 = f.Histo1D(...) h2 = f.Histo2D(...) g = f.Graph(...) Local … … … … … CPU … … … … CPU … … … … CPU … … … … CPU … … … … CPU … … … … CPU … … … … CPU Computation Graph #SAISExp1
- 29. Spark Cluster Executor Spark Backend for RDataFrame • Map-reduce workflow where every mapper runs the RDataFrame computation graph on a range of collision events • Run analysis in C++ with Spark – Through Python bindings 29 Spark Backend Mapper Driver px py pz eMake ranges Read ranges Schedule tasks Reducer #SAISExp1
- 30. Real Example: TOTEM Analysis • TOTEM experiment analysis coded with RDataFrame • Spark backend • 4.7 TB dataset on EOS • Launched from SWAN to a dedicated Spark cluster • Get to physics results faster! 30#SAISExp1
- 32. LHC: Huge machine, highly sophisticated Control and monitoring are crucial 32
- 33. Hardware Controls • Complex control system for monitoring • 1000s of devices, 100s of properties each 33#SAISExp1
- 34. Controls Log Data Growth 34 Now 1.5 TB/day #SAISExp1
- 35. CERN Accelerator Logging Service • Old system based on SQL databases – Hard to scale horizontally – Slow data extraction • New system (NXCALS) – Data pumped into HBase and HDFS (Parquet) – Spark to extract and process data – SWAN to visualise + analyse 35#SAISExp1
- 36. NXCALS Data Analysis in SWAN • NXCALS rely on SWAN as their data analysis platform • Connection to Spark clusters • Access to software (data science Python ecosystem) 36#SAISExp1
- 38. Challenges • The increase in physics and controls data volumes and complexity is pushing software at CERN – Adoption of Spark and other big data technologies still in its early stages • Large codebase developed over decades – Cannot change overnight • Changing the mindset of programmers takes time – Declarative analysis – Pushing computations to data 38#SAISExp1
- 39. Future Directions • Bridge the gap between data processing needs and technology evolution – Complement traditional ways with new strategies • Combine interactive analysis with easy access to more processing power – Higher-level programming models – Pluggable computing resources More on CERN and Spark: stay tuned for Luca’s and Prasanth’s presentations 39#SAISExp1
- 41. JupyterLab • Jupyter is evolving towards a desktop- like environment – Notebook, terminal, file browser, editors, … – Highly customisable 41#SAISExp1