Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

PEARC17: Modernizing GooFit: A Case Study

150 views

Published on

This is a talk covering the CUDA/OpenMP fitting package GooFit, and its modernization to be easy to install and run in any environment.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

PEARC17: Modernizing GooFit: A Case Study

  1. 1. Modernizing GOOFIT: A Case Study Henry Schreiner July 12, 2017 goofit.github.io/pearc17.pdf
  2. 2. History of GOOFIT Timeline This talk 2013 2014 2015 2016 2017 0.1 0.2 0.3 OpenMP 0.4 Work in forks Minor updates 1.0 CMAKE 2.0 /GooFit/GooFit • Developed by Rolf Andreassen in 2013 • A lot of duplicated work in 2014-2016 • We will cover the 2016-2017 modernization Henry Schreiner Modernizing GOOFIT: A Case Study 1/17
  3. 3. GOOFIT: Probability Density Function (PDF) Fitting width mean −3 −2 −1 0 1 2 3 0 0.2 0.4 x Design • Resembles the popular ROOFIT package in ROOT • Built with CUDA/OpenMP using THRUST • Includes 30+ High Energy Physics (HEP) PDFs and examples Henry Schreiner Modernizing GOOFIT: A Case Study 2/17
  4. 4. Features of a Fit Composition Variables PDF1 PDF2 CombinePDF DataSet Fitter Composition • Changed often • 100+ Variables possible PDFs • Many provided • Users may add more Backend • Managed by core team Henry Schreiner Modernizing GOOFIT: A Case Study 3/17
  5. 5. Why GOOFIT? Charm Physics at CERN’s LHCb experiment • 1,000,000+ events in dataset • 5 or more independent variables common • 20+ PDFs with complex coefficients • Some analyses run 1,000+ fits Henry Schreiner Modernizing GOOFIT: A Case Study 4/17
  6. 6. Why GOOFIT? Charm Physics at CERN’s LHCb experiment • 1,000,000+ events in dataset • 5 or more independent variables common • 20+ PDFs with complex coefficients • Some analyses run 1,000+ fits GOOFIT transforms fitting with 950x speedup over single-core ROOFIT Henry Schreiner Modernizing GOOFIT: A Case Study 4/17
  7. 7. Why GOOFIT? 1 2 4 12 24 48 36 60 118 336 625 1,240 OpenMP Threads Time[s] GOOFIT is fast • 142576 events • Unbinned fit • 24 physical Xeon cores K40 96.6 seconds P100 23.5 seconds Henry Schreiner Modernizing GOOFIT: A Case Study 5/17
  8. 8. GOOFIT 0.4: A State of Disrepair Research level code • Written for CUDA 4.0 and Compute Architecture 2.0 • Hardcoded paths in Makefiles • Little file organization • Forked 10+ times, new features not in master • Cludges: fake nvcc, globbing, fake ROOT, ... Henry Schreiner Modernizing GOOFIT: A Case Study 6/17
  9. 9. Build System Updates Iterative approach 1. Makefile cleanup and consolidation 2. Organization of file structure • ModernizeGooFit.py script 3. Adding CMAKE • Coexisted with makefiles for a while #include "CompositePdf.hh" abortWithCudaPrintFlush(__FILE__, __LINE__, "Failed"); ModernizeGooFit.py Uses regular expressions and Plumbum #include <goofit/PDFs/combine/CompositePdf.h> GooFit::abort(__FILE__, __LINE__, "Failed"); Henry Schreiner Modernizing GOOFIT: A Case Study 7/17
  10. 10. CMAKE CMAKE CUDA support • Require CMAKE 3.4+ • Backported FindCUDA from CMAKE 3.7 • Keyword vs. standard targets • CUDA in 3.8 massively improved Features • IDE support (XCODE, QTCREATOR) • Library discovery / configuration • Multiple compiler support • Integration with other tools • Download datafiles from GitHub releases Git submodules • Libraries as submodules • Automatic checkout by CMAKE build • Separate CMAKE folder ( /CLIUtils/cmake) and external libraries Henry Schreiner Modernizing GOOFIT: A Case Study 8/17
  11. 11. Automation and Testing Travis CI • Verify OpenMP build • Verify PRs • Upload coverage reports • Upload documentation Challenges C++11 • ROOT • Docs Tests • Run all examples with script • Slowly added verification • Unit-tests added with GOOGLETEST (evaluating CATCH) (NVIDIA-)DOCKER • Added images with ROOT+Utils • From scratch install Henry Schreiner Modernizing GOOFIT: A Case Study 9/17
  12. 12. Modernization C++11 • Limited to CUDA 7.0+ • Reduced # of lines / simplified • Used CLANG-TIDY to convert (CMAKE 3.6+ integration) Major updates nullptr • foreach • override Variadic templates • Initializer lists Cleanup • Readability: CLANG-FORMAT • Moved all code to namespace • Compile-time logging choice /fmtlib/fmt • Smart color output /agauniyal/rang • Removed custom classes and iterators (complex, etc) Henry Schreiner Modernizing GOOFIT: A Case Study 10/17
  13. 13. Command Line Parsing ./MyAnalysis generate_toy --params=file.ini --release_K892_mass --A12=0.3 --plot Recurring theme • Analyses require 40+ options and multiple procedures • Found a lot of duplicated code for argument parsing • Many bugs related to parsing (usually segfaults) • Needed powerful solution, with direct access to values Henry Schreiner Modernizing GOOFIT: A Case Study 11/17
  14. 14. CLI11 /CLIUtils/CLI11 • No dependencies • Compiles to single header file Features • Nested subcommands • Configuration files • 100% test coverage • CI tests on macOS/Linux/Windows • + GOOFIT’s features Use in GOOFIT • Testbed for new build features GooFit::Application • Auto logging • Optimization warnings • GPU switches • MPI support • Completely optional Henry Schreiner Modernizing GOOFIT: A Case Study 12/17
  15. 15. Improvements Expanded physics tools • Three body time-dependent amplitude analyses • Four body time-integrated and time-dependent amplitude analyses • Toy Monte Carlo generation using MCBOOSTER Caching: /bryancatanzaro/generics • Support for LDG caching • LDG generalized form • Performance boost for mid-age cards MPI • Available for Application • Supports multiple GPUs /MultithreadCorner/MCBooster is deprecated in favor of /MultithreadCorner/Hydra Henry Schreiner Modernizing GOOFIT: A Case Study 13/17
  16. 16. MINUIT 2 MINUIT in HEP • MINUIT is a standard HEP parameter search algorithm (CPU) • MINUIT 1 and 2 available in ROOT • Old copy of MINUIT 2 was available stand-alone Status in GOOFIT 0.4 • Internal MINUIT 1 copy • Required manual upkeep /GooFit/Minuit2 • Newly forked from ROOT 6.08 • CMAKE build, no other changes • Already being used outside GOOFIT Henry Schreiner Modernizing GOOFIT: A Case Study 14/17
  17. 17. PYTHON Bindings (GOOFIT 2.1 feature) Pre-release syntax from goofit import * xvar = Variable("xvar", 0, 10) xdata = UnbinnedDataSet(xvar) xdata.from_numpy(np.random.exponential(size=100000)) alpha = Variable("alpha", -2, 0.1, -10, 10) exppdf = ExpPdf("exppdf", xvar, alpha) exppdf.fitTo(data) /pybind/pybind11 • Only uses advanced C++11 • Full working example in GOOFIT 2.0 • Expanding for GOOFIT 2.1 • pip install with SCIKIT-BUILD Current challenges • CUDA vs. C++ • Pythonic syntax • Example conversion Henry Schreiner Modernizing GOOFIT: A Case Study 15/17
  18. 18. GOOFIT Future /GooFit/GooFit GOOFIT 2.0: Released • Source code and docs on GitHub GOOFIT 2.1: Coming soon • Drastically expanded Python bindings by Himadri Pandey Plans • HYDRA integration • Use in D 0 → K −π−π+π+ amplitude analysis • GOOFIT 2torial under development at henryiii.gitbook.io/goofit Henry Schreiner Modernizing GOOFIT: A Case Study 16/17
  19. 19. Build it yourself docker run -it alpine apk add --no-cache make cmake g++ git git clone --branch=stable https://github.com/GooFit/GooFit.git cd GooFit make Henry Schreiner Modernizing GOOFIT: A Case Study 17/17
  20. 20. Acknowledgments • A. Augusto Alves Jr. • Christoph Hasse • Bradley Hittle • Zachary Huard • Brian Maddock • Himadri Pandey • Michael Sokoloff • Karen Tomko Development of GOOFIT 2.x is supported under NSF grant PHY-1414736. GOOFIT was originally developed under PHY-1005530. Any opinions, findings, and conclusions or recom- mendations expressed in this presentation are those of the develop- ers and do not necessarily reflect the views of the National Science Foundation. CERN image from project-hl-lhc-industry.web.cern.ch. goofit.github.io Henry Schreiner Modernizing GOOFIT: A Case Study 18/17
  21. 21. Questions? henry.schreiner@uc.edu Henry Schreiner Modernizing GOOFIT: A Case Study 19/17

×