Download as PDF, PPTX
Uploaded byinside-BigData.com
Assisting User’s Transition to Titan’s Accelerated Architecture
The document discusses the transition of users to Titan's hybrid computing architecture at Oak Ridge Leadership Computing Facility, highlighting the challenges and strategies for optimizing performance across manycore systems. Key lessons include the necessity of code refactoring for porting applications and the importance of hierarchical parallelism and data locality in achieving improved performance. It emphasizes the evolving programming models and directives that support efficient application development on future architectures.
More Related Content
Similar to Assisting User’s Transition to Titan’s Accelerated Architecture
![[Harvard CS264] 03 - Introduction to GPU Computing, CUDA Basics](https://cdn.slidesharecdn.com/ss_thumbnails/cs264201103-cudabasicsshare-110209024624-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
Assisting User’s Transition to Titan’s Accelerated Architecture
- 1. ORNL is managedby UT-Battelle for the US Department of Energy Leveraging Leadership Computing Facilities: Assisting User's Transition to Titan's Accelerated Architecture Fernanda Foertter HPC User Assistance Team Oak Ridge Leadership Computing Facility Oak Ridge National Laboratory Workshop on “Directives and Tools for Accelerators: A Seismic Programming Shift” Center for Advanced Computing and Data Systems, University of Houston 20 October 2014
- 2. 2 Outline • OLCF CenterOverview • Manycore is here to stay • The Titan Project: Lessons Learned • Coding for future architectures
- 3. 3 OLCF Services Liasons User Assistance Viz Tech Ops Outreach Oak RidgeLeadership Computing Facility Everest Future Tours Internships Tools Collaboration Scaling Performance Advocacy Training Software Communications
- 4. 4 Increased our systemcapability by 10,000X
- 5. 5 No more freelunch: Moore’s Law continues, Denard Scaling is over Herb Sutter: Dr. Dobb’s Journal: http://www.gotw.ca/publications/concurrency-ddj.htm
- 6. 6 Per core performancedown, cores up
- 7. 7 Kogge and Shalf,IEEE CISE Watts per Sq Cm
- 8.
- 9. 9 4,352 ft2 404 m2 SYSTEMSPECIFICATIONS: • Peak performance of 27.1 PF (24.5 & 2.6) • 18,688 Compute Nodes each with: • 16-Core AMD Opteron CPU (32 GB) • NVIDIA Tesla “K20x” GPU (6 GB) • 512 Service and I/O nodes • 200 Cabinets • 710 TB total system memory • Cray Gemini 3D Torus Interconnect ORNL’s “Titan” Hybrid System: Cray XK7 with AMD Opteron and NVIDIA Tesla processors
- 10. 10 Titan Compute Nodes(Cray XK7) Node AMD Opteron 6200 Interlagos (16 cores) 2.2 GHz 32 GB (DDR3) Accelerator Tesla K20x (2688 CUDA cores) 732 MHz 6 GB (DDR5) HT 3HT 3 PCIe Gen2
- 11. 11 Shift into HierarchicalParallelism • Expose more parallelism through code refactoring and source code directives – Doubles CPU performance of many codes • Use right type of processor for each task • Data locality: Keep data near processing – GPU has high bandwidth to local memory for rapid access – GPU has large internal cache • Explicit data management: Explicitly manage data movement between CPU and GPU memories CPU GPU Accelerator • Optimized for sequential multitasking • Optimized for many simultaneous tasks • 10× performance per socket • 5× more energy- efficient systems
- 12. 12 Old Programming Models Node Core MPI MPI Node Core Node Core
- 13. 13 1 Old Programming Models Node MPI MPI MPI Collectives Node MPI MPI MPI MPI Node MPI MPI MPI MPI Collectives MPI
- 14. 14 1 Directive Programming Models Node OpenMP MPI MPI Node OpenMP Node OpenMP
- 15.
- 16.
- 17.
- 18. 18 Path to Exascale Hierarchicalparallelism Improve scalability of applications Expose more parallelism Code refactoring and source code directives can double performance Explicit data management Between CPU and GPU memories Data locality: Keep data near processing GPU has high bandwidth to local memory and large internal cache Heterogeneous multicore processor architecture Using right type of processor for each task
- 19.
- 20. 20 All Codes WillNeed Refactoring To Scale! • Up to 1-2 person-years required to port each code from Jaguar to Titan • We estimate possibly 70-80% of developer time was spent in code restructuring, regardless of whether using OpenMP / CUDA / OpenCL / OpenACC / … – Experience shows this is a one-time investment • Each code team must make its own choice of using OpenMP vs. CUDA vs. OpenCL vs. OpenACC, based on the specific case—may be different conclusion for each code • Our users and their sponsors must plan for this expense.
- 21. 21 Center for AcceleratedApplication Readiness (CAAR) • Prepare applications for accelerated architectures • Goals: – Create applications teams to develop and implement strategies for exposing hierarchical parallelism for our users applications – Maintain code portability across modern architectures – Learn from and share our results • We selected six applications from across different science domains and algorithmic motifs
- 22. 22 CAAR: SElected LessonsLearned • Repeated themes in the code porting work • finding more threadable work for the GPU • Improving memory access patterns • making GPU work (kernel calls) more coarse-grained if possible • making data on the GPU more persistent • overlapping data transfers with other work (leverage HyperQ) • use as much asynchronicity as possible (CPU, GPU, MPI, PCIe-2)
- 23. 23 CAAR: SElected LessonsLearned • The difficulty level of the GPU port was in part determined by: • Structure of the algorithms—e.g., available parallelism, high computational intensity • Code execution profile—flat or hot spots • The code size (LOC)
- 24. 24 CAAR: SElected LessonsLearned • More available flops on the node should lead us to think of new science opportunities enabled • We may need to look in unconventional places to get another ~30X thread parallelism that may be needed for exascale—e.g., parallelism in time
- 25. 25 Co-designing Future ProgrammingModels • Evolutionary vs. Revolutionary approaches: – Message Passing and PGAS • MPI, UPC, OpenSHMEM, Fortran 2008 CoArrays, Chapel – Shared Memory Models • OpenMP, Pthreads – Acceletator-based models • OpenACC, OpenMP 4.0, OpenCL, CUDA – Hybrid Models • MPI+OpenACC ,MPI + OpenMP 4.0, OpenSHMEM + OpenACC, etc • New runtime models: Legion, OCR, Express, ParSeC, – Asychronous task based models • How to efficiently map the model to the hardware while meeting application requirements?
- 26. 26 • Serve instandard’s committees • Gather requirements from users • Translate users’ needs and use cases Directives collaboration
- 27. 27 App Language Datastructure Issues LSMS 3 C++ Templated Matrix class with bare pointer to data. Either owns the data or is an alias to another Matrix object. STL::vector and STL::complex needed on device CAM-SE F90 Array of structs. A struct member of the struct has a multidimensional array member of which sections must be transferred at different times. Mini-FE C Vector of pointers transferred to the device. Pointers are to the same data structure. LAMMPS C / C++ Flat C arrays requiring transfer ICON (CSCS) F95 array of structs of allocatable arrays. Need selective deep copy of derived type members. UPACS F90 structs of allocatable arrays. GENESIS F90 structs of allocatable arrays, these arrays accessed by pointers that are set before entering the parallel region HFODD F90 Require better support for Fortran derived types Delta5D F77 / F90 vectors, indexing arrays; no derived types XGC1 F90 array of derived types with pointers to other nested derived types. block(b) %grp(g)%p. Need deep copy. DFTB F77 / F90 dense linear algebra NIM/FIM F90 Multidimensional arrays, no structs Requirements Gathering Example
- 28. 28 Challenges with Directive-based programmingmodels • How to specify the in-node parallelism in the application – Loop based parallelism is not enough for future systems • How to efficiently map the parallelism of the application to the hardware – How to schedule work to multiple accelerators within the node? – How to schedule work to within accelerators while being portable? • How to transfer data across different types of memory – Problem may go away but is important for data locality • How to specify different memory hierarchies in the programming model – Shared memory within GPU, etc
- 29. 29 Future is DescriptiveProgramming • Large number of small cores • Data parallelism is key • PCIe to CPU connection AMD Discrete GPU AMD APU • Integrated CPU+GPU cores • Target power efficient devices at this stage • Shared memory system with partitions INTEL Many Integrated Cores • 50+ number of x86 cores • Support conventional programming • Vectorization is key • Run as an accelerator or standalone NVIDIA GPU • Large number of small cores • Data parallelism is key • Support nested and dynamic parallelism • PCIe to host CPU or low power ARM CPU (CARMA) Directives help describe data layout, parallelism
- 30. 30 OpenACC influence àOpenMP • Compare OpenMP 4.0 accelerator extension with OpenACC – Understand mapping – Understand impact of newer OpenACC features • OpenACC is evolving with new features which may impact OpenMP 4.1 or 5. • OpenACC interoperability with OpenMP is important for the transition OpenACC 2.0 OpenMP 4.0 parallel target parallel/gang/workers/vector target teams/parallel/simd data target data parallel loop teams/distribute/parallel for update target update cache wait OpenMP 4.1 proposal declare declare target data enter/exit OpenMP 4.1 proposal routine declare target async wait OpenMP 4.1 proposal device type tile host data
- 31. 31 Training at OLCF • Webinars/Remote • Hands on • Lectures • Open to public!!
- 32.
- 33. 33 Conclusions • There’s noavoiding manycore • Rethink algorithms to expose more parallelism • Directives are morphing into Descriptive Programming • Memory placement is important • Flops are free, avoid reads/writes • Standards built from application requirements • Training events are open to the public • Looking for domain specific communities
- 34. 34 Acknowledgements OpenACC and OpenMPStandards Committees OLCF-3 CAAR Team: • Bronson Messer, Wayne Joubert, Mike Brown, Matt Norman, Markus Eisenbach, Ramanan Sankaran OLCF-3 Vendor Partners: Cray, AMD, NVIDIA, CAPS, Allinea This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE- AC05-00OR22725.
- 35.