Unveiling the Early Universe with Intel Xeon Processors and Intel Xeon Phi at COSMOS (University of Cambridge)

Uploaded on


More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide
  • Background for "Walls" code. The discovery of the Higgs particle at the LHC confirms that the Universe has a complicated underlying structure with broken symmetries.   A more familiar type of broken symmetry is seen in a ferromagnet with its magnetised domains; the boundaries between each of these misaligned regions has additional energy which are called domain walls.  The same can happen in our Universe: Higgs-like fields in different regions become aligned in different directions with domain walls separating them (or indeed other defects, such as cosmic strings).    Walls have potentially important implications in the very early universe, as well as the late universe today.   The Planck satellite recently observed asymmetries (or anomalies) in the cosmic microwave sky; one possibility is that these are due to low-energy walls stretching across the Universe.  The Walls code creates random initial conditions on a 3D grid for a network of domain walls in the early universe.  It then solves dynamical field equations to find out how they evolve, counting up the wall area.  Typically walls 'scale' as the Universe expands, with a fixed number of them stretching across the observable universe at any one time.   The largest defect simulations to date have been performed on COSMOS using the Walls code, which has been developed into a number of variants.  For example, complex hybrid networks with up to 100 different types of walls and strings have been investigated, as well as wall evolution in four and even five spatial dimensions inspired by fundamental theory; this is only possible because of the large available memory on COSMOS.    The scalable OpenMP Walls code has also been used to benchmark shared-memory machines on several occasions.     
  • Laplacian stencil is standard 3D stencil. In 4D you have two extra additions (for l-1, l+1) and the 6 becomes an 8. This is a good example to refer to when discussing the “3D specialization” optimization; the original code relied on the periodic boundary condition to simplify this down to 3D. (i.e. stencil = 6 useful directions + ijkl + ijkl – 8ijkl = 6 useful directions – 6ijkl)The single loop optimization depends on a rearrangement of Eq. 2: we can rewrite to find n-1/2 from n and n-1. Also useful when discussing division hoisting: (1+delta) is a constant that we divide by every iteration.Eq 3 is weird-looking, but what the code basically does is looks for a change of sign between two adjacent grid-points in all dimensions (e.g. ijkl = +1, ijkl-1 = -1) to detect walls, then computes their area.
  • We use KMP_AFFINITY=compact,granularity=fine. This ensures the 4 threads on the same KNC core work on the same area of the stencil, and encourages cache re-use.If somebody asks about the reason for 480^3, it’s to fairly divide the same way on both pieces of hardware (i.e. 480 / 16 and 480 / 240 are both “good” decompositions).Other problem sizes would be unfair to the coprocessor. Coprocessor can’t ever have an advantage, because 240 / 16 = 15.Early work with #pragma omp collapse suggests we can actually get the same performance benefits for any problem size, by increasing task granularity. A few threads having one or two extra grid-points each has much less impact on performance than a few threads having one or two extra tiles.
  • “Optimizations” are largely hardware- or compiler-driven; improving an algorithm’s implementation on IA.“Modernizations” are code changes that are algorithmic; not specific to IA.Halo regions help because then reading “one to the left” or “one to the right” is a contiguous chunk. The original code used modulo division to determine the correct index (and then pulled the value from the other side of the array).
  • This is not the only thing we did to improve auto-vectorization, but it’s a good example of where the tools helped.We had two % operations (for +1 and -1) in each of the four dimensions.Also had to change some variable types (code tried to add doubles to a long) and help the compiler realise the variable it thought had a dependency issue was actually being used for a reduction.
  • Again, not the only instance of this – just an example.Think we actually hoisted a few other divisions out.
  • Just an example (but an easy one to follow).There’s some similar stuff in the area calculation.
  • This shows the speed-up for each version of the code running on the processor and coprocessor, relative to the original code running on the processor.A few interesting points: 1) auto-vectorization alone was sufficient to make coprocessor beat processor; 2) performance difference between the two is pretty striking; 3) because of this investigation, a single coprocessor runs the code 8x faster than dual-socket processor ran the baseline code, and 1.38x faster than it runs the final code.
  • This shows the speed-up for each version of the code running on the processor and coprocessor, relative to the original code running on the processor.A few interesting points: 1) auto-vectorization alone was sufficient to make coprocessor beat processor; 2) performance difference between the two is pretty striking; 3) because of this investigation, a single coprocessor runs the code 8x faster than dual-socket processor ran the baseline code, and 1.38x faster than it runs the final code.
  • None of the optimizations employed to date are really very complicated; they’re all either pragmas or simple code transformations. The code is readable and looks much like it did before.Future optimization work is likely to be a little more destructive, but we’re not interested in going to intrinsics or inline assembly.Bulk of future work is actually in moving away from a single coprocessor in order to run much larger problems.


  • 1. Unveiling the Early Universe with Intel® Xeon® Processors and Intel® Xeon Phi™ Coprocessors James Briggs, Carlos Martins, Paul Shellard COSMOS (University of Cambridge) John Fu, Karl Feind, Mike Woodacre John Pennycook, Jim Jeffers SGI Intel
  • 2. COSMOS @ DiRAC - University of Cambridge Supercomputing facility originally dedicated to cosmology research:  Founded in January 1997, now part of the UK DiRAC facility  Consortium brought together by Prof. Stephen Hawking  History of large, shared-memory Intel® Architecture machines COSMOS-IX:  SGI Altix UV2000, delivered in July 2012  1856 Intel® Xeon® processor cores (Sandy Bridge)  14.5 TB of globally shared memory  31 Intel® Xeon Phi™ coprocessors introduced in December 2012 2
  • 3. What is the SGI UV? The most flexible compute platform in the industry SSI  SuperPC  Huge Coherent Memory  Scalable IO/Coprocessors     PGAS UPC, SHMEM, etc. 8 PB Global Address Space Optimized for Small Transfers Rich Synchronization Cluster  MPI  Connectionless  Small message optimized 3
  • 4. The Code – “Walls”  Simulates the evolution of domain wall networks in the early universe:  Adjacent regions misalign over time (Higgs)  Domain „energy‟ walls form to separate them  „Observable” analogy is ferromagnet domains  Press-Ryden-Spergel algorithm [1]  Compiled for 3D or 4D simulations  Used at COSMOS for over 10 years, developed for complex hybrid networks  Stencil code, targeting SMP with OpenMP  Benchmark for acceptance testing on previous machines  To find out more about domain walls see the Hawking Cosmology Centre public pages: www.ctc.cam.ac.uk/outreach/origins/cosmic_structures_two.php Video courtesy of Dr. Carlos Martins, University of Porto. 4
  • 5. The Algorithm (in 3D) 5
  • 6. Baseline Performance “Out-of-the-Box” Comparison: 1.2  Processor is ~2x faster than coprocessor!  Why?  Poor vectorization  Poor memory behavior  etc Experimental Setup:  480 x 480 x 480 problem  2 x Intel® Xeon® E5-4650L processor 1 x Intel® Xeon Phi™ 5110P coprocessor  icc version 14.0.0 1 0.8 0.6 0.4 0.2 0 2 x Processor 1 x Coprocessor 6
  • 7. Optimization and Modernization (1/3) The Strategy:  Use straightforward parallel tuning techniques  vectorize, scale, memory  Use tools/compiler guidance features  Maintain readability and platform portability The Result:  Significant performance improvements in ~3-4 weeks  Single, clear, readable code-base  “Template” stencil code transferable to other simulations 7
  • 8. Optimization and Modernization (2/3) Optimizations  Improve auto-vectorization (using -vec-report3 and -guide-vec) int ip1 = (i+1) % Nx; loop was not vectorized: operator unsuited for vectorization. int ip1 = (i < Nx-1) ? i+1 : 0; LOOP WAS VECTORIZED 8
  • 9. Optimization and Modernization (2/3) Optimizations  Improve auto-vectorization (using -vec-report3 and -guide-vec)  Introduce halo regions, to improve cache behavior (and remove gathers) int ip1 = i+1; Data from i = 0 is replicated at i = Nx; all loads become contiguous 9
  • 10. Optimization and Modernization (2/3) Optimizations  Improve auto-vectorization (using -vec-report3 and -guide-vec)  Introduce halo regions, to improve cache behavior (and remove gathers)  Swap division by constants for multiplication by pre-computed reciprocals P2[i][j][k][l] = … / (1-delta); One division per stencil point P2[i][j][k][l] = … * i1mdelta; One division reused for all stencil points 10
  • 11. Optimization and Modernization (3/3) 11
  • 12. Optimization and Modernization (3/3) Modernizations  Reduce memory footprint (2x) by removing redundant arrays  Remove unnecessary 4D calculations and array lookups from 3D simulations Lphi = P[i-1][j][k][l] + P[i+1][j][k][l] + P[i][j-1][k][l] + … - 8 * P[i][j][k][l]; Lphi = P[i][j-1][k][l] + P[i][j+1][k][l] + … - 6 * P[i][j][k][l]; Saves two reads/additions per stencil point 12
  • 13. Optimization and Modernization (3/3) Modernizations  Reduce memory footprint (2x) by removing redundant arrays  Remove unnecessary 4D calculations and array lookups from 3D simulations  Combine three algorithmic stages into a single loop Before: solve(t), timestep(), area(t+1) After: solve(t), area(t), timestep() Allows for one pass through the data each timestep. 13
  • 14. Final Performance 8 7 6 5 4 3 2 1 0 Baseline Auto-vec Memory Reduction Halo (Inner) 2 x Processor 3D Specialisation Halo (All) Single Loop Division Hoisting 1 x Coprocessor 14
  • 15. Final Performance 1.38x 8 7 1.44x 6 1.62x 5 1.52x 4 1.28x 1.18x 3 1.12x Auto-vec Memory Reduction 2 1 0.45x 0 Baseline Halo (Inner) 2 x Processor 3D Specialisation Halo (All) Single Loop Division Hoisting 1 x Coprocessor 15
  • 16. Future Work  Exploration of other optimizations (e.g. cache blocking)  Stream larger problems through coprocessors  Work sharing between multiple processors and coprocessors  Incorporate stencil template into other key codes 16
  • 17. Conclusions Modernizing Code for Parallelism Works!  Straightforward code changes -> Dramatic Performance Impact  Dual-tuning advantage –> Single Source  Processor ~6x ; Coprocessor ~18x over baseline  Future benefits -> ready to take advantage of increasing parallelism Welcome to the Parallel Universe! 17
  • 18. Intel® Xeon Phi™ Coprocessor Starter Kits 3120A OR 5110P software.intel.com/xeon-phi-starter-kit Other brands and names are the property of their respective owners. *Pricing and starter kit configurations will vary. See software.intel.com/xeon-phi-starter-kit and provider websites for full details and disclaimers. Stated currency is US Dollars.
  • 19. References [1] W.H. Press, B.S. Ryden and D.N. Spergel, “Dynamical Evolution of Domain Walls in an Expanding Universe”, Astrophys. J. 347 (1989) [2] A.M.M. Leite and C.J.A.P Martins, “Scaling Properties of Domain Wall Networks”, Physical Review D 84 (2011) [3] A.M.M. Leite, C.J.A.P Martins and E.P.S. Shellard, “Accurate Calibration of the Velocity-Dependent One-Scale Model for Domain Walls”, Physics Letters B 7 18 (2013) 19
  • 20. Legal Disclaimer INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death. SHOULD YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS. Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined". Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information. The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order. Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800548-4725, or go to: http://www.intel.com/design/literature.htm Knights Landing and other code names featured are used internally within Intel to identify products that are in development and not yet publicly announced for release. Customers, licensees and other third parties are not authorized by Intel to use code names in advertising, promotion or marketing of any product or services and any such use of Intel's internal code names is at the sole risk of the user Intel, Cilk, VTune, Xeon, Xeon Phi and the Intel logo are trademarks of Intel Corporation in the United States and other countries. *Other names and brands may be claimed as the property of others. Copyright ©2013 Intel Corporation.
  • 21. Risk Factors The above statements and any others in this document that refer to plans and expectations for the third quarter, the year and the future are forwardlooking statements that involve a number of risks and uncertainties. Words such as “anticipates,” “expects,” “intends,” “plans,” “believes,” “seeks,” “estimates,” “may,” “will,” “should” and their variations identify forward-looking statements. Statements that refer to or are based on projections, uncertain events or assumptions also identify forward-looking statements. Many factors could affect Intel‟s actual results, and variances from Intel‟s current expectations regarding such factors could cause actual results to differ materially from those expressed in these forward-looking statements. Intel presently considers the following to be the important factors that could cause actual results to differ materially from the company‟s expectations. Demand could be different from Intel's expectations due to factors including changes in business and economic conditions; customer acceptance of Intel‟s and competitors‟ products; supply constraints and other disruptions affecting customers; changes in customer order patterns including order cancellations; and changes in the level of inventory at customers. Uncertainty in global economic and financial conditions poses a risk that consumers and businesses may defer purchases in response to negative financial events, which could negatively affect product demand and other related matters. Intel operates in intensely competitive industries that are characterized by a high percentage of costs that are fixed or difficult to reduce in the short term and product demand that is highly variable and difficult to forecast. Revenue and the gross margin percentage are affected by the timing of Intel product introductions and the demand for and market acceptance of Intel's products; actions taken by Intel's competitors, including product offerings and introductions, marketing programs and pricing pressures and Intel‟s response to such actions; and Intel‟s ability to respond quickly to technological developments and to incorporate new features into its products. The gross margin percentage could vary significantly from expectations based on capacity utilization; variations in inventory valuation, including variations related to the timing of qualifying products for sale; changes in revenue levels; segment product mix; the timing and execution of the manufacturing ramp and associated costs; start-up costs; excess or obsolete inventory; changes in unit costs; defects or disruptions in the supply of materials or resources; product manufacturing quality/yields; and impairments of long-lived assets, including manufacturing, assembly/test and intangible assets. Intel's results could be affected by adverse economic, social, political and physical/infrastructure conditions in countries where Intel, its customers or its suppliers operate, including military conflict and other security risks, natural disasters, infrastructure disruptions, health concerns and fluctuations in currency exchange rates. Expenses, particularly certain marketing and compensation expenses, as well as restructuring and asset impairment charges, vary depending on the level of demand for Intel's products and the level of revenue and profits. Intel‟s results could be affected by the timing of closing of acquisitions and divestitures. Intel's results could be affected by adverse effects associated with product defects and errata (deviations from published specifications), and by litigation or regulatory matters involving intellectual property, stockholder, consumer, antitrust, disclosure and other issues, such as the litigation and regulatory matters described in Intel's SEC reports. An unfavorable ruling could include monetary damages or an injunction prohibiting Intel from manufacturing or selling one or more products, precluding particular business practices, impacting Intel‟s ability to Rev. its products, or requiring other remedies such as compulsory licensing of intellectual property. A detailed discussion of these and other factors design 7/17/13 that could affect Intel‟s results is included in Intel‟s SEC filings, including the company‟s most recent reports on Form 10-Q, Form 10-K and earnings release.