DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
Unveiling the Early Universe with Intel Xeon Processors and Intel Xeon Phi at COSMOS (University of Cambridge)
1. Unveiling the Early Universe with
Intel® Xeon® Processors and Intel® Xeon Phi™
Coprocessors
James Briggs,
Carlos Martins, Paul Shellard
COSMOS (University of Cambridge)
John Fu, Karl Feind, Mike Woodacre John Pennycook, Jim Jeffers
SGI
Intel
2. COSMOS @ DiRAC - University of Cambridge
Supercomputing facility originally dedicated to cosmology
research:
Founded in January 1997, now part of the UK DiRAC
facility
Consortium brought together by Prof. Stephen Hawking
History of large, shared-memory Intel® Architecture
machines
COSMOS-IX:
SGI Altix UV2000, delivered in July 2012
1856 Intel® Xeon® processor cores (Sandy Bridge)
14.5 TB of globally shared memory
31 Intel® Xeon Phi™ coprocessors introduced in
December 2012
2
3. What is the SGI UV?
The most flexible
compute platform in the
industry
SSI
SuperPC
Huge Coherent
Memory
Scalable
IO/Coprocessors
PGAS
UPC, SHMEM, etc.
8 PB Global Address
Space
Optimized for Small
Transfers
Rich Synchronization
Cluster
MPI
Connectionless
Small
message
optimized
3
4. The Code – “Walls”
Simulates the evolution of domain wall networks
in the early universe:
Adjacent regions misalign over time (Higgs)
Domain „energy‟ walls form to separate
them
„Observable” analogy is ferromagnet
domains
Press-Ryden-Spergel algorithm [1]
Compiled for 3D or 4D simulations
Used at COSMOS for over 10 years, developed
for complex hybrid networks
Stencil code, targeting SMP with OpenMP
Benchmark for acceptance testing on previous
machines
To find out more about domain walls see the
Hawking Cosmology Centre public pages:
www.ctc.cam.ac.uk/outreach/origins/cosmic_structures_two.php
Video courtesy of Dr. Carlos Martins, University of Porto.
4
6. Baseline Performance
“Out-of-the-Box” Comparison:
1.2
Processor is ~2x faster than coprocessor!
Why?
Poor vectorization
Poor memory behavior
etc
Experimental Setup:
480 x 480 x 480 problem
2 x Intel® Xeon® E5-4650L processor
1 x Intel® Xeon Phi™ 5110P coprocessor
icc version 14.0.0
1
0.8
0.6
0.4
0.2
0
2 x Processor
1 x Coprocessor
6
7. Optimization and Modernization (1/3)
The Strategy:
Use straightforward parallel tuning techniques
vectorize, scale, memory
Use tools/compiler guidance features
Maintain readability and platform portability
The Result:
Significant performance improvements in ~3-4 weeks
Single, clear, readable code-base
“Template” stencil code transferable to other simulations
7
8. Optimization and Modernization (2/3)
Optimizations
Improve auto-vectorization (using -vec-report3 and -guide-vec)
int ip1 = (i+1) % Nx;
loop was not vectorized: operator unsuited for vectorization.
int ip1 = (i < Nx-1) ? i+1 : 0;
LOOP WAS VECTORIZED
8
9. Optimization and Modernization (2/3)
Optimizations
Improve auto-vectorization (using -vec-report3 and -guide-vec)
Introduce halo regions, to improve cache behavior (and remove gathers)
int ip1 = i+1;
Data from i = 0 is replicated at i = Nx; all loads become contiguous
9
10. Optimization and Modernization (2/3)
Optimizations
Improve auto-vectorization (using -vec-report3 and -guide-vec)
Introduce halo regions, to improve cache behavior (and remove gathers)
Swap division by constants for multiplication by pre-computed reciprocals
P2[i][j][k][l] = … / (1-delta);
One division per stencil point
P2[i][j][k][l] = … * i1mdelta;
One division reused for all stencil points
10
12. Optimization and Modernization (3/3)
Modernizations
Reduce memory footprint (2x) by removing redundant arrays
Remove unnecessary 4D calculations and array lookups from 3D simulations
Lphi = P[i-1][j][k][l] + P[i+1][j][k][l] + P[i][j-1][k][l] + …
- 8 * P[i][j][k][l];
Lphi = P[i][j-1][k][l] + P[i][j+1][k][l] + …
- 6 * P[i][j][k][l];
Saves two reads/additions per stencil point
12
13. Optimization and Modernization (3/3)
Modernizations
Reduce memory footprint (2x) by removing redundant arrays
Remove unnecessary 4D calculations and array lookups from 3D simulations
Combine three algorithmic stages into a single loop
Before: solve(t), timestep(), area(t+1)
After: solve(t), area(t), timestep()
Allows for one pass through the data each timestep.
13
16. Future Work
Exploration of other optimizations (e.g. cache blocking)
Stream larger problems through coprocessors
Work sharing between multiple processors and coprocessors
Incorporate stencil template into other key codes
16
17. Conclusions
Modernizing Code for Parallelism Works!
Straightforward code changes -> Dramatic Performance Impact
Dual-tuning advantage –> Single Source
Processor ~6x ; Coprocessor ~18x over baseline
Future benefits -> ready to take advantage of increasing parallelism
Welcome to the Parallel Universe!
17
18. Intel® Xeon Phi™ Coprocessor Starter Kits
3120A
OR
5110P
software.intel.com/xeon-phi-starter-kit
Other brands and names are the property of their respective owners.
*Pricing and starter kit configurations will vary. See software.intel.com/xeon-phi-starter-kit and provider websites for full details and disclaimers. Stated currency is US Dollars.
19. References
[1] W.H. Press, B.S. Ryden and D.N. Spergel, “Dynamical Evolution of Domain
Walls in an Expanding Universe”, Astrophys. J. 347 (1989)
[2] A.M.M. Leite and C.J.A.P Martins, “Scaling Properties of Domain Wall
Networks”, Physical Review D 84 (2011)
[3] A.M.M. Leite, C.J.A.P Martins and E.P.S. Shellard, “Accurate Calibration of
the Velocity-Dependent One-Scale Model for Domain Walls”, Physics Letters B 7
18 (2013)
19
23. Risk Factors
The above statements and any others in this document that refer to plans and expectations for the third quarter, the year and the future are forwardlooking statements that involve a number of risks and uncertainties. Words such as “anticipates,” “expects,” “intends,” “plans,” “believes,” “seeks,”
“estimates,” “may,” “will,” “should” and their variations identify forward-looking statements. Statements that refer to or are based on
projections, uncertain events or assumptions also identify forward-looking statements. Many factors could affect Intel‟s actual results, and variances
from Intel‟s current expectations regarding such factors could cause actual results to differ materially from those expressed in these forward-looking
statements. Intel presently considers the following to be the important factors that could cause actual results to differ materially from the company‟s
expectations. Demand could be different from Intel's expectations due to factors including changes in business and economic conditions; customer
acceptance of Intel‟s and competitors‟ products; supply constraints and other disruptions affecting customers; changes in customer order patterns
including order cancellations; and changes in the level of inventory at customers. Uncertainty in global economic and financial conditions poses a risk
that consumers and businesses may defer purchases in response to negative financial events, which could negatively affect product demand and
other related matters. Intel operates in intensely competitive industries that are characterized by a high percentage of costs that are fixed or difficult to
reduce in the short term and product demand that is highly variable and difficult to forecast. Revenue and the gross margin percentage are affected by
the timing of Intel product introductions and the demand for and market acceptance of Intel's products; actions taken by Intel's competitors, including
product offerings and introductions, marketing programs and pricing pressures and Intel‟s response to such actions; and Intel‟s ability to respond
quickly to technological developments and to incorporate new features into its products. The gross margin percentage could vary significantly from
expectations based on capacity utilization; variations in inventory valuation, including variations related to the timing of qualifying products for sale;
changes in revenue levels; segment product mix; the timing and execution of the manufacturing ramp and associated costs; start-up costs; excess or
obsolete inventory; changes in unit costs; defects or disruptions in the supply of materials or resources; product manufacturing quality/yields; and
impairments of long-lived assets, including manufacturing, assembly/test and intangible assets. Intel's results could be affected by adverse
economic, social, political and physical/infrastructure conditions in countries where Intel, its customers or its suppliers operate, including military
conflict and other security risks, natural disasters, infrastructure disruptions, health concerns and fluctuations in currency exchange rates.
Expenses, particularly certain marketing and compensation expenses, as well as restructuring and asset impairment charges, vary depending on the
level of demand for Intel's products and the level of revenue and profits. Intel‟s results could be affected by the timing of closing of acquisitions and
divestitures. Intel's results could be affected by adverse effects associated with product defects and errata (deviations from published
specifications), and by litigation or regulatory matters involving intellectual property, stockholder, consumer, antitrust, disclosure and other
issues, such as the litigation and regulatory matters described in Intel's SEC reports. An unfavorable ruling could include monetary damages or an
injunction prohibiting Intel from manufacturing or selling one or more products, precluding particular business practices, impacting Intel‟s ability to
Rev. its products, or requiring other remedies such as compulsory licensing of intellectual property. A detailed discussion of these and other factors
design 7/17/13
that could affect Intel‟s results is included in Intel‟s SEC filings, including the company‟s most recent reports on Form 10-Q, Form 10-K and earnings
release.
Editor's Notes
Background for "Walls" code. The discovery of the Higgs particle at the LHC confirms that the Universe has a complicated underlying structure with broken symmetries. A more familiar type of broken symmetry is seen in a ferromagnet with its magnetised domains; the boundaries between each of these misaligned regions has additional energy which are called domain walls. The same can happen in our Universe: Higgs-like fields in different regions become aligned in different directions with domain walls separating them (or indeed other defects, such as cosmic strings). Walls have potentially important implications in the very early universe, as well as the late universe today. The Planck satellite recently observed asymmetries (or anomalies) in the cosmic microwave sky; one possibility is that these are due to low-energy walls stretching across the Universe. The Walls code creates random initial conditions on a 3D grid for a network of domain walls in the early universe. It then solves dynamical field equations to find out how they evolve, counting up the wall area. Typically walls 'scale' as the Universe expands, with a fixed number of them stretching across the observable universe at any one time. The largest defect simulations to date have been performed on COSMOS using the Walls code, which has been developed into a number of variants. For example, complex hybrid networks with up to 100 different types of walls and strings have been investigated, as well as wall evolution in four and even five spatial dimensions inspired by fundamental theory; this is only possible because of the large available memory on COSMOS. The scalable OpenMP Walls code has also been used to benchmark shared-memory machines on several occasions.
Laplacian stencil is standard 3D stencil. In 4D you have two extra additions (for l-1, l+1) and the 6 becomes an 8. This is a good example to refer to when discussing the “3D specialization” optimization; the original code relied on the periodic boundary condition to simplify this down to 3D. (i.e. stencil = 6 useful directions + ijkl + ijkl – 8ijkl = 6 useful directions – 6ijkl)The single loop optimization depends on a rearrangement of Eq. 2: we can rewrite to find n-1/2 from n and n-1. Also useful when discussing division hoisting: (1+delta) is a constant that we divide by every iteration.Eq 3 is weird-looking, but what the code basically does is looks for a change of sign between two adjacent grid-points in all dimensions (e.g. ijkl = +1, ijkl-1 = -1) to detect walls, then computes their area.
We use KMP_AFFINITY=compact,granularity=fine. This ensures the 4 threads on the same KNC core work on the same area of the stencil, and encourages cache re-use.If somebody asks about the reason for 480^3, it’s to fairly divide the same way on both pieces of hardware (i.e. 480 / 16 and 480 / 240 are both “good” decompositions).Other problem sizes would be unfair to the coprocessor. Coprocessor can’t ever have an advantage, because 240 / 16 = 15.Early work with #pragma omp collapse suggests we can actually get the same performance benefits for any problem size, by increasing task granularity. A few threads having one or two extra grid-points each has much less impact on performance than a few threads having one or two extra tiles.
“Optimizations” are largely hardware- or compiler-driven; improving an algorithm’s implementation on IA.“Modernizations” are code changes that are algorithmic; not specific to IA.Halo regions help because then reading “one to the left” or “one to the right” is a contiguous chunk. The original code used modulo division to determine the correct index (and then pulled the value from the other side of the array).
This is not the only thing we did to improve auto-vectorization, but it’s a good example of where the tools helped.We had two % operations (for +1 and -1) in each of the four dimensions.Also had to change some variable types (code tried to add doubles to a long) and help the compiler realise the variable it thought had a dependency issue was actually being used for a reduction.
Again, not the only instance of this – just an example.Think we actually hoisted a few other divisions out.
Just an example (but an easy one to follow).There’s some similar stuff in the area calculation.
This shows the speed-up for each version of the code running on the processor and coprocessor, relative to the original code running on the processor.A few interesting points: 1) auto-vectorization alone was sufficient to make coprocessor beat processor; 2) performance difference between the two is pretty striking; 3) because of this investigation, a single coprocessor runs the code 8x faster than dual-socket processor ran the baseline code, and 1.38x faster than it runs the final code.
This shows the speed-up for each version of the code running on the processor and coprocessor, relative to the original code running on the processor.A few interesting points: 1) auto-vectorization alone was sufficient to make coprocessor beat processor; 2) performance difference between the two is pretty striking; 3) because of this investigation, a single coprocessor runs the code 8x faster than dual-socket processor ran the baseline code, and 1.38x faster than it runs the final code.
None of the optimizations employed to date are really very complicated; they’re all either pragmas or simple code transformations. The code is readable and looks much like it did before.Future optimization work is likely to be a little more destructive, but we’re not interested in going to intrinsics or inline assembly.Bulk of future work is actually in moving away from a single coprocessor in order to run much larger problems.