In this deck from the HPC User Forum, Zachary Cobell from ARCADIS-US presents: High Performance Computing for the Louisiana Coastal Master Plan. Watch the video presentation: http://insidehpc.com/2015/09/video-hpc-for-the-louisiana-coastal-master-plan/ Learn more: http://hpcuserforum.com Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

1. Imagine the result
High Performance Computing for the
2017 Louisiana Coastal Master Plan
Z. Cobell

2. 9 September 20152
Acknowledgements
*This project was funded by The Water Institute of the Gulf under project award number
CPRA-2012-TO3-EM. The views expressed in this publication are those of the author
and do not necessarily represent the views of The Water Institute of the Gulf

3. 9 September 20153
Acknowledgements

4. 9 September 20154
Motivation • Coastal Louisiana is in a land loss crisis.
• 4800 sq km lost since 1930*
• 4500 sq km potential loss by 2060*
• More loss, more damage
• $23 billion additional damage from storm surge due to
loss by 2060*
*Peyronnin et al, 2013, Louisiana’s 2012 Coastal Master Plan: Overview of a Science-
Based and Publicly Informed Decision-Making Process

5. 9 September 20155
Agenda • Numerical Models Employed
• 2007 LACPR
• 2012 Master Plan
• 2017 Master Plan
• Conclusions

6. 6 9 September 2015
Numerical Models

7. 9 September 20157
Numerical
Models
• ADCIRC
• Circulation model
• Solves shallow water equations
• Unstructured finite element mesh
• Serial or MPI
• SWAN
• Spectral wave model
• Solves action density equation for non-stationary waves
• Unstructured, curvilinear, or structured mesh
• Serial, OpenMP, or MPI

8. 9 September 20158
METIS Decomposition

9. 9 September 20159
Model Communication

10. 9 September 201510
Coupled Simulation Schematic
Fx Fy
ζ u v
wx wy
ADCIRC
UnSWAN
0 600 1200
TIME
Note: This is a single model run

11. 9 September 201511
Model Efficiency

12. 9 September 201512
Model Efficiency

13. 9 September 201513
Model Efficiency

14. 9 September 201514
Model Efficiency

15. 9 September 201515
Model Domain

16. 9 September 201516
Model Domain

17. 9 September 201517
Model Domain

18. 9 September 201518
Model Validation

19. 9 September 201519
Model Validation

20. 20 9 September 2015
LACPR 2007

21. 9 September 201521
2007 LACPR • 2007 FEMA study in Louisiana created suite of 446 synthetic
hurricanes
• These 446 storms used to define the statistical returns for
water surface elevation

22. 9 September 201522
2007 LACPR • One of the first studies of its kind
• Developed methods still used today
• Computers simply unable to analyze large numbers of
projects
• Large storm set
• Computational time required per simulation
• Conducted at USACE supercomputing facilities

23. 23 9 September 2015
2012 Master Plan
Framework

24. 9 September 201524
2012 Master
Plan Framework
• Framework to analyze proposed restoration projects
• How do these projects effect the landscape?
• How do these projects effect each other?
• How do these projects reduce damage during storms?

25. 9 September 201525
2012 Master
Plan Framework
• Modeling system developed to aid in project selection
process
• Includes models for:
• Vegetation
• Water Quality
• Ecology/Hydrology
• Wetland Morphology
• Barrier Island Morphology
• Fish/Wildlife
• Storm Surge/Waves
• Damage/Risk Analysis

26. 9 September 201526
Model Interaction

27. 9 September 201527
Model Interaction

28. 9 September 201528
2012 Master
Plan
• Premise of analyzing large number of projects and large
number of environmental scenarios
• Storm suite reduced while maintaining acceptable levels of
accuracy

29. 9 September 201529
2012 Master
Plan
• 2012
• Project level screening analysis – 40 storm suite
• 34 hurricane protection projects analyzed
• Projects grouped into 7 non-interacting groups to
isolate impacts

30. 9 September 201530
Project Groups

31. 9 September 201531
Simulation Results

32. 9 September 201532
2012 Master
Plan
• 2012
• Project level screening analysis – 40 storm suite
• 34 hurricane protection projects analyzed
• Projects grouped into non-interacting groups to isolate
impacts
• 7 groupings and base condition
• Portfolio analysis – 40 storm suite
• Future conditions with different suites of projects
implemented

33. 9 September 201533
Model Interaction

34. 9 September 201534
2012 Master
Plan
• 2012
• Project level screening analysis – 40 storm suite
• 34 hurricane protection projects analyzed
• Projects grouped into non-interacting groups to isolate
impacts
• 7 groupings and base condition
• Portfolio analysis – 40 storm suite
• Future conditions with different suites of projects
implemented

35. 9 September 201535
Simulation Results

36. 9 September 201536
2012 Master
Plan
• 2012
• Total simulations conducted:
• Project Level
40 storms
8 groupings (7+1 base condition)
3 scenarios
840 simulations
• Alternative Level
40 storms
3 groupings
3 scenarios
2* time levels (0*, 25, 50 years)
800 simulations
1640 simulations
~6MM CPU Hours

37. 37 9 September 2015
2017 Master Plan

38. 9 September 201538
2017 Simulation
Suite
• 2017 – In progress
• Number of storms increased
• 60 storm project screening level suite
– 3 time levels
– 3 scenarios
• 154 storm alternatives analysis suite
– 3 time levels
– 3 scenarios
• Total planned simulations: 7712
• Total planned CPU hours: ~10MM

39. 39 9 September 2015
Computational
Resources

40. 9 September 201540
Computational
Resources
• 2007 LACPR Resource – USACE “Sapphire”
• Cray XT3
• Dual AMD Opteron Processors @ 2.6GHz
• 8192 total cores
• Shared government resource
• 2012 Master Plan Resource – “Athos”
• Online 2010
• Dual 6 core Nehalem Processors @ 2.67GHz
• Qlogic QDR Infiniband, 2:1 Oversubscription
• 996 total cores
• Dedicated resource
• 2017 Master Plan Resource – “Aegaeon”
• Online 2015
• Dual 12 core Sandy Bridge Processors @ 2.5GHz
• Mellanox FDR Infiniband, 2:1 Oversubscription
• 1488 total cores
• Dedicated resource

41. 9 September 201541
Computational
Resources • Aegaeon (2015) provides nearly a 2X speedup versus the
machine developed in 2010
5000
50000
500000
10 100 1000 10000
Scaling Analaysis
Aegaeon (sec) Athos (sec)

42. 9 September 201542
Conclusions

43. 9 September 201543
Conclusions • 2007 – 446 simulations
• 2012 – 1640 simulations (x3.6) – 4MM CPU
• 2017 – 7712 simulations (x4.7) – 8MM CPU
• The number of simulations that can be reasonably completed
has grown drastically
• CPUs, Interconnects, Storage, Availability
• The “value” of a CPU hour has increased
• These simulations heavily influence policy decisions
• The computational power we can harness, the more detail we
can provide to policy makers

44. 9 September 201544
Conclusions • Computer speed has increased to allow high performance
models to derive forecasts

45. Imagine the result