This document discusses high performance cloud computing on Amazon Web Services. It describes how AWS EC2 allows on-demand access to compute capacity in the cloud to elastically scale up and quickly provision resources. It also covers cluster computing using EC2, with examples of using EC2 for computational fluid dynamics, molecular dynamics, and machine learning workloads. Customer case studies demonstrate how biotech and energy companies use AWS for high performance computing.

1. High Performance
Cloud Computing
Deepak Singh
P r i n c i p a l P r o d u c t M a n a g e r

2. Via butteryflysha under a CC-BY license

3. Image: Simon Cockell under CC-BY

4. High Scale Computing

5. using a large number of
computers at the same
time to solve a problem

6. 2

7. 1
High Throughput
Computing

8. scale out

11. “embarassingly
parallel”

12. constraints

13. constrained by
capacity

14. More molecules
Bigger systems
constrained by
capacity
More simulations
More dimensions

15. constrained by time

16. Upcoming conference
Grant submissions
constrained by time
Impatience!
Exploratory “spike” run

17. EC2

18. EC2
Elastic Compute Cloud

19. elastic

20. programmatic

22. ec2-run-instances

24. AWS CloudFormation

25. EC2 instance types

26. s
pe
ty
ce
an
standard “m1”
st
in
2
EC
high cpu “c1”
high memory “m2”
http://aws.amazon.com/ec2/instance-types/

27. s
pe
ty
ce
an
standard “m1”
st
in
2
EC
high cpu “c1”
high memory “m2”
http://aws.amazon.com/ec2/instance-types/

29. ec2-terminate-instances

31. rapid provisioning

32. 10K in 45 minutes

33. design patterns

34. optimize for
throughput

35. Tasks
Instances

36. Tasks
Queue
Instances

37. Tasks
Queue
Instances

38. vertical scaling

39. Tasks
Queue
Instances
Increase
instance
size

40. Tasks
Queue
Instances
Increase
instance
size

41. horizontal scaling

42. Tasks
Queue
Instances
Increase
instance
count

43. Tasks
Queue
Instances
Results
Store

44. Tasks
Queue
On-premise
Instances
Results
Store

45. Tasks
Queue
On-premise
Instances
Results
Store

46. Tasks
Queue
On-premise
Instances
Results
Store

47. optimize for cost

48. optimize for cost
maximize bang for buck

49. on-demand instances

50. reserved instances

51. spot instances

53. ideal for batch

54. persistent requests

55. all or nothing

56. use cases galore

58. Credit: Angel Pizzaro, U. Penn

60. 2
Cluster
Computing

61. tightly coupled

62. MPI

63. Dua l Intel 23GB RA
X 5570 GPGPU
M
“Neha lem”
HVM
1.7TB
scratch
Cluster Compute

64. 10 gig E
Cluster Compute

65. Placement
Group

66. Placement
group
Cluster Compute

67. 231

68. 450

69. Cores 7040
R max
41.82
R peak
82.51

70. GPGPU

71. 2 x Tesla
M2050

72. Getting Started

73. http://aws.amazon.com/hpc

75. 4 steps

76. 15 minutes

77. http://aws.amazon.com/ec2

83. performance

85. WIEN2K Parallel
Performance
H size 56,000 (25GB)
Runtime (16x8 processors)
Local (Infiniband) 3h:48
Cloud (10Gbps) 1h:30 ($40)
1200 atom unit cell; SCALAPACK+MPI
diagonalization, matrix size 50k-100k
Credit: K. Jorissen, F. D. Villa, and J. J. Rehr (U. Washington)

86. customer examples

87. Example Use Case #1
Computational Fluid Dynamics
Dynamic Clusters
40-180 CC1 instances

88. Example Use Case #2
Molecular Dynamics
Steady Usage
32-40 CG1 instances

89. Example Use Case #3
Machine Learning
Spiky, Experimental
8-20 CG1 instances

90. Customer Case Study: Bioproximity
http://aws.amazon.com/solutions/case-studies/bioproximity/

91. Customer Case Study: cyclopic energy
OpenFOAM®
http://aws.amazon.com/solutions/case-studies/cyclopic-energy/

92. Customer Case Study: PSR
Stochastic Dual Dynamic Programming
44,000 CPU hrs in Oct 2010
http://aws.amazon.com/solutions/case-studies/psr/

93. familiar tools

94. Oracle Grid Engine

95. MIT StarCluster

96. LSF

97. Moab/Torque

98. Condor

99. StackIQ Rocks+

101. Slurm

102. deesingh@amazon.com
Twitter:@mndoci
http://slideshare.net/mndoci
http://mndoci.com
Inspiration and ideas from
Matt Wood, James Hamilton
& Larry Lessig
Credit” Oberazzi under a CC-BY-NC-SA license