My keynote from Bio-IT World 2010

1. There
is
no
magic,
there
is
only
awesome
Scien&fic
compu&ng
with
Amazon
Web
Services
Deepak
Singh
Bio-­‐IT
World
2010

3. By ~Prescott under a CC-BY-NC license

4. <1>

5. data

6. Image: Wikipedia

8. Source: http://www.nature.com/news/specials/bigdata/index.html

12. implications

13. data management
data processing
data sharing

14. <2>

15. !"#$%&'()*+',-./0)
1%$.#'$$)

16. !"#$%&'()*+',-./0)
1%$.#'$$)
!"##"$%
&'()*"((%

17. massive scale
highly available
efficient
service oriented
secure
Amazon Infrastructure

19. infrastructure as a service

20. undifferentiated heavy lifting

22. *+,-./-01$23,4567-$89$
:6+;3,$<78$=7>?/@74$
*+,-./-01$23,4567-$89$
:6+;3,A4$BC38+C$<784/074$
!""#$ !""!$ !""%$ !""&$ !""'$ !""($ !"")$ !""D$

23. Amazon S3 Momentum
102 Billion
Peak Requests:
40 Billion
14 Billion
2.9 Billion
!"#$%%&# !"#$%%'# !"#$%%(# !"#$%%)#
Total Number of Objects Stored in Amazon S3

25. Compute Storage
Amazon Elastic Compute Database
Amazon Simple Amazon RDS and
Cloud (EC2) Storage Service (S3)
- Elastic Load Balancing SimpleDB
- AWS Import/Export
- Auto Scaling

26. Messaging
Amazon Simple Payments On-Demand
Parallel Processing
Content Delivery Queue Service (SQS) Amazon Flexible Workforce
Amazon Elastic
Amazon CloudFront Amazon Simple Payments Service Amazon Mechanical
MapReduce
Notification Service (FPS) Turk
(SNS)
Compute Storage
Amazon Elastic Compute Database
Amazon Simple Amazon RDS and
Cloud (EC2) Storage Service (S3)
- Elastic Load Balancing SimpleDB
- AWS Import/Export
- Auto Scaling

27. Tools Isolated Networks
Monitoring Management
AWS Toolkit for Eclipse Amazon Virtual Private
Amazon CloudWatch AWS Management Console
AWS SDK for .NET Cloud
Messaging
Amazon Simple Payments On-Demand
Parallel Processing
Content Delivery Queue Service (SQS) Amazon Flexible Workforce
Amazon Elastic
Amazon CloudFront Amazon Simple Payments Service Amazon Mechanical
MapReduce
Notification Service (FPS) Turk
(SNS)
Compute Storage
Amazon Elastic Compute Database
Amazon Simple Amazon RDS and
Cloud (EC2) Storage Service (S3)
- Elastic Load Balancing SimpleDB
- AWS Import/Export
- Auto Scaling

28. Your Custom Applications and Services
Tools Isolated Networks
Monitoring Management
AWS Toolkit for Eclipse Amazon Virtual Private
Amazon CloudWatch AWS Management Console
AWS SDK for .NET Cloud
Messaging
Amazon Simple Payments On-Demand
Parallel Processing
Content Delivery Queue Service (SQS) Amazon Flexible Workforce
Amazon Elastic
Amazon CloudFront Amazon Simple Payments Service Amazon Mechanical
MapReduce
Notification Service (FPS) Turk
(SNS)
Compute Storage
Amazon Elastic Compute Database
Amazon Simple Amazon RDS and
Cloud (EC2) Storage Service (S3)
- Elastic Load Balancing SimpleDB
- AWS Import/Export
- Auto Scaling

33. scalability

34. > 1PB of data in S3

35. elasticity

36. 3000 CPU’s for one firm’s risk management application
3444JJ'
!"#$%&'()'*+,'-./01.2%/'
344'+567/'(.'
8%%9%.:/'
344'JJ'
I%:.%/:1=' ;<"&/:1=' A&B:1=' C10"&:1=' C".:1=' E(.:1=' ;"%/:1='
>?,,?,44@' >?,3?,44@' >?,>?,44@' >?,H?,44@' >?,D?,44@' >?,F?,44@' >?,G?,44@'

38. highly available

40. “Everything fails, all the time”
-- Werner Vogels

41. 2-4% of servers & 1-5% of
disk drives will die annually
Source: Jeff Dean, LADIS 2009

42. human errors

43. human errors
~20% admin issues have unintended consequences
Source: James Hamilton

44. scalable & available

45. assume sw/hw failure

46. design apps to be resilient

47. automation & alarming

48. Image: Chris Dagdigian

49. US East Region !"#$%&'()*+
T T
Availability Availability
Zone A Zone B
Availability Availability T
Zone C Zone D

51. elastic load balancing
CloudWatch
auto scaling
SNS
SQS
elastic IP
elastic block store

52. "#$%&!'()*+#,$! !"#$%"&&'%()( !"#$%"&&'%(*( !"#$%"&&'%(!(
+ ( +! +!
-((./!'()*+#,$!
).#,$!0)/)/.! !"#$%"&&'%()( !"#$%"&&'%(*( !"#$%"&&'%(!(

53. cost effective
pay as you go
economies of scale
choices in pricing

54. on-demand instances
reserved instances
spot instances

55. on-demand instances
!"#$"%$#&'$(&$
%)"*+,($")$-./.0123&'*$
45%)$56&*)$7"!"78)#$

56. reserved instances
!"#$%&$'(")*$'++$
!"#+($"&+(,-).$/"0*$
.%,(,)*++1$/,&,/2*3$

57. spot instances
!"#$%&$'&'()#$*+,+*"-.$
,/"*)$!+()#$%&$(',,0.$+&#$#)1+&#$
.%'$,+.$1+/2)-$,/"*)$
"&(-+&*)($*+&$#"(+,,)+/$

58. http://cloudexchange.com

60. physical is free
network is easy
rest can be added

61. Customer 1 Customer 2 !" Customer n
Hypervisor
Virtual Interfaces
Customer 1
Security Groups
Customer 2
Security Groups ! Customer n
Security Groups
Firewall
Physical Interfaces

62. A MAZON
V PC
A RCHITECTURE
Customer’s isolated
AWS resources
10
.32
. 2. 0
/24 Subnets
10 10
.32 .32
.1.0 . 3. 0
/24 /24
VPN Gateway
Amazon
Web Services
Cloud
Secure VPN Connection
over the Internet
External
Your Network Customers

63. http://aws.amazon.com/security

64. <3>

65. AWS + science

66. data management

67. Biomarker Warehouse
pre-clinical, clinical, 3rd party data and publications
;<./5'=>?6@' !)*(%"&&'
23,3415'61789:1' !#%&$(%&&&'
+,'-./01' !"#$%"&&'
6178170' 6A.7341' B817-135'
Estimated cost: 10 TB warehouse over 3 years

68. data processing

71. http://cyclecomputing.com

72. sudo gem install cloud-crowd
http://cyclecomputing.com
http://wiki.github.com/documentcloud/cloud-crowd

73. http://www.rightscale.com

74. Amazon Elastic MapReduce
Amazon EC2 Instances
End
Deploy Application
Hadoop Hadoop Hadoop
Elastic Elastic
MapReduce MapReduce
Hadoop Hadoop Hadoop Notify
Web Console, Command
line tools Input output
dataset results
Input
S3
Output
S3
Get Results
Input Data
bucket bucket
Amazon S3

76. Protein interactions @ U. Washington
EC2!
Simple Python scripts automate the
management of 1000s of simultaneous
experiments using the EC2 API
http://faculty.washington.edu/danielt/
Source: Ed Lazowska

77. BLAT @ U. Penn
Map 100 million, 100 base paired end reads
Quad core with 5 GB of RAM would take 16 days
:..2#>*120-#D#1+*@01#
$%# C*120-#
!67>(2#2A0#:;<=#9.7#
E,+.*?#(',621# !+*@0# &'()*+#,-./011#
1,+(21#3*12*#4+05#
F.G'+.*?#-016+21# !6710860'2#9.71#
!+*@0# :;<=#1>*++0-#4+01#
!"# *'?#1*@0#0*/A#
-016+2#*1#(2#B.01#
!+*@0#
!+*@0#
30 high-memory instances; 32 hours; $195
Source: Angel Pizzaro/John Hogenesch

78. 200 instances
60000 structures
4 hours
http://bioteam.net/aws

79. Crossbow: Rapid whole genome SNP analysis
Preprocessed reads
Map: Bowtie
Sort: Bin and partition
Reduce: SoapSNP
Langmead B, Schatz MC, Lin, J, Pop M, Salzberg SL. Genome Biol 10(11): R134.

80. data storage & distribution
public & private

81. http://aws.amazon.com/publicdatasets/

83. sharing and collaboration

84. http://www.elasticr.net
Elastic-R Collaborative Research Environment

85. software/pipeline distribution

86. http://www.cloudbiolinux.com/

87. http://usegalaxy.org/cloud

88. http://clovr.igs.umaryland.edu/

89. platforms & services

90. http://heroku.com

91. http://chempedia.com/

92. http://dnanexus.com

93. NexusDB
NexusDB
NexusDB Storage
Server (S3)
SSL
NexusDB https
Client
http://www.biodiscovery.com/index/biod-nexusdb-action

94. Cloud-based biosemantic apps
http://syapse.com/

95. <4>

96. 2006 comparison. Large Service vs. Mid Size
Source: James Hamilton

97. datacenter design efficiency
PUE < 1.2 - 1.5*
Source: James Hamilton * average > 2.0 (Source: EPA)

98. multiple datacenters
Source: James Hamilton

99. h/w costs
efficiency optimization
Source: James Hamilton

100. investments in automation
investments in s/w
investments in special skills

101. server utilization
infrastructure as opex
pay as you go/grow

102. • Boot from EBS
• AWS Multi Factor Authentication • US West Region
• Virtual Private Cloud private beta • VPC Unlimited Beta
• Lower Reserved Instance Pricing • ELB Support in Console
• Reserved Instances in EU
• Console Support for CloudWatch • CloudFront streaming
• Elastic MapReduce
• SQS in EU • EC2 Spot Instances
• Windows 2008 Support
• RDS Launched • Lowered Prices
• New SimpleDB Features • AWS Security Center • High Memory Instances • AWS Economics Center
• Console support for Cloudfront • Reduced EC2 Pricing
• FPS General Availability
• EMR Apache Hive support
• EC2 Reserved Instances
• Elastic MapReduce in EU • SAS 70 Type II Audit
• EC2 with Windows
• AWS SDK for .NET
• EC2 in EU
• CloudFront Private Content
• AWS Toolkit for Eclipse • EBS Shared Snapshots • APAC announced
• SimpleDB in EU
• Monitoring in EU
• AWS Import/Export • Auto Scaling in EU
• Lower pricing tiers for Cloudfront • Elastic Load Balancing in EU
• Monitoring, Auto Scaling,
• AWS Management Console • AWS Solutions Provider program
and Elastic Load Balancing
• CloudFront adds access logging

103. there is no magic
there is only awesome

104. Thank
you!
deesingh@amazon.com
Twi;er:@mndoci
Presenta?on
ideas
from
James
Hamilton,
@mza
and
@lessig