Scien&fic	
  Compu&ng	
  with	
  Amazon	
  Web	
  Services
Deepak	
  Singh




NHGRI	
  Cloud	
  Compu&ng	
  Mee&ng,	
  Bal...
AWS + science = win
scale has implications
data management
data processing
data sharing
Image: Chris Dagdigian
amazon web services
the cloud
has_many :definitions
infrastructure as a service
Your Custom Applications and Services

                                                                          Tools    ...
• Boot from EBS
                                                                • AWS Multi Factor Authentication         ...
elasticity
3000 CPU’s for one firm’s risk management application
     3444JJ'
!"#$%&'()'*+,'-./01.2%/'




                          ...
scale
> 1PB of data in S3
highly availability
Image: Chris Dagdigian
“Everything fails, all the time”
                   -- Werner Vogels
“Things will crash. Deal with it”
                        -- Jeff Dean
2-4% of servers
                                will die annually



Source: Jeff Dean, LADIS 2009
1-5% of disk drives
                                 will die every year



Source: Jeff Dean, LADIS 2009
human errors
human errors
             ~20% admin issues have unintended consequences




Source: James Hamilton
scalable & available
assume sw/hw failure
design apps to be resilient
automation & alarming
US East Region               !"#$%&'()*+


                                T                 T
Availability     Availabili...
elastic load balancing


                           CloudWatch
auto scaling

                              SQS
  elastic I...
flexibility
on-demand instances
 reserved instances
   spot instances
some implications
computing platforms
sudo gem install cloud-crowd

     http://cyclecomputing.com
http://wiki.github.com/documentcloud/cloud-crowd
http://www.rightscale.com
Amazon Elastic
                                    MapReduce

                                     Amazon EC2 Instances
  ...
application platforms
http://heroku.com
http://chempedia.com/
Image: O’Reilly Radar
software distribution
http://www.cloudbiolinux.com/
http://bitbucket.org/galaxy/galaxy-central/wiki/Home
data distribution
http://aws.amazon.com/publicdatasets/
to conclude
built for scale
built for availability
shared dataspaces
common namespaces
task-based resources
new software architectures
new computing platforms
Data Platform




App Platform
available today
http://aws.amazon.com/education
Thank	
  you!




deesingh@amazon.com	
  Twi2er:@mndoci	
  
     Presenta7on	
  ideas	
  from	
  James	
  Hamilton,	
  @mz...
NHGRI Cloud Computing talk
NHGRI Cloud Computing talk
NHGRI Cloud Computing talk
NHGRI Cloud Computing talk
NHGRI Cloud Computing talk
NHGRI Cloud Computing talk
NHGRI Cloud Computing talk
NHGRI Cloud Computing talk
NHGRI Cloud Computing talk
NHGRI Cloud Computing talk
NHGRI Cloud Computing talk
NHGRI Cloud Computing talk
NHGRI Cloud Computing talk
NHGRI Cloud Computing talk
NHGRI Cloud Computing talk
NHGRI Cloud Computing talk
NHGRI Cloud Computing talk
NHGRI Cloud Computing talk
Upcoming SlideShare
Loading in …5
×

NHGRI Cloud Computing talk

3,341
-1

Published on

My talk from the 2010 NHGRI Cloud Computing meeting

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,341
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
64
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

NHGRI Cloud Computing talk

  1. 1. Scien&fic  Compu&ng  with  Amazon  Web  Services Deepak  Singh NHGRI  Cloud  Compu&ng  Mee&ng,  Bal&more,  2010
  2. 2. AWS + science = win
  3. 3. scale has implications
  4. 4. data management
  5. 5. data processing
  6. 6. data sharing
  7. 7. Image: Chris Dagdigian
  8. 8. amazon web services
  9. 9. the cloud
  10. 10. has_many :definitions
  11. 11. infrastructure as a service
  12. 12. Your Custom Applications and Services Tools Isolated Networks Monitoring Management AWS Toolkit for Eclipse Amazon Virtual Private Amazon CloudWatch AWS Management Console AWS Toolkit for .NET Cloud Payments On-Demand Parallel Processing Messaging Content Delivery Amazon Flexible Workforce Amazon Elastic Amazon Simple Amazon CloudFront Payments Service Amazon Mechanical MapReduce Queue Service (SQS) (FPS) Turk Compute Storage Amazon Elastic Compute Database Amazon Simple Amazon RDS and Cloud (EC2) Storage Service (S3) - Elastic Load Balancing SimpleDB - AWS Import/Export - Auto Scaling
  13. 13. • Boot from EBS • AWS Multi Factor Authentication • US West Region • Virtual Private Cloud private beta • VPC Unlimited Beta • Lower Reserved Instance Pricing • ELB Support in Console • Reserved Instances in EU • Console Support for CloudWatch • CloudFront streaming • Elastic MapReduce • SQS in EU • EC2 Spot Instances • Windows 2008 Support • RDS Launched • Lowered Prices • New SimpleDB Features •  AWS Security Center • High Memory Instances • AWS Economics Center • FPS General Availability • Console support for Cloudfront • Reduced EC2 Pricing • EMR Apache Hive support • EC2 Reserved Instances • Elastic MapReduce in EU • SAS 70 Type II Audit • EC2 with Windows • AWS SDK for .NET • EC2 in EU • CloudFront Private Content • AWS Toolkit for Eclipse • EBS Shared Snapshots • APAC announced • SimpleDB in EU • Monitoring in EU • AWS Import/Export • Auto Scaling in EU • Lower pricing tiers for Cloudfront • Elastic Load Balancing in EU • AWS Management Console • Monitoring, Auto Scaling, and Elastic Load Balancing • AWS Solutions Provider program • CloudFront adds access logging
  14. 14. elasticity
  15. 15. 3000 CPU’s for one firm’s risk management application 3444JJ' !"#$%&'()'*+,'-./01.2%/' 344'+567/'(.' 8%%9%.:/' 344'JJ' I%:.%/:1=' ;<"&/:1=' A&B:1=' C10"&:1=' C".:1=' E(.:1=' ;"%/:1=' >?,,?,44@' >?,3?,44@' >?,>?,44@' >?,H?,44@' >?,D?,44@' >?,F?,44@' >?,G?,44@'
  16. 16. scale
  17. 17. > 1PB of data in S3
  18. 18. highly availability
  19. 19. Image: Chris Dagdigian
  20. 20. “Everything fails, all the time” -- Werner Vogels
  21. 21. “Things will crash. Deal with it” -- Jeff Dean
  22. 22. 2-4% of servers will die annually Source: Jeff Dean, LADIS 2009
  23. 23. 1-5% of disk drives will die every year Source: Jeff Dean, LADIS 2009
  24. 24. human errors
  25. 25. human errors ~20% admin issues have unintended consequences Source: James Hamilton
  26. 26. scalable & available
  27. 27. assume sw/hw failure
  28. 28. design apps to be resilient
  29. 29. automation & alarming
  30. 30. US East Region !"#$%&'()*+ T T Availability Availability Zone A Zone B Availability Availability T Zone C Zone D
  31. 31. elastic load balancing CloudWatch auto scaling SQS elastic IP elastic block store
  32. 32. flexibility
  33. 33. on-demand instances reserved instances spot instances
  34. 34. some implications
  35. 35. computing platforms
  36. 36. sudo gem install cloud-crowd http://cyclecomputing.com http://wiki.github.com/documentcloud/cloud-crowd
  37. 37. http://www.rightscale.com
  38. 38. Amazon Elastic MapReduce Amazon EC2 Instances End Deploy Application Hadoop Hadoop Hadoop Elastic Elastic MapReduce MapReduce Hadoop Hadoop Hadoop Notify Web Console, Command line tools Input output dataset results Input  S3   Output  S3   Get Results Input Data bucket bucket Amazon S3
  39. 39. application platforms
  40. 40. http://heroku.com
  41. 41. http://chempedia.com/
  42. 42. Image: O’Reilly Radar
  43. 43. software distribution
  44. 44. http://www.cloudbiolinux.com/
  45. 45. http://bitbucket.org/galaxy/galaxy-central/wiki/Home
  46. 46. data distribution
  47. 47. http://aws.amazon.com/publicdatasets/
  48. 48. to conclude
  49. 49. built for scale
  50. 50. built for availability
  51. 51. shared dataspaces common namespaces
  52. 52. task-based resources
  53. 53. new software architectures
  54. 54. new computing platforms
  55. 55. Data Platform App Platform
  56. 56. available today
  57. 57. http://aws.amazon.com/education
  58. 58. Thank  you! deesingh@amazon.com  Twi2er:@mndoci   Presenta7on  ideas  from  James  Hamilton,  @mza,  and  @lessig
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×