A presentation given at the 2011 Amazon AWS Genomics meeting held in Seattle, WA.
This is a 30 minute talk I gave focusing mainly on practical tools, tips and methods for bootstrapping and orchestration on the cloud.
Covers examples of:
Ubuntu Cloud Init
AWS Cloud Formation
Opscode Chef
MIT StarCluster
26. If servers, storage and systems can be
deployed in minutes …
image: shanelin via flickr
27. … why does it still take days, several
helpdesk tickets a team of humans
to load software and configure my
systems to actually do science?
image: shanelin via flickr
32. “… the quality that makes you go to great
effort to reduce overall energy expenditure. It
makes you write labor-saving programs that
other people will find useful, and document
what you wrote so you don't have to answer
so many questions about it.”
65. Important to understand:
• ami-8c1fece5 is Amazon Linux public AMI
• No web server pre-installed
• Never before been ‘touched’ by me
• Cloud Init does it all via the script I injected at
instance launch time
69. Treat complex infrastructure as single resource
• Cliché example - In a single “stack” you can
define and then start/stop:
• Elastic database cluster +
• Elastic webserver cluster +
• Monitoring auto-scaling triggers
• Event error notification
• Elastic load balancer
70. My live demo of CloudFormation
• Using the example WordPress Blog template
• It does a ton of cool stuff:
• RDS backend for mySQL database, elastic
webserver cluster with auto-scaling, security
group setup, automatic scaling, automatic
alarm notices
• It all sits behind an elastic load balancer
71. My CloudFormation blog demo:
• Actual stack file at http://biote.am/6d
• Check it out …
• .JSON formatted but still quite readable
• It lets me define and then control a ton of
different related AWS services all at once.
89. Chef lets you:
Manage configuration as idempotent Resources.
Group resources as idempotent Recipes.
Group recipes into Roles.
Track it all like Source Code.
Search your infrastructure like a ninja. Ohai!
Configure your systems, software pipelines
90. http://www.opscode.com/chef/
• Several flavors
• Open source
• Commercial / Managed
• Commercial / ‘Behind your Firewall’
• No time today for even a short description
of how it works. You should check it out.
102. MIT Starcluster
• http://web.mit.edu/stardev/cluster
• Ready to use Linux compute farm on AWS
• Grid Engine, MPI, NFS filesystems
• Libraries, tools, applications
• Easy to use, easy to extend
• Integrates well with Chef
103. If you have not built Linux clusters from
scratch before …
104. It’s hard to really appreciate everything that
StarCluster does behind the scenes.
105. MIT Starcluster – More Info
• Live demo (time permitting)
• StarCluster Spot Instances Screencast
• http://biote.am/6c
• http://aws.amazon.com/ec2/spot-and-
science/