Chris Dagdigian discusses the advantages of using AWS for cloud and workflow orchestration in genomics, emphasizing the need for efficient automated provisioning and configuration management to speed up research processes. He highlights tools like Cloud Init and AWS CloudFormation that facilitate infrastructure management and orchestration, making IT infrastructures fully virtual and scriptable. Dagdigian also covers the role of Chef for configuration management and MIT StarCluster for creating Linux compute farms on AWS.

1. Practical Cloud & Workflow Orchestration
2011 Amazon Genomics Event
Chris Dagdigian
chris@bioteam.net

3. Twitter: @chris_dag
I’m Chris.
I’m an infrastructure
geek.
I work for the
BioTeam.

4. Disclaimer.

5. I’m not an Amazon shill.

6. Really.

7. The IaaS competition just can’t compete.

8. AWS lets me build useful stuff.

9. When stuff gets built, I get paid.

10. Installing VMware & excreting a
press release does not turn a
company into a cloud provider.

11. I need more than just virtual compute
and block storage. AWS has tons of glue
and many useful IaaS building blocks.

12. IaaS competitors lag far behind in
features and service offerings.

13. Speaking of pretenders…

14. No APIs?
Not a cloud.

15. No self-service?
Not a cloud.

16. I have to email a human?
Not a cloud.

17. 50% failure rate on server launch?
Lame cloud.

18. Virtual servers & block storage only?
Barely a cloud.

19. I’m getting insufferable, huh?
Moving on …

20. Three Topics Today.

21. Time, Laziness Beauty.

22. Tick …
Tick
Tick…
image: shanelin via flickr

23. User expectations are changing.
image: shanelin via flickr

24. Automated provisioning
can shrink the time between
“I want to do some science”
“I’m ready to do some science”.
image: shanelin via flickr

25. However…
image: shanelin via flickr

26. If servers, storage and systems can be
deployed in minutes …
image: shanelin via flickr

27. … why does it still take days, several
helpdesk tickets a team of humans
to load software and configure my
systems to actually do science?
image: shanelin via flickr

28. It shouldn’t.
image: shanelin via flickr

29. If provisioning gets faster,
configuration management
also needs to keep pace.
image: shanelin via flickr

30. Laziness.

31. Larry Wall’s 1st Great Virtue

32. “… the quality that makes you go to great
effort to reduce overall energy expenditure. It
makes you write labor-saving programs that
other people will find useful, and document
what you wrote so you don't have to answer
so many questions about it.”

33. It’s all scriptable.

34. • Servers
• Storage
• Network
• Bootstrapping
• Provisioning
• Configuration
• Management
• Monitoring
• Scaling
• Accounting
audit trails

35. Not hype. Real.

36. I can do it from my ipad.

37. No cubicle required.

38. Our research IT infrastructures can now be
100% virtual and 100% scriptable

39. And it’s pretty easy to understand.

40. Anyone can drive this stuff.

41. Especially motivated researchers.

42. Stuff like this is a big deal.

43. 5GB managed MySQL in the cloud.
$.011 / hour

44. Database Administrator not required.

45. Automatic patching, backups clustering

46. Anyone with a web browser can launch one.

47. Beauty.

48. Scriptable infrastructure is just the beginning.

49. The really cool stuff is what we build on top.

50. With good tools …

51. We can orchestrate complex systems,
pipelines and workflows.

52. Orchestrated systems working in concert
are a beautiful thing.

53. Let me show you a few of the tools we like.

54. Cloud Init

55. Cloud Init
• https://help.ubuntu.com/community/UEC
• Developed by Ubuntu
• Baked into all Ubuntu UEC releases
• Also baked into Amazon Linux AMIs
• Works on Eucalyptus clouds as well

56. Cloud Init gives you a hook into freshly
booted systems.

57. It’s a great and easy-to-comprehend way to
bootstrap or customize generic server images.

58. When you launch a server, you can inject a
YAML formatted file into the environment.

59. Cloud init files are parsed and executed right
after the node boots for the first time.

60. You can run scripts, install software, load SSH
keys, etc. to ‘bootstrap’ a generic node.

61. #cloud-config!
packages:!
- httpd!
!
runcmd:!
- /etc/init.d/httpd start !
- echo h1Hello Amazon Genomics Event!/h1”
! /var/www/html/index.html!
!

62. Previous real-world example does this:
1. Download/install Apache web server
2. Turn on the web server
3. Create a cheezy index.html

63. This is the script I ran moments before this talk …

64. #!/bin/sh!
!
ec2-run-instances ami-8c1fece5 !
-n 1 !
-t m1.small !
-g dagdemo-SG !
-k dagdemo-sshkeypair !
--user-data-file ./cloudInit-config.txt!
!

65. Important to understand:
• ami-8c1fece5 is Amazon Linux public AMI
• No web server pre-installed
• Never before been ‘touched’ by me
• Cloud Init does it all via the script I injected at
instance launch time

66. Lets see if it worked …

67. Amazon CloudFormation

68. Amazon CloudFormation
• http://aws.amazon.com/cloudformation/
• AWS specific
• Sweet way to turn on|off entire stacks of
related and dependent AWS services

69. Treat complex infrastructure as single resource
• Cliché example - In a single “stack” you can
define and then start/stop:
• Elastic database cluster +
• Elastic webserver cluster +
• Monitoring auto-scaling triggers
• Event error notification
• Elastic load balancer

70. My live demo of CloudFormation
• Using the example WordPress Blog template
• It does a ton of cool stuff:
• RDS backend for mySQL database, elastic
webserver cluster with auto-scaling, security
group setup, automatic scaling, automatic
alarm notices
• It all sits behind an elastic load balancer

71. My CloudFormation blog demo:
• Actual stack file at http://biote.am/6d
• Check it out …
• .JSON formatted but still quite readable
• It lets me define and then control a ton of
different related AWS services all at once.

72. #!/bin/sh!
# Launch Stack
!
cfn-create-stack AWSGenomics-demoStack !
--template-file cf-wordpress.json.txt!
!
!

73. #!/bin/sh!
# Check state status!
!
cfn-describe-stacks AWSGenomics-demoStack!
echo !
cfn-describe-stack-events !
AWSGenomics-demoStack --headers!

74. 10 AWS Services/Resources orchestrated as one.

76. Cloudwatch.

77. Auto-scaling triggers.

78. SNS Endpoints for Alarms.

79. Alarm triggers.

80. RDS Database Security Group.

81. Elastic Load Balancer.

82. EC2 Security Group.

83. Cool, huh?

84. { in case the demo fails! }

86. Opscode Chef

87. Chef enables Infrastructure as Code

88. It’s freaking awesome.

89. Chef lets you:
Manage configuration as idempotent Resources.
Group resources as idempotent Recipes.
Group recipes into Roles.
Track it all like Source Code.
Search your infrastructure like a ninja. Ohai!
Configure your systems, software pipelines

90. http://www.opscode.com/chef/
• Several flavors
• Open source
• Commercial / Managed
• Commercial / ‘Behind your Firewall’
• No time today for even a short description
of how it works. You should check it out.

91. Chef demo via ‘knife’ command line …

92. knife ec2 server create !
-N aws-genomicsDemo !
-I ami-63be790a !
-f t1.micro !
-G default !
-S bioteam-IAM-admins-v1 !
-r 'recipe[getting-started]' !
-i ./bioteam-IAM-admins-v1.pem !
-x ubuntu!

93. Fully automatic remote bootstrapping …

94. Done!

96. Search-driven, parallel remote SSH execution

97. knife ssh name:aws-genomicsDemo !
-a cloud.public_hostname !
-x ubuntu !
-i bioteam-IAM-admins-v1.pem !
'sudo chef-client; !
cat /tmp/chef-getting-started.txt'!

98. Lets install some genomics tools
• Our Maq short read assembler cookbook:
• Installs all dependencies (compilers, etc.)
• Puts application source on node
• Builds maq from source
• Installs it

99. $ knife node !
run_list add !
aws-genomicsDemo !
'recipe[maq]'!

100. It really is that easy.

101. MIT StarCluster

102. MIT Starcluster
• http://web.mit.edu/stardev/cluster
• Ready to use Linux compute farm on AWS
• Grid Engine, MPI, NFS filesystems
• Libraries, tools, applications
• Easy to use, easy to extend
• Integrates well with Chef

103. If you have not built Linux clusters from
scratch before …

104. It’s hard to really appreciate everything that
StarCluster does behind the scenes.

105. MIT Starcluster – More Info
• Live demo (time permitting)
• StarCluster Spot Instances Screencast
• http://biote.am/6c
• http://aws.amazon.com/ec2/spot-and-
science/

106. Phew. That’s a lot of slides.

107. Time to explore the demos?

108. Questions?

109. Thanks!
Related talk slides:
http://biote.am/6a
“Mapping Informatics to the Cloud”