1. DevOps at Wayin
Orchestration with AWS Cloud Formation
David M. Johnson
Software Developer - Wayin, Inc.
Friday, June 21, 13

2. Agenda
•What is Cloud Orchestration
• Quick intro to Wayin Hub
• Wayin Hub deployment & application architecture
• How we scale, deploy and do backups
• About Cloud Formation Templates
• About vendor lock-in
• Designing for orchestration
Friday, June 21, 13

3. “Orchestration describes the
automated arrangement,
coordination, and management
of complex computer systems,
middleware, and services.
It is often discussed as having
an inherent intelligence or even
implicitly autonomic control, but
those are largely aspirations or
analogies rather than technical
descriptions.”
http://en.wikipedia.org/wiki/Orchestration_(computing)
Friday, June 21, 13

4. “cloud orchestration =
automation
+ integration
+ best practices”
https://www.ibm.com/developerworks/mydeveloperworks/blogs/9e696bfa-94af-4f5a-ab50-
c955cca76fd0/entry/orchestrating_the_cloud_to_simplify_and_accelerate_service_delivery1
Friday, June 21, 13

5. Cloud Orchestration is...
Automating the infrastructure
needed to provide a computing
service so that resources are
automatically allocated and de-
allocated to meet changing demand
Friday, June 21, 13

6. The Orchestra Analogy
Friday, June 21, 13

7. Benefits
•Saves money
•Saves time
•Repeatable, version-able process
•Less human intervention
•Fewer stupid mistakes
Friday, June 21, 13

8. Agenda
• What is Cloud Orchestration
•Quick intro to Wayin Hub
• Wayin Hub deployment & application architecture
• How we scale, deploy and do backups
• About Cloud Formation Templates
• About vendor lock-in
• Designing for orchestration
Friday, June 21, 13

9. Friday, June 21, 13

10. Friday, June 21, 13

11. Friday, June 21, 13

12. Friday, June 21, 13

13. Friday, June 21, 13

14. Friday, June 21, 13

15. Friday, June 21, 13

16. Agenda
• What is Cloud Orchestration
• Quick intro to Wayin Hub
•Wayin Hub deployment & app architecture
• How we scale, deploy and do backups
• About Cloud Formation Templates
• About vendor lock-in
• Designing for orchestration
Friday, June 21, 13

17. Glossary
• EC2 - server virtualization
• EBS - mountable block storage
• ELB - load balancer
• R53 - domain name services (DNS)
• S3 - file / bucket storage service
• CloudFront - content distribution network (CDN)
• Autoscale - automatically stop & start servers
• CloudWatch / Cloud Alarms - monitoring & triggering
• Security Group - restrict port access on group of instances
• SQS - Simple Queuing Service
Friday, June 21, 13

18. Application Architecture
Browser
Front
end
HTML5
jQuery
Backbone
Ubuntu
Back
end
Java, Jetty
Jersey
Guice
REST
HTTP
Ubuntu
Cassandra
Redis
Memcache
Queue Service
Twitter Feeds
Other Soc.
Nets
Friday, June 21, 13

19. Ubuntu
Jenkins
API Security Group
S3
R53CloudFront
ELB
EC2
Browser
Front
end
DB Security Group
EBS
SQS
TwitterFeeds
Tracker Security Group
Deployment Architecture
Cloud Formation
// CFN template
{
"Parameters": {
},
"Mappings": {
},
"Resources": {
}
}
Friday, June 21, 13

20. Cloud Formation
“An easy way to create and manage a collection of related
AWS resources, provisioning and updating them in an orderly
and predictable fashion.
You can use AWS CloudFormation’s sample templates or create
your own. Once deployed, you can modify and update the AWS
resources in a controlled and predictable way allowing you to
version control your AWS infrastructure in the same way as
you version control your software.
You can deploy and update a template and its associated collection
of resources (called a stack) via the AWS Management Console,
CloudFormation command line tools or APIs. CloudFormation is
available at no additional charge, and you pay only for the
AWS resources needed to run your applications.”
Friday, June 21, 13

21. Agenda
• What is Cloud Orchestration
• Quick intro to Wayin Hub
• Wayin Hub deployment & application architecture
•How we scale, deploy and do backups
• About Cloud Formation Templates
• About vendor lock-in
• Designing for orchestration
Friday, June 21, 13

22. Separate auto-scale groups
Ubuntu
API server
WAR
Jetty
Ubuntu
API server
WAR
Jetty
Ubuntu
API server
WAR
Jetty
Min: 3
Max: 9
Scale up (quickly):
CPU > X% forY minutes
Scale down (slowly):
CPU < A% for B minutes
Ubuntu
Cassandra
Redis
Memcache
DB serverUbuntu
Cassandra
Redis
Memcache
DB server
Min: 2
Max: 2
We decide when to scale
up and scale down our
DB server instances
Ubuntu
Tracker
WAR
Jetty
Ubuntu
Tracker
WAR
Jetty
Ubuntu
Tracker
WAR
Jetty
Min: 2
Max: 20
Scale up (quickly):
Queue has > X
messages forY minutes
Scale down (slowly):
Queue has < X message
forY minutes
Friday, June 21, 13

23. CFN Launches
new API server
instances
1
Self-configuring API servers
DB Security Group
S3 ELB
CloudInit script
downloads the
deployment bundle
2
CloudInit scripts
find DB servers by
Security Group,
starts Jetty
3
CloudInit
scripts put
server into
ELB rotation
4
API Security Group
CFN
Cloud
Watch
Friday, June 21, 13

24. Self-configuring DB servers
DB Security Group
S3
CloudInit scripts finds other
DB servers by Server Group
3
CloudInit scripts
download latest
deployment bundle
2Human runs the
command to increase
DB servers by one
1
Ubuntu
New DB server
New Cassandra node joins
cluster when ready
7
CFN
CloudInit scripts write
Memcache config, start it
4
MemcacheCloudInit scripts write Redis config,
assume first server is master, start it
5
Redis
CloudInit scripts write
Cassandra config, start it
6
Cassandra
Friday, June 21, 13

25. Automated deployment
API Security Group
GitDeveloper merges code
into deployment branch
1
Developer kicks off
Jenkins build
2
S3
Jenkins copies
bundle to S3 bucket
3
Developer kicks off
deployment build
4
Jenkins runs
deploy command
via SSH
5
On each server the script
runs locally, downloads
and installs the bundle
6
Ubuntu
Jenkins
Tracker Security Group
Friday, June 21, 13

26. Incremental backups
copied to S3 bucket
5
Automated Backups
S3
Jenkins runs
daily backup job
1
Backup script
creates EBS snapshot 3
DB Security Group
Cronjob kicks off
incremental backups
4
Cassandra snapshot
backups happen
simultaneously
2
Ubuntu
Jenkins
EBS
Friday, June 21, 13

27. Agenda
• What is Cloud Orchestration
• Quick intro to Wayin Hub
• Wayin Hub deployment & application architecture
• How we scale, deploy and do backups
•About Cloud Formation Templates
• About vendor lock-in
• Designing for orchestration
Friday, June 21, 13

28. CFN Template skeleton
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "Example stack",
"Parameters": {
},
"Mappings": {
},
"Resources": {
}
}
Friday, June 21, 13

29. CFN Parameters
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "WayinHub Three Server-type Prod Stack",
"Parameters": {
"DnsName" : {
"Description" : "DNS name to be created",
"Type" : "String",
"Default": "stack-test"
},
"DnsDomain" : {
"Description" : "DNS domain to be used",
"Type" : "String",
"Default": "twt.pl"
},
"KeyPair" : {
"Description" : "EC2 Key Pair to allow SSH...",
"Type" : "String",
"Default": "wayin"
},
Friday, June 21, 13

30. CFN Parameters
"ReleaseBucket" : {
"Description" : "S3 Bucket of release bundle",
"Type" : "String",
"Default": "hub-deployment-release"
},
"ApiMinServers" : {
"Description" : "Minimum number of API servers",
"Type" : "Number",
"Default": "2",
"MinValue": "2"
},
"ApiMaxServers" : {
"Description" : "Maximum number API servers",
"Type" : "Number",
"Default": "5",
"MinValue": "2"
},
"ApiInstanceType": {
"Description": "EC2 instance type for API servers",
"Type": "String",
"Default": "m1.large",
"AllowedValues": ["m1.xlarge", "m3.xlarge"],
"ConstraintDescription": "must be a valid EC2 type."
},
Friday, June 21, 13

31. CFN Mappings
"Mappings": {
"AWSRegionArch2AMI": {
"us-east-1": { "64": "ami-e864da81" },
"us-west-2": { "64": "ami-50f97060" }
},
"TwoAZs": {
"us-east-1": {
"AZ1" : "us-east-1a", "AZ2" : "us-east-1b" },
"us-west-2": {
"AZ1" : "us-west-2a", "AZ2" : "us-west-2b" }
}
},
Friday, June 21, 13

32. CFN Resources
"Resources": {
"DbAutoScalingLaunchConfiguration": {
"Type": "AWS::AutoScaling::LaunchConfiguration",
},
"DbAutoScalingGroup": {
"Type": "AWS::AutoScaling::AutoScalingGroup",
},
"ApiAutoScalingLaunchConfiguration": {
"Type": "AWS::AutoScaling:: LaunchConfiguration",
},
"ApiAutoScalingGroup": {
"Type": "AWS::AutoScaling::AutoScalingGroup",
},
"ApiServerScaleUpPolicy" : {
"Type" : "AWS::AutoScaling::ScalingPolicy",
},
"ApiServerScaleDownPolicy" : {
"Type" : "AWS::AutoScaling::ScalingPolicy",
},
"CPUAlarmHigh": {
},
"CPUAlarmLow": {
},
Friday, June 21, 13

33. CFN Resources
"ApiAutoScalingLaunchConfiguration": {
"Type": "AWS::AutoScaling::LaunchConfiguration",
"Properties": {
"UserData": {
"Fn::Base64" : { "Fn::Join": [ "", [
"#!/bin/bash -exn",
"# Log all output from this scriptn",
"exec >/var/log/wayin-bootstrap.log 2>&1n",
"# Expose various environmental variables, etc.n",
"cat >/etc/profile.d/wayin-env.sh <<EOFn",
"export ELB_NAME=", { "Ref" : "ApiElasticLoadBalancer" }, "n",
"export EC2_REGION=", { "Ref" : "AWS::Region" }, "n",
"export DNS_NAME=", { "Ref" : "DnsName" }, "n",
"export DNS_DOMAIN=", { "Ref" : "DnsDomain" }, "n",
"EOFn",
"# Install various packagesn",
"apt-get updaten",
"apt-get -y install s3cmdn",
"n",
"# Download our application filesn",
"cd /usr/share/wayinhubn",
"s3cmd --config=/etc/s3cfg get s3://",{"Ref":"ReleaseBucket"},"/twithub.tgzn",
"tar xf twithub.tgzn",
"rm -fr twithub.tgzn",
"cd ./init_instancen",
"sh ./init_api_server.shn"
]]}
},
Friday, June 21, 13

34. CFN Console
Friday, June 21, 13

35. Agenda
• What is Cloud Orchestration
• Quick intro to Wayin Hub
• Wayin Hub deployment & application architecture
• How we scale, deploy and do backups
• About Cloud Formation Templates
•About vendor lock-in
• Designing for orchestration
Friday, June 21, 13

36. Lock-in in automation
• Cloud Formation itself is lock-in
• Plenty of AWS dependencies in our:
• Instance init scripts
• Deployment scripts
• Backup scripts
Friday, June 21, 13

37. No lock-in in app code
• Bulk of our code is in our application
• AWS dependencies are pluggable
Friday, June 21, 13

38. Agenda
• What is Cloud Orchestration
• Quick intro to Wayin Hub
• Wayin Hub deployment & application architecture
• How we scale, deploy and do backups
• About Cloud Formation Templates
• About vendor lock-in
•Designing for orchestration
Friday, June 21, 13

39. Choose right components
• Good
• Peer-to-peer
• Linear scalability
• Bad
• Master-slave
• “Special” servers
• Single point-of-failure
Friday, June 21, 13

40. Bootstrap everything
• Every instance must self-configure
• Make services easy to discover
Friday, June 21, 13

41. Keep it simple
• Minimize number of different...
• Server types
• External services
• Learning curves
Friday, June 21, 13

42. Questions?
Friday, June 21, 13