Scaling DevOps from the Seattle Scalability Meetup, December 2013

1. Scaling DevOps
Jeffrey Hulten
Whitepages, Inc.
Wednesday, December 4, 13

2. Who Am I?
Jeffrey Hulten - @jhulten
Software Engineer, Architecture Improvement at
WhitePages, Inc.
Organizer of Seattle DevOps MeetUp
Too Ops for Dev, Too Dev for Ops
Wednesday, December 4, 13

3. Scaling DevOps
Your Tools Aren’t Enough
Your First 100 Machines
Watching the Horizon
Wednesday, December 4, 13

4. Your Tools Aren’t Enough
Culture
Deploy a Package
Manage the Metal
Minimize the Surface Area
Growing Pains
Wednesday, December 4, 13

5. Culture
Not Just for Bacteria Anymore
Starts with one person
Infect your friends
Try changing things
Mutation must allow for
failure
Requirement for survival
Wednesday, December 4, 13

6. Deploy a Package
TGZ
OS Package (.deb / .rpm)
Machine Image (AMI)
LXC Image (Docker)
Wednesday, December 4, 13

7. OS Packages Made Easier
FPM
Package Config Separately
Package Repository
Wednesday, December 4, 13

8. FPM: Effing Package
Management
https://github.com/jordansissel/fpm
$ gem install fpm
Make RPM, DEB, and Solaris packages and Puppet
module
Create from directories, RPM, DEB, Gem, PyPi, Tarball
and more
Wednesday, December 4, 13

9. Example: Jenkins
NAME=jenkins
VERSION=1.396
.PHONY: package
package:
rm -f jenkins.war
wget http://ftp.osuosl.org/pub/hudson/war/$(VERSION)/jenkins.war
fpm -s dir -t deb -n $(NAME) -v $(VERSION)
--prefix /opt/jenkins jenkins.war
Wednesday, December 4, 13

10. Machine Images &
Containers
Machine Image
Entire OS
All Services Enabled
Containers
Minimal OS Image
Run One Process (Replace `init`)
Wednesday, December 4, 13

11. Manage the Metal
Application as Inventory...
Wednesday, December 4, 13

12. Minimize the Surface Area
What is Surface Area?
Libraries?
Versions?
Components?
Data Stores?
Wednesday, December 4, 13

13. Minimize the Surface Area
What is Surface Area?
Libraries?
Versions?
Components?
Data Stores?
Deployables...
Wednesday, December 4, 13

14. Growing Pains
Don’t bottle knowledge
There are no
specialists
Think about the
smallest solution
Delay your decisions
Wednesday, December 4, 13

15. Your First 20 Machines
DEMO TIME!
Source at:
https://github.com/whitepages/scaledemo-20-machines
AWS CloudFormation, cloud-init...
Wednesday, December 4, 13

16. Application: Random
Numbers as a Service
Wednesday, December 4, 13

17. CloudFormation
Deploy Stacks
Templates in JSON
Parameters for Reuse
Wednesday, December 4, 13

18. Template: Parameters
"Parameters": {
"OperatorEmail": {
"Type": "String",
"Description": "Email address to notify if there are any scaling operations"
},
"InstanceType": {
"Description": "WebServer EC2 instance type",
"Type": "String",
"AllowedValues": [ "t1.micro", "m1.small", ...],
"Default": "m1.small",
"ConstraintDescription": "must be a valid EC2 instance type."
},
...
"SSHLocation": {
"Default": "0.0.0.0/0",
"Type": "String",
"MaxLength": "18",
"MinLength": "9",
"AllowedPattern": "(d{1,3}).(d{1,3}).(d{1,3}).(d{1,3})/(d{1,2})",
"Description": "The IP address range that can be used to SSH to the EC2 instances",
"ConstraintDescription": "must be a valid IP CIDR range of the form x.x.x.x/x."
}
},
Wednesday, December 4, 13

19. Template: Parameters
"SSHLocation": {
"Default": "0.0.0.0/0",
"Type": "String",
"MaxLength": "18",
"MinLength": "9",
"AllowedPattern": "(d{1,3}).(d{1,3}).../(d{1,2})",
"Description": "...",
"ConstraintDescription": "..."
}
Wednesday, December 4, 13

20. Template: Mappings
"Mappings": {
"AWSRegionArch2AMI": {
"us-east-1": {
"64": "ami-955b79fc"
},
"us-west-1": {
"64": "ami-6ca89929"
},
"us-west-2": {
"64": "ami-a8395c98"
}
},
...
},
Wednesday, December 4, 13

21. Template: Resources
"Resources": {
"NotificationTopic": {
"Type": "AWS::SNS::Topic",
"Properties": {
"Subscription": [
{
"Protocol": "email",
"Endpoint": {
"Ref": "OperatorEmail"
}
}
]
}
},
...
}
Wednesday, December 4, 13

22. AutoScaling Group
"RandServerGroup": {
"Type": "AWS::AutoScaling::AutoScalingGroup",
"Properties": {
"AvailabilityZones": { "Fn::GetAZs": "" },
"NotificationConfiguration": {
"NotificationTypes": [
"autoscaling:EC2_INSTANCE_LAUNCH",
"autoscaling:EC2_INSTANCE_LAUNCH_ERROR",
"autoscaling:EC2_INSTANCE_TERMINATE",
"autoscaling:EC2_INSTANCE_TERMINATE_ERROR"
],
"TopicARN": { "Ref": "NotificationTopic" }
},
"MinSize": "1",
"MaxSize": "100",
"LaunchConfigurationName": { "Ref": “..." }
}}
Wednesday, December 4, 13

23. Launch Config
"RandLaunchConfig": {
"Type": "AWS::AutoScaling::LaunchConfiguration",
"Properties": {
"SecurityGroups": [{ "Ref": InstanceSecurityGroup" }],
"InstanceType": { "Ref": "InstanceType" },
"KeyName": { "Ref": "KeyName" },
"UserData": {
"Fn::Base64": { "Fn::Join" : ["",[
"#includen",
"https://.../cloud-config.txtn"
]]}},
"ImageId": {"Fn::FindInMap": [...] }
}
},
Wednesday, December 4, 13

24. Cloud Init
#cloud-config
packages:
- ruby1.9.3
- git
- build-essential
runcmd:
- /usr/bin/gem install bundler
- /usr/bin/git clone "https://github.com/whitepages/
scaledemo-20-machines.git" /opt/rand
- cd /opt/rand && /usr/local/bin/bundle install
- cd /opt/rand &&/usr/local/bin/bundle exec randserver &
Wednesday, December 4, 13

25. Watching the Horizon
Docker / LXC
SmartStack
Akka Cluster / Riak
Core
Wednesday, December 4, 13

26. Questions?
jhulten@whitepages.com
Twitter: @jhulten
Github: jhulten
whitepages
whitepages.com/careers
Wednesday, December 4, 13