Scaling DevOps

713 views
573 views

Published on

Scaling DevOps from the Seattle Scalability Meetup, December 2013

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
713
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
23
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Scaling DevOps

  1. 1. Scaling DevOps Jeffrey Hulten Whitepages, Inc. Wednesday, December 4, 13
  2. 2. Who Am I? Jeffrey Hulten - @jhulten Software Engineer, Architecture Improvement at WhitePages, Inc. Organizer of Seattle DevOps MeetUp Too Ops for Dev, Too Dev for Ops Wednesday, December 4, 13
  3. 3. Scaling DevOps Your Tools Aren’t Enough Your First 100 Machines Watching the Horizon Wednesday, December 4, 13
  4. 4. Your Tools Aren’t Enough Culture Deploy a Package Manage the Metal Minimize the Surface Area Growing Pains Wednesday, December 4, 13
  5. 5. Culture Not Just for Bacteria Anymore Starts with one person Infect your friends Try changing things Mutation must allow for failure Requirement for survival Wednesday, December 4, 13
  6. 6. Deploy a Package TGZ OS Package (.deb / .rpm) Machine Image (AMI) LXC Image (Docker) Wednesday, December 4, 13
  7. 7. OS Packages Made Easier FPM Package Config Separately Package Repository Wednesday, December 4, 13
  8. 8. FPM: Effing Package Management https://github.com/jordansissel/fpm $ gem install fpm Make RPM, DEB, and Solaris packages and Puppet module Create from directories, RPM, DEB, Gem, PyPi, Tarball and more Wednesday, December 4, 13
  9. 9. Example: Jenkins NAME=jenkins VERSION=1.396 .PHONY: package package: rm -f jenkins.war wget http://ftp.osuosl.org/pub/hudson/war/$(VERSION)/jenkins.war fpm -s dir -t deb -n $(NAME) -v $(VERSION) --prefix /opt/jenkins jenkins.war Wednesday, December 4, 13
  10. 10. Machine Images & Containers Machine Image Entire OS All Services Enabled Containers Minimal OS Image Run One Process (Replace `init`) Wednesday, December 4, 13
  11. 11. Manage the Metal Application as Inventory... Wednesday, December 4, 13
  12. 12. Minimize the Surface Area What is Surface Area? Libraries? Versions? Components? Data Stores? Wednesday, December 4, 13
  13. 13. Minimize the Surface Area What is Surface Area? Libraries? Versions? Components? Data Stores? Deployables... Wednesday, December 4, 13
  14. 14. Growing Pains Don’t bottle knowledge There are no specialists Think about the smallest solution Delay your decisions Wednesday, December 4, 13
  15. 15. Your First 20 Machines DEMO TIME! Source at: https://github.com/whitepages/scaledemo-20-machines AWS CloudFormation, cloud-init... Wednesday, December 4, 13
  16. 16. Application: Random Numbers as a Service Wednesday, December 4, 13
  17. 17. CloudFormation Deploy Stacks Templates in JSON Parameters for Reuse Wednesday, December 4, 13
  18. 18. Template: Parameters "Parameters": { "OperatorEmail": { "Type": "String", "Description": "Email address to notify if there are any scaling operations" }, "InstanceType": { "Description": "WebServer EC2 instance type", "Type": "String", "AllowedValues": [ "t1.micro", "m1.small", ...], "Default": "m1.small", "ConstraintDescription": "must be a valid EC2 instance type." }, ... "SSHLocation": { "Default": "0.0.0.0/0", "Type": "String", "MaxLength": "18", "MinLength": "9", "AllowedPattern": "(d{1,3}).(d{1,3}).(d{1,3}).(d{1,3})/(d{1,2})", "Description": "The IP address range that can be used to SSH to the EC2 instances", "ConstraintDescription": "must be a valid IP CIDR range of the form x.x.x.x/x." } }, Wednesday, December 4, 13
  19. 19. Template: Parameters "SSHLocation": { "Default": "0.0.0.0/0", "Type": "String", "MaxLength": "18", "MinLength": "9", "AllowedPattern": "(d{1,3}).(d{1,3}).../(d{1,2})", "Description": "...", "ConstraintDescription": "..." } Wednesday, December 4, 13
  20. 20. Template: Mappings "Mappings": { "AWSRegionArch2AMI": { "us-east-1": { "64": "ami-955b79fc" }, "us-west-1": { "64": "ami-6ca89929" }, "us-west-2": { "64": "ami-a8395c98" } }, ... }, Wednesday, December 4, 13
  21. 21. Template: Resources "Resources": { "NotificationTopic": { "Type": "AWS::SNS::Topic", "Properties": { "Subscription": [ { "Protocol": "email", "Endpoint": { "Ref": "OperatorEmail" } } ] } }, ... } Wednesday, December 4, 13
  22. 22. AutoScaling Group "RandServerGroup": { "Type": "AWS::AutoScaling::AutoScalingGroup", "Properties": { "AvailabilityZones": { "Fn::GetAZs": "" }, "NotificationConfiguration": { "NotificationTypes": [ "autoscaling:EC2_INSTANCE_LAUNCH", "autoscaling:EC2_INSTANCE_LAUNCH_ERROR", "autoscaling:EC2_INSTANCE_TERMINATE", "autoscaling:EC2_INSTANCE_TERMINATE_ERROR" ], "TopicARN": { "Ref": "NotificationTopic" } }, "MinSize": "1", "MaxSize": "100", "LaunchConfigurationName": { "Ref": “..." } }} Wednesday, December 4, 13
  23. 23. Launch Config "RandLaunchConfig": { "Type": "AWS::AutoScaling::LaunchConfiguration", "Properties": { "SecurityGroups": [{ "Ref": InstanceSecurityGroup" }], "InstanceType": { "Ref": "InstanceType" }, "KeyName": { "Ref": "KeyName" }, "UserData": { "Fn::Base64": { "Fn::Join" : ["",[ "#includen", "https://.../cloud-config.txtn" ]]}}, "ImageId": {"Fn::FindInMap": [...] } } }, Wednesday, December 4, 13
  24. 24. Cloud Init #cloud-config packages: - ruby1.9.3 - git - build-essential runcmd: - /usr/bin/gem install bundler - /usr/bin/git clone "https://github.com/whitepages/ scaledemo-20-machines.git" /opt/rand - cd /opt/rand && /usr/local/bin/bundle install - cd /opt/rand &&/usr/local/bin/bundle exec randserver & Wednesday, December 4, 13
  25. 25. Watching the Horizon Docker / LXC SmartStack Akka Cluster / Riak Core Wednesday, December 4, 13
  26. 26. Questions? jhulten@whitepages.com Twitter: @jhulten Github: jhulten whitepages whitepages.com/careers Wednesday, December 4, 13

×