Wayin devops-2013

  • 818 views
Uploaded on

 

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
818
On Slideshare
0
From Embeds
0
Number of Embeds
5

Actions

Shares
Downloads
8
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. DevOps at WayinOrchestration with AWS Cloud FormationDavid M. JohnsonSoftware Developer - Wayin, Inc.Friday, June 21, 13
  • 2. Agenda•What is Cloud Orchestration• Quick intro to Wayin Hub• Wayin Hub deployment & application architecture• How we scale, deploy and do backups• About Cloud Formation Templates• About vendor lock-in• Designing for orchestrationFriday, June 21, 13
  • 3. “Orchestration describes theautomated arrangement,coordination, and managementof complex computer systems,middleware, and services.It is often discussed as havingan inherent intelligence or evenimplicitly autonomic control, butthose are largely aspirations oranalogies rather than technicaldescriptions.”http://en.wikipedia.org/wiki/Orchestration_(computing)Friday, June 21, 13
  • 4. “cloud orchestration =automation+ integration+ best practices”https://www.ibm.com/developerworks/mydeveloperworks/blogs/9e696bfa-94af-4f5a-ab50-c955cca76fd0/entry/orchestrating_the_cloud_to_simplify_and_accelerate_service_delivery1Friday, June 21, 13
  • 5. Cloud Orchestration is...Automating the infrastructureneeded to provide a computingservice so that resources areautomatically allocated and de-allocated to meet changing demandFriday, June 21, 13
  • 6. The Orchestra AnalogyFriday, June 21, 13
  • 7. Benefits•Saves money•Saves time•Repeatable, version-able process•Less human intervention•Fewer stupid mistakesFriday, June 21, 13
  • 8. Agenda• What is Cloud Orchestration•Quick intro to Wayin Hub• Wayin Hub deployment & application architecture• How we scale, deploy and do backups• About Cloud Formation Templates• About vendor lock-in• Designing for orchestrationFriday, June 21, 13
  • 9. Friday, June 21, 13
  • 10. Friday, June 21, 13
  • 11. Friday, June 21, 13
  • 12. Friday, June 21, 13
  • 13. Friday, June 21, 13
  • 14. Friday, June 21, 13
  • 15. Friday, June 21, 13
  • 16. Agenda• What is Cloud Orchestration• Quick intro to Wayin Hub•Wayin Hub deployment & app architecture• How we scale, deploy and do backups• About Cloud Formation Templates• About vendor lock-in• Designing for orchestrationFriday, June 21, 13
  • 17. Glossary• EC2 - server virtualization• EBS - mountable block storage• ELB - load balancer• R53 - domain name services (DNS)• S3 - file / bucket storage service• CloudFront - content distribution network (CDN)• Autoscale - automatically stop & start servers• CloudWatch / Cloud Alarms - monitoring & triggering• Security Group - restrict port access on group of instances• SQS - Simple Queuing ServiceFriday, June 21, 13
  • 18. Application ArchitectureBrowserFrontendHTML5jQueryBackboneUbuntuBackendJava, JettyJerseyGuiceRESTHTTPUbuntuCassandraRedisMemcacheQueue ServiceTwitter FeedsOther Soc.NetsFriday, June 21, 13
  • 19. UbuntuJenkinsAPI Security GroupS3R53CloudFrontELBEC2BrowserFrontendDB Security GroupEBSSQSTwitterFeedsTracker Security GroupDeployment ArchitectureCloud Formation// CFN template{"Parameters": {},"Mappings": {},"Resources": {}}Friday, June 21, 13
  • 20. Cloud Formation“An easy way to create and manage a collection of relatedAWS resources, provisioning and updating them in an orderlyand predictable fashion.You can use AWS CloudFormation’s sample templates or createyour own. Once deployed, you can modify and update the AWSresources in a controlled and predictable way allowing you toversion control your AWS infrastructure in the same way asyou version control your software.You can deploy and update a template and its associated collectionof resources (called a stack) via the AWS Management Console,CloudFormation command line tools or APIs. CloudFormation isavailable at no additional charge, and you pay only for theAWS resources needed to run your applications.”Friday, June 21, 13
  • 21. Agenda• What is Cloud Orchestration• Quick intro to Wayin Hub• Wayin Hub deployment & application architecture•How we scale, deploy and do backups• About Cloud Formation Templates• About vendor lock-in• Designing for orchestrationFriday, June 21, 13
  • 22. Separate auto-scale groupsUbuntuAPI serverWARJettyUbuntuAPI serverWARJettyUbuntuAPI serverWARJettyMin: 3Max: 9Scale up (quickly):CPU > X% forY minutesScale down (slowly):CPU < A% for B minutesUbuntuCassandraRedisMemcacheDB serverUbuntuCassandraRedisMemcacheDB serverMin: 2Max: 2We decide when to scaleup and scale down ourDB server instancesUbuntuTrackerWARJettyUbuntuTrackerWARJettyUbuntuTrackerWARJettyMin: 2Max: 20Scale up (quickly):Queue has > Xmessages forY minutesScale down (slowly):Queue has < X messageforY minutesFriday, June 21, 13
  • 23. CFN Launchesnew API serverinstances1Self-configuring API serversDB Security GroupS3 ELBCloudInit scriptdownloads thedeployment bundle2CloudInit scriptsfind DB servers bySecurity Group,starts Jetty3CloudInitscripts putserver intoELB rotation4API Security GroupCFNCloudWatchFriday, June 21, 13
  • 24. Self-configuring DB serversDB Security GroupS3CloudInit scripts finds otherDB servers by Server Group3CloudInit scriptsdownload latestdeployment bundle2Human runs thecommand to increaseDB servers by one1UbuntuNew DB serverNew Cassandra node joinscluster when ready7CFNCloudInit scripts writeMemcache config, start it4MemcacheCloudInit scripts write Redis config,assume first server is master, start it5RedisCloudInit scripts writeCassandra config, start it6CassandraFriday, June 21, 13
  • 25. Automated deploymentAPI Security GroupGitDeveloper merges codeinto deployment branch1Developer kicks offJenkins build2S3Jenkins copiesbundle to S3 bucket3Developer kicks offdeployment build4Jenkins runsdeploy commandvia SSH5On each server the scriptruns locally, downloadsand installs the bundle6UbuntuJenkinsTracker Security GroupFriday, June 21, 13
  • 26. Incremental backupscopied to S3 bucket5Automated BackupsS3Jenkins runsdaily backup job1Backup scriptcreates EBS snapshot 3DB Security GroupCronjob kicks offincremental backups4Cassandra snapshotbackups happensimultaneously2UbuntuJenkinsEBSFriday, June 21, 13
  • 27. Agenda• What is Cloud Orchestration• Quick intro to Wayin Hub• Wayin Hub deployment & application architecture• How we scale, deploy and do backups•About Cloud Formation Templates• About vendor lock-in• Designing for orchestrationFriday, June 21, 13
  • 28. CFN Template skeleton{"AWSTemplateFormatVersion": "2010-09-09","Description": "Example stack","Parameters": {},"Mappings": {},"Resources": {}}Friday, June 21, 13
  • 29. CFN Parameters{"AWSTemplateFormatVersion": "2010-09-09","Description": "WayinHub Three Server-type Prod Stack","Parameters": {"DnsName" : {"Description" : "DNS name to be created","Type" : "String","Default": "stack-test"},"DnsDomain" : {"Description" : "DNS domain to be used","Type" : "String","Default": "twt.pl"},"KeyPair" : {"Description" : "EC2 Key Pair to allow SSH...","Type" : "String","Default": "wayin"},Friday, June 21, 13
  • 30. CFN Parameters"ReleaseBucket" : {"Description" : "S3 Bucket of release bundle","Type" : "String","Default": "hub-deployment-release"},"ApiMinServers" : {"Description" : "Minimum number of API servers","Type" : "Number","Default": "2","MinValue": "2"},"ApiMaxServers" : {"Description" : "Maximum number API servers","Type" : "Number","Default": "5","MinValue": "2"},"ApiInstanceType": {"Description": "EC2 instance type for API servers","Type": "String","Default": "m1.large","AllowedValues": ["m1.xlarge", "m3.xlarge"],"ConstraintDescription": "must be a valid EC2 type."},Friday, June 21, 13
  • 31. CFN Mappings"Mappings": {"AWSRegionArch2AMI": {"us-east-1": { "64": "ami-e864da81" },"us-west-2": { "64": "ami-50f97060" }},"TwoAZs": {"us-east-1": {"AZ1" : "us-east-1a", "AZ2" : "us-east-1b" },"us-west-2": {"AZ1" : "us-west-2a", "AZ2" : "us-west-2b" }}},Friday, June 21, 13
  • 32. CFN Resources"Resources": {"DbAutoScalingLaunchConfiguration": {"Type": "AWS::AutoScaling::LaunchConfiguration",},"DbAutoScalingGroup": {"Type": "AWS::AutoScaling::AutoScalingGroup",},"ApiAutoScalingLaunchConfiguration": {"Type": "AWS::AutoScaling:: LaunchConfiguration",},"ApiAutoScalingGroup": {"Type": "AWS::AutoScaling::AutoScalingGroup",},"ApiServerScaleUpPolicy" : {"Type" : "AWS::AutoScaling::ScalingPolicy",},"ApiServerScaleDownPolicy" : {"Type" : "AWS::AutoScaling::ScalingPolicy",},"CPUAlarmHigh": {},"CPUAlarmLow": {},Friday, June 21, 13
  • 33. CFN Resources"ApiAutoScalingLaunchConfiguration": {"Type": "AWS::AutoScaling::LaunchConfiguration","Properties": {"UserData": {"Fn::Base64" : { "Fn::Join": [ "", ["#!/bin/bash -exn","# Log all output from this scriptn","exec >/var/log/wayin-bootstrap.log 2>&1n","# Expose various environmental variables, etc.n","cat >/etc/profile.d/wayin-env.sh <<EOFn","export ELB_NAME=", { "Ref" : "ApiElasticLoadBalancer" }, "n","export EC2_REGION=", { "Ref" : "AWS::Region" }, "n","export DNS_NAME=", { "Ref" : "DnsName" }, "n","export DNS_DOMAIN=", { "Ref" : "DnsDomain" }, "n","EOFn","# Install various packagesn","apt-get updaten","apt-get -y install s3cmdn","n","# Download our application filesn","cd /usr/share/wayinhubn","s3cmd --config=/etc/s3cfg get s3://",{"Ref":"ReleaseBucket"},"/twithub.tgzn","tar xf twithub.tgzn","rm -fr twithub.tgzn","cd ./init_instancen","sh ./init_api_server.shn"]]}},Friday, June 21, 13
  • 34. CFN ConsoleFriday, June 21, 13
  • 35. Agenda• What is Cloud Orchestration• Quick intro to Wayin Hub• Wayin Hub deployment & application architecture• How we scale, deploy and do backups• About Cloud Formation Templates•About vendor lock-in• Designing for orchestrationFriday, June 21, 13
  • 36. Lock-in in automation• Cloud Formation itself is lock-in• Plenty of AWS dependencies in our:• Instance init scripts• Deployment scripts• Backup scriptsFriday, June 21, 13
  • 37. No lock-in in app code• Bulk of our code is in our application• AWS dependencies are pluggableFriday, June 21, 13
  • 38. Agenda• What is Cloud Orchestration• Quick intro to Wayin Hub• Wayin Hub deployment & application architecture• How we scale, deploy and do backups• About Cloud Formation Templates• About vendor lock-in•Designing for orchestrationFriday, June 21, 13
  • 39. Choose right components• Good• Peer-to-peer• Linear scalability• Bad• Master-slave• “Special” servers• Single point-of-failureFriday, June 21, 13
  • 40. Bootstrap everything• Every instance must self-configure• Make services easy to discoverFriday, June 21, 13
  • 41. Keep it simple• Minimize number of different...• Server types• External services• Learning curvesFriday, June 21, 13
  • 42. Questions?Friday, June 21, 13