Hi!
Stratalux is a cloud-basedManaged Services ProviderWe build highly scalable &available websites inthe public cloud7/20/201...
A match made in the               Hall of Justice.7/20/2012                         3
Infrastructure as code ™7/20/2012                              4
Consistent builds and configurations7/20/2012                                          5
Backups!7/20/2012              6
Things break!7/20/2012                   7
Reduced costs!7/20/2012                    8
7/20/2012   9
•   Hosted Chef•   Attributes•   Cookbooks•   Data Bags•   Roles•   Environments7/20/2012          10
$ knife node list web01.stratalux.com web02.stratalux.com web03.stratalux.com web04.stratalux.com web05.stratalux.com app0...
cookbook_versions: amazon_dns_tools:    = 0.0.3 amazon_ses:          = 0.1.5 apache2:             = 0.99.5 apt:           ...
override_attributes: java:  install_flavor: openjdk   7/20/2012                13
comment:        Justin Sysadminftp_enabled: trueftp_passwd: passwordgroups:      sysadminid:       justinshell:     /bin/b...
$knife role show application_serverchef_type:       roledescription:     Application server role.json_class:      Chef::Ro...
$ knife environment show Production            name:          Production            cookbook_versions:             java:  ...
• Needed AWS Resources      –     IAM User      –     IAM Access Key      –     Bucket Policy      –     Auto Scaling Grou...
•   Parameters•   Mappings•   Resources•   Functions7/20/2012        18
"KeyName": {             "Type": "String",             "Description": "Name of an existing EC2 KeyPair to enable SSH acces...
"ChefOrganization": {             "Description": "The Organization name used in Hosted Chef.",             "Type": "String...
"DomainName": {             "Description": "Server Domain Name.",             "Type": "String",             "Default": “st...
"Mappings" : {            "RegionMap" : {                "us-east-1" : {                  "AMI" : "ami-76f0061f"          ...
• Resources "Resources" :{   "MySimpleImage" : {     "Type" : "AWS::EC2::Image",     "Properties" : {       "ImageId" : "m...
"ChefClientUser": { "Type": "AWS::IAM::User", "Properties": {   "Path": "/",   "Policies": [{      "PolicyName": "root",  ...
"HostKeys": {  "Type": "AWS::IAM::AccessKey",  "DependsOn": "ChefClientUser",  "Properties": {   "UserName": {      "Ref":...
"BucketPolicy": { "Type": "AWS::S3::BucketPolicy", "DependsOn": "HostKeys", "Properties": {  "PolicyDocument": {    "State...
"AppServerGroup": { "Type": "AWS::AutoScaling::AutoScalingGroup", "Properties": {   "AvailabilityZones": {     "Fn::GetAZs...
"ChefClient": {  "Type": "AWS::AutoScaling::LaunchConfiguration",  "DependsOn": "BucketPolicy",  "Metadata": {    "AWS::Cl...
"files": {  "/etc/chef/first-boot.json": {    "content": {      "run_list": {        "Ref": "ChefRunList“        },       ...
"Properties": {   "SecurityGroups": [    {      "Ref": “AppServerAccess"    } ], "ImageId": {   "Fn::FindInMap": ["AWSRegi...
"UserData": {   "Fn::Base64": {   "Fn::Join": ["",     ["#!/bin/bash -vn", "function error_exitn",       "{n", " cfn-signa...
“AppServerAccess ": {  "Type": "AWS::EC2::SecurityGroup",  "Properties": {    "GroupDescription": “AppServerAccess",      ...
“appserverstrata1uxit": {  "Type": "AWS::ElasticLoadBalancing::LoadBalancer",  "Properties": {     "AvailabilityZones": { ...
"AppServerScaleUpPolicy": {  "Type": "AWS::AutoScaling::ScalingPolicy",  "Properties": {    "AdjustmentType": "ChangeInCap...
"AppServerScaleDownPolicy": {  "Type": "AWS::AutoScaling::ScalingPolicy",  "Properties": {    "AdjustmentType": "ChangeInC...
"CPUAlarmHigh": {  "Type": "AWS::CloudWatch::Alarm",  "Properties": {    "AlarmDescription": "Scale-up if CPU > 75% for 2 ...
"CPUAlarmLow": {  "Type": "AWS::CloudWatch::Alarm",  "Properties": {    "AlarmDescription": "Scale-down if CPU < 75% for 1...
"ChefClientWaitHandle": {  "Type": "AWS::CloudFormation::WaitConditionHandle“},"ChefClientWaitCondition": {  "Type": "AWS:...
7/20/2012   39
7/20/2012   40
7/20/2012   41
7/20/2012   42
7/20/2012   43
Jeremy Przygodejp@stratalux.comwww.stratalux.com@stratalux Special thanks to JD Hanson for the awesome images!7/20/2012   ...
Upcoming SlideShare
Loading in …5
×

Stratalux Cloud Formation and Chef Integration Presentation

1,382 views

Published on

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,382
On SlideShare
0
From Embeds
0
Number of Embeds
18
Actions
Shares
0
Downloads
0
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • Welcome.Introduce HeatherIntroduce AlexIntroduce FlorianIntroduce ColoftSince it’s Summer in Los Angeles Spiderman release last weekBatman release tonightTechnology can be boring.I thought I would spice it up my presentation with some action heros.
  • We are Amazon partnerOpscode PartnerRightscale PartnerCustomers includeWorld Golf TourActive NetworkCrownPeakServices we provide.Professional services.Build websites on public cloud resources like amazon.Chef development services for all needs. Both windows and Unix.Managed services We provide managed services for systems built by us or any systems in the public cloud.We provide 24/7 NOC services.
  • Why cloud formation and Chef?In order to leverage public cloud resources to fullest advantage you need to be able to automate your infrastructureCloud formation automates Amazon’s infrastructureChef automates server configurations.
  • Opscode says “Infrastructure as code” What does this mean? With code and data you should be able to recreate your infrastructure.Apply development methodologies to IT Revision control. (use git, svnetc). Agile methodologies Lean methodologies use only what you need now, scale as necessary laterDeploy oftenImproved quality Ensure consistent dev, test and production environments. Automation you need automation to scale.
  • Don’t fix it, rebuild it. Cloud formation &amp; Chef let’s you rebuild systems.Improved deploymentsDeploy entire stacks not just new code.No Gumby Servers
  • Cloud formation and Chef means you can recreate your environment. You now have backups!Rebuild your infrastructure anytime anywhere.
  • Don’t fix problems, build new servers.Things break in the cloud. Commodity hardware fails. You need automation to rebuild systems.
  • Using cloud formation and chef let’s you implement technologies like auto scaling.You can scale your service with customer traffic.You can build DR systems without the cost of running the entire stack.
  • We use hosted chef to configure our servers.
  • We use hosted chef and configure our servers.You can get a free hosted chef account by visiting opscode.comHosted chef is a centralized and searchable database where all your chef configurations reside and are made available to your nodes. Hosted chef holds all your attributes, cookbooks, recipes and data bags.ExampleApp servers search hosted chef for names of DB servers and configure themselves appropriately.
  • Cookbooks are the fundamental units of distribution in Chef. They encapsulate all the resources you need to automate your infrastructure and are easily sharable with other Chef users.We will use the jetty and java cookbooks to configure our application server.
  • Attributes are Node data such as the IP address, hostname, loaded kernel modules, version of programming languages available on the system and more. New attributes can be dynamically added to the node in a variety of ways.Attributes are used by chef recipes to configure systems. Example:You may use a Java cookbook but set flavor of Java to openJDK or Oracle Java depending upon your server’s role.
  • Data bags provide an arbitrary stores of globally available JSON data.Data Bags are not directly associated with Node or Role attributes. When using Data Bags with a chef server, they are stored on the server and indexed for searching. When using Data Bags with chef-solo, data bags are stored in a directory hierarchy on the machine running chef-solo. Example:the users cookbook searches the user data bag for users to add to the system. You need a new user? Update the data_bag and run chef-client.
  • A role provides a means of grouping similar features of similar nodes, providing a mechanism for easily composing sets of functionality.Roles allow you to scale…We have created an application_server role. That is configured to install jetty and java. Among other things.
  • With environments, you can specify per environment run lists in roles, per environment cookbook versions, and environment attributes.Environments allows us to protect production cookbooks and configurations from cookbooks in development. Environments allows us to promote updates in our cookbooks in a controlled fashion.Review how Chef-client runs. Connects to hosted chef.Get’s role,Runs recipes, etc.
  • In order to build WGT an auto scaling group using Cloud Formation we need to create the following Cloud Formation resources.
  • Cloud Formation is JSON.Consisting of Parameters mappings resources and functions.
  • Parameters are inputs that allow you to control your stack.Here we Define the Host KeypairAnd the instance type.
  • In order for Cloud formation to boot strap the server using chef we have to define some Parameters that are used to communicate with Hosted chef.OrganizationEnvironmentRunlist (basically the role)
  • Next we define the DomainName for the server. The hostname is automatically generated based upon what Amazon provides.Then the name of the bucket that holds Chef’s validator.pem and encrypted data bag secrets. Basically keys to authenticate with Hosted chef.
  • Mappings match a key to a corresponding set of named values. For example, if you want to set values based on a region, you can create a mapping that uses the region name as a key and contains the values you want to specify for each specific region.Here we are matching Amazon Regions with AMI’s we want to use for our application servers.
  • Resources are the actual amazon services that we will need to configure.More later but they include everything from Instances to SQS, SES, Load balancers, security groups etc.
  • Here we setup a chefclient user to access our S3 bucket with the chef validator.pem and other files.
  • Here we define the hostkeyNotice the dependson directive.
  • Bucket policy sets permissions to only allow the chefclient access to our bucket of secured items.Notice the function Join and Ref.
  • Here we define the autoscaling group.Availability Zones.Launch configurationLoad balancer.Min/max
  • Launch configuration.Here we configure the instance to install packages and gems necessary for chef to run.
  • Here we lay down configuration files for chef to use And for S3cmd to get the files from the bucket.
  • Here we set the security group, instnance type and Image id.Ie which AMI and what size instance we are going to use.
  • Userdata allows us to run arbitrary code on the instance once it’s up and running.Here we are actually going to run chef-client. Which connects to hosted chef.Notice the cfn-signal command. This gives status information to cloudformation on if command ran successfully or not.Notice the WaitHandle. WE are telling cloudformation to wait for chef run to finish before continuing.
  • Security groups.
  • Create our load balancers.Choses which Azs to use.Healthcheck and other things.
  • Scale up policy.How many servers to add as we scale up.
  • Scale down does the oposite.
  • Now we configure our envrionment to monitor the servers.We are monitoring CPU utilization and then sacaling up and down as encessary. CPU &gt; 75% scales up.
  • Here we scale down.
  • Here is the wait definitiondefinition used earlier.
  • Now that we have a complete cloud formation template we need to load it into amazon.
  • Load your Template.
  • Here are the default values for parameters for input.You can update those inputs now if necessary.
  • Templates define stacks
  • Templates define stacks
  • Thank you!There are many things I didn’t mention like. Chef:Chef search, Resources, primitives.AWS cloud formation: Simplified JSONfunctions
  • Stratalux Cloud Formation and Chef Integration Presentation

    1. 1. Hi!
    2. 2. Stratalux is a cloud-basedManaged Services ProviderWe build highly scalable &available websites inthe public cloud7/20/2012 2
    3. 3. A match made in the Hall of Justice.7/20/2012 3
    4. 4. Infrastructure as code ™7/20/2012 4
    5. 5. Consistent builds and configurations7/20/2012 5
    6. 6. Backups!7/20/2012 6
    7. 7. Things break!7/20/2012 7
    8. 8. Reduced costs!7/20/2012 8
    9. 9. 7/20/2012 9
    10. 10. • Hosted Chef• Attributes• Cookbooks• Data Bags• Roles• Environments7/20/2012 10
    11. 11. $ knife node list web01.stratalux.com web02.stratalux.com web03.stratalux.com web04.stratalux.com web05.stratalux.com app01.stratalux.com app02.stratalux.com proxy01.stratalux.com proxy02.stratalux.com 7/20/2012 11
    12. 12. cookbook_versions: amazon_dns_tools: = 0.0.3 amazon_ses: = 0.1.5 apache2: = 0.99.5 apt: = 1.1.2 aws: = 0.99.0 aws_accounts: = 0.0.4 aws_tools: = 0.1.6 backup-manager: = 0.0.2 bluepill: = 0.2.0 build-essential: = 1.0.0 cacti: = 0.0.5 collectd: = 1.0.1 confluence: = 0.0.1 7/20/2012 12
    13. 13. override_attributes: java: install_flavor: openjdk 7/20/2012 13
    14. 14. comment: Justin Sysadminftp_enabled: trueftp_passwd: passwordgroups: sysadminid: justinshell: /bin/bashuid: 2304vsftpd_configs: guest_username: justin local_root: /home/justin 7/20/2012 14
    15. 15. $knife role show application_serverchef_type: roledescription: Application server role.json_class: Chef::Rolename: application_serverrun_list: role[stratalux_base_prod] recipe[jetty] recipe[java]default_attributes: jetty: admin_server: false bucket: bucket123 caching.servers: server1.stratalux.com:11211 server2.stratalux.com:11211 7/20/2012 15
    16. 16. $ knife environment show Production name: Production cookbook_versions: java: = 1.0.0 users: = 0.3.5 ntp: = 0.0.2 wordpress: = 0.8.4 apache2: = 1.0.1 jetty: = 0.0.8 default_attributes: ntp: servers: 0.pool.ntp.org 1.pool.ntp.org override_attributes: {}7/20/2012 © 2012 – Stratalux, Inc. 16
    17. 17. • Needed AWS Resources – IAM User – IAM Access Key – Bucket Policy – Auto Scaling Group – Launch Configuration – Security Groups – Load Balancer – Scaling Policy – Cloudwatch Alarms – WaitConditions7/20/2012 17
    18. 18. • Parameters• Mappings• Resources• Functions7/20/2012 18
    19. 19. "KeyName": { "Type": "String", "Description": "Name of an existing EC2 KeyPair to enable SSH access to the web server", "Default": “stratalux-key" }, "InstanceType": { "Description": "Application Server EC2 instance type", "Type": "String", "Default": "m1.small", "AllowedValues": ["t1.micro", "m1.small", "m1.medium", "m1.large", "m1.xlarge", "m2.xlarge", "m2.2xlarge", "m2.4xlarge", "c1.medium", "c1.xlarge", "cc1.4xlarge", "cc2.8xlarge", "cg1.4xlarge"], "ConstraintDescription": "must be a valid EC2 instance type." },7/20/2012 19
    20. 20. "ChefOrganization": { "Description": "The Organization name used in Hosted Chef.", "Type": "String", "Default": “stratalux" }, "ChefEnvironment": { "Description": "The Chef Environment.", "Type": "String", "Default": “Production" }, "ChefRunList": { "Description": "The Chef runlist.", "Type": "String", "Default": "role[application_server]" },7/20/2012 20
    21. 21. "DomainName": { "Description": "Server Domain Name.", "Type": "String", "Default": “stratalux.com" }, "ChefServerPrivateKeyBucket": { "Description": "S3 bucket containing validation private key for Chef Server", "Type": "String", "Default": "stratalux_cloudformation" }7/20/2012 21
    22. 22. "Mappings" : { "RegionMap" : { "us-east-1" : { "AMI" : "ami-76f0061f" }, "us-west-1" : { "AMI" : "ami-655a0a20" }, "eu-west-1" : { "AMI" : "ami-7fd4e10b" }, "ap-southeast-1" : { "AMI" : "ami-72621c20" } } }7/20/2012 22
    23. 23. • Resources "Resources" :{ "MySimpleImage" : { "Type" : "AWS::EC2::Image", "Properties" : { "ImageId" : "myLinuxBundle-2011-12-30", } } }7/20/2012 23
    24. 24. "ChefClientUser": { "Type": "AWS::IAM::User", "Properties": { "Path": "/", "Policies": [{ "PolicyName": "root", "PolicyDocument": { "Statement": [ {"Effect": "Allow", "Action": ["cloudformation:DescribeStackResource", "s3:Get"], "Resource": "*"}]}}]}}, 7/20/2012 24
    25. 25. "HostKeys": { "Type": "AWS::IAM::AccessKey", "DependsOn": "ChefClientUser", "Properties": { "UserName": { "Ref": "ChefClientUser" } }},7/20/2012 25
    26. 26. "BucketPolicy": { "Type": "AWS::S3::BucketPolicy", "DependsOn": "HostKeys", "Properties": { "PolicyDocument": { "Statement": [{ "Sid": "ReadAccess", "Action": ["s3:GetObject"], "Effect": "Allow", "Resource": { "Fn::Join": ["", ["arn:aws:s3:::", { "Ref": "ChefServerPrivateKeyBucket"}, "/*"]]}, "Principal": { "AWS": { "Fn::GetAtt": ["ChefClientUser", "Arn"] }}}]}, "Bucket": { "Ref": "ChefServerPrivateKeyBucket"}}}, 7/20/2012 26
    27. 27. "AppServerGroup": { "Type": "AWS::AutoScaling::AutoScalingGroup", "Properties": { "AvailabilityZones": { "Fn::GetAZs": ""}, "LaunchConfigurationName": { "Ref": "ChefClient"}, "LoadBalancerNames": [ {"Ref": “appserverstrataluxcom"} ], "MinSize": "2", "MaxSize": "10"}}, 7/20/2012 27
    28. 28. "ChefClient": { "Type": "AWS::AutoScaling::LaunchConfiguration", "DependsOn": "BucketPolicy", "Metadata": { "AWS::CloudFormation::Init": { "config": { "packages": { "rubygems": { "chef": [],"ohai": ["0.6.4"] }, "apt": { "build-essential": [],"ruby": [],"rubygems": [] }},7/20/2012 28
    29. 29. "files": { "/etc/chef/first-boot.json": { "content": { "run_list": { "Ref": "ChefRunList“ }, "mode": "000644", "owner": "root", "group": "root" }}, "/home/ubuntu/.s3cfg": { "content": { "Fn::Join": ["", ["[default]n", "access_key = ", { "Ref": "HostKeys" }, "n", "secret_key = ", { "Fn::GetAtt": ["HostKeys", "SecretAccessKey"] }, "n", "use_https = Truen"]]}, "mode": "000644", "owner": "ubuntu", "group": "ubuntu" },7/20/2012 29
    30. 30. "Properties": { "SecurityGroups": [ { "Ref": “AppServerAccess" } ], "ImageId": { "Fn::FindInMap": ["AWSRegionArch2AMI", { "Ref": "AWS::Region" }, { "Fn::FindInMap": ["AWSInstanceType2Arch", { "Ref": "InstanceType" }, "Arch“ ]}]}, "KeyName": { "Ref": "KeyName" }, "InstanceType": { "Ref": "InstanceType" }7/20/2012 30
    31. 31. "UserData": { "Fn::Base64": { "Fn::Join": ["", ["#!/bin/bash -vn", "function error_exitn", "{n", " cfn-signal -e 1 -r "$1" ", { "Ref": "ChefClientWaitHandle“ }, "chef-client -j /etc/chef/first-boot.json ||”, “error_exit Failed to initialize host via chef client n", "# If all went well, signal successn", "cfn-signal -e $? -r Chef Server configuration ", { "Ref": "ChefClientWaitHandle" }, "n“ }7/20/2012 31
    32. 32. “AppServerAccess ": { "Type": "AWS::EC2::SecurityGroup", "Properties": { "GroupDescription": “AppServerAccess", "SecurityGroupIngress": [ { "IpProtocol": "tcp", "FromPort": "8080", "ToPort": "8080", "CidrIp": "0.0.0.0/0"} ]}},7/20/2012 32
    33. 33. “appserverstrata1uxit": { "Type": "AWS::ElasticLoadBalancing::LoadBalancer", "Properties": { "AvailabilityZones": { "Fn::GetAZs": ""}, "HealthCheck": { "HealthyThreshold": "10", "Interval": "30", "Target": "HTTP:8080/healthcheck", "Timeout": "2", "UnhealthyThreshold": "2" }, "Listeners": [{ "InstancePort": "8080", "LoadBalancerPort": "80", "Protocol": "HTTP", "PolicyNames": [] }]}},7/20/2012 33
    34. 34. "AppServerScaleUpPolicy": { "Type": "AWS::AutoScaling::ScalingPolicy", "Properties": { "AdjustmentType": "ChangeInCapacity", "AutoScalingGroupName": { "Ref": "AppServerGroup" }, "Cooldown": "60", "ScalingAdjustment": "1"} },7/20/2012 34
    35. 35. "AppServerScaleDownPolicy": { "Type": "AWS::AutoScaling::ScalingPolicy", "Properties": { "AdjustmentType": "ChangeInCapacity", "AutoScalingGroupName": { "Ref": "AppServerGroup" }, "Cooldown": "60", "ScalingAdjustment": "-1"} },7/20/2012 35
    36. 36. "CPUAlarmHigh": { "Type": "AWS::CloudWatch::Alarm", "Properties": { "AlarmDescription": "Scale-up if CPU > 75% for 2 minutes", "MetricName": "CPUUtilization", "Namespace": "AWS/EC2", "Statistic": "Average", "Period": "60", "EvaluationPeriods": "2", "Threshold": "75", "AlarmActions": [{ "Ref": "AppServerScaleUpPolicy"}], "Dimensions": [{ "Name": "AutoScalingGroupName", "Value": { "Ref": "AppServerGroup"}}], "ComparisonOperator": "GreaterThanThreshold“ }},7/20/2012 36
    37. 37. "CPUAlarmLow": { "Type": "AWS::CloudWatch::Alarm", "Properties": { "AlarmDescription": "Scale-down if CPU < 75% for 10 minutes", "MetricName": "CPUUtilization", "Namespace": "AWS/EC2", "Statistic": "Average", "Period": "300", "EvaluationPeriods": "2", "Threshold": "75", "AlarmActions": [{ "Ref": "AppServerScaleDownPolicy"}], "Dimensions": [{ "Name": "AutoScalingGroupName", "Value": { "Ref": "AppServerGroup"}}], "ComparisonOperator": "LessThanThreshold" }},7/20/2012 37
    38. 38. "ChefClientWaitHandle": { "Type": "AWS::CloudFormation::WaitConditionHandle“},"ChefClientWaitCondition": { "Type": "AWS::CloudFormation::WaitCondition", "DependsOn": "ChefClient", "Properties": { "Handle": { "Ref": "ChefClientWaitHandle"}, "Timeout": "1200“}} 7/20/2012 38
    39. 39. 7/20/2012 39
    40. 40. 7/20/2012 40
    41. 41. 7/20/2012 41
    42. 42. 7/20/2012 42
    43. 43. 7/20/2012 43
    44. 44. Jeremy Przygodejp@stratalux.comwww.stratalux.com@stratalux Special thanks to JD Hanson for the awesome images!7/20/2012 © 2012 – Stratalux, Inc. 44

    ×