Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

EC2 AMI Factory with Chef, Berkshelf, and Packer

12,061 views

Published on

Presentation accompanying a Live Demo at the AWS Pop-Up Loft in San Francisco on using Chef + Berks + Packer to create an AWS EC2 AMI Factory.

Demo Repo available here -- https://github.com/gmiranda23/chef-ami-factory

  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://soo.gd/qURD } ......................................................................................................................... Download Full EPUB Ebook here { https://soo.gd/qURD } ......................................................................................................................... Download Full doc Ebook here { https://soo.gd/qURD } ......................................................................................................................... Download PDF EBOOK here { https://soo.gd/qURD } ......................................................................................................................... Download EPUB Ebook here { https://soo.gd/qURD } ......................................................................................................................... Download doc Ebook here { https://soo.gd/qURD } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THIS can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THIS is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THIS Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THIS the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THIS Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL eBOOK INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF eBook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB eBook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc eBook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. PDF eBook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB eBook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc eBook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, CookeBOOK Crime, eeBOOK Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL. BOOKS INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

EC2 AMI Factory with Chef, Berkshelf, and Packer

  1. 1. AWS Provisioning with Chef Creating an AMI Factory George Miranda Partner Integration Engineer AWS Pop-up Loft June 23, 2014
  2. 2. Introductions • George Miranda ! • Partner Integration Engineer • Consultant • Infrastructure guy ! • gmiranda@getchef.com • gmiranda23 Twitter | GitHub | IRC | *:*
  3. 3. Deploying to AWS EC2 • A pretty BIG topic • FAQ: where does Chef fit? • Here’s what we’ll cover • Brief intro to Chef • Basic intro to provisioning options • AWS specific challenge: auto-scaling • Focus on AMI management • Live Demo • Next Steps
  4. 4. Chef Basics In 90 seconds or less • Configuration Management & Automation Framework • Scalable, Extensible, Testable • Infrastructure as Code • Data Driven Cookbooks • Run the same code: Dev/Stg/Prod, On-Prem or AWS • “Recipes” are a collection of “resources” • “Cookbooks” are a collection of “recipes” • 1,400+ Community Cookbooks Available • “Roles” are an ordered list of recipes to apply • “Nodes” can have one or more “roles” • “Knife” is a Chef’s go-to tool
  5. 5. Provisioning a node with knife-ec2 ! knife  ec2  server  create  -­‐r  "role[webserver]"  -­‐I  ami-­‐2d4aa444          -­‐-­‐flavor  m1.small  -­‐S  aws_ssh_key  -­‐Z  us-­‐east-­‐1b  -­‐N  my_webserver1 • Provisions a new m1.small in us-east-1b • Waits for instance-id & network settings from AWS • Waits for ssh or winrm • Issues remote commands • Install Chef • Configure Chef • Run chef-client
  6. 6. Provisioning with chef-metal require  'chef_metal_fog'   ! with_machine_options  :bootstrap_options  =>  {      :key_name  =>  'aws_ssh_key',      :image_id  =>  'ami-­‐2d4aa444',      :flavor_id  =>  'm1.small',      :region  =>  'us-­‐east-­‐1b'   }   ! machine  'my_database1'  do      role  'database'      tag  'mysql_master'   end   ! num_webservers  =  5   ! 1.upto(num_webservers)  do  |i|      machine  "my_webserver#{i}"  do          role  'webserver'    end   end • Declare infrastructure topology in a recipe • Configure multiple VM "drivers" • Ensure my_database1 is present • Then ensure 5 web servers are present • Self-healing (convergent) • Version your infrastructure • Almost 1.0.0
  7. 7. {        ~500  lines  of  JSON   !      "UserData"  :  {  "Fn::Base64"  :  {  "Fn::Join"  :  ["",  [                "#!/bin/bash  -­‐vn",                "function  error_exitn",                "{n",                "  cfn-­‐signal  -­‐e  1  -­‐r  "$1"  '",  {  "Ref"  :   "ChefClientWaitHandle"  },  "'n",                "exit  1n",                "}n",     !            #{Bunch  of  stuff  to  get  Chef  installed/configured}   !            "#  If  all  went  well,  signal  successn",                "cfn-­‐signal  -­‐e  $?  -­‐r  'Chef  Server  configuration'   '",  {  "Ref"  :  "ChefClientWaitHandle"  },  "'n"            ]]}},   !      ~500  more  lines  of  JSON   } Provisioning with CloudFormation
  8. 8. Why CloudFormation? JSON All The Things! • Create new Security Groups • Provision EC2 stack topology • Elastic Load Balancers • Elastic IPs • Provision AWS PaaS connections • RDS • SQS • S3 Buckets • etc • Create EC2 Auto-Scaling Groups
  9. 9. The Auto-Scaling Challenge
 Meeting Demand Fast • Typically triggered by utilization alerts • You are already falling behind by the time your infrastructure responds • You need capacity… FAST! • You already lose response time during instantiation • 3-5 minutes is an eternity when your application is choking
  10. 10. • Flexibility • Managing Configuration Drift • Rapid incremental change • Avoid golden image sprawl • Systems are not always disposable • Avoid asynchronous VM configuration • Infrastructure as Code Deploying fully-baked AMIs
 Why Would You Still Use Configuration Management?
  11. 11. Pre-Loaded AMIs
 Flexibility and Performance • Bake in the most expensive components • Minimal Configuration occurs after initialization (e.g. systems integration tasks) • Manage your pre-loaded AMI creation in a Chef Recipe • Chef Recipes are idempotent • Include those same recipes in your node’s run_list
  12. 12. Today’s Demo
 A Live Demo? SRSLY? • Creating an AMI Factory with Chef • Chef Recipe • Packer • Berkshelf ! • For today’s Demo, we’ll pre-bake a simple Nginx Web Server* ! * (you would probably never bake in something that inexpensive in “the real world”)
  13. 13. Packer • Packer is a tool for creating identical machine images for multiple platforms from a single source configuration • A HashiCorp Project • http://www.packer.io
  14. 14. Berkshelf • Berkshelf is a bundler-like dependency manager for Chef cookbooks • Encourages treating your cookbooks as Libraries or Applications • Makes it very easy to automatically download and use Chef Community Cookbooks • Started at Riot Games • http://berkshelf.com
  15. 15. Live Demo Creating a Pre-Loaded AMI Instance-Store AMI & EBS-Backed AMI: building in parallel
  16. 16. A Tour of Today’s Demo
 The Recipe include_recipe  'apt'   include_recipe  'nginx'   ! directory  '/var/www/nginx-­‐default'  do      owner  'www-­‐data'      group  'www-­‐data'      mode  '0755'      recursive  true      action  :create   end   ! file  '/var/www/nginx-­‐default/index.html'  do      owner  'www-­‐data'      group  'www-­‐data'      mode  '0755'      content  'Hello  World  from  the  AWS  Pop-­‐up  Loft!'      action  :create   end
  17. 17. A Tour of Today’s Demo
 The Berksfile site  :opscode   ! cookbook  'apt'   cookbook  'nginx'   ! metadata
  18. 18. A Tour of Today’s Demo
 The packer.json file — building Instance-Store AMIs    "builders":  [          {  "type":  "amazon-­‐instance",              "access_key":  "{{user  `aws_access_key_id`}}",              "secret_key":  "{{user  `aws_secret_key`}}",              "region":  "us-­‐east-­‐1",              "source_ami":  "ami-­‐6f969506",              "instance_type":  "m1.xlarge",              "ssh_username":  "ubuntu",              "ami_name":  "packer-­‐instance_store_nginx_{{timestamp}}",   !            "bundle_vol_command":  "sudo  -­‐n  #{very  long  ami  tool  e2-­‐bundle-­‐vol  command}”,              "bundle_upload_command":  "sudo  -­‐n  #{very  long  ami  tool  e2-­‐upload-­‐bundle  command}”,   !            "account_id":  "{{user  `account_id`}}",              "s3_bucket":  "{{user  `s3_bucket`}}",              "x509_cert_path":  "{{user  `x509_cert_path`}}",              "x509_key_path":  "{{user  `x509_key_path`}}"          },
  19. 19. A Tour of Today’s Demo
 The packer.json file — building EBS-backed AMIs builders  (continued)   !        {  "type":  "amazon-­‐ebs",              "access_key":  "{{user  `aws_access_key_id`}}",              "secret_key":  "{{user  `aws_secret_key`}}",              "region":  "us-­‐east-­‐1",              "source_ami":  "ami-­‐2b0b1442",              "instance_type":  "m1.xlarge",              "ssh_username":  "ubuntu",              "ami_name":  "packer-­‐ebs_nginx_{{timestamp}}"          }      ],
  20. 20. A Tour of Today’s Demo
 The packer.json file — provisioners    "provisioners"  :  [          {  "type":  "chef-­‐solo",              "cookbook_paths":  [  "../vendor/cookbooks"  ],              "run_list":  [  "packer::default"  ]          },   !        {  "type":  "shell",              "inline":  [                  "sudo  apt-­‐get  install  -­‐y  ruby  unzip  kpartx",                  "wget  http://s3.amazonaws.com/ec2-­‐downloads/ec2-­‐ami-­‐tools.zip",                  "sudo  mkdir  -­‐p  /usr/local/ec2",                  "sudo  unzip  ec2-­‐ami-­‐tools.zip  -­‐d  /usr/local/ec2",                  "sudo  mv  /usr/local/ec2/ec2-­‐ami-­‐tools-­‐*  /usr/local/ec2/ec2-­‐ami-­‐tools/"              ],              "only":  [  "amazon-­‐instance"  ]          }    ]
  21. 21. A Tour of Today’s Demo
 The packer command (script) #!/bin/bash   rm  -­‐r  ../vendor/cookbooks   berks  install  -­‐-­‐path  ../vendor/cookbooks   packer  build        -­‐var  "account_id=$AWS_ACCOUNT_ID"        -­‐var  "aws_access_key_id=$AWS_ACCESS_KEY_ID"        -­‐var  "aws_secret_key=$AWS_SECRET_ACCESS_KEY"        -­‐var  "x509_cert_path=$AWS_X509_CERT_PATH"        -­‐var  "x509_key_path=$AWS_X509_KEY_PATH"        -­‐var  "s3_bucket=chef-­‐packer-­‐bucket"        packer.json
  22. 22. Live Demo Checkpoint Creating a Pre-Loaded AMI Instance-Store AMI & EBS-Backed AMI: building in parallel
  23. 23. Ready to Use Our AMIs
 Run it the best way for your environment • Use these AMIs in your CloudFormation Templates • Define UserData to run "chef-client -j run_list.json" • Example run_list.json { "run_list": [ "role[webserver]" ] } ! • Leave integration tasks to Chef • Recommended: run chef-client periodically
  24. 24. Pitfalls
 Or How I Learned to Stop Worrying and Not Rage Against Pre-Baked AMIs • Beware Image Sprawl • Pre-Load AMIs only for critical Autoscaling Instances • Just Enough OS everywhere else • Generate AMIs with a CI Pipeline • Purge previous AMIs methodically • Cycle running instances methodically ! • Note: I am NOT suggesting everyone start making pre-loaded AMIs
  25. 25. Further Reading Next Steps From here • https://github.com/gmiranda23/chef-ami-factory • Unit testing your AMIs (e.g. serverspec) • AWS Chef Cookbook • ebs_volume • ebs_raid • elastic_ip • elastic_lb • resource_tag • http://packer.io • http://community.opscode.com/cookbooks • http://learnchef.com/
  26. 26. Thank You! George Miranda ! gmiranda@getchef.com @gmiranda23 ! Demo Repo — https://github.com/gmiranda23/chef-ami-factory ! Packer Demo primer found at — http://engineering.cotap.com

×