Real World AWS Deployment With Boto, Fabric, and Cloud Formation


Published on

A real world use case of spinning up EC2 instances using a Python based frameworks (Fabric and Boto) and Cloud Formation. Source code is available on

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • IaaS – use the cloud in an “elastic fashion”, resources on demand. Turn it off when you don’t need it!Regions/availability zones – redundancy and availability. PD stack spread across multiple availability zonesAmi - “disk images” for your virtual machines (EC2 instances). Keep ‘em simple. We opt to
  • As pertains to demo:EC2 - mongo instancesEBS - root and data storageElastic IP – makes life easier with mongo rs starting and stopping and for failoverImportance of tagging – (Region Code)_(Environment)_(Tier) used as a selectorSecurity Groups – mongo: port 22, and 27017 internallyAccessing Instances – SSH
  • servers serving same function with different OS’s, software installation locations, ssh keys, etc
  • From the Zen of PythonIf the implementation is hard to explain, it’s a bad idea.If the implementation is easy to explain, it may be a good idea.Great API, easy to use and if used correctly trends towards “good lookin’” code
  • - Note that connections are to REGIONS use of metadata as filters
  • Fabric is
  • Tasks are run against specific environments. Can be a single or multiple environments. Also supports parallel execution.
  • The “bene” – you don’t write any code.Allows you to allocate AWS resources in a declarative fashion w/JSON.Composition – have a template for Mongo Replica Set which references a Data Node template and Arbiter template. Reduces duplication and indirectly “consolidates” the configuration.Can do a lot with thempackage support (apt-get, yum, git)Execute bash commands (whatever scripting environment is supported in the shell)
  • Verbose, but rather simple
  • All in one file . . .bash, package management in JSON stringsRegion specific if you want to support multiple regions you’ll need to have copies of the template in each region. Also each template must be configured to work in any enviornment, unless you want tohave separate copies of the template for each region (doesn’t scale)
  • AWS UI for CF template parameters
  • Real World AWS Deployment With Boto, Fabric, and Cloud Formation

    1. 1. Real World AWS DeploymentDeploying MongoDB with Fabric, Boto, and Cloud Formation{}, @nixonhughes
    2. 2. Who Am I ?Software Engineer @ PokitdokTasked with scaling out infrastructureDeveloped custom provisioning process for AWS (Boto, Fabric, Cloud Formation)
    3. 3. What we’ll cover . . .Amazon Web Services Overview (EC2, ESB, Regions, etc)REALISTIC provisioningBuilding a provisioning solution with Boto, Fabric, and Cloud Formation to deploy a MongoDB Replica Set
    4. 4. AWS OverviewCloud Computing and Infrastructure as a Service (Iaas)AWS Regions and Availability ZonesAmazon Machine Images (AMI)
    5. 5. AWS Overview (continued)Core Services Elastic Compute (EC2) Elastic Block Storage (EBS) Elastic IPTaggingSecurity GroupsAccessing Instances
    6. 6. Realistic ProvisioningBase process off of configuration standards: directory layouts, users, etc.Processes are repeatable (scripted) but may not be 100% automatedIterate to improve automation %
    7. 7. the alternative . . .
    8. 8. Boto – Python Interface for AWSHosted on githubActivate developer community on Google GroupsSupports additional services such as OpenStack, Eucalyptus, and Google Storage
    9. 9. Boto ExamplesStop all instances in us-west-2Get all running Mongo Instances
    10. 10. Fabric – Python SSH LibraryHosted on githubEncapsulates common ssh/scp operationsExecutable tasks are plain ‘ol Python functions decorated with @task
    11. 11. Fabric ExamplesStart servicesCommand line task execution
    12. 12. AWS Cloud FormationDeclarative provisioning and deployment tool based on JSON templatesSupports composition/aggregation with template referencesTemplates are parameterized, may include scripts, and support updates
    13. 13. AWS Cloud Formation Snippet
    14. 14. The Dark Side of Cloud FormationUser Data, BASH, and JSON
    15. 15. The DesignCloud Formation is used for provisioning not configurationBoto creates AMIs and configures instancesFabric as an execution framework
    16. 16. The ProcessCreate AMIs In the appropriate regionsUpload Cloud Formation templates to S3Spin up instances using Cloud FormationConfigure instances with Fabric
    17. 17. Create AMIsUse the command:fab create_amiSupports saving AMI to multiple regionsDifferentiates between general and specific functions using ami_typeUpdate CF Templates with AMI ID
    18. 18. Upload to CF Templates to S3Publish templates using:fab push_cf_templates_to_s3Templates are stored in S3 buckets per regionPublish for changes/updates to templates
    19. 19. Cloud Formation
    20. 20. Wrap-UpDoes this scale?Python/Boto ease of us vs Cloud Formation’s declarative verbosityNext iteration – fabric toolchain wrapping boto APIs
    21. 21. FREE CODE !!!!! No copyright No terms of use No strings attached
    22. 22. Questions?