Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DevOps, Common use cases, Architectures, Best Practices

12,589 views

Published on

DevOps, Common use cases, Architectures, Best Practices

Published in: Internet

DevOps, Common use cases, Architectures, Best Practices

  1. 1. DevOps Daniel Zoltak & Shiva N Use Cases,Architectures & Best Practices
  2. 2. Agenda •  Culture •  Technology •  Your experiences
  3. 3. •  Breakdown the barriers •  Work as one team end to end •  Support business and IT agility •  Collaboration & Communication •  Treat Infrastructure as code •  Automate everything •  Test everything •  Measure & monitor everything Agenda… Culture Technology
  4. 4. Competing Forces Business Development Operations Build it faster Keep it stable
  5. 5. Change is inevitable •  Change is the root cause of most outages •  This leaves two approaches: -  Make it a big deal -  Small iterative non events
  6. 6. “Do painful things more frequently, so you can make it less painful…” - Adrian Cockcroft, Architect, Netflix
  7. 7. Deploying More Frequently Lowers Risk Smaller Effort “Minimized Risk” Frequent Release Events: “Agile Methodology” Time Change Rare Release Events: “Waterfall Methodology” Larger Effort “Increased Risk” Time Change
  8. 8. Knowledge is power Requirements Gathering Release Development and Check In Test and QA Some Learning Lots of Learning Minimal Learning Cycle Time
  9. 9. “How long would it take your organization to deploy a change that involves just one single line of code?” “Do you do this on a repeatable, reliable basis?” - Mary & Tom Poppendieck, Lean Software Development
  10. 10. Waterfall Analyze Design Build Test Deploy Agile Sprint Test Deploy Agile-er A D B A D B Sprint Sprint A D B Sprint A D B T D Sprint Sprint A D B T D A D B T D Agile-er Dev QA Ops Dev QA Ops Dev/QA/Ops Dev/QA/Ops Dev/QA/Ops
  11. 11. What is it? •  A philosophy? Cultural change? Paradigm shift ? •  Alignment of development and IT operations with better communication and collaboration ? •  Improvement in software deployment ? •  Breaking down the barriers between development and IT operations ? •  Akin to Agile software development applied to infrastructure and IT operations •  Set of tools and processes •  It’s all of the above! Code W a l l Developer IT Operations DevOps
  12. 12. Building Teams •  Make DevOps everyone’s job - No silo/walled off DevOps teams •  Encourage developers to participate openly in the automation of operations code •  Embolden operations participation in testing and automation of application code •  “Take pride” in how fast and frequent you deploy
  13. 13. DevOps Team Anti Patterns Dev Ops Separate Silos Dev Ops DevOps Separate DevOps Silo Dev Ops DevOps We don’t need Ops http://www.slideshare.net/matthewskelton/devops-patterns-team-topologies
  14. 14. DevOps is a Re-Org! - Adrian Cockcroft, Architect, Netflix RE
  15. 15. DevOps Team Patterns Dev Ops Collaboration Dev Ops Embedded Dev Ops DevOps-as-a-service DevOps Dev Ops DevOps Temporary DevOps team http://www.slideshare.net/matthewskelton/devops-patterns-team-topologies
  16. 16. Business Case Requirements Use Case Features Plan Go to market Business Design Code Refactor Unit Test Bug Fix Deliver Developers (application) Provision Configure Orchestrate Deploy Report Monitor IT Operations (infrastructure) Agile Development DevOps •  Continuous Integration •  Continuous Deployment •  IT Automation •  Application Management •  Iterative development •  Scrum, sprints, stories •  Velocity Business Agility IT Agility Business and IT agility
  17. 17. Collaboration and Communication Standups Pairing Rotation Visibility
  18. 18. Everyone knows what production looks like.
  19. 19. Building a DevOps Culture •  Encourage a fail fast, learn quick mindset •  Foster innovation and accountability •  Promote open and honest sharing of lessons learned •  Build trust across organizational boundaries - Include all the stakeholders •  Expect that it will be a work-in-progress
  20. 20. The DevOps Stack
  21. 21. The DevOps Stack Continuous Deployment Delivery Pipelines Deployment Automation Continuous Integration Automated Testing Configuration Management Agile Communication
  22. 22. •  Infrastructure as code •  IT Automation •  Continuous Integration -  Application §  Compile, test, optimize (code coverage) -  Infrastructure §  Logical, valid, secure DevOps Practices
  23. 23. •  Continuous Deployment -  Application -  Rollout & Rollback •  Version control integration •  Application and Infrastructure version management •  Monitoring and logging DevOps Practices
  24. 24. Continuous Integration •  Every developer check-in initiates a build -  Builds are fully automated -  Tests validate every check-in -  Always check-in to the trunk •  Instant feedback for developers - The build must remain fast •  Minimizes the impact of broken builds - Bugs don’t accumulate •  Drives the culture of small, frequent releases
  25. 25. Continuous Integration Tools
  26. 26. Automated Testing •  Robust, automated testing drives confidence in the entire process •  Testing is not viewed as a “phase” of a project - Testing starts on day one •  View failures as successes •  Don’t overlook testing the infrastructure - Are you ready for the Simian Army? •  Testing is everyone’s responsibility
  27. 27. Configuration Management •  Everything must go into version control •  Implement an effective strategy for branching and merging •  Don’t let tools get in the way
  28. 28. Deployment Automation •  Invest in streamlining and unifying all aspects of the deployment process •  An application’s environment and configuration should be managed through code - Remove any dependencies on human intervention •  Every manual step represents an opportunity for future failure - Console tweaks = future bugs •  Value repeatability - How fast can you completely rebuild your environment from scratch?
  29. 29. One Step Environment Creation •  Require a single build process for all environments - Try to build on horizontal concepts •  Expect the environments to be constantly evolving - Fragile automation becomes fragile environments •  “The longer you wait to have a common environment build process, the harder it is to create one”
  30. 30. Version Control Build/ Compile Code Dev Unit Test App Code IT Ops DR Env Test Env Prod Env Dev Env Application Write App Code Infrastructure CloudFormation tar, war, zip yum, rpm Deploy App Package Applicatio n Deploy application only Deploy infrastructure only AMI Build AMIs Validate Template s Write Infra Code Deploy Infras Automate Deployme nt Artifact Repository Continuous Integration / Deployment & Automation
  31. 31. Delivery Pipelines Commit Unit Test System Test •  Defines the automated lifecycle of delivery •  Enables collaboration between the groups •  Provides everyone with visibility into the flow of change Commit Unit Test System Test UAT Staging ProdCommit Unit Test System Test
  32. 32. Continuous Deployment •  Code that makes it through the pipeline goes straight to production •  Requires continuous integration and delivery •  Relies heavily on small, constantly tested releases •  Often a stretch goal for many organizations
  33. 33. Continuous Deployment vs Delivery
  34. 34. • Deploy in-place - Deploy all at once (Service outage) -  Rolling updates Deployment Approaches Demo
  35. 35. •  Blue-Green Deployment - Discrete environment - Multiple environments from branches - Support A/B testing - “Rolling DNS” •  Alternate Blue-Green (Red-Black?) deployment - Attach and detach Auto Scaling groups from Elastic Load balancers - Avoid messing with DNS Deployment Approaches
  36. 36. Blue-Green Deployment Amazon Route 53 EC2 Instances ELB 100% DynamoDB MySQL RDS Instance ElastiCache Cache Node •  Stand up duplicate infrastructure and slowly cut traffic over to it - Shift via DNS - Simplifies feature testing - Simplifies roll back •  As we shift more traffic over, let auto-scaling grow/shrink our instances - Shut down the old when no traffic there
  37. 37. Blue-Green Deployment Amazon Route 53 EC2 Instances ELB EC2 Instances ELB 90% 10% DynamoDB MySQL RDS Instance ElastiCache Cache Node •  Stand up duplicate infrastructure and slowly cut traffic over to it - Shift via DNS - Simplifies feature testing - Simplifies roll back •  As we shift more traffic over, let auto-scaling grow/shrink our instances - Shut down the old when no traffic there
  38. 38. Blue-Green Deployment Amazon Route 53 EC2 Instances ELB EC2 Instances ELB 0% 100% DynamoDB MySQL RDS Instance ElastiCache Cache Node •  Stand up duplicate infrastructure and slowly cut traffic over to it - Shift via DNS - Simplifies feature testing - Simplifies roll back •  As we shift more traffic over, let auto-scaling grow/shrink our instances - Shut down the old when no traffic there
  39. 39. Blue-Green Deployment •  Stand up duplicate infrastructure and slowly cut traffic over to it - Shift via DNS - Simplifies feature testing - Simplifies roll back •  As we shift more traffic over, let auto-scaling grow/shrink our instances - Shut down the old when no traffic there Amazon Route 53 EC2 Instances ELB EC2 Instances ELB 0% 100% DynamoDB MySQL RDS Instance ElastiCache Cache Node
  40. 40. ASG V1 ELB Red-Black Deployment RDS
  41. 41. Red-Black Deployment ELB ASG V1 ASG V2 UAT RDS
  42. 42. Red-Black Deployment ELB ASG V1 ASG V2 RDS
  43. 43. Red-Black Deployment ELB RDS ASG V2
  44. 44. Agile Everywhere •  Agile tenets are core to all dimensions of the DevOps lifecycle - Small, iterative releases - Quality is everyone’s job - Customer focused - Fast, constant feedback - Done means “released”
  45. 45. Automation Everything •  Automation is vital to smaller, more frequent releases •  All steps that add human intervention also add fragility - “Small compromises can undermine your success” •  Use knowledge gathered from frequent, automated releases to constantly evolve the process
  46. 46. If you have to login to production, your DevOps-fu is sub optimal
  47. 47. •  Scalability (anything manual is not scalable) •  Reliability •  Reproduction/Duplication •  Environment consistency •  Auditability/Record Keeping •  Security •  Governance Infrastructure as code – why?
  48. 48. "WebServer": { "Type": "AWS::EC2::Instance", "Metadata" : { "AWS::CloudFormation::Init" : { "config" : { "packages" : { "yum" : { "httpd" : [], "php" : [], "php-mysql" : [], "php-gd" : [], "php-xml" : [], "php-mbstring" : [], "mysql" : [] } }, "sources" : { "/var/www/html" : "http://ftp.drupal.org/files/projects/drupal-7.8.tar.gz", "/home/ec2-user" : "http://ftp.drupal.org/files/projects/drush-7.x-4.5.tar.gz" }, AWS CloudFormation template Here’s some infrastructure as Code
  49. 49. Describing Infrastructure with Code Developers & Operations Internal Git CI Server Pre-commit Hook Testing Environment Subnet CI Workers Dev Environment VPC Subnet DEV WEB ELBDev Stack Tier 1 Dev Stack Tier 2 Dev MySQL DB Instance DEV APP ELB VPN TUNNEL VPN facing VPC Subnet Internet Gateway VPN Endpoint Dev Admin Instance NAT Instance Amazon S3 Amazon DynamoDB Amazon SQS Amazon CloudFront Amazon Route 53 This entire infrastructure stack can be constructed, configured, and deployed with code:
  50. 50. Infrastructure Code The code that describes infrastructure should inherit the same values applied to application code •  Not JUST revision control •  Make use of bug tracking/ticketing systems •  Peer reviews of changes before they happen •  Establish infrastructure code patterns/designs •  Test infrastructure changes like code changes
  51. 51. Declarative Approach to: •  Provisioning •  Configuration •  Orchestration •  Reporting Elastic Beanstalk CloudFormation OpsWorks Automation and configuration management
  52. 52. Automation Tools AWS Tools •  CloudFormation •  Elastic Beanstalk •  OpsWorks Third-Party Tools •  Chef •  Puppet
  53. 53. Monitoring & Metrics : The DevOps Pulse •  Track every resource possible •  Alert on services, availability, and response times •  Capture, learn, and improve •  Share access with everyone on the team •  Plan metrics and monitoring into the DevOps lifecycle
  54. 54. If it moves graph it, or it didn’t happen
  55. 55. Monitoring/Logging Tools •  Logging - Logstash Ø  Check out ELK! -  Sumo Logic -  Splunk -  CloudWatch logs •  Metrics - CloudWatch - Ganglia - StatsD + Graphite + Grafana - NewRelic -  Monitoring - Nagios - Munin - Sensu - Hyperion
  56. 56. Discussion
  57. 57. Thank You

×