Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Our Journey To Continuous Delivery

Presentation for ITAKE Unconference 2015

  • Login to see the comments

  • Be the first to like this

Our Journey To Continuous Delivery

  1. 1. Virgil Chereches & Robert Mircea May 28th, 2015 Our Journey to Continuous Delivery
  2. 2. 2 “How long would it take your organization to deploy a change that involved just one single line of code? Do you do this on a repeatable, reliable basis?” Mary and Tom Poppendieck, Implementing Lean Software Development
  3. 3. 3 “Failure is not an option”  Learning opportunity  Detectable quickly  Restore service quickly Failure is inevitable Orange
  4. 4. 4 IT Performance and DevOps Practices “Our data shows that IT performance and well- know DevOps practices, such as those that enable continuous delivery, are predictive of organizational performance. As IT performance increases, profitability, market share and productivity also increase. IT is a competitive advantage, not just a utility, and it’s more critical now than ever to invest in IT performance.”
  5. 5. 5 The Need for Continuous Delivery Agility for our Business Eliminate wasted time due to hand offs between teams
  6. 6. 6 Enable Architectural Decomposition of Systems
  7. 7. 7 Do Continuous Delivery to be Ready for… Microservices Swarm
  8. 8. 8 Share the Vision of CD Spread the “WHY” the “WHAT” ensure they “GET IT” Started organic, inspired by a few enthusiasts. Understand that it is about changing the way we work
  9. 9. 9 How We Joined the Army Dev Ops Enthusiasts 7 x DevOps Wannabes One project One opportunity R&D style Managers jumped on board from the start DevOps
  10. 10. 10 What We Want to Achieve Push to release at Orange scale
  11. 11. 11 JBoss cluster standalone JBoss cluster standalone v1.0 v1.1 JBoss cluster standalone v1.1 … VMWare VCenter Infrastructure JBoss cluster standalone v1.0 ProdDev/UAT v1.1 commit
  12. 12. 12 From What We Start Business requests in the hundreds/year 200+ apps in Java, C#, C, a few PHP. Lots of PL/SQL ~150 people – Dev, QA, BA, Ops, PM Mélange of proprietary and lots of custom development apps
  13. 13. 13 What is Continuous Delivery? Small, frequent changes delivered constantly A method to reduce risk of failure A way to give to our business a glimpse of the shape of product being built A set of practices enabled by a toolset
  14. 14. 14 Many deployments eventually lead to a Product Launch What is the difference between a Deployment and a Product Launch?
  15. 15. 15 Changes flowing through deployment pipeline Push the button “Approval”
  16. 16. 16 Orange Deployment Pipeline Architecture (Atlassian Stash Git / SVN) (Artifactory) (Jenkins/Teamcity/Sonar) (Selenium) (Enterprise Tester) (JMeter) Rundeck /Puppet
  17. 17. 17 Continuous Integration Server Purpose: detect code problems or quality issues quickly
  18. 18. 18 Best Practices for CD  build application only once and use it in all environments  keep application releasable all the time – master is releasable at all times  hide new functionality until it is finished – use feature flags  make small changes incrementally – commit and integrate often  use “branching by abstraction” – use a “proxy” component  use snapshots (in Maven) with care
  19. 19. 19 Techniques for achieving Continuous Deployment
  20. 20. 20 Blue-Green Deployments v1.0-WL App Server v1.0-Jboss Weblogic JBoss
  21. 21. 21 Feature flags Integrating with production is a test  On/Off  Orange IPs  List of Users  Time interval  Release date  Gradual Rollout (%)  Blacklist/whitelist IPs Enable/disable features quickly (no restart) ORO testers check feature in live
  22. 22. 22 Prefer Non-Breaking Expansion in Database Database migration tools FluentMigrator (.NET)
  23. 23. 23 Code Based Migration - FluentMigrator Run migration .migrate --conn "server=oracle-server; database=SmsDb" -db OracleManaged -a "..MigrationsMigrations.dll" --task migrate:up Rollback change .migrate --conn "server=oracle-server; database=SmsDb" -db OracleManaged -a "..MigrationsMigrations.dll" --task migrate:down
  24. 24. 24 Automated tests AFTER deploy Run smoke tests automatically to validate deployment success Examples:  make a HTTP request to homepage and look for a string  login with a test user  make a recharge with a test PrePay
  25. 25. 25 Metrics Server Metrics  CPU utilization  disk space Application Metrics  JMS messages sent/received,  memcached hits Business Metrics  Logins / Registration  Orders  Recharges  Options activations
  26. 26. 26 Naming Metrics Establish consistent rules in naming metrics Env App name Server name Module name Message type Target Action prod.apps.maco.platini.jms.vantive.customers.request.sent
  27. 27. 27 Graphite & Grafana - real-time metrics
  28. 28. 28 Instrumenting Code for Metrics Collection Trivial counters with Statsd ( Types of counters:  Counters  Timers  Gauges  Histograms
  29. 29. 29 Centralized Logging Apache Logs App logs Logstash Lumberjack Logstash Log shipping Logstash (parse log into fields) Elasticsearch (index logs) Statsd + Graphite (metrics dashboard) (log search & analysis)
  30. 30. 30 Infrastructure Automation
  31. 31. 31 “At an abstract level, a deployment pipeline is an automated manifestation of your process for getting software from version control into the hands of your users.” Deployment Pipeline
  32. 32. 32 Provisioning managers Configuration managers Workflow managers
  33. 33. 33 Configuration Manager Choice
  34. 34. 34 Software components HIERA
  35. 35. 35 Coding best practices Data/code separation r10k HIERA Keep code in version control reaktor Unit & acceptance testing Development workflow feature branch workflow puppet-rspec server-spec
  36. 36. 36 Node breeds Several items give node breed:  puppet role class Foreman host group (one-to-one relationship)  stage (dev/prod/qa/test)  Foreman compute profile (compute provider, CPUs, RAM, Disk)
  37. 37. 37 Tag-based everything Nodes Workflow steps Security
  38. 38. 38 Node herds Pre-provisioned nodes Shorten deployment time Breed matters Hostname does not matter
  39. 39. 39 Don’t be afraid to get dirty Rundeck API Foreman API PuppetDB API BigIP iControl Atlassian Stash API
  40. 40. 40 Scaling Puppet & Foreman
  41. 41. 41 What’s Next • Sponsor Rundeck development for SCM integration and sensitive options encryption • Evaluate network orchestration • Containers • Acceptance testing of puppet code with beaker (rspec+serverspec)
  42. 42. multumim