Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Machine Learning for Continuous Delivery


Published on

Let our friends, the machines, help with building confidence in your Continuous Delivery pipeline. We can compare doughnuts to software to make this more delicious.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Machine Learning for Continuous Delivery

  1. 1. Continuous Delivery & Machine Learning – SEA Edition Ravi Lachhman @ravilach
  2. 2. Who am I? <3 SRE
  3. 3. 5 Dozens of Flavors People in need of SUGAR
  4. 4. Eaten No Negative Yelp Reviews More are bought What is Winning Doughnuts?
  5. 5. Doughnut Deployment Pipeline Idea You VerifyTestBuild Manual RollbackDeploy Yum New Treat Hours MinsDays Mins Ask co-workers about taste Blast on Social Media Gather Raw Material Prepare New Doughnut Flavor Eat / Instagram Doughnut Yelp / Google Map Krispy Kreme Subjective Look Here Ouch
  6. 6. Wait we are in Software…
  7. 7. Software Deployment Pipeline Code Customer Verify Manual Test Kinda- Automated Build Automated Rollback Manual #WTF Deploy Manual Custom Scripts Custom Scripts Custom Scripts HoursMins Hours/DaysChange
  8. 8. One BIG Difference
  9. 9. Copyright © Harness 2018 Confidential 11 Deploying & Verifying Apps Is A Little Different Press Monthly Deployment Console
  10. 10. Confidential Traditional Verification “Application is up so everything is fine” “Homepage Works” “Not seeing an alerts” “No-one has complained yet”
  11. 11. Continuous Delivery = Automate Everything Dev Stage 1 Static Code Analysis Unit Tests Integration Tests Depend/license Tests Deployment Pipeline CustomerCode QA Stage 2 Smoke Tests Regression Tests Security Tests Browser Tests Load/Perf Tests On Success Production Stage 3 Availability Tests Performance Tests Quality Tests Security Tests Canary Verification On Success On Commit
  12. 12. Enter Machine Learning
  13. 13. Disclaimer: Machine Learning Is Not A Silver Bullet (But its better at doing some things better than humans)
  14. 14. Unsupervised Machine Learning Algorithms identify meaning and context of data on its own E.g. Natural Language Processing (NLP), Neural Nets Supervised Machine Learning Algorithms require human assistance Neural Feedback (Ignore, Dismiss, Ratings, Yes, No) NOT….Rules Regex, Thresholds, Configuration or Standard Deviation! Machine Learning 101
  15. 15. Applying Machine Learning Performance Quality Security Availability Deploy Rollback Continuous Verification Performance Quality Business Availability Unsupervised Machine Learning Automate Deployment Test/Verify/Rollback $ Revenue / Throughput
  16. 16. Context Application Artifact (Service) Infrastructure Deployment Context, Control and Test Data Control Data (Baseline) Previous Tests Previous Deployments Previous Canary Phases Test Data (New) New Tests New Deployments New Canary Phases
  17. 17. Verify your Metrics + Custom Metrics
  18. 18. Confidential 20 Performance Good
  19. 19. Confidential 21 Performance Regression
  20. 20. Revenue Regression
  21. 21. Metric = Timestamp + Name + Value E.g. “1/1/19 10:01:34, CPU, 99.9%” This is what Dynatrace, AppDynamics, New Relic, Datadog and Prometheus Provides Analyze Time-Series Metrics from APM/Monitoring
  22. 22. Symbolic aggregate representation (SAX) Machine Learning Algorithms Used (time-series)
  23. 23. Machine Learning Algorithms Used (time-series) Hidden Markov Models Predict when data is incomplete
  24. 24. Verify your Logs
  25. 25. Confidential 27 Stack Trace Error Regressions Quality Regression
  26. 26. Copyright © Harness 2018 Confidential 28 KMeans Clustering Group Events By Natural Language Similarity
  27. 27. Automate Verification (Unstructured Events) Usual crap AnomaliesUsual crapAnomalies
  28. 28. Entropy (Frequency of Attributes) 0 1 High EntropyLow Entropy
  29. 29. Application Log File has 100 Events 10 of those are DoughnutNotFound Exceptions (Cluster) DNF are thrown in 10 different places of the application (different positions relative to each other) Simple Example
  30. 30. Jaccard & Cosine Distance
  31. 31. Simple Example DoughnutNotFoundException ArraryOutOfBoundsException BusinessException ClassNotFoundException
  32. 32. KMeans Clustering isn’t Perfect Event Data can contain noise (guids, urls, ip addresses, cookies, …)
  33. 33. Use Trained Neural Nets to Reduce False Positives Non-Neural Net This is why Engineers Don’t Trust Machine Learning Trained Neural Net
  34. 34. Rollback in 32 Seconds
  35. 35. Build confidence, let machines help Fold early, fold often Doughnuts are amazing What did we learn?
  36. 36. Copyright © Harness 2019 Confidential 38 Questions? Free 15-Day Trial at
  37. 37. Grab the slides! Free 15-Day Trial at