Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Ship code like a keptn

41 views

Published on

Our team just released Keptn (https://keptn.sh/), an open source framework for event-based, automated continuous operations in cloud-native environments. In this session, we will talk about WHY we built Keptn, HOW we implemented it (Architecture) and where we want the community to take it.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Ship code like a keptn

  1. 1. Shipping code like a @keptnProject https://keptn.sh https://github.com/keptn Rob.Jahn@Dynatrace.com Technical Partner Manager & Keptn Advocate
  2. 2. 2 Every CIOs vision of their Cloud Native investment
  3. 3. 3 The unfortunate reality for most…. 1. Complexity 2. Manual operations 3. Lack information 4. People 5. Identifying root cause
  4. 4. 4* based on our ACM survey
  5. 5. 5 Maturity Questions Mean Time to Innovation (MTTI) 1. Are you on two week sprints?​ 2. Do you also release 25 times a year?​ 3. How long does it take you to deploy a new release to production?​ 4. Do you measure feature adoption? Mean Time to Remediate (MTTR) 1. How long does it take to rollback/roll forward a faulty release?​ 2. To release any software in your organization, how many manual steps does it take?​
  6. 6. 6 MTTI Mean Time to Innovation MTTR Mean Time to Remediate 4.8 days 4 hours ~ 10min 12.5 days 2 days ~ 1 hour The reality and evidence supports it! https://dynatrace.ai/acsurvey Only < 5% is „Cloud Native“
  7. 7. 7 That is why we are building Because cloud native delivery and operations is a BIG challenge for enterprises! Cloud Native
  8. 8. 8 . https://dzone.com/articles/keep-calms-and-devops-c-is-for-culture Culture first, but change is not easy…
  9. 9. 9 Common Pipeline delivery challenges 1. Pipelines are semi-automated 2. Each team building duplicate delivery & integration mechanisms 3. Changing tooling causes a ton pipeline re-work 4. Lack software architecture for DevOps automation 5. Lack auditing support for customized tool integrations 6. Lack of standards 7. Too many technologies to keep up with 8. What else ????
  10. 10. 10 So what is this thing? • Open source initiative to make continuous delivery and operations automation a commodity for modern cloud-native applications • Enterprise-grade event-based framework for shipping and operating cloud- native applications built to run on any Kubernetes cluster • With Keptn ... • Developers can focus on code instead of creating YAML files • DevOps can focus on tools instead of building pipelines • SREs can enforce processes instead of debugging problems
  11. 11. 11 Automated OperationsContinuous Deployment Core Capabilities -- accelerates building autonomous clouds Event-drivenrunbook automation Productionproblemscan beautomatically remediatedin real-timebyexecuting runbooksthatrequirenomanual intervention. Self-healing blue/green deployments Deploymentsthatfollowthe“Operationsas Code”paradigmautomaticallyremediate problemsandgetyourdeploymentpipeline workingagainin underaminute. Automatedmulti-stage unbreakable pipelines GitOps-enableddeliverypipelineswith automatedqualitygatessupportautomated testingandmonitoring-as-a-service.
  12. 12. 12 First: solves the Continuous Delivery Problem! Many approaches today are brittle, error-prone and hard to maintain
  13. 13. 13 Quote: “We spend more time in manual communicaton than remediating issues“ Second: has Continuous Operations at its Core! ENGAGE TRIAGE FIND & ASSEMBLE RESOLVE RESTORE MANUAL COMMUNICATION MANUAL COMM Before After RESTORERESOLVE NUMBER OF ISSUES BEFORE: mostly manual AFTER: mostly automated
  14. 14. 14 Increase Quality & Level of Automation Increase Speed & Reduce Costs Automated Testing Continuous Performance Auto Quality Gates AUTOMATE OPERATIONS AUTOMATE DEPLOYMENT AUTOMATE MONITORING Feature Flagging Adaptive Scaling Auto Roll-Back Canary Releases Blue / Green Deploymen ts Auto- Remediation AUTOMATE QUALITY Strategically Used as Pipeline Feature Building Blocks for ACM/Cloud Natives!
  15. 15. 15 Design Principals – follows patterns for modern applications GitOps-based collaboration Operator patterns for all logic components Monitoring and operations ascode Built onandfor Kubernetes Event-drivenandserverless Pluggabletooling
  16. 16. 16 Keptn architecture Platform keptn cli api eventbroker REST API bridge datastore gitconfiguration-service internal- keptn- services distributor distributor keptn services Continuous Delivery Container Registry Operations Automation AIOps GitOps Test Automation ChatOps Data Provider dev staging production keptnNamespaces: distributor Mini-kube AWS EKS Azure AKS Google GKE Openshift Pivotal PKS
  17. 17. 17 Keptn Use Cases - Installation 1. One-Line Installation: on most popular k8s platforms 2. Zero-Touch Toolchain Integration: No custom tool integrations needed 3. Zero-Touch Cloud Native Services: Enables GitOps event-driven CD/CO for your services
  18. 18. 18 Config ChatOps IT Autom Deploy Test Observe One-Line Installation: $ keptn install --platform=[aks, gke, openshift] 1. Keptn installer pod 2. Keptn pods and services(such as logging, keptn bridge,event broker) 3. Keptn routes (such asauthenticator, bridge, gate-keeper,helm service) 4. Keptn channels (such as configuration-changed, deployment-finished, evaluation-done) 5. Istio and knative pods
  19. 19. 19 Config ChatOps IT Autom Deploy Test Observe Zero-Touch Toolchain Integration: $ keptn wear uniform uniform.yaml uniform.yaml services: - name: "dockerhub" - name: "jenkins-service" subscribedchannels: - new-artifact - deployment-finished -name: "slack-service”
  20. 20. 20 Zero-Touch Toolchain Integration: Config ChatOps IT Autom Deploy Test Observe Pluggable tooling All tools leveraged by keptn can be replaced based on your tool preferences.
  21. 21. Keptn Orders Demo -- tool chain ”uniform” Config ChatOps IT Autom Deploy Test Observe • Configuration:GitHub • Deploy:Helm • Chat:Slack& MS Teams • IT Automation:None • Test:Jmeter • Observe:PitometerQualityGates
  22. 22. Keptn Orders Demo Setup DockerRegistrywithpre-builtimages GitHub Org Staging Production Update GitHub repositories - front-end - catalog-service - customer-service - catalog-service Dev https://github.com/keptn-orders
  23. 23. 23 Keptn Use Cases – Continuous Delivery 1. Re-Think Pipelines: Gone are the days of custom pipeline coding! 2. Automated Multi-Stage Delivery: Risk-Free auto deployment through multi-stage delivery pipelines 3. Automated Quality Gates: stops bad changes before production using Pitometer 4. Self-Healing Blue/Green Deployments: reverts bad changes before impacting end- users 5. Zero-Touch Toolchain Updates: Add/Remove/Replace tools without custom coding 6. Mastering Continuous Delivery: Risk-Free Automated Deployments
  24. 24. 24 Config ChatOps IT Autom Deploy Test Observe Re-Think Pipelines: $ keptn create project keptn-sample shipyard.yaml S T A G I N G P R O D DirectUpdateC D Blue/GreenUpdateC D
  25. 25. 25 Config ChatOps IT Autom Deploy Test Observe Zero-Touch Cloud Native Services: $ keptn onboard service myservice [values.yaml] S T A G I N G P R O D DirectUpdateC D Blue/GreenUpdateC D PLACEHOLDER PLACEHOLDER
  26. 26. As a developer, I want to safely deploy a new version by just providing my docker image
  27. 27. 27 Config ChatOps IT Autom Deploy Test Observe Automated Multi-Stage Delivery: $ keptn new artifact myservice:1.0.0 S T A G I N G P R O D ScoreDirect PerformanceUpdate Promote?C D T O ScoreBlue/GreenUpdate Keep?C D T O PLACEHOLDER PLACEHOLDER 1.0.0 1 1 90 / 100 1.0.0 1 1 1 75 / 100 P R O M O T E K E E P
  28. 28. 28 Config ChatOps IT Autom Deploy Test Observe Automated Quality Gates: $ keptn new artifact myservice:2.0.0 S T A G I N G P R O D ScoreDirect PerformanceUpdate Promote?C D T O ScoreBlue/GreenUpdate Keep?C D T O 1.0.0 1 1 45 / 100 1.0.0 1 1 1 2.0.0 2 2 A B O R T
  29. 29. Pitometer ??? devices used to measure a ship's speed relative to the water. They are used on both surface ships and submarines. Data from the pitometer log is usually fed directly into the ship's navigation system. (Wikipedia)
  30. 30. 30 A Quick word on Keptn Quality Gates: Automated Deployment Validation Metric Source & Query Grading Details & Metric Score Pitometer Specfile Total Scoring Objectives 2GB Allocated Bytes (from Prometheus) > 2GB: 0 Points < 2GB: 20 Points 5% 2% < 2%: 0 Points < 5%: 10 Points > 5%: 20 Points Conversion Rate (Dynatrace) GraderSource If value: 3GB Score: 0 If value: 3.9% Score: 10 Total Score: 10
  31. 31. 31 Automated Quality Gates with Keptn’s Pitometer #1 Call Pitometer Library Pass / Fail Promote? Quality Gates as Code • First extensions by https://www.dynatrace.com/news/blog/automated-deployment-and-architectural-validation-with-pitometer-and-keptn/
  32. 32. 32 Pitometer Client  Evaluates & Scores results  Community can build more  1 to many modules as required  Queries data for requested indicators  Community can build more Pitometer Module  Core library for Data, sources, graders and perfspec file parsing PerfSpec JSON Grader Module Graders Graders Data Module(s) Custom Code  Logic to assemble and configure modules. Read file and pass to Pitometer functions Pitometer JSON Results Keptn Pitometer: Extensible design using NodeJS Modules https://github.com/keptn/pitometer
  33. 33. 33 Quality Gate Examples and Metrics Check 1 • Is bad coding leading to higher costs? Check 2 • New dependencies? On Purpose? • Services connecting accurately? • Number of container instances needed? Check 3 • Are we jeopardizing our SLAs? • Does load balancing work? • Difference between Canaries? Check 4 • Did we introduce new “hidden” exceptions? Metrics  Memory usage  Bytes sent / received  Overall CPU  CPU per transaction type Metrics  Number of incoming / outgoing dependencies  Number of instances running on containers  Metrics  Response Time (Percentiles)  Throughput & Perf per Instance / Canary  Metrics  Total Exceptions  Exceptions by Class & Service
  34. 34. 34 Config ChatOps IT Autom Deploy Test Observe Self-Healing Blue/Green Deployments: $ keptn new artifact myservice:3.0.0 S T A G I N G P R O D ScoreDirect PerformanceUpdate Promote?C D T O ScoreBlue/GreenUpdate Keep?C D T O 1.0.0 1 1 85 / 100 1.0.0 1 1 1 2.0.0 2 2 3.0.0 3 3 3.0.0 3 3 3 80 / 100 P R O M O T E R E V E R T
  35. 35. As a DevOps Engineer, I want switch tools without writing custom pipeline code
  36. 36. 36 Config ChatOps IT Autom Deploy Test Observe Zero-Touch Toolchain Updates: $ keptn update uniform <+neo,+end2end,+spinnaker> S T A G I N G P R O D ScoreDirect PerformanceUpdate Promote?C D T O ScoreBlue/GreenUpdate Keep?C D O 1.0.0 1 1 1.0.0 1 1 1 2.0.0 2 2 3.0.0 3 3 End2EndT
  37. 37. 37 Config ChatOps IT Autom Deploy Test Observe Mastering Continuous Delivery: $ keptn new artifact myservice:4.0.0 S T A G I N G P R O D ScoreDirect PerformanceUpdate Promote?C D T O ScoreBlue/GreenUpdate Keep?C D O 1.0.0 1 1 1.0.0 1 1 1 2.0.0 2 2 3.0.0 3 3 4 4 4 End2EndT 4.0.0 4.0.0 4 4 95 / 100 90 / 100 K E E P P R O M O T E
  38. 38. 38 Keptn Use Cases – Continuous Operations 1. Self-Healing Production: Automated Problem Remediation (Scale-Up, Scale- Down, ...) 2. Self-Healing/Continuous BizDevOps: Automated Business Operations Optimization, Turn on/off feature flags based on conversion rates ... 3. Auto-Protect Production & Business: Stop DDOS attacks, Redirect Bot Traffic ... 4. Chaos-Driven Operation Readiness: Chaos Engineering to validate your production self-healing
  39. 39. As an SRE, I want production to auto- remediate problems detected by monitioring
  40. 40. 40 4 4 Self-Healing Production: $ keptn new problem <Services, Root Cause> P R O D Evaluate Decide Act Notify 4 Escalate 4 4 #1 Evaluate Problem #3 Execute Action #4 Notify on Success #5 Escalate on Failure#2 Decide on Remediation
  41. 41. 41 • https://github.com/keptn/community • Twitter - @keptnproject • Slack - keptn.slack.com • Biweekly community meetings • Zoom https://dynatrace.zoom.us/j/578828410 • Notes https://tinyurl.com/keptn-meeting-notes • Blogs– http://medium.com/keptn • Web – http://keptn.sh • Email- keptn@dynatrace.com Jointhe community Rob.Jahn@Dynatrace.com Technical PartnerManager & KeptnAdvocate
  42. 42. 42
  43. 43. 43 Keptn Orders Application • Order processing application • Web UI with 3 java spring boot microservices with embedded databases • Components pre-compiled, built as a Docker image, and staged on dockerhub.com • All source code can be found at https://github.com/keptn-orders
  44. 44. 44 Keptn-Orders GitHub Repo Structure App Repos front-end customer- service order-service catalog- service /jmeter/ basiccheck.jmx front-end_load.jmx /perfspec/ perfspec.json Other files * Source Code * CI Build pipeline files * Dockerfile * Build scripts Same structure as front-end Same structure as front-end Same structure as front-end orders- project Dev Branch /Helm-Charts/ chart values files /templates/ istio virtual gateway YAML service YAML deployment YAML environment gateway YAML Staging Branch Same structure and files as Dev Branch Production Branch Same structure and files as Dev Branch Folders &Files Keptn Repo Folders &Files https://github.com/jahn-keptn-orders-aks2
  45. 45. 45 VM Worker Pool NAME SPACES Kubernetes Demo Cluster Setup with Keptn & Deployed Keptn-Orders Application Node 1 Dynatrace Keptn Staging Production front-end front-end customer- service order-service catalog- service bridge OneAgent Operator OneAgent (node 1) front-end customer- service order-service catalog- service gatekeeper Jmeter- service helm-service … Dev front-end customer- service order-service catalog- service Node X

×