Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
What to Upload to SlideShare
Next

Share

PAC 2020 Santorin - Andreas Grabner

SLO Validation as a Self Service with Keptn Quality Gates

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

PAC 2020 Santorin - Andreas Grabner

  1. 1. PERFORMANCE IS NOT A MYTH P E R F O R M A N C E A D V I S O R Y C O U N C I L SANTORINI GREECE FEBRUARY 26 - 27 2020 SLO Validation as a Self Service with Quality Gates Andreas Grabner - @grabnerandi www.keptn.sh - @keptnProject – github.com/keptn
  2. 2. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Agenda • Inspiration • Use Cases we provide in Keptn • How it works • How you can leverage & contribute
  3. 3. P E R F O R M A N C E A D V I S O R Y C O U N C I L Motivation / Inspiration What's the problem we want to solve
  4. 4. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Inspiration #1 – Mark Tomlinson 1 Create JIRA Ticket for Performance Feedback Next Continuous Test Cycle picks up Request Performance Feedback back on JIRA Ticket 2 3 1 2 3
  5. 5. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Inspiration #2 – Thomas Steinmaurer “Performance Signature” for Build Nov 16 “Performance Signature” for Build Nov 17 “Performance Signature” for every Build “Multiple Metrics” compared to prev Timeframe Simple Regression Detection per Metric
  6. 6. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Inspiration #3 – Previous PACs
  7. 7. P E R F O R M A N C E A D V I S O R Y C O U N C I L What we want to deliver with Use Cases for Developers
  8. 8. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L UC #1 – Automate SLI Evaluation 1 $ keptn start-evaluation timeframe service sli slo Keptn pulls data from different data sources and Evaluates SLIs Keptn delivers values and Score based on SLOs 2 3 1 2 3 7/8 (87.5%) 4/8 (50%)
  9. 9. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L UC #2 – Performance as a Self-Service 1 $ keptn new-artifact mycontainer:2 Keptn Deploys Keptn Delivers Score based on SLO 2 3 1 2 and Evaluates SLIs, Runs Tests 3 7/8 (87.5%) 4/8 (50%) 1 2 2
  10. 10. PERFORMANCE IS NOT A MYTH P E R F O R M A N C E A D V I S O R Y C O U N C I L SANTORINI GREECE FEBRUARY 26 - 27 2020 Reminder: Start the Demo NOW!
  11. 11. P E R F O R M A N C E A D V I S O R Y C O U N C I L How it works Keptn underneath the hood
  12. 12. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Keptn Quality Gate: SLIs & SLOs • YAML based definition of SLIs • SLI Provider specific queries, e.g: Response Time Quality Gates ... Dynatrace Prometheus Neoload Scores SLIs Queries SLI Providers with SLI Definitions & Timeframe • YAML based definition of SLOs: • List of objectives with fixed or relative pass & warn criteria indicators: error_rate: "builtin:service.errors.total.count:merge(0):avg" count_dbcalls: "calc:service.toptestdbcalls:merge(0):sum" jvm_memory: "builtin:tech.jvm.memory.pool.committed:merge(0):sum" objective: - sli: error_rate - sli: jvm_memory - sli: count_dbcalls pass: - criteria: - "=+2%" # We allow a 2% increase in DB Calls to previous runs warning: - criteria: - "<=10" # We expect no more than 10 DB Calls per TX total_score: pass: "90%" warning: "75%" 0.5 1.0 0.0 info 7/8 (87.5%) 4/8 (50%) $ keptn start-evaluation timeframe service sli slo 5 DB Calls 360MB 4.3% 123SLI Value: SLI Score: Total Score 1 2 3 Tool X
  13. 13. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L SLI / SLO Validation Example SLI Criterion Scoring Objective ResponseTime (ms) <500: 0.5 <200: 1 <210: 1 <204: 2 # SQLs / min Value: 400 Score: 0.5 Value: 0% 2 2.5/3 (83%) >190: 1 Criteria per SLI Weighted Grading SLIValues Total spec_version: "1.0" comparison: compare_with: "single_result" objectives: - sli: response_time_95 pass: - criteria - "<200" warning: - criteria - "<500" - sli: sql_statements weight: 2 pass: - criteria: - "<+2%" warning: - criteria: - "<+5%" - ">-5%" total_score: pass: "90%" warning: "75%" SLI Criterion 500 200 Strategy # other possible options: # compare_with: "several_results" 400ms 202/min 500 200 Base:200 204(+2%) 190(-5%) 210(+5%) >500: 0 <190: 0 >210: 0
  14. 14. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Detailed SLIs & SLOs example Overall Failure Rate Source: Dynatrace Query: builtin:service.errors.total Response Time LOGIN Tx Source: Dynatrace Query: calc:service.responsetime:filter(tx, LOGIN) # DB Calls LOGIN Tx Source: Dynatrace Query: calc:service.dbcalls:filter(tx, LOGIN) <= 5% <= 2% <=150ms & <=+10% <= 400ms <= +0% <= +1 Build 1 0% 80ms 100ms SLO: Overall Score Goal 90% 75% Response Time 95th Perc Source: Dynatrace Query: builtin:service.responsetime(p95) <=100ms <= 250ms SLOSLIs (Service Level Indicators) warn pass 3 100% Build 2 4% 120ms 90ms 3 75% Build 3 1% 90ms 120ms 6 62.5% Build 4 0% 95ms 95ms 3 100% Build 1 Build 2 Build 3 Build 4 $ keptn send event start-evaluation myproject myservice starttime=build1_deploy endtime=build1_testsdone$ keptn send event start-evaluation myproject myservice starttime=build2_deploy endtime=build2_testsdone$ keptn send event start-evaluation myproject myservice starttime=build3_deploy endtime=build3_testsdone$ keptn send event start-evaluation myproject myservice starttime=build4_deploy endtime=build4_testsdone
  15. 15. P E R F O R M A N C E A D V I S O R Y C O U N C I L Keptn in Action Demo www.keptn.sh
  16. 16. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Installing Keptn Quality Gate Only $ keptn install --use-case=quality-gates api.keptn.* bridge.keptn.* Quality Gate Only Installation $ kubectl apply –f yoursli/service.yaml Upstream Evalulation Notify ... ... ...$ keptn create project myproj upstream.url $ keptn add resource myproj sli.yaml slo.yaml $ keptn create service myservice $ kubectl apply –f notificationservice.yaml Optionals
  17. 17. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Integrating with existing Pipelines $ keptn send event start-evaluation --project= --timeframe= ... ... Evaluating keptn-context 12345678 $ keptn get event evaluation-done --keptn-context=12345678 SLO validation not finished yet curl -X POST "http://api.keptn.*/v1/event" –d {"start" : "xxx", ... "type": "sh.keptn.event.start-evaluation"} {"keptnContext":"12345678..."} Integration via Keptn CLI – more via https://keptn.sh/docs/0.6.0/reference/cli/ Integration via Keptn API – more via https://api.keptn.*/swagger-ui/ curl -X GET "http://api.keptn.*/v1/event?keptnContext=12345678 &type=sh.keptn.events.evaluation-done" {no result yet} (1)start (3)done? Your Pipeline (2)Retrieve SLIs 87.5% (4)Evaluate SLO(5)done!$ keptn get event evaluation-done --keptn-context=12345678 ... Here is your result { score : 87.5 } curl -X GET "http://api.keptn.*/v1/event?keptnContext=12345678 &type=sh.keptn.events.evaluation-done" {jsonresult} Your Monitored App Environment 87.5%
  18. 18. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Example Integration with GitLab https://gitlab.com/checkelmann/dynatrace-pipeline 2 Monitored Int-Environment (1) s t a r t (3) d o n e ? (2)Retrieve SLIs 87.5% (4)Evaluate SLO (5) d o n e ! 87.5%
  19. 19. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Keptn Bridge Visualization
  20. 20. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Monitoring Installing Full Keptn $ keptn install api.keptn.* bridge.keptn.* $ kubectl apply –f yoursli/service.yaml Upstream Full Install Evalulation Testing Deploy Notify ... ... ... ... ... $ kubectl apply –f yourdeploytool/service.yaml $ keptn create project myproj upstream.url $ keptn add resource myproj sli.yaml slo.yaml $ keptn onboard service myservice helmchart $ keptn add resource myproj mytests.neo $ kubectl apply –f notificationservice.yaml Optionals $ kubectl apply –f yourtesttool/service.yaml $ keptn configure monitoring yourtool
  21. 21. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Full Keptn Performance as a Service 1 2 Deploy Run Tests Evaluate SLI SLO 4/8 (50%) 7/8 (87.5%) 1 $ keptn new-artifact V2 Push Results (via Slack, UI …) 6 2 4 3 ConfigMonitoring 5
  22. 22. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Building custom SLI providers • https://keptn.sh/docs/0.6.0/usecases/custom-service/#sli-provider
  23. 23. PERFORMANCE IS NOT A MYTH P E R F O R M A N C E A D V I S O R Y C O U N C I L SANTORINI GREECE FEBRUARY 26 - 27 2020 Andreas Grabner - @grabnerandi www.keptn.sh - @keptnProject – github.com/keptn SLO Validation as a Self Service with Quality Gates
  • ecassamc

    Sep. 4, 2021

SLO Validation as a Self Service with Keptn Quality Gates

Views

Total views

164

On Slideshare

0

From embeds

0

Number of embeds

0

Actions

Downloads

0

Shares

0

Comments

0

Likes

1

×