SlideShare a Scribd company logo

OSMC 2023 | Know your data: The stats behind your alerts by Dave McAllister

NETWAYS
NETWAYS
NETWAYSNETWAYS

Quick, what’s the difference between the mean, the mode and the median? Do you need a Gaussian or a normal distribution And does your choice impact the alerts and observations you get from your observability tools? Come get refreshed on the impact some basic choices in statistical behavior can have on what gets triggered. Learn why a median might be the choice for historical anomaly or sudden change. Jump into Gaussian distributions, data alignment challenges, and the trouble with sampling. Walk out with a deeper understanding of your metrics and what they might tell you.

OSMC 2023 | Know your data: The stats behind your alerts by Dave McAllister

1 of 38
Download to read offline
Know Your Data:
The stats behind the alerts
Dave McAllister
Sr. OSS Technologist
NGINX
©2023 F5
2
Quick:
What’s the difference between
Mean, Median and Mode?
©2023 F5
3
Quick:
What’s the difference between
Mean, Median and Mode?
And for extra credit, What’s the 9th Dedekind number?
©2023 F5
5
Monitoring is a numbers game
• Metrics are numbers that represent selected
behavior
• Generally
• Timestamped
• Key-Values
• Data, to be useful, must be
• Aggregated
• Analyzed
• Visualized
©2023 F5
6
• How do you deal with outliers (spikes) in
monitoring?
• How do you get a representative value when
values build on each other?
• How do you arrive at values to represent rate of
change over time?
Do you know what your alert is
really showing you?
Some questions to ponder
©2023 F5
7
Mean, Median, Mode
Median:
Represents the middle
value in a set of ordered
data
Mean:
Measure of central
tendency, represents
average value of a set of
data
Mode:
Value that appears most
frequently in a set of data
Data:
2, 6, 4, 9, 5, 1, 7, 8, 1, 9, 9, 1, 10, 2, 9, 6, 7, 2, 1, 4, 7, 1, 10, 9, 2, 7, 1, 1, 4, 3, 5, 6,
3, 8, 1, 8, 4, 7, 6, 3, 9, 9, 9, 4, 9, 1, 4, 1, 9, 8, 10, 10, 1, 1, 1, 7, 10, 9, 7, 3, 7, 4
Median = 6 Mode = 1
Mean = 5.444

Recommended

Know Your Data: The stats behind your alerts
Know Your Data: The stats behind your alertsKnow Your Data: The stats behind your alerts
Know Your Data: The stats behind your alertsAll Things Open
 
Performance OR Capacity #CMGimPACt2016
Performance OR Capacity #CMGimPACt2016 Performance OR Capacity #CMGimPACt2016
Performance OR Capacity #CMGimPACt2016 Alex Gilgur
 
Credit Card Fraudulent Transaction Detection Research Paper
Credit Card Fraudulent Transaction Detection Research PaperCredit Card Fraudulent Transaction Detection Research Paper
Credit Card Fraudulent Transaction Detection Research PaperGarvit Burad
 
SPC Training by D&H Engineers
SPC Training by D&H EngineersSPC Training by D&H Engineers
SPC Training by D&H EngineersD&H Engineers
 
Six Sigma Complete Guide
Six Sigma Complete GuideSix Sigma Complete Guide
Six Sigma Complete GuideNitish Nagar
 
Six Sigma Nitish Nagar
Six Sigma Nitish NagarSix Sigma Nitish Nagar
Six Sigma Nitish NagarNitish Nagar
 
Short Data Rules for Observability.pdf
Short Data Rules for Observability.pdfShort Data Rules for Observability.pdf
Short Data Rules for Observability.pdfDave McAllister
 

More Related Content

Similar to OSMC 2023 | Know your data: The stats behind your alerts by Dave McAllister

Performance Issue? Machine Learning to the rescue!
Performance Issue? Machine Learning to the rescue!Performance Issue? Machine Learning to the rescue!
Performance Issue? Machine Learning to the rescue!Maarten Smeets
 
Using Time Series for Full Observability of a SaaS Platform
Using Time Series for Full Observability of a SaaS PlatformUsing Time Series for Full Observability of a SaaS Platform
Using Time Series for Full Observability of a SaaS PlatformDevOps.com
 
DMM9 - Data Migration Testing
DMM9 - Data Migration TestingDMM9 - Data Migration Testing
DMM9 - Data Migration TestingNick van Beest
 
Designing apps for resiliency
Designing apps for resiliencyDesigning apps for resiliency
Designing apps for resiliencyMasashi Narumoto
 
Time Series Anomaly Detection with .net and Azure
Time Series Anomaly Detection with .net and AzureTime Series Anomaly Detection with .net and Azure
Time Series Anomaly Detection with .net and AzureMarco Parenzan
 
Bridging the Gap: Machine Learning for Ubiquitous Computing -- Evaluation
Bridging the Gap: Machine Learning for Ubiquitous Computing -- EvaluationBridging the Gap: Machine Learning for Ubiquitous Computing -- Evaluation
Bridging the Gap: Machine Learning for Ubiquitous Computing -- EvaluationThomas Ploetz
 
Dependable Systems - Structure-Based Dependabiilty Modeling (6/16)
Dependable Systems - Structure-Based Dependabiilty Modeling (6/16)Dependable Systems - Structure-Based Dependabiilty Modeling (6/16)
Dependable Systems - Structure-Based Dependabiilty Modeling (6/16)Peter Tröger
 
Heuristic design of experiments w meta gradient search
Heuristic design of experiments w meta gradient searchHeuristic design of experiments w meta gradient search
Heuristic design of experiments w meta gradient searchGreg Makowski
 
GLM & GBM in H2O
GLM & GBM in H2OGLM & GBM in H2O
GLM & GBM in H2OSri Ambati
 
Observability - the good, the bad, and the ugly
Observability - the good, the bad, and the uglyObservability - the good, the bad, and the ugly
Observability - the good, the bad, and the uglyAleksandr Tavgen
 
Chapter 9 learning more about sample data(1)
Chapter 9 learning more about sample data(1)Chapter 9 learning more about sample data(1)
Chapter 9 learning more about sample data(1)Celumusa Godfrey Nkosi
 
Amaya_Presentation
Amaya_PresentationAmaya_Presentation
Amaya_PresentationIsaias Amaya
 
Monte Carlo and Schedule Risk Analysis
Monte Carlo and Schedule Risk AnalysisMonte Carlo and Schedule Risk Analysis
Monte Carlo and Schedule Risk AnalysisIntaver Insititute
 

Similar to OSMC 2023 | Know your data: The stats behind your alerts by Dave McAllister (20)

Performance Issue? Machine Learning to the rescue!
Performance Issue? Machine Learning to the rescue!Performance Issue? Machine Learning to the rescue!
Performance Issue? Machine Learning to the rescue!
 
Using Time Series for Full Observability of a SaaS Platform
Using Time Series for Full Observability of a SaaS PlatformUsing Time Series for Full Observability of a SaaS Platform
Using Time Series for Full Observability of a SaaS Platform
 
Time series
Time seriesTime series
Time series
 
DMM9 - Data Migration Testing
DMM9 - Data Migration TestingDMM9 - Data Migration Testing
DMM9 - Data Migration Testing
 
Process Control
Process ControlProcess Control
Process Control
 
Training - What is Performance ?
Training  - What is Performance ?Training  - What is Performance ?
Training - What is Performance ?
 
Designing apps for resiliency
Designing apps for resiliencyDesigning apps for resiliency
Designing apps for resiliency
 
Time Series Anomaly Detection with .net and Azure
Time Series Anomaly Detection with .net and AzureTime Series Anomaly Detection with .net and Azure
Time Series Anomaly Detection with .net and Azure
 
Time-resource v&v for complex systems
Time-resource v&v for complex systemsTime-resource v&v for complex systems
Time-resource v&v for complex systems
 
Bridging the Gap: Machine Learning for Ubiquitous Computing -- Evaluation
Bridging the Gap: Machine Learning for Ubiquitous Computing -- EvaluationBridging the Gap: Machine Learning for Ubiquitous Computing -- Evaluation
Bridging the Gap: Machine Learning for Ubiquitous Computing -- Evaluation
 
Dependable Systems - Structure-Based Dependabiilty Modeling (6/16)
Dependable Systems - Structure-Based Dependabiilty Modeling (6/16)Dependable Systems - Structure-Based Dependabiilty Modeling (6/16)
Dependable Systems - Structure-Based Dependabiilty Modeling (6/16)
 
Heuristic design of experiments w meta gradient search
Heuristic design of experiments w meta gradient searchHeuristic design of experiments w meta gradient search
Heuristic design of experiments w meta gradient search
 
GLM & GBM in H2O
GLM & GBM in H2OGLM & GBM in H2O
GLM & GBM in H2O
 
Observability - the good, the bad, and the ugly
Observability - the good, the bad, and the uglyObservability - the good, the bad, and the ugly
Observability - the good, the bad, and the ugly
 
Multivariate Analysis
Multivariate AnalysisMultivariate Analysis
Multivariate Analysis
 
Statistics
StatisticsStatistics
Statistics
 
Chapter 9 learning more about sample data(1)
Chapter 9 learning more about sample data(1)Chapter 9 learning more about sample data(1)
Chapter 9 learning more about sample data(1)
 
Amaya_Presentation
Amaya_PresentationAmaya_Presentation
Amaya_Presentation
 
Six Sigma Overview
Six Sigma OverviewSix Sigma Overview
Six Sigma Overview
 
Monte Carlo and Schedule Risk Analysis
Monte Carlo and Schedule Risk AnalysisMonte Carlo and Schedule Risk Analysis
Monte Carlo and Schedule Risk Analysis
 

Recently uploaded

Ammunition and it's types and use in forensic ballistics
Ammunition and it's types and use in forensic ballisticsAmmunition and it's types and use in forensic ballistics
Ammunition and it's types and use in forensic ballisticsshubhamdwivedi7521
 
1.2 Ingredients Used for Sandwiches 1.3 Culinary Terms.pptx
1.2 Ingredients Used for Sandwiches 1.3 Culinary Terms.pptx1.2 Ingredients Used for Sandwiches 1.3 Culinary Terms.pptx
1.2 Ingredients Used for Sandwiches 1.3 Culinary Terms.pptxNinia
 
"SACRED WAY" Athens - VENUE- presentation
"SACRED WAY" Athens - VENUE- presentation"SACRED WAY" Athens - VENUE- presentation
"SACRED WAY" Athens - VENUE- presentationVicky Evangeliou
 
West Africa Scene Setting African Continental Master Plan (CMP) for electrici...
West Africa Scene Setting African Continental Master Plan (CMP) for electrici...West Africa Scene Setting African Continental Master Plan (CMP) for electrici...
West Africa Scene Setting African Continental Master Plan (CMP) for electrici...Francois Stepman
 
KKrish - DOVE Leadership Program Concept
KKrish - DOVE Leadership Program ConceptKKrish - DOVE Leadership Program Concept
KKrish - DOVE Leadership Program ConceptKarthik Krishna
 
Scaling up renewable energy investments in West Africa
Scaling up renewable energy investments in West AfricaScaling up renewable energy investments in West Africa
Scaling up renewable energy investments in West AfricaFrancois Stepman
 
AWS RDS Data API and CloudTrail. Who drop the table_.pdf
AWS RDS Data API and CloudTrail. Who drop the table_.pdfAWS RDS Data API and CloudTrail. Who drop the table_.pdf
AWS RDS Data API and CloudTrail. Who drop the table_.pdfVladimir Samoylov
 
ONLINE RESORT BOOKING SYSTEM WEBSITE 1.pptx
ONLINE RESORT BOOKING SYSTEM WEBSITE 1.pptxONLINE RESORT BOOKING SYSTEM WEBSITE 1.pptx
ONLINE RESORT BOOKING SYSTEM WEBSITE 1.pptxDivyaPatel621561
 
Forklift Telehandler ONLY Operator Training Revised_7.6.2022.pptx
Forklift Telehandler ONLY Operator Training Revised_7.6.2022.pptxForklift Telehandler ONLY Operator Training Revised_7.6.2022.pptx
Forklift Telehandler ONLY Operator Training Revised_7.6.2022.pptxbriancriswell1979
 
Git and Github.pptx
Git and Github.pptxGit and Github.pptx
Git and Github.pptxaymanessam16
 
DAY 05 Book of Revelation 2-18-24 PPT.pptx
DAY 05 Book of Revelation 2-18-24 PPT.pptxDAY 05 Book of Revelation 2-18-24 PPT.pptx
DAY 05 Book of Revelation 2-18-24 PPT.pptxFamilyWorshipCenterD
 

Recently uploaded (13)

Ammunition and it's types and use in forensic ballistics
Ammunition and it's types and use in forensic ballisticsAmmunition and it's types and use in forensic ballistics
Ammunition and it's types and use in forensic ballistics
 
1.2 Ingredients Used for Sandwiches 1.3 Culinary Terms.pptx
1.2 Ingredients Used for Sandwiches 1.3 Culinary Terms.pptx1.2 Ingredients Used for Sandwiches 1.3 Culinary Terms.pptx
1.2 Ingredients Used for Sandwiches 1.3 Culinary Terms.pptx
 
"SACRED WAY" Athens - VENUE- presentation
"SACRED WAY" Athens - VENUE- presentation"SACRED WAY" Athens - VENUE- presentation
"SACRED WAY" Athens - VENUE- presentation
 
West Africa Scene Setting African Continental Master Plan (CMP) for electrici...
West Africa Scene Setting African Continental Master Plan (CMP) for electrici...West Africa Scene Setting African Continental Master Plan (CMP) for electrici...
West Africa Scene Setting African Continental Master Plan (CMP) for electrici...
 
KKrish - DOVE Leadership Program Concept
KKrish - DOVE Leadership Program ConceptKKrish - DOVE Leadership Program Concept
KKrish - DOVE Leadership Program Concept
 
Becoming a Member of Society.pptx
Becoming a Member of Society.pptxBecoming a Member of Society.pptx
Becoming a Member of Society.pptx
 
Scaling up renewable energy investments in West Africa
Scaling up renewable energy investments in West AfricaScaling up renewable energy investments in West Africa
Scaling up renewable energy investments in West Africa
 
AWS RDS Data API and CloudTrail. Who drop the table_.pdf
AWS RDS Data API and CloudTrail. Who drop the table_.pdfAWS RDS Data API and CloudTrail. Who drop the table_.pdf
AWS RDS Data API and CloudTrail. Who drop the table_.pdf
 
ONLINE RESORT BOOKING SYSTEM WEBSITE 1.pptx
ONLINE RESORT BOOKING SYSTEM WEBSITE 1.pptxONLINE RESORT BOOKING SYSTEM WEBSITE 1.pptx
ONLINE RESORT BOOKING SYSTEM WEBSITE 1.pptx
 
Forklift Telehandler ONLY Operator Training Revised_7.6.2022.pptx
Forklift Telehandler ONLY Operator Training Revised_7.6.2022.pptxForklift Telehandler ONLY Operator Training Revised_7.6.2022.pptx
Forklift Telehandler ONLY Operator Training Revised_7.6.2022.pptx
 
Git and Github.pptx
Git and Github.pptxGit and Github.pptx
Git and Github.pptx
 
An Inviting Church
An Inviting ChurchAn Inviting Church
An Inviting Church
 
DAY 05 Book of Revelation 2-18-24 PPT.pptx
DAY 05 Book of Revelation 2-18-24 PPT.pptxDAY 05 Book of Revelation 2-18-24 PPT.pptx
DAY 05 Book of Revelation 2-18-24 PPT.pptx
 

OSMC 2023 | Know your data: The stats behind your alerts by Dave McAllister

  • 1. Know Your Data: The stats behind the alerts Dave McAllister Sr. OSS Technologist NGINX
  • 2. ©2023 F5 2 Quick: What’s the difference between Mean, Median and Mode?
  • 3. ©2023 F5 3 Quick: What’s the difference between Mean, Median and Mode? And for extra credit, What’s the 9th Dedekind number?
  • 4. ©2023 F5 5 Monitoring is a numbers game • Metrics are numbers that represent selected behavior • Generally • Timestamped • Key-Values • Data, to be useful, must be • Aggregated • Analyzed • Visualized
  • 5. ©2023 F5 6 • How do you deal with outliers (spikes) in monitoring? • How do you get a representative value when values build on each other? • How do you arrive at values to represent rate of change over time? Do you know what your alert is really showing you? Some questions to ponder
  • 6. ©2023 F5 7 Mean, Median, Mode Median: Represents the middle value in a set of ordered data Mean: Measure of central tendency, represents average value of a set of data Mode: Value that appears most frequently in a set of data Data: 2, 6, 4, 9, 5, 1, 7, 8, 1, 9, 9, 1, 10, 2, 9, 6, 7, 2, 1, 4, 7, 1, 10, 9, 2, 7, 1, 1, 4, 3, 5, 6, 3, 8, 1, 8, 4, 7, 6, 3, 9, 9, 9, 4, 9, 1, 4, 1, 9, 8, 10, 10, 1, 1, 1, 7, 10, 9, 7, 3, 7, 4 Median = 6 Mode = 1 Mean = 5.444
  • 7. ©2023 F5 8 Mean, Median, Mode Data: 2, 6, 4, 9, 5, 1, 7, 8, 1, 9, 9, 1, 10, 2, 9, 6, 7, 2, 1, 4, 7, 1, 10, 9, 2, 7, 1, 1, 4, 3, 5, 6, 3, 8, 1, 8, 4, 7, 6, 3, 9, 9, 9, 4, 9, 1, 4, 1, 9, 8, 10, 10, 1, 1, 1, 7, 10, 9, 7, 3, 7, 4 Mean = 5.444 Or is it 4.130 or 2.791? Median: Represents the middle value in a set of ordered data Mean: Measure of central tendency, represents average value of a set of data Mode: Value that appears most frequently in a set of data
  • 8. ©2023 F5 9 Arithmetic, Harmonic, Geometric, Trimmed, Weighted, Moving Means to an End • Each has potential uses and drawbacks • Often already implemented in monitoring software • Can give very different results • Can make like and unlike comparisons easier
  • 9. ©2023 F5 10 Also often called Average • Most common • Is the central point in a normal distribution • This is not the 50% mark (mostly) • Useful for comparing current to previous conditions • May be aggregated into groups (time series) In a time series, we usually calculate constantly to incorporate new data Arithmetic Internet Load Balancer 300 Request/sec 200 Request/sec 100 Request/sec Amean = (100 + 200 + 300) / 3 Amean = 200 Requests per second
  • 10. ©2023 F5 11 • Multiply all the items together, take the nth root • Often used for things growing exponentially • In DevOps • Average number of deploys per unit of time • Average lead time for changes • MTTR • Throughput Geometric DevOps team: optimizing app deployment Sprint 1: 5% reduction Sprint 2: 10% reduction Sprint 3: 3% reduction Sprint 4: 7% reduction How well are they reducing deployment times? X1=1-0.05 = 0.95, x2 = 0.90, X3 = 0.97, x4 = 0.93 GM = 0.937 or Deployment has a 6.3% improvement
  • 11. ©2023 F5 12 Harmonic • Divide n by the sum of the reciprocals • Measure the performance where multiple systems are involved • Weights the lowest figure the highest • In DevOps • Performance within range • Overall indication of latency or thruput • Use in complex environments • Especially useful for outliers Internet Load Balancer 300 Request/sec 200 Request/sec 100 Request/sec HM = 150 Requests per second
  • 12. ©2023 F5 13 Take a comparison look What’s more important to the equation? • Latency • Thruput • GM is the moral equivalent • AM weights the larger number • HM weights the lower value Average Latency ms/sec Average Count/sec Suggested metric AM GM HM 250 40 10000 145.00 100.00 68.97 200 50 10000 125.00 100.00 80.00 400 25 10000 212.50 100.00 47.06 198 70 13860 134.00 117.73 103.43 105 99 10395 102.00 101.96 101.91 474 73 34602 273.50 186.02 126.52 195 97 18915 146.00 137.53 129.55 196 138 27048 167.00 164.46 161.96 133 10 1330 71.50 36.47 18.60 298 55 16390 176.50 128.02 92.86 0.00 50.00 100.00 150.00 200.00 250.00 300.00 1 2 3 4 5 6 7 8 9 10 Means overlay AM GM HM
  • 13. ©2023 F5 14 • Amazingly underutilized! • Center value of a sorted list • Median is always the 50% point of a normal curve Median Mean 25.25 Median 26
  • 14. ©2023 F5 15 Choosing Between Mean and Median • Mean can be impacted by outliers • Resilience is better in median • In DevOps • Response time monitoring • Anomaly Detection • Capacity Planning Old New Outlier 250 25.54 Mean 31.15 26 Median 26.5
  • 15. ©2023 F5 16 Old New Outlier 291 21.90 Mean 28.63 22 Median 22.5 18.61 Harmonic 19.06
  • 16. ©2023 F5 17 If you are using P95 you are using using a percentage value Congrats!
  • 17. ©2023 F5 18 Slight sidetrack: Measure of Variability How the numbers behave. • Standard Deviation • Range • InterQuartile Range (IQR) • Variance • Clusters • Outliers Properly used, variability can help you target outliers
  • 18. ©2023 F5 19 Those outside edges are good candidates for outliers Outlier Formula, visually Q2 IQR Outlier Formula Q3 Q1
  • 19. ©2023 F5 20 How about the Mode? • The most commonly recurring value in the set • Often presented as a histograph • Not commonly used in DevOps, mostly inferential • Log Analysis • Security Monitoring • User Behavior Analysis 0 2 4 6 8 10 12 14 one two three four five six seven eight nine ten Mode Chart
  • 20. ©2023 F5 21 Slight sidetrack: Descriptive versus Inferential stats • Descriptive uses the whole data set to draw statistical conclusions • Used for visualization • Can define and extract trends • Inferential uses a sampled set to draw conclusions • Used for predictions or hypotenuse testing • Can also visualize • But this leads us to sampling
  • 21. ©2023 F5 22 Dealing with the data • Monitoring is now a data problem • Observability signals: Metrics, Traces, Logs • Analysis is often • Aggregated or Analyzed in segments: Time-defined • Sampled and inferential • Random sampling • Stratified sampling • Cluster sampling • Systematic sampling • Purposive sampling
  • 23. ©2023 F5 24 • Changes behavior from Descriptive to Inferential • Can hide outlier behavior • Metrics are not usually sampled • May make forensics tougher • Lack of direct correlation • A necessary evil • But understand what you are giving up Sampling
  • 24. ©2023 F5 25 Distributions • Normal • Data equally distributed • Poisson • used to model the occurrence of rare events • Beta • Success/failure of binomial events • Exponential • Time between async events • Weibull • Likelihood of failure • Log-normal • Values based on many small events
  • 25. ©2023 F5 26 Slight sidetrack: Standard deviation • Measures the variability of your data • Identifies trends and outliers • NOT percentage based • Except with coefficient of variability • CV=Mean / std dev X 100 • Useful for measurement ignoring range • SRE cases • Lead times • Recovery times • Anomalies (alerts) • SLO / SLI This Photo by Unknown Author is licensed under CC BY-NC-ND
  • 26. ©2023 F5 27 Deeper dive: Weibull • Usually used for time-to-failure • Defined by a Shape and a Scale parameter • This can be challenging • Don’t ask the math • F(t)=1−e−(t/λ)β • R does it for you! Component Time-to-Failure Spinning Rust 500 hours Memory 1000 hours Power Supply 1500 hours CPU 2000 hours SSD 2500 hours library(fitdistrplus) data <- c(500, 1000, 1500, 2000, 2500) fit.weib <- fitdist(data, "weibull") summary(fit.weib) Fitting of the distribution ' weibull ' by maximum likelihood Parameters : shape 1.0624082 scale 2158.256 p.failure <- pweibull(300, shape = fit.weib$estimate[1], scale = fit.weib$estimate[2]) 1 - p.failure Failure (disk, memory) = 64.81% Failure (entire system) = 75.52% Fdisk(300)≈1−e−0.6918≈0.50025 Fmemory(300)≈1− −0.3490≈0.2959Fmemory(300)≈1−e−0.3490≈0.2959 Fpower(300)≈1− −0.2327≈0.2092Fpower(300)≈1−e−0.2327≈0.2092 FCPU(300)≈1− −0.1745≈0.1593FCPU(300)≈1−e−0.1745≈0.1593 FSSD(300)≈1− −0.1396≈0.1298FSSD(300)≈1−e−0.1396≈0.1298
  • 27. ©2023 F5 28 Deeper dive: Exponential • Models the “rate” (time between events that are unrelated) • Use cases • Network performance • User Requests (traces) • Messaging service • System failures • f(x) = me–mx • λ = 1/µ • e ~ 2.71828182846 Average Latency ms/sec Average Count/sec Suggested metric AM GM HM 250 40 10000 145.00 100.00 68.97 200 50 10000 125.00 100.00 80.00 400 25 10000 212.50 100.00 47.06 198 70 13860 134.00 117.73 103.43 105 99 10395 102.00 101.96 101.91 474 73 34602 273.50 186.02 126.52 195 97 18915 146.00 137.53 129.55 196 138 27048 167.00 164.46 161.96 133 10 1330 71.50 36.47 18.60 298 55 16390 176.50 128.02 92.86 Latency λ = 1/244.9 Probability density of 158 ms = 0.2142% Cumulative density = 47.5422% Expdist median ≈ 169.7 ms Throughput λ = 1/65.7 Probability density at 58 count = 0.628% Cumulative density = 58.7% Expdist median ≈ 45.6
  • 28. ©2023 F5 29 You may stumble upon: “On scale, statistics are not your friend” WRONG On scale, probability is not your friend. Slight Sidetrack and a pet peeve
  • 29. ©2023 F5 30 Conditional Probability – The Monty Hall Problem Bayes Theorem • You have three doors • Behind one of them is a car • The other two are goats • You pick Door # 1 1 3 2 ✅
  • 30. ©2023 F5 31 Conditional Probability – The Monty Hall Problem Bayes Theorem • You have three doors • Behind one of them is a car • The other two are goats • You pick Door # 1 • The host shows you door # 2 and offers to let you switch • Do you switch or not? 1 3 2 ✅
  • 31. ©2023 F5 32 1 3 2 𝟏 𝟑 𝟏 𝟑 𝟏 𝟑 1 3 2 ✔ 𝟏 𝟑 𝟐 𝟑 𝟏 𝟑 𝟏 𝟑 + = { } 1 3 2 ✔ 𝟏 𝟑 𝟐 𝟑 𝟎 𝟑 • Mathematical framework for updating probability for an event as new information becomes available • Based on this theorem • Initial probability of Door 1 = 1/3 • Initial probability of Doors 2 or 3 = 2/3 • Since it isn’t behind Door # 2 then Door # 3 is the remaining 2/3 • Conditional Probability Bayes Theorem
  • 32. ©2023 F5 33 Bayes Theorem in DevOps Incident Troubleshooting • You're seeing an increase in error rates a<er a recent deployment. Historically, 80% of such incidents have been Ged to database issues a<er major updates. • Bayes' theorem suggests that the current issue is database-related considering the recent deployment and historical data. Predictive Monitoring • Historically there's a 5% chance that your application will crash on any given day. Now, you get alerts about increasing memory consumption. • Bayes' theorem indicates the probability of an impending application crash. Optimizing A/B Testing • You're testing a new feature against a baseline to determine its impact on server load. Initially, both variants might be assumed equally likely to have an effect. • Data from the test allows you to update the probability distribution of the effect of the new feature on server load. Bayes' theorem provides a mathematical framework for incorporating new evidence into existing beliefs
  • 33. ©2023 F5 34 Common Pitfalls in Statistics Looking at the wrong central measure Ignoring Scale Confusing correlation with causation Getting causation backwards Failing to see biases
  • 35. ©2023 F5 36 • Statistics are how we tend to analyze our metrics • Statistics are aggregation and reduction to reveal central tendencies • They do not show individual behavior • Most choices make use of very few basics • But other choices may show amazing inferential results • And finally Summary
  • 36. ©2023 F5 37 • Statistics are how we tend to analyze our metrics • Statistics are aggregation and reduction to reveal central tendencies • They do not show individual behavior • Most choices make use of very few basics • But other choices may show amazing inferential results • And finally Summary The most effective debugging tool is still careful thought, coupled with judiciously placed print statements. -Brian Kernighan Unix for Beginners 1979
  • 37. ©2023 F5 38 D(9) = 286 386 577 668 298 411 128 469 151 667 598 498 812 366