Adaptive Fault Detection

Adaptive Fault DetectionBaron Schwartz • Percona Live NYC 2012Date

Me Optimization, Backups, 3r e rs V Replication, and more Co d e rs v Ed io Author of High Performance MySQL iti n 5.✤ on 5✤ Creator of some tools that you might use✤ I love hearing from people just like you: High @xaprb on Twitter Performance MySQL ✤ ✤ http://www.linkedin.com/in/xaprb Baron Schwartz, Peter Zaitsev & Vadim Tkachenko

Conventional Fault DetectionMetrics, Thresholds, and Actions

Nagios and ThresholdsIs there a right answer?

Motivations✤ Detect unknown failure mode✤ Capture diagnostic data automatically✤ Surface relevant information

Six Sigmas99.7% of measurements fall within ±3 sigmas of mean in a normal distribution

Abnormality DetectionStatistical process control, operations research, and intuition

Shewhart Control ChartsMetrics that fall ± too many standard deviations from the mean are out of bounds

Holt-Winters ForecastingPredict the future based on history, trend, and seasonality

Brownian MotionA random walk shouldn’t go the same way for long

Probability of Increase/Decrease Increase 49.88% Same 0.21% Decrease 49.91%Coin TossingQPS increases and decreases with ~equal probability

Length of QPS Runs 100000 75000 50000 25000 0 -9 -8 -7 -6 -5 -4 -3 -2 -1 1 2 3 4 5 6 7 8 9I Feel Normal TodayLong runs of QPS increases/decreases behave like a coin toss

Run Improbability Length 1 50.10% 75.09% 2 75.10% 93.79% 3 87.57% 98.46% 4 93.80% 99.62% 5 96.91% 99.90% 6 98.46% 99.98% 7 99.23% 99.99% 8 99.62% 9 99.90% 10 99.95%Oddly EvenHow long is an unusually long random walk?

Run Improbability Length 1 50.10% 75.09% 2 75.10% 93.79% 3 87.57% 98.46% 4 93.80% 99.62% Two ! 5 96.91% 99.90% 6 98.46% 99.98% 7 99.23% 99.99% 8 99.62% 9 99.90% 10 99.95%Oddly EvenHow long is an unusually long random walk?

Run Improbability Length 1 50.10% 75.09% 2 75.10% 93.79% 3 87.57% 98.46% 4 93.80% 99.62% Two ! 5 96.91% 99.90% 6 98.46% 99.98% 7 99.23% 99.99% 8 99.62% Three ! 9 99.90% 10 99.95%Oddly EvenHow long is an unusually long random walk?

Run Improbability Length 1 50.10% 75.09% 2 75.10% 93.79% 3 87.57% 98.46% 4 93.80% 99.62% Two ! 5 96.91% 99.90% 6 98.46% 99.98% 7 99.23% 99.99% 8 99.62% Three ! 9 99.90% 10 99.95% Same thing, butOddly Even two variablesHow long is an unusually long random walk?

Houston, We Have an OpportunityThese techniques fall short of what’s needed

God hath chosen the foolish things of theworld to confound the wise [1 Cor 1:27]

Unexpected Things Happen... but who says abnormal is bad?

Bottleneck DetectionFind abnormalities, then determine whether they are system faults

Metrics That MatterThroughput, concurrency, and change—but not response time (why?)

AlgorithmsVarious combinations of severity, directionality, run length, duration, and more

Out of BoundsHow often each algorithm detected abnormalities

Drilling DownOne of the algorithms triggered at the gray line

Another ExampleThis one from a more selective algorithm

Brownian CommotionJust because it’s long and aimed the right way doesn’t mean it’s scary

QPS Cxn RunsSee Spots RunThere’s a clear run of decreasing QPS and increasing connections—but no stall/lockup

A New ApproachCombinations of algorithms to avoid run-based false positives

I’d Like Fries With ThatAnd supersize my Threads_running, please

Why Not Response Time?We are legion and we want your carat patch

Mass ApplicationDoes it generalize to many different settings?

Dataset #2Looks reasonable on a different workload—so far

Stall Detection In ActionSorry, no witty comment here

Oh Crud.What does “bad” mean on a workload like this?

He taketh the wise in their own craftiness.[1 Cor 3:19]

Different Like Everybody ElseVariance-to-mean ratio / index of dispersion to the rescue?

Still Life With Purple LineIf you think this is art, I am happy to sell it to you

When In Doubt, Get EmpiricalMeasure 99.7th% V:M on a well-behaved dataset. Good enough for government work.

Workload #1 ReduxStill ﬁnds lots of the same “bad spots” with the V:M ratio ﬁlter

If you can read this, ﬂip me over!Workload #2 With New Filter

Conclusions✤ Workloads differ. GIGO.✤ Unlikely events are not necessarily bad.✤ Naive techniques fail; more sophisticated methods are required.✤ Thresholds are too simplistic, but reappear.✤ Common sense beats elaborate math.✤ Automated problem detection is a good thing?

Jobs✤ Want to work and live in America’s #1 City?✤ I am hiring DBAs and developers. Talk to me.

Questions? @xaprb • http://www.linkedin.com/in/xaprb

Image Credits✤ http://www.flickr.com/photos/katej/853418592/✤ http://www.flickr.com/photos/domesticat/2963393184/✤ http://www.flickr.com/photos/markho/481969187/✤ http://www.flickr.com/photos/exquisitur/3502317741/✤ http://www.flickr.com/photos/calleephoto/4952091078/✤ http://www.flickr.com/photos/paperpariah/4150220583/✤ http://www.flickr.com/photos/stevewall/6057281066/✤ http://www.flickr.com/photos/mybloodyself/5879425774/✤ http://www.flickr.com/photos/josephrobertson/92849605/✤ http://en.wikipedia.org/

http://www.xaprb.com	754
http://www.cvilleblogs.com	53
http://newsblur.com	5
http://www.newsblur.com	1

Adaptive Fault Detection

by xaprb on Oct 02, 2012

Accessibility

Categories

Upload Details

Usage Rights

4 Embeds 813

Statistics

Adaptive Fault Detection Presentation Transcript