Mitigating User Experience from 'Breaking Bad': The Twitter Approach [Velocity New York 2014]

Mitigating User Experience from 'Breaking Bad’
@Twitter | PK, AK 2014 1
The Twitter Approach
Piyush Kumar, Arun Kejariwal
(@pi_kumar, @arun_kejariwal)
Performance and Capacity Engineering @ Twitter
September 2014

Internet Usage
• Increasing User Base and Data Traffic
 >4 Billion Internet users by 2018, up from 2.5 B in 2013 [1]
 9.2 B mobile subscriptions by 2019 [2]
 10x growth in mobile data traffic between 2013 and 2019 [2]
 Sales of wearable devices to reach 111.9 million units by 2018 [3]
• Organic growth
 Over 271 M monthly active users*
 Over 173 B Timeline Views*
[1] http://www.cisco.com/c/en/us/solutions/service-provider/visual-networking-index-vni/index.html#~overview (June 2014)
[2] http://www.ericsson.com/res/docs/2014/ericsson-mobility-report-june-2014.pdf
[3] http://online.wsj.com/articles/apple-plans-multiple-designs-for-smartwatch-1403245062
* https://investor.twitterinc.com/releasedetail.cfm?ReleaseID=862505

Internet Usage (contd.)
• Real-time consumption
[2]
[1]
[1] http://socialtimes.com/twitter-succeeded-espn-failed_b199578 [2] http://online.wsj.com/articles/airlines-take-to-twitter-to-keep-world-informed-1406125253

Systematic Capacity Planning: Why Bother?
• Evolving product landscape
 Agile development
 New products
 Experimentation
 New features
 A/B Testing
• Organic user growth
• Growing user engagement

Factors Impacting Capacity Planning
• Breakout(s)
 Unexpected ramp up

Factors Impacting Capacity Planning (contd.)
• Breakout(s)
 Unexpected mean shift

Breakout Detection: How?
• Visual
 Large number of services
 Not scalable
 @ Twitter – Algorithmic approach
 Detect breakout(s) via statistical learning

Breakout Detection: Why Bother?
• Detect cold and hot services
 Cold
 Optimization
 Hot
 Higher traffic
 Operational
 Efficiency
 Minimize performance impact
• Detect Performance Regressions
• Characterize user engagement
 Assess an A/B Test
 Roll out for a new platform

Breakout Detection (contd.)
• Key Questions
 How do we characterize a breakout?
 Input
o Time Series
 Magnitude
o Sustained increase or decrease
 Direction
o Positive – Rising services
o Negative – Cold services
 Account for characteristics of time series
 Local spikes
o Removed using rolling median
• Commonly used in computer vision for noise reduction, called edge thinning
 Seasonality
o May induce false positive(s)

Existing Approaches
• Support and Resistance
 Donchian Channels
• Limitations
 Establishing support and resistance zones

Existing Approaches (contd.)
• MACD (Moving Average Convergence Divergence)
 Finance
 MACD: Moving Average Convergence Divergence. Used in finance.
o Difference between fast (12 day) and slow (26 day) EMA (Exponential Moving Average)
 Signal
o 9 day EMA of MACD
 Histogram
o MACD - Signal

Existing Approaches (contd.)
• MACD (Moving Average Convergence Divergence)
• Limitations
 How to determine the appropriate period of MACD?
 Inaccurate

Our Approach
• Applying the change point detection algorithm
 Rolling Median of the raw time series
 Raw time series
 Very susceptible to the noise or outliers
 Moving average
 SMA or EMA
 Mean is susceptible to outliers
o SMA and EMA are sensitive to noise for a small window size [1]
[1] http:
//www.mathworks.com/help/signal/examples/signal-smoothing.html
Useful
Not useful
Not useful

Our Approach (contd.)
• Step 1: Compute rolling median
 Definition
 Take the median of a given window size, use it as a single point. That window slides over every point in
the time series.
 Properties of rolling median
 Robust to anomalies
o Better capture the underlying trend
o Strong noise attenuation property
o Produces a consistent and unbiased estimate of input distribution [1]
[1] Non Linear Signal Processing – Ganzalo R. Arce, Page 84

• Hypothesis testing
To determine if a significant change point exists or not.
 H0 – Null Hypothesis
 No change point exists
 H1 – Alternative Hypothesis
 One significant change point exists
 Test Statistic
 Test statistic is a quantity calculated from our sample of data.
 Its value is used to decide whether or not the null hypothesis should be rejected in our hypothesis test
 Critical Value/Threshold
 Threshold to which test statistic is compared to determine if the null hypothesis should be rejected.

• Step 2: Change-point detection
 Hypothesis testing [Change point Detection - David Hinkley, 1970]
 Maximize Log-likelihood Estimation (MLE)
 Incoming data is represented by data, y1:n ∈ (y1, …, yn )
 For a model with a change point at τ1where τ1 ∈ {1, 2, ….n-1}, the maximum likelihood is
 p(.) is the probability density function, Θ – MLE of parameter
 Inference
o Test Statistic
H1 – Alternative
Hypothesis
o If λ > c, then we reject the null hypothesis, where ‘c’ is the threshold.
o Value of c is still an open question in statistical research

Input time series
• Illustration – Detecting mean shift
Anomaly, not mean shift
Rolling median, input to
change-point algorithm
Input Time series

Comparison evaluation
Time Series Input
Change point Algorithm vs. MACD and Donchian Channels
Donchian Channels
MACD
Change point Algorithm

• Detecting positive/negative breakouts
 Hot services (initial mean < final mean)
 Cold services (initial mean > final mean)
• Minimizing false positives
 Define mean shift threshold
 Measuring mean shift in the presence of anomalous data

Our Approach: Evaluation (contd.)
• Ramp up

• Mean shift

• Detecting Breakout in presence of anomalies
Anomaly
Mean shift happening here,
not visible to the human eye

• How things look like after anomaly removal
~17% jump!

• Robust in the presence of anomalies
Input time series with lots of spikes
Rolling median of the input time series

• Other Data – Stocks

• Gold Price

• Boundary conditions and Limitations
 Address sensitivity during the training period
 Minimize false positives
 Threshold change
 Non normally distributed data

Future Work
• Heuristic
 Length of breakout segment >10% of total length
• Learn the threshold over time

Acknowledgments
• Capacity Engineering Team at Twitter

Mitigating User Experience from 'Breaking Bad': The Twitter Approach [Velocity New York 2014]

Recommended

Recommended

More Related Content

What's hot

What's hot (18)

Viewers also liked

Viewers also liked (20)

Similar to Mitigating User Experience from 'Breaking Bad': The Twitter Approach [Velocity New York 2014]

Similar to Mitigating User Experience from 'Breaking Bad': The Twitter Approach [Velocity New York 2014] (20)

Recently uploaded

Recently uploaded (20)

Mitigating User Experience from 'Breaking Bad': The Twitter Approach [Velocity New York 2014]