Twitter QCon NY 2013: Isolating Events from the Fail Whale

@Twitter | QCon NY 2013 1
Isolating Events from the Fail Whale
Arun Kejariwal, Bryce Yan
(@arun_kejariwal, @bryce_yan)
Capacity Engineering @ Twitter
June 2013

Delivering Best User Experience
•  Performance
  Real time!
  Latency tolerance of end-users has nose dived
  Average, p99, p999
  Variability on large clusters
  Tolerate faults when using commodity hardware
•  Availability
  Anytime, Anywhere, Any Device
•  Organic Growth
  Over 200M monthly active users
•  Events
  Planned, Unplanned
[3] https://twitter.com/twitter/status/281051652235087872
[2] http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/people/jeff/Berkeley-Latency-Mar2012.pdf
[1] Xu et al. NSDI 2013 - https://www.usenix.org/system/ﬁles/conference/nsdi13/nsdi13-ﬁnal77.pdf
[2]
[3]
[1]

High Performance, Availability
•  Capacity Planning
  Throw hardware at the problem
  Operationally inefﬁcient
  Even otherwise
o  How much?
o  What kind? (Inventory management etc.)
  Reactive approach
  Degraded user experience
o  Impact bottomline
  Overall goal
  Deliver best user experience
  Minimal operational footprint
o  Factor in organic growth and lead times for provisioning additional capacity

Capacity Planning is Non-trivial
•  Behavioral response is unpredictable
•  Multiplier Effect
  # Retweets x Followers of each retweeter
Large fan-out

Capacity Planning is Non-trivial (cont’d)
•  Unforeseen events
  Power failure
  “Hurricane Sandy takes data centers offline with flooding, power outages”
  Network issues
  “Amazon's compute cloud has a networking hiccup”
•  Evolving product development landscape
  New features
  New products
  New partners
  “Twitter Arrives on Wall Street, Via Bloomberg”
[1] http://arstechnica.com/information-technology/2012/10/hurricane-sandy-takes-data-centers-offline-with-flooding-power-outages/
[2] http://www.zdnet.com/amazons-compute-cloud-has-a-networking-hiccup-7000005776/
[4] http://dealbook.nytimes.com/2013/04/04/twitter-arrives-on-wall-street-via-bloomberg/
[3] Ballani et al. NSDI 2013 - https://www.usenix.org/system/files/conference/nsdi13/nsdi13-final186.pdf.
[1]
[2] [3]
[4]
14 June 2013

Capacity Planning is Non-trivial (cont’d)
•  New hardware platforms
  Purchase pipeline
  How much and when to buy – Cost performance trade-off

Events
•  Planned

  Still, trafﬁc pattern subject to, say,
  Nature of the event
  Behavioral response
  Community effect
  Demographics

Events (cont’d)
•  Unplanned

  Intensity of the event
  Population density
Japan Tsunami
New Zealand Earthquake
Hurricane Sandy
Flash Crash
Egyptian Revolution
Iran’s Disputed Election
Boston Explosion
Remembering Steve Jobs

Events (cont’d)
•  Unplanned (transient)

  Duration
  Type of the transient event
White House Rumor: AP account being hacked

[1]
[1] http://ﬁnance.yahoo.com/news/stocks-brieﬂy-drop-recover-fake-172814328.html

Events (cont’d)
•  Black Swans (ala Nassim Taleb)
  Planned events, but…
Superbowl’13 Blackout
Zidane in “Action”
“Hand of God”
Usain Bolt’s 100m World Record

Events (cont’d)
•  Events timeline
Time

Events’ Impact
•  Differ in characteristics
  Tweets
  Photos
  Vines
  Now, Music
•  Consequently, tax different services
  Different capacity requests

Capacity Modeling Overview

Capacity Modeling
•  Takes core drivers as inputs to generate usage demand
  Forecasts the amount of work based on core driver projections
•  Relates the work metric to a primary resource to identify the capacity
threshold
  Primary resources
  Computing power (CPU, RAM)
  Storage (disk I/O, disk space)
  Network (network bandwidth)
•  Generate hardware demand based on the limiting primary resource

Core Drivers
•  Underlying business metrics that drive demand for more capacity
  Active Users
  Tweets per second (TPS)
  Favorites per second (FPS)
  Requests per second (RPS)
•  Normalized by Active Users to isolate user engagement
•  Project user engagement and Active Users independently

Active Users aka User Growth
Normalized Core Drivers for Engagement
Core Drivers (cont’d)
PerActiveUserValues
Time
Favorites
Retweets
Poly. (Favorites)
Linear (Retweets)
ActiveUserCount
Time
Active
Users
Linear (Active
Users)

Core Drivers (cont’d)
Time
User Growth: Active Users
Active
Users
Linear (Active
Users)
Time
Engagement: Photos/Active User
Photos
Linear (Photos)
Time
Core Driver: Photos per Day
Photos
Photos
Forecast

Capacity Threshold
•  Primary resource scalability threshold
  Determined by load testing
  Synthetic load
  Replaying production trafﬁc
  Real-time production trafﬁc
  Test systems may be
  Isolated replicas of production
  Staging systems in production
  Production systems
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
ServiceResponseTime
CPU
Average Response Times vs CPU
X

Hardware Demand
•  Core driver  capacity threshold  scaling formula  server count
•  Example
  Core driver: Requests per Second
  Per server request throughput determined by
capacity threshold
  Scaling formula for Sizing
  Number of Servers = (RPS) / Per Server Threshold
CoreDriver(RPS)/ServerCount
Time
RPS (Actuals)
RPS (Forecast)
# Servers (Actuals)
# Servers (Forecast)

Deep Dive and Superbowl 2013

Events: High Level Methodology
•  Goal
  Handle trafﬁc “spike”
•  Predict expected trafﬁc based on historical and temporal statistical analysis
  Statistical Metrics
  Average
  Standard deviation
  Max
•  Limitations
  Changing usage patterns
  Organic growth, behavioral, cultural
  Event driven
  How a game would turn out?

Statistical Time Series Analysis
•  Time window
  Week over Week (WoW)
  Month over Month (MoM)
  Year over Year (YoY)
•  Data Distribution
  Normal, Log Normal, Multi-modal
  Has implications on model selection
•  Forecasting
  Regression model
  Linear, Spline
  ARIMA
  Trending, Seasonal, Residuals

Superbowl 2013: Capacity Planning
•  Assess capacity requirement based 2011, 2012 Superbowl trafﬁc patterns

•  Core driver selection
  RPS (Reads)
  TPS (Writes)

•  What time granularity to use?
  Avg TPS (Tweets per sec)
  1s/10s/15s/30s Max TPS
  1 min/5 min/10 min Max TPS
  1 hr Max TPS

Superbowl 2013: Capacity Planning (cont’d)
•  Which metric to use?
Time
Highly correlated

•  Which metric to use?
  Time sensitive – correlation may change YoY
Time
Highly correlated

•  Approaches
  TPSSuperbowl (denote by Tn)
  d-Day historical window
  TPSn-1, TPSn-2, …, TPSn-d
  Ratio Analysis
  Rn = Tn/Max(TPSn-1, TPSn-2, …, TPSn-d)
  Distribution Analysis
  αn = (Tn - AVG(TPSn-1, TPSn-2, …, TPSn-d))/STDEV(TPSn-1, TPSn-2, …, TPSn-d)

•  Ratio Analysis (Rn)
  1s Max TPS
14 Day
28 day
45 Day
2011
0.791
0.791
1.007
2012
1.062
0.858
0.580

μ
•  Distribution Analysis (αn)
  AVG (μ), STDEV(σ)
  μ increased YoY (expected)
  σ also increased YoY
  1s Max TPS
Tn /μ
(Tn – μ)/σ
2011
1.448
1.746
2012
1.517
2.756
TPS during Superbowl has been
moving right YoY
2011
2012

•  Distribution Analysis
  YoY movement of TPSSuperbowl further into the right tail
  Expectation: Progressive moves would be smaller

  Overestimate α
  Handle unplanned events
  Business decision

•  Historical component
  Determine extent of movement (αexpected) of TPSSuperbowl into right tail

•  Temporal component
  Current μc
  Current σc

•  Capacity planning
  Plan capacity corresponding to μc + αexpected * σc
  Scenario Analysis (ala Global Macro Hedge Funds)
  αexpected
o  αn-1 (same as last year)
o  αn-1 + (αn-1 + αn-2)/2 (extrapolate from last two years)

•  Capacity planning
  1s Max TPS
  αn-1  20K+
  αn-1 + (αn-1 + αn-2)/2  22K+

•  Validation
  1s Max TPS
  αobserved < αexpected

  Twitter was highly available during Superbowl 2013
  Over-allocation concerns?
  Minimal
  Limited to few services
  Seamlessly handled trafﬁc spike due to the Superbowl 2013 Blackout

Join the Flock
•  We are hiring!
  https://twitter.com/JoinTheFlock
  https://twitter.com/jobs

Twitter QCon NY 2013: Isolating Events from the Fail Whale

Recommended

Recommended

More Related Content

Similar to Twitter QCon NY 2013: Isolating Events from the Fail Whale

Similar to Twitter QCon NY 2013: Isolating Events from the Fail Whale (20)

More from Arun Kejariwal

More from Arun Kejariwal (16)

Recently uploaded

Recently uploaded (20)

Twitter QCon NY 2013: Isolating Events from the Fail Whale