• Save
QCon NYC 2012: Twitter's Real Time Architecture
Upcoming SlideShare
Loading in...5
×
 

QCon NYC 2012: Twitter's Real Time Architecture

on

  • 27,947 views

With hundreds of millions of users, Twitter operates one of the world's largest real-time delivery systems, large enough and pervasive enough to exert noticeable "pressure" on the overall internet ...

With hundreds of millions of users, Twitter operates one of the world's largest real-time delivery systems, large enough and pervasive enough to exert noticeable "pressure" on the overall internet itself. At steady state, Twitter receives thousands of tweets a second that it needs to deliver to disks, in-memory timelines, email, and mobile devices. The name of the game for Twitter is "now", so those deliveries, which multiply according to the graph of who follows whom, need to occur in real-time. In this session, we will dive into both the "write path" and "read path" of Twitter to understand the architecture which supports those tweets, and also how Twitter serves them through one of the world's largest web sites.

animated slides available at animated slides available at http://www.youtube.com/watch?v=l55jFAGsgbs

Statistics

Views

Total Views
27,947
Slideshare-icon Views on SlideShare
26,381
Embed Views
1,566

Actions

Likes
86
Downloads
0
Comments
5

25 Embeds 1,566

http://cptl.corp.yahoo.co.jp 1004
http://www.scoop.it 350
https://twitter.com 63
http://test.localhost 30
http://bizpremium.nikkeibp.co.jp 21
http://us-w1.rockmelt.com 14
http://cms.logly.jp 14
http://pinterest.com 11
http://www.twylah.com 11
https://si0.twimg.com 10
http://local.cf.com 7
http://younghands.org 6
http://artegoticod.blogspot.com 4
http://tweetedtimes.com 4
http://real.commons.co.kr 3
http://www.pinterest.com 3
http://local.microhint.com 2
http://translate.googleusercontent.com 2
http://www.linkedin.com 1
http://blog.geekple.com 1
http://www.pradtimes.com.cp-28.webhostbox.net 1
http://staging2.intratuin.nl 1
https://twimg0-a.akamaihd.net 1
http://safe.tumblr.com 1
https://www.linkedin.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

15 of 5 Post a comment

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • good
    Are you sure you want to
    Your message goes here
    Processing…
  • pretty good
    Are you sure you want to
    Your message goes here
    Processing…
  • neighbor's step-aunt brought in $19301 the previous month. she is making income on the internet and got a $549400 house. All she did was get blessed and put into action the advice revealed on this web site snipr.com/2400vf5
    Are you sure you want to
    Your message goes here
    Processing…
  • nice
    Are you sure you want to
    Your message goes here
    Processing…
  • great
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    QCon NYC 2012: Twitter's Real Time Architecture QCon NYC 2012: Twitter's Real Time Architecture Presentation Transcript

    • real-time delivery architecture @raffi qcon nyc 2012
    • http://twitpic.com/135xa -There’s a plane in theHudson. I’m on a ferrygoing to pick up thepeople. Crazy.15 Jan 09Janis Krums @jkrums
    • this is barack - lets getthis started! -bo24 May 12The White House@whitehouse
    • what are the goals?⇢ evolve from being solely a web stack
    • ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Darkwing Flock(s)
    • what are the goals?⇢ evolve from being solely a web stack⇢ isolate responsibilities and concerns⇢ site speed and reliability⇢ developer innovation speed
    • Pull PushTargeted twitter.com User / Site Streams home_timeline API Mobile Push (SMS, etc.)Queried Search API Track / Follow Streams
    • Write APIIngester Fanout Batch Compute Timeline Cache Push Compute Search Cache HTTP PushRedis Redis Redis Redis Hadoop Earlybird Redis Mobile Push TimelineBlender Service
    • Write API Social Ingester Fanout Graph Serviceinsert Batch Compute Timeline Cache Push Compute Search Cache HTTP Push Redis Redis Redis Redis Hadoop Earlybird Redis⇢ keyed off Mobile Push “recipient” Timeline⇢ pipelined 4k Blender Service “destinations” at a time⇢ replicated
    • Write API Ingester Fanoutusing redis Batch Compute Timeline Cache Push Compute Search Cache Tweet IDPush User ID HTTP Bits Redis Redis Redis Redis Hadoop Earlybird Redis 8 bytes 8 bytes 4 bytes⇢ native list Mobile Push structure Timeline⇢ RPUSHX to Blender Service only add to cached timelines
    • Write API Ingester Fanoutusing redis Batch Compute Timeline Cache Push Compute Search Cache Tweet IDPush User ID HTTP Bits Redis Redis Redis Redis Tweet ID User ID Bits Hadoop Earlybird Redis⇢ native list Mobile Tweet ID User ID Bits Push structure Tweet ID User ID Bits Tweet ID User ID Bits Timeline⇢ RPUSHX to Blender Tweet ID User ID Bits Service Tweet ID User ID Bits only add to Tweet ID User ID Bits cached Tweet ID User ID Bits timelines Tweet ID User ID Bits
    • Write APIIngester Fanout Batch Compute Timeline Cache Push Compute Search Cache HTTP PushRedis Redis Redis Redis Hadoop Earlybird Redis Mobile Push TimelineBlender Service
    • Pull PushTargeted twitter.com User / Site Streams home_timeline API Mobile Push (SMS, etc.)Queried Search API Track / Follow Streams
    • Write APIIngester Fanout Batch Compute Timeline Cache Push Compute Search Index HTTP PushRedis RedisEarlybird Redis Hadoop Earlybird Redis Mobile Push TimelineBlender Service
    • Write APIIngester Fanout blender Batch Compute Timeline Cache Push Compute Search Index HTTP PushRedis RedisEarlybird Redis Hadoop Earlybird Redis ⇢ queries one Mobile Push replica of allBlender Timeline Service indexes ⇢ merges & ranks results
    • Write APIIngester Fanout Batch Compute Timeline Cache Push Compute Search Index HTTP PushRedis RedisEarlybird Redis Hadoop Earlybird Redis Mobile Push TimelineBlender Service
    • Pull PushTargeted twitter.com User / Site Streams home_timeline API Mobile Push (SMS, etc.)Queried Search API Track / Follow Streams
    • Write APIIngester Fanout Batch Compute Timeline Cache Push Compute Search Index HTTP PushRedis RedisEarlybird Redis Hadoop Earlybird Redis Mobile Push TimelineBlender Service
    • http push / hosebird⇢ maintains persistent connections with end clients⇢ processes tweet & social graph events⇢ event-based “router”
    • Hosebird Firehose Write API Hosebird User Streams Hosebird Track / Followevent propagation⇢ write API sends all events into hosebird; sees content creation events, social graph changes, etc.⇢ different queues for public tweets, protected tweets, social events, etc.
    • Hosebird Firehose Write API Hosebird User Streams Hosebird Track / Followevent cascading⇢ bandwidth management⇢ simultaneous connection management (~1m long lived & open connections to this cluster)
    • Hosebird Firehose Write API Hosebird User Streams Hosebird Track / Followfirehose⇢ edge machine simply outputs the public tweet queue⇢ only allow a limited number of firehoses per hosebird box for bandwidth management
    • Hosebird Firehose Write API Hosebird Track / Follow Hosebird User Streamstrack / follow⇢ simple query based on tweet content⇢ keeps list of terms / users of interest⇢ parses public tweets at the edge, and if term matches a token, or user is of interest, then route
    • Hosebird Firehose Write API Hosebird Track / Follow Hosebird User Streamsuser streams⇢ replicate home timeline experience⇢ upon login, obtain “following” list⇢ keep cached following list coherent by seeing social graph updates⇢ route tweet if from a followed user
    • Write APIIngester Fanout Batch Compute Timeline Cache Push Compute Search Index HTTP PushRedis RedisEarlybird Redis Hadoop Earlybird Redis Mobile Push TimelineBlender Service
    • Pull PushTargeted twitter.com User / Site Streams home_timeline API Mobile Push (SMS, etc.)Queried Search API Track / Follow Streams
    • Write APIIngester Fanout Batch Compute Timeline Cache Push Compute Search Index HTTP PushRedis RedisEarlybird Redis Hadoop Earlybird Redis Mobile Push TimelineBlender Service
    • Write APIIngester Fanout Batch Compute Timeline Cache Push Compute Search Index HTTP PushRedis RedisEarlybird Redis Hadoop Earlybird Redis Social Mobile Graph Push Service TimelineBlender Service
    • Write APIIngester Fanout Batch Compute Timeline Cache Push Compute Search Index HTTP PushRedis RedisEarlybird Redis Hadoop Earlybird Redis Social Mobile Graph Push Service TimelineBlender Service
    • Write APIIngester Fanout Batch Compute Timeline Cache Push Compute Search Index HTTP PushRedis RedisEarlybird Redis Hadoop Earlybird Redis Mobile Push TimelineBlender Service
    • Pull PushTargeted twitter.com User / Site Streams home_timeline API Mobile Push (SMS, etc.)Queried Search API Track / Follow Streams
    • Write APIIngester Fanout Batch Compute Timeline Cache Push Compute Search Index HTTP PushRedis RedisEarlybird Redis Hadoop Earlybird Redis Mobile Push TimelineBlender Service
    • Write APIIngester Fanout Batch Compute Timeline Cache Push Compute Search Index HTTP PushRedis RedisEarlybird Redis Hadoop Earlybird Redis Mobile Push TimelineBlender Service
    • Synchronous Path Write APIIngester Fanout Asynchronous Path Batch Compute Timeline Cache Push Compute Search Index HTTP PushRedis RedisEarlybird Redis Hadoop Earlybird Redis Mobile Push TimelineBlender Query Path Service
    • Synchronous Path Write APIIngester Fanout Asynchronous Path Batch Compute Timeline Cache Push Compute Search Index HTTP PushRedis RedisEarlybird Redis Hadoop Earlybird Redis Mobile Push TimelineBlender Query Path Service
    • Synchronous Path Write APIIngester Fanout Asynchronous Path Batch Compute Timeline Cache Push Compute Search Index HTTP PushRedis RedisEarlybird Redis Hadoop Earlybird Redis Mobile Push TimelineBlender Query Path Service
    • Write APIRead Path Write Path Ingester Fanout Batch Compute Timeline Cache Push Compute Search Index HTTP Push Redis Redis Earlybird Redis Hadoop Earlybird Redis Mobile Push Timeline Blender Service
    • Write APIRead Path Write Path Ingester Fanout Batch Compute Timeline Cache Push Compute Search Index HTTP Push Redis Redis Earlybird Redis Hadoop Earlybird Redis Mobile Push Timeline Blender Service
    • things we’re trying...
    • Write APIIngester Fanout Timeline CacheRedis Search Index RedisEarlybird Redis Earlybird Redis
    • Write API Ingester Fanout Timeline Cache Redis Search Index Redis Earlybird Redis Earlybird Redissearch index fanout index⇢[‘hello’,‘world’] ⇢[@danadanger, ...]
    • User Intent Query Expansion “Hello, world” “Hello” AND “world”@raffi’s home timeline home_timeline:raffi
    • User Intent Query Expansion “Hello, world” “Hello” AND “world” user_timeline:nelson@raffi’s home timeline OR user_timeline:danadanger
    • Write APIfan-in fan-out⇢ O(1) write ⇢ O(n) write Ingester Fanout⇢ O(n) read ⇢ O(1) read Timeline Cache Redis Search Index Redis Earlybird Redis Earlybird Redis
    • User Intent Query Expansion “Hello, world” “Hello” AND “world”@raffi’s home timeline home_timeline:raffi
    • User Intent Query Expansion “Hello, world” “Hello” AND “world” home_timeline:raffi@raffi’s home timeline OR user_timeline:taylorswift13
    • streaming compute⇢ continuous computation⇢ driven by the events that come into twitter⇢ generalizing the push mechanism
    • Write APIIngester Fanout Batch Compute Timeline Cache Push Compute Search Index HTTP PushRedis RedisEarlybird Redis Hadoop Earlybird Redis Mobile Push TimelineBlender Service
    • timeline query statistics⇢ >150m active users worldwide⇢ 300k qps poll-based timelines @ 1ms p50 / 4ms p99⇢ 30k qps search-based timelines
    • tweet input⇢ ~340m tweets per day⇢ ~4K/sec daily average⇢ ~6K/sec daily peak⇢ >10K/sec during large events
    • followed by following
    • timeline delivery statistics⇢ 26b deliveries / day (~18m / min)⇢ 3.5 seconds @ p50 to deliver to 1m⇢ ~300k deliveries / sec
    • thanks!