• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Raffi Krikorian, Twitter Timelines at Scale
 

Raffi Krikorian, Twitter Timelines at Scale

on

  • 8,746 views

Charla de Raffi Krikorian, el VP de Ingeniería de Twitter, sobre la infraestructura de los Timelines de twitter

Charla de Raffi Krikorian, el VP de Ingeniería de Twitter, sobre la infraestructura de los Timelines de twitter

Statistics

Views

Total Views
8,746
Views on SlideShare
5,741
Embed Views
3,005

Actions

Likes
6
Downloads
35
Comments
1

14 Embeds 3,005

http://www.uberbin.net 2414
http://cloud.feedly.com 467
http://platformweb100 56
http://reader.aol.com 17
http://digg.com 13
http://hal9000 12
http://www.newsblur.com 6
http://www.google.com 6
http://feedproxy.google.com 4
https://drive.jolicloud.com 3
http://www.feedspot.com 3
http://newsblur.com 2
http://feeds.feedburner.com 1
http://beta.jolicloud.co 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

11 of 1 previous next

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • erda q video de chebere soi colonbiano
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Raffi Krikorian, Twitter Timelines at Scale Raffi Krikorian, Twitter Timelines at Scale Presentation Transcript

    • timelines at scale @raffi qcon sf 2012
    • Pull Push Targeted twitter.com home_timeline API User / Site Streams Mobile Push (SMS, etc.) Queried Search API Track / Follow Streams
    • the challenge ⇢> 150M world wide active users ⇢> 300K QPS for timelines ⇢naïve timeline “materialization” can be slow
    • Timeline Service Ingester SearchCache Redis Redis Earlybird Blender PushCompute HTTP Push Mobile Push BatchCompute Hadoop Write API Fanout Redis Redis TimelineCache Redis
    • Timeline Service Ingester SearchCache Redis Redis Earlybird Blender PushCompute HTTP Push Mobile Push BatchCompute Hadoop Write API Fanout Redis Redis TimelineCache Redis Social Graph Service
    • Timeline Service Ingester SearchCache Redis Redis Earlybird Blender PushCompute HTTP Push Mobile Push BatchCompute Hadoop Write API Fanout Redis Redis TimelineCache Redis Social Graph Service insert ⇢keyed off “recipient” ⇢pipelined 4k “destinations” at a time ⇢replicated
    • Timeline Service Ingester SearchCache Redis Redis Earlybird Blender PushCompute HTTP Push Mobile Push BatchCompute Hadoop Write API Fanout Redis Redis TimelineCache Redis using redis ⇢native list structure Tweet ID BitsUser ID 8 bytes 4 bytes8 bytes
    • Timeline Service Ingester SearchCache Redis Redis Earlybird Blender PushCompute HTTP Push Mobile Push BatchCompute Hadoop Write API Fanout Redis Redis TimelineCache Redis using redis ⇢native list structure ⇢RPUSHX to only add to cached timelines Tweet ID BitsUser ID Tweet ID BitsUser ID Tweet ID BitsUser ID Tweet ID BitsUser ID Tweet ID BitsUser ID Tweet ID BitsUser ID Tweet ID BitsUser ID Tweet ID BitsUser ID Tweet ID BitsUser ID Tweet ID BitsUser ID Tweet ID Tweet ID Tweet ID
    • Ingester SearchCache Redis Redis Earlybird Blender PushCompute HTTP Push Mobile Push BatchCompute Hadoop Write API Fanout Redis Redis TimelineCache Timeline Service Redis
    • Timeline Service Write API Fanout Redis Redis TimelineCache Redis TweetyPieGizmoduck
    • Pull Push Targeted twitter.com home_timeline API User / Site Streams Mobile Push (SMS, etc.) Queried Search API Track / Follow Streams
    • Ingester SearchCache Redis Redis Earlybird Blender PushCompute HTTP Push Mobile Push BatchCompute Hadoop Write API Fanout Redis Redis TimelineCache Timeline Service Redis
    • PushCompute HTTP Push Mobile Push BatchCompute HadoopSearchIndex Blender Redis Timeline Service Ingester Earlybird Write API Fanout Redis Redis TimelineCache RedisEarlybird blender ⇢queries one replica of all indexes ⇢merges & ranks results
    • PushCompute HTTP Push Mobile Push BatchCompute HadoopSearchIndex Blender Redis Timeline Service Ingester Earlybird Write API Fanout Redis Redis TimelineCache RedisEarlybird
    • Write API Redis Redis Redis Write API Earlybird Earlybird Earlybird API Cache Read API Redis Redis Redis Read API Earlybird Earlybird Earlybird API Cache ⇢O(n) write ⇢O(1) write ⇢O(1) read ⇢O(n) read
    • the challenge (part #2) ⇢fanout can be really slow! ⇢...especially for high follower counts
    • @barackobama 23 million followers 31 million followers @katyperry 28 million followers @justinbieber 28 million followers @raffi 0.019 million followers @ladygaga
    • there are over 400 million tweets a day
    • a second 4600 tweets 0.2 m a twe ≈
    • Write API Ingester Fanout SearchIndexRedis Earlybird Earlybird Redis Redis Redis TimelineCache search index ⇢[‘hello’,‘world’] fanout index ⇢[@danadanger, ...]
    • User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi
    • User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline user_timeline:nelson OR user_timeline:danadanger
    • User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi
    • User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi OR user_timeline:taylorswift13
    • BatchCompute Hadoop PushCompute HTTP Push SearchIndex Blender Redis Timeline Service Ingester Earlybird Write API Fanout Redis Redis TimelineCache RedisEarlybird Mobile Push
    • Asynchronous Path Query Path BatchCompute Hadoop Synchronous Path PushCompute HTTP Push SearchIndex Blender Redis Timeline Service Ingester Earlybird Write API Fanout Redis Redis TimelineCache RedisEarlybird Mobile Push
    • Synchronous Path Query Path BatchCompute Hadoop Asynchronous Path PushCompute HTTP Push SearchIndex Blender Redis Timeline Service Ingester Earlybird Write API Fanout Redis Redis TimelineCache RedisEarlybird Mobile Push
    • Asynchronous Path Synchronous Path BatchCompute Hadoop Query Path PushCompute HTTP Push SearchIndex Blender Redis Timeline Service Ingester Earlybird Write API Fanout Redis Redis TimelineCache RedisEarlybird Mobile Push
    • timeline query statistics ⇢>150m active users worldwide ⇢>300k qps poll-based timelines @ 1ms p50 / 4ms p99 ⇢>30k qps search-based timelines
    • tweet input ⇢~400m tweets per day ⇢~5K/sec daily average ⇢~7K/sec daily peak ⇢>12K/sec during large events
    • timeline delivery statistics ⇢30b deliveries / day (~21m / min) ⇢3.5 seconds @ p50 to deliver to 1m ⇢~300k deliveries / sec
    • thanks!