Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

[B1]real time large data at twitter

5,044 views

Published on

Published in: Technology, Business

[B1]real time large data at twitter

  1. 1. real-time large data @raffi deview - 17 september 2012
  2. 2. there are over400 million tweetsa day
  3. 3. 4600 tweets a twe a second ≈ 0.2 m
  4. 4. Pull PushTargeted twitter.com User / Site Streams home_timeline API Mobile Push (SMS, etc.)Queried Search API Track / Follow Streams
  5. 5. Redis IngesterSearch Cache Redis Fanout Write APITimeline Cache HTTP PushPush Compute HadoopBatch Compute
  6. 6. Write API SocialIngester Fanout Graph Service Batch Compute Timeline Cache Push Compute Search Cache HTTP PushRedis Redis Redis Redis Hadoop Earlybird Redis Mobile Push TimelineBlender Service
  7. 7. Write API SocialIngester Fanout Graph Service Batch Compute Timeline Cache Push Compute Search Cache HTTP PushRedis Redis Redis Redis Hadoop Earlybird Redis Mobile Push TimelineBlender Service
  8. 8. Write APIIngester Fanout Batch Compute Timeline Cache Push Compute Search Cache HTTP PushRedis Redis Redis Redis Hadoop Earlybird Redis Mobile Push TimelineBlender Service
  9. 9. Write APIIngester Fanout Batch Compute Timeline Cache Push Compute Search Index HTTP PushRedis RedisEarlybird Redis Hadoop Earlybird Redis Mobile Push TimelineBlender Service
  10. 10. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Darkwing Flock(s)
  11. 11. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL Tweetypie Monorail T-Bird Gizmoduck T-Flock + Woodstar Haplo TFE TLS Macaw Darkwing +Swift Social Graph Service Macaw Flock(s) +Disco Story Service
  12. 12. timeline delivery statistics⇢ 30b deliveries / day (~20m / min)⇢ 3.5 seconds @ p50 to deliver to 1m⇢ ~350k deliveries / sec
  13. 13. #JoinTheFlock

×