Q con shanghai2013-[jains krums]-[real-time-delivery-archiecture]

2,153 views
2,051 views

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,153
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Q con shanghai2013-[jains krums]-[real-time-delivery-archiecture]

  1. 1. real-time delivery architecture @raffi qcon shanghai 2013
  2. 2. @raffi qcon shanghai 2013
  3. 3. http://twitpic.com/135xa There’s a plane in the Hudson. I’m on a ferry going to pick up the people. Crazy. ! 15 Jan 09 Janis Krums @jkrums @raffi qcon shanghai 2013
  4. 4. what are the goals? ⇢ evolve from being solely a web stack @raffi qcon shanghai 2013
  5. 5. @raffi qcon shanghai 2013
  6. 6. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Gizmoduck Flock(s) @raffi qcon shanghai 2013
  7. 7. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Gizmoduck Flock(s) @raffi qcon shanghai 2013
  8. 8. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Gizmoduck Flock(s) @raffi qcon shanghai 2013
  9. 9. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Gizmoduck Flock(s) @raffi qcon shanghai 2013
  10. 10. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Gizmoduck Flock(s) @raffi qcon shanghai 2013
  11. 11. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Gizmoduck Flock(s) @raffi qcon shanghai 2013
  12. 12. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Darkwing Flock(s) @raffi qcon shanghai 2013
  13. 13. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Darkwing Flock(s) @raffi qcon shanghai 2013
  14. 14. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Darkwing Flock(s) @raffi qcon shanghai 2013
  15. 15. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Darkwing Flock(s) @raffi qcon shanghai 2013
  16. 16. what are the goals? ⇢ evolve from being solely a web stack ⇢ isolate responsibilities and concerns ⇢ site speed and reliability ⇢ developer innovation speed @raffi qcon shanghai 2013
  17. 17. @raffi qcon shanghai 2013
  18. 18. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
  19. 19. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
  20. 20. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
  21. 21. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
  22. 22. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
  23. 23. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
  24. 24. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
  25. 25. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
  26. 26. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
  27. 27. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
  28. 28. @raffi qcon shanghai 2013
  29. 29. @raffi qcon shanghai 2013
  30. 30. @raffi qcon shanghai 2013
  31. 31. @raffi qcon shanghai 2013
  32. 32. @raffi qcon shanghai 2013
  33. 33. Write API Ingester Hadoop Batch Compute HTTP Push Push Compute Redis Timeline Cache Search Cache Redis Fanout @raffi qcon shanghai 2013
  34. 34. Write API Ingester @raffi qcon shanghai 2013
  35. 35. Write API @raffi qcon shanghai 2013
  36. 36. Write API Ingester Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout @raffi qcon shanghai 2013
  37. 37. Write API Ingester Hadoop Batch Compute Push Compute Search Cache Fanout @raffi qcon shanghai 2013
  38. 38. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  39. 39. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  40. 40. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  41. 41. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  42. 42. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  43. 43. Write API Blender HTTP Push Mobile Push Hadoop Batch Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Push Compute Ingester Social Graph Service Timeline Service @raffi qcon shanghai 2013
  44. 44. Write API ⇢ pipelined 4k “destinations” at a time Blender ⇢ replicated HTTP Push Mobile Push Hadoop Batch Compute ⇢ keyed off “recipient” Redis Redis Redis Timeline Cache Redis Redis Earlybird Search Cache insert Fanout Push Compute Ingester Social Graph Service Timeline Service @raffi qcon shanghai 2013
  45. 45. Write API Ingester Blender ⇢ RPUSHX to only add to cached timelines Tweet IDPush User ID HTTP 8 bytes 8 bytes Mobile Push Bits Hadoop 4 bytes Batch Compute ⇢ native list structure Redis Redis Redis Push Compute Redis Redis Earlybird Search Cache using redis Timeline Cache Fanout Timeline Service @raffi qcon shanghai 2013
  46. 46. Write API Ingester Blender ⇢ RPUSHX to only add to cached timelines Tweet IDPush User ID HTTP Bits Tweet ID Bits Hadoop User ID Bits Tweet ID User ID Bits Tweet ID Timeline Service Mobile Tweet ID User ID Push User ID Bits Tweet ID User ID Bits Tweet ID User ID Bits Tweet ID User ID Bits Tweet ID User ID Bits Tweet ID User ID Batch Compute ⇢ native list structure Redis Redis Redis Push Compute Redis Redis Earlybird Search Cache using redis Timeline Cache Fanout Bits @raffi qcon shanghai 2013
  47. 47. Write API Ingester Blender ⇢ RPUSHX to only add to cached timelines Tweet IDPush User ID HTTP Bits Tweet ID Bits Hadoop ID Tweet User ID Bits Tweet ID User ID Bits Tweet ID Timeline Service Mobile Tweet ID User ID Push User ID Bits Tweet ID User ID Bits Tweet ID User ID Bits Tweet ID User ID Bits Tweet ID User ID Bits Tweet ID User ID Batch Compute ⇢ native list structure Redis Redis Redis Push Compute Redis Redis Earlybird Search Cache using redis Timeline Cache Fanout Bits Tweet ID Tweet ID @raffi qcon shanghai 2013
  48. 48. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  49. 49. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  50. 50. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  51. 51. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  52. 52. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
  53. 53. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  54. 54. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  55. 55. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  56. 56. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  57. 57. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  58. 58. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  59. 59. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  60. 60. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  61. 61. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  62. 62. Write API Ingester Blender Timeline Service HTTP Push Hadoop ⇢ queries one replica of all indexes Mobile Push Batch Compute blender Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout ⇢ merges & ranks results @raffi qcon shanghai 2013
  63. 63. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  64. 64. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  65. 65. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  66. 66. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  67. 67. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
  68. 68. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  69. 69. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  70. 70. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  71. 71. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  72. 72. http push / hosebird ⇢ maintains persistent connections with end clients ⇢ processes tweet & social graph events ⇢ event-based “router” @raffi qcon shanghai 2013
  73. 73. Hosebird Write API Hosebird Firehose User Streams Hosebird Track / Follow Text @raffi qcon shanghai 2013
  74. 74. Hosebird Write API Hosebird Firehose User Streams Hosebird Track / Follow @raffi qcon shanghai 2013
  75. 75. Hosebird Write API Hosebird Firehose User Streams Hosebird Track / Follow event propagation ⇢ write API sends all events into hosebird; sees content creation events, social graph changes, etc. ⇢ different queues for public tweets, protected tweets, social events, etc. @raffi qcon shanghai 2013
  76. 76. Hosebird Write API Hosebird Firehose User Streams Hosebird Track / Follow @raffi qcon shanghai 2013
  77. 77. Hosebird Write API Hosebird Firehose User Streams Hosebird Track / Follow event cascading ⇢ bandwidth management ⇢ simultaneous connection management (~1m long lived & open connections to this cluster) @raffi qcon shanghai 2013
  78. 78. Hosebird Write API Hosebird Firehose User Streams Hosebird Track / Follow @raffi qcon shanghai 2013
  79. 79. Hosebird Write API Hosebird Firehose User Streams Hosebird Track / Follow firehose ⇢ edge machine simply outputs the public tweet queue ⇢ only allow a limited number of firehoses per hosebird box for bandwidth management @raffi qcon shanghai 2013
  80. 80. Hosebird Write API Hosebird Firehose Track / Follow Hosebird User Streams @raffi qcon shanghai 2013
  81. 81. Hosebird Write API Hosebird Firehose Track / Follow Hosebird User Streams track / follow ⇢ simple query based on tweet content ⇢ keeps list of terms / users of interest ⇢ parses public tweets at the edge, and if term matches a token, or user is of interest, then route @raffi qcon shanghai 2013
  82. 82. Hosebird Write API Hosebird Firehose Track / Follow Hosebird User Streams @raffi qcon shanghai 2013
  83. 83. Hosebird Write API Hosebird Firehose Track / Follow Hosebird User Streams user streams ⇢ replicate home timeline experience ⇢ upon login, obtain “following” list ⇢ keep cached following list coherent by seeing social graph updates ⇢ route tweet if from a followed user @raffi qcon shanghai 2013
  84. 84. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  85. 85. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  86. 86. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  87. 87. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
  88. 88. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  89. 89. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  90. 90. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  91. 91. Write API Ingester Blender Mobile Push Hadoop Social Graph Service Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  92. 92. Write API Ingester Blender Mobile Push Hadoop Social Graph Service Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  93. 93. Write API Ingester Blender Mobile Push Hadoop Social Graph Service Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  94. 94. Write API Ingester Blender Mobile Push Hadoop Social Graph Service Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  95. 95. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  96. 96. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
  97. 97. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  98. 98. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  99. 99. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  100. 100. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  101. 101. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  102. 102. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  103. 103. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  104. 104. Synchronous Path Write API Ingester Blender Timeline Service Mobile Push Asynchronous Path Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Query Path @raffi qcon shanghai 2013
  105. 105. Synchronous Path Write API Ingester Blender Timeline Service Mobile Push Asynchronous Path Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Query Path @raffi qcon shanghai 2013
  106. 106. Synchronous Path Write API Ingester Blender Timeline Service Mobile Push Asynchronous Path Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Query Path @raffi qcon shanghai 2013
  107. 107. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  108. 108. Blender Write Path HTTP Push Mobile Push Hadoop Batch Compute Redis Redis Redis Timeline Cache Redis Earlybird Earlybird Fanout Push Compute Ingester Search Index Read Path Write API Timeline Service @raffi qcon shanghai 2013
  109. 109. Blender Write Path HTTP Push Mobile Push Hadoop Batch Compute Redis Redis Redis Timeline Cache Redis Earlybird Earlybird Fanout Push Compute Ingester Search Index Read Path Write API Timeline Service @raffi qcon shanghai 2013
  110. 110. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  111. 111. things we’re trying... @raffi qcon shanghai 2013
  112. 112. @raffi qcon shanghai 2013
  113. 113. Write API Redis Earlybird Earlybird Fanout Search Index Ingester Redis Redis Redis @raffi qcon shanghai 2013
  114. 114. Write API Redis Redis Redis Timeline Cache Redis Earlybird Earlybird Fanout Search Index Ingester @raffi qcon shanghai 2013
  115. 115. Write API Redis Redis Redis Timeline Cache Redis Earlybird Earlybird Fanout Search Index Ingester @raffi qcon shanghai 2013
  116. 116. Write API Redis Redis Redis Timeline Cache Redis Earlybird Earlybird Fanout Search Index Ingester @raffi qcon shanghai 2013
  117. 117. Write API Redis Redis Redis Timeline Cache Redis Earlybird Earlybird Fanout Search Index Ingester @raffi qcon shanghai 2013
  118. 118. Write API Redis Redis Redis Timeline Cache Redis Earlybird Earlybird Fanout Search Index Ingester @raffi qcon shanghai 2013
  119. 119. Write API Redis Redis Redis Timeline Cache Redis Earlybird Earlybird Fanout Search Index Ingester @raffi qcon shanghai 2013
  120. 120. Write API search index ⇢[‘hello’,‘world’] Redis Redis Redis Timeline Cache Redis Earlybird Earlybird Fanout Search Index Ingester fanout index ⇢[@danadanger, ...] @raffi qcon shanghai 2013
  121. 121. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi @raffi qcon shanghai 2013
  122. 122. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi @raffi qcon shanghai 2013
  123. 123. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi @raffi qcon shanghai 2013
  124. 124. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi @raffi qcon shanghai 2013
  125. 125. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi @raffi qcon shanghai 2013
  126. 126. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi @raffi qcon shanghai 2013
  127. 127. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi @raffi qcon shanghai 2013
  128. 128. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi @raffi qcon shanghai 2013
  129. 129. Write API fan-in ⇢ O(n) read Redis Earlybird Earlybird ⇢ O(n) write Fanout Redis Redis Redis Timeline Cache Ingester Search Index ⇢ O(1) write fan-out ⇢ O(1) read @raffi qcon shanghai 2013
  130. 130. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi @raffi qcon shanghai 2013
  131. 131. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi OR user_timeline:taylorswift13 @raffi qcon shanghai 2013
  132. 132. streaming compute ⇢ continuous computation ⇢ driven by the events that come into twitter ⇢ generalizing the push mechanism @raffi qcon shanghai 2013
  133. 133. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
  134. 134. timeline query statistics ⇢ >150m active users worldwide ⇢ 300k qps poll-based timelines 
 @ 1ms p50 / 4ms p99 ⇢ 30k qps search-based timelines @raffi qcon shanghai 2013
  135. 135. tweet input ⇢ ~340m tweets per day ⇢ ~4K/sec daily average ⇢ ~6K/sec daily peak ⇢ >10K/sec during large events @raffi qcon shanghai 2013
  136. 136. @raffi qcon shanghai 2013
  137. 137. followed by following @raffi qcon shanghai 2013
  138. 138. @raffi qcon shanghai 2013
  139. 139. timeline delivery statistics ⇢ 26b deliveries / day (~18m / min) ⇢ 3.5 seconds @ p50 to deliver to 1m ⇢ ~300k deliveries / sec @raffi qcon shanghai 2013
  140. 140. thanks!

×