Q con shanghai2013-[jains krums]-[real-time-delivery-archiecture]
•
0 likes•2,281 views
The document describes Twitter's real-time delivery architecture. It aims to evolve beyond being just a web stack by isolating responsibilities into components like routing, presentation, logic, storage and retrieval. The architecture uses various services and technologies like the Write API, Ingester, Blender, Hadoop, Redis, and Earlybird to ingest tweets in real-time, cache timelines and search indexes, and deliver tweets to users via pull and push methods. The goal is to improve site speed, reliability, and developer innovation speed.
Recommended
More Related Content
Viewers also liked
Similar to Q con shanghai2013-[jains krums]-[real-time-delivery-archiecture]
Similar to Q con shanghai2013-[jains krums]-[real-time-delivery-archiecture] (20)
More from Michael Zhang
More from Michael Zhang (20)
Recently uploaded
Recently uploaded (20)
Q con shanghai2013-[jains krums]-[real-time-delivery-archiecture]
- 1. real-time delivery architecture @raffi qcon shanghai 2013
- 3. http://twitpic.com/135xa There’s a plane in the Hudson. I’m on a ferry going to pick up the people. Crazy. ! 15 Jan 09 Janis Krums @jkrums @raffi qcon shanghai 2013
- 4. what are the goals? ⇢ evolve from being solely a web stack @raffi qcon shanghai 2013
- 6. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Gizmoduck Flock(s) @raffi qcon shanghai 2013
- 7. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Gizmoduck Flock(s) @raffi qcon shanghai 2013
- 8. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Gizmoduck Flock(s) @raffi qcon shanghai 2013
- 9. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Gizmoduck Flock(s) @raffi qcon shanghai 2013
- 10. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Gizmoduck Flock(s) @raffi qcon shanghai 2013
- 11. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Gizmoduck Flock(s) @raffi qcon shanghai 2013
- 12. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Darkwing Flock(s) @raffi qcon shanghai 2013
- 13. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Darkwing Flock(s) @raffi qcon shanghai 2013
- 14. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Darkwing Flock(s) @raffi qcon shanghai 2013
- 15. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Darkwing Flock(s) @raffi qcon shanghai 2013
- 16. what are the goals? ⇢ evolve from being solely a web stack ⇢ isolate responsibilities and concerns ⇢ site speed and reliability ⇢ developer innovation speed @raffi qcon shanghai 2013
- 18. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
- 19. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
- 20. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
- 21. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
- 22. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
- 23. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
- 24. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
- 25. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
- 26. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
- 27. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
- 33. Write API Ingester Hadoop Batch Compute HTTP Push Push Compute Redis Timeline Cache Search Cache Redis Fanout @raffi qcon shanghai 2013
- 35. Write API @raffi qcon shanghai 2013
- 36. Write API Ingester Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout @raffi qcon shanghai 2013
- 37. Write API Ingester Hadoop Batch Compute Push Compute Search Cache Fanout @raffi qcon shanghai 2013
- 38. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 39. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 40. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 41. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 42. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 43. Write API Blender HTTP Push Mobile Push Hadoop Batch Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Push Compute Ingester Social Graph Service Timeline Service @raffi qcon shanghai 2013
- 44. Write API ⇢ pipelined 4k “destinations” at a time Blender ⇢ replicated HTTP Push Mobile Push Hadoop Batch Compute ⇢ keyed off “recipient” Redis Redis Redis Timeline Cache Redis Redis Earlybird Search Cache insert Fanout Push Compute Ingester Social Graph Service Timeline Service @raffi qcon shanghai 2013
- 45. Write API Ingester Blender ⇢ RPUSHX to only add to cached timelines Tweet IDPush User ID HTTP 8 bytes 8 bytes Mobile Push Bits Hadoop 4 bytes Batch Compute ⇢ native list structure Redis Redis Redis Push Compute Redis Redis Earlybird Search Cache using redis Timeline Cache Fanout Timeline Service @raffi qcon shanghai 2013
- 46. Write API Ingester Blender ⇢ RPUSHX to only add to cached timelines Tweet IDPush User ID HTTP Bits Tweet ID Bits Hadoop User ID Bits Tweet ID User ID Bits Tweet ID Timeline Service Mobile Tweet ID User ID Push User ID Bits Tweet ID User ID Bits Tweet ID User ID Bits Tweet ID User ID Bits Tweet ID User ID Bits Tweet ID User ID Batch Compute ⇢ native list structure Redis Redis Redis Push Compute Redis Redis Earlybird Search Cache using redis Timeline Cache Fanout Bits @raffi qcon shanghai 2013
- 47. Write API Ingester Blender ⇢ RPUSHX to only add to cached timelines Tweet IDPush User ID HTTP Bits Tweet ID Bits Hadoop ID Tweet User ID Bits Tweet ID User ID Bits Tweet ID Timeline Service Mobile Tweet ID User ID Push User ID Bits Tweet ID User ID Bits Tweet ID User ID Bits Tweet ID User ID Bits Tweet ID User ID Bits Tweet ID User ID Batch Compute ⇢ native list structure Redis Redis Redis Push Compute Redis Redis Earlybird Search Cache using redis Timeline Cache Fanout Bits Tweet ID Tweet ID @raffi qcon shanghai 2013
- 48. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 49. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 50. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 51. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 52. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
- 53. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 54. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 55. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Cache Redis Redis Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 56. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 57. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 58. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 59. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 60. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 61. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 62. Write API Ingester Blender Timeline Service HTTP Push Hadoop ⇢ queries one replica of all indexes Mobile Push Batch Compute blender Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout ⇢ merges & ranks results @raffi qcon shanghai 2013
- 63. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 64. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 65. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 66. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 67. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
- 68. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 69. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 70. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 71. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 72. http push / hosebird ⇢ maintains persistent connections with end clients ⇢ processes tweet & social graph events ⇢ event-based “router” @raffi qcon shanghai 2013
- 73. Hosebird Write API Hosebird Firehose User Streams Hosebird Track / Follow Text @raffi qcon shanghai 2013
- 74. Hosebird Write API Hosebird Firehose User Streams Hosebird Track / Follow @raffi qcon shanghai 2013
- 75. Hosebird Write API Hosebird Firehose User Streams Hosebird Track / Follow event propagation ⇢ write API sends all events into hosebird; sees content creation events, social graph changes, etc. ⇢ different queues for public tweets, protected tweets, social events, etc. @raffi qcon shanghai 2013
- 76. Hosebird Write API Hosebird Firehose User Streams Hosebird Track / Follow @raffi qcon shanghai 2013
- 77. Hosebird Write API Hosebird Firehose User Streams Hosebird Track / Follow event cascading ⇢ bandwidth management ⇢ simultaneous connection management (~1m long lived & open connections to this cluster) @raffi qcon shanghai 2013
- 78. Hosebird Write API Hosebird Firehose User Streams Hosebird Track / Follow @raffi qcon shanghai 2013
- 79. Hosebird Write API Hosebird Firehose User Streams Hosebird Track / Follow firehose ⇢ edge machine simply outputs the public tweet queue ⇢ only allow a limited number of firehoses per hosebird box for bandwidth management @raffi qcon shanghai 2013
- 80. Hosebird Write API Hosebird Firehose Track / Follow Hosebird User Streams @raffi qcon shanghai 2013
- 81. Hosebird Write API Hosebird Firehose Track / Follow Hosebird User Streams track / follow ⇢ simple query based on tweet content ⇢ keeps list of terms / users of interest ⇢ parses public tweets at the edge, and if term matches a token, or user is of interest, then route @raffi qcon shanghai 2013
- 82. Hosebird Write API Hosebird Firehose Track / Follow Hosebird User Streams @raffi qcon shanghai 2013
- 83. Hosebird Write API Hosebird Firehose Track / Follow Hosebird User Streams user streams ⇢ replicate home timeline experience ⇢ upon login, obtain “following” list ⇢ keep cached following list coherent by seeing social graph updates ⇢ route tweet if from a followed user @raffi qcon shanghai 2013
- 84. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 85. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 86. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 87. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
- 88. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 89. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 90. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 91. Write API Ingester Blender Mobile Push Hadoop Social Graph Service Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 92. Write API Ingester Blender Mobile Push Hadoop Social Graph Service Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 93. Write API Ingester Blender Mobile Push Hadoop Social Graph Service Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 94. Write API Ingester Blender Mobile Push Hadoop Social Graph Service Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 95. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 96. Pull Targeted twitter.com home_timeline API Queried Search API Push User / Site Streams Mobile Push (SMS, etc.) Track / Follow Streams @raffi qcon shanghai 2013
- 97. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 98. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 99. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 100. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 101. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 102. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 103. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 104. Synchronous Path Write API Ingester Blender Timeline Service Mobile Push Asynchronous Path Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Query Path @raffi qcon shanghai 2013
- 105. Synchronous Path Write API Ingester Blender Timeline Service Mobile Push Asynchronous Path Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Query Path @raffi qcon shanghai 2013
- 106. Synchronous Path Write API Ingester Blender Timeline Service Mobile Push Asynchronous Path Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Query Path @raffi qcon shanghai 2013
- 107. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 108. Blender Write Path HTTP Push Mobile Push Hadoop Batch Compute Redis Redis Redis Timeline Cache Redis Earlybird Earlybird Fanout Push Compute Ingester Search Index Read Path Write API Timeline Service @raffi qcon shanghai 2013
- 109. Blender Write Path HTTP Push Mobile Push Hadoop Batch Compute Redis Redis Redis Timeline Cache Redis Earlybird Earlybird Fanout Push Compute Ingester Search Index Read Path Write API Timeline Service @raffi qcon shanghai 2013
- 110. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 111. things we’re trying... @raffi qcon shanghai 2013
- 113. Write API Redis Earlybird Earlybird Fanout Search Index Ingester Redis Redis Redis @raffi qcon shanghai 2013
- 114. Write API Redis Redis Redis Timeline Cache Redis Earlybird Earlybird Fanout Search Index Ingester @raffi qcon shanghai 2013
- 115. Write API Redis Redis Redis Timeline Cache Redis Earlybird Earlybird Fanout Search Index Ingester @raffi qcon shanghai 2013
- 116. Write API Redis Redis Redis Timeline Cache Redis Earlybird Earlybird Fanout Search Index Ingester @raffi qcon shanghai 2013
- 117. Write API Redis Redis Redis Timeline Cache Redis Earlybird Earlybird Fanout Search Index Ingester @raffi qcon shanghai 2013
- 118. Write API Redis Redis Redis Timeline Cache Redis Earlybird Earlybird Fanout Search Index Ingester @raffi qcon shanghai 2013
- 119. Write API Redis Redis Redis Timeline Cache Redis Earlybird Earlybird Fanout Search Index Ingester @raffi qcon shanghai 2013
- 120. Write API search index ⇢[‘hello’,‘world’] Redis Redis Redis Timeline Cache Redis Earlybird Earlybird Fanout Search Index Ingester fanout index ⇢[@danadanger, ...] @raffi qcon shanghai 2013
- 121. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi @raffi qcon shanghai 2013
- 122. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi @raffi qcon shanghai 2013
- 123. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi @raffi qcon shanghai 2013
- 124. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi @raffi qcon shanghai 2013
- 125. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi @raffi qcon shanghai 2013
- 126. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi @raffi qcon shanghai 2013
- 127. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi @raffi qcon shanghai 2013
- 128. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi @raffi qcon shanghai 2013
- 129. Write API fan-in ⇢ O(n) read Redis Earlybird Earlybird ⇢ O(n) write Fanout Redis Redis Redis Timeline Cache Ingester Search Index ⇢ O(1) write fan-out ⇢ O(1) read @raffi qcon shanghai 2013
- 130. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi @raffi qcon shanghai 2013
- 131. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi OR user_timeline:taylorswift13 @raffi qcon shanghai 2013
- 132. streaming compute ⇢ continuous computation ⇢ driven by the events that come into twitter ⇢ generalizing the push mechanism @raffi qcon shanghai 2013
- 133. Write API Ingester Blender Mobile Push Hadoop Batch Compute HTTP Push Push Compute Redis Redis Redis Timeline Cache Search Index Redis Earlybird Earlybird Fanout Timeline Service @raffi qcon shanghai 2013
- 134. timeline query statistics ⇢ >150m active users worldwide ⇢ 300k qps poll-based timelines @ 1ms p50 / 4ms p99 ⇢ 30k qps search-based timelines @raffi qcon shanghai 2013
- 135. tweet input ⇢ ~340m tweets per day ⇢ ~4K/sec daily average ⇢ ~6K/sec daily peak ⇢ >10K/sec during large events @raffi qcon shanghai 2013
- 137. followed by following @raffi qcon shanghai 2013
- 139. timeline delivery statistics ⇢ 26b deliveries / day (~18m / min) ⇢ 3.5 seconds @ p50 to deliver to 1m ⇢ ~300k deliveries / sec @raffi qcon shanghai 2013
- 140. thanks!