Successfully reported this slideshow.
Your SlideShare is downloading. ×

QCon London 2020: Streaming a million likes/second Real-time interactions on Live Video

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 168 Ad

QCon London 2020: Streaming a million likes/second Real-time interactions on Live Video

Akhilesh Gupta does a technical deep-dive into how Linkedin uses the Play/Akka Framework and a scalable distributed system to enable live interactions like likes/comments at massive scale at extremely low costs across multiple data centers.

https://www.infoq.com/presentations/linkedin-play-akka-distributed-systems/

Akhilesh Gupta does a technical deep-dive into how Linkedin uses the Play/Akka Framework and a scalable distributed system to enable live interactions like likes/comments at massive scale at extremely low costs across multiple data centers.

https://www.infoq.com/presentations/linkedin-play-akka-distributed-systems/

Advertisement
Advertisement

More Related Content

Advertisement

QCon London 2020: Streaming a million likes/second Real-time interactions on Live Video

  1. 1. Streaming a million likes/second Real-time interactions on Live Video Akhilesh Gupta Realtime Engineer, LinkedIn 1
  2. 2. tiny.cc/qconlive 2 Login with LinkedIn credentials
  3. 3. Which was the largest live stream in the world? (largest number of concurrent viewers) ? 3
  4. 4. INDIA VS NEW ZEALAND WORLD CUP Largest Live Stream >25.3M viewers © The Indian Express https://images.indianexpress.com/2019/06/13th-june_india-vs-new-zealand_759.jpg 4
  5. 5. 5
  6. 6. D i s t r i b u t e d S y s t e m s Start small and add simple layers to your architecture 6
  7. 7. 7
  8. 8. What is a Live Video? 8
  9. 9. What are real-time interactions on Live Video? 9
  10. 10. Mobile LINKEDIN LIVE Likes/Comments Concurrent Viewer Counts 10
  11. 11. LINKEDIN LIVE Desktop Likes/Comments Concurrent Viewer Counts 11
  12. 12. Distribution of Likes A S 12
  13. 13. Distribution of Likes A S Simple HTTP Request 13
  14. 14. Distribution of Likes A S Simple HTTP Request ? 14
  15. 15. Challenge 1:The delivery pipe 15
  16. 16. The delivery pipe Real-time Delivery A S Likes Backend (Publisher) 16
  17. 17. The delivery pipe Real-time Delivery A S Likes Backend (Publisher) Simple HTTP Request 17
  18. 18. The delivery pipe Real-time Delivery A S Likes Backend (Publisher) Persistent Connection 18
  19. 19. The delivery pipe Real-time Delivery A S Likes Backend (Publisher) Persistent Connection 19
  20. 20. Persistent Connection Real-time Delivery 20
  21. 21. TRY IT! Login at linkedin.com Persistent Connection tiny.cc/realtime 21
  22. 22. 22
  23. 23. EVENTSOURCE INTERFACE CLIENT REQUEST HTTP Long Poll with Server Sent Events GET /connect HTTP/1.1 Accept: text/event-stream 23
  24. 24. EVENTSOURCE INTERFACE SERVER RESPONSE HTTP Long Poll with Server Sent Events HTTP/1.1 200 OK - RESPONSE HEADERS - Content-Type: text/event-stream 24
  25. 25. EVENTSOURCE INTERFACE SERVER RESPONSE HTTP Long Poll with Server Sent Events HTTP/1.1 200 OK - RESPONSE HEADERS - Content-Type: text/event-stream - RESPONSE BODY - data: {“like” object} 25
  26. 26. EVENTSOURCE INTERFACE SERVER RESPONSE HTTP Long Poll with Server Sent Events HTTP/1.1 200 OK - RESPONSE HEADERS - Content-Type: text/event-stream - RESPONSE BODY - data: {“like” object} .. data: {“comment” object} 26
  27. 27. EVENTSOURCE INTERFACE SERVER RESPONSE HTTP Long Poll with Server Sent Events HTTP/1.1 200 OK - RESPONSE HEADERS - Content-Type: text/event-stream - RESPONSE BODY - data: {“like” object} .. data: {“comment” object} .. data: {“like” object} 27
  28. 28. WEB EventSource Client Libraries var evtSource = new EventSource("https://www.linkedin.com/realtime/connect"); evtSource.onmessage = function(e) { var likeObject = JSON.parse(e.data); } 28
  29. 29. ANDROID & IOS SUPPORT EventSource Client Libraries Android: https://github.com/tylerjroach/eventsource-android iOS Swift: https://github.com/inaka/EventSource 29
  30. 30. Next Challenge? Real-time Delivery 30
  31. 31. Next Challenge: Multiple Connections Real-time Delivery 31
  32. 32. Challenge 2: Connection Management 32
  33. 33. PLAY & AKKA Connection Management Akka is a toolkit for building highly concurrent, distributed, and resilient message-driven applications 33
  34. 34. © Disney, All Rights Reserved, Pirates of the Caribbean 34
  35. 35. AN AKKA ACTOR Connection Management State Behavior 35
  36. 36. AN AKKA ACTOR Connection Management State Behavior A B C Mailbox One per Actor 36
  37. 37. AN AKKA ACTOR Connection Management State Behavior A B C Mailbox One per Actor 37
  38. 38. AN AKKA ACTOR Connection Management State Behavior A B C Mailbox One per Actor Millions of Actors 38
  39. 39. AN AKKA ACTOR Connection Management State Behavior A B C Mailbox One per Actor No. of threads proportional to no. of cores Millions of Actors 39
  40. 40. AN AKKA ACTOR Connection Management Connection Publish Event A B C Events to be published Millions of Connections 40
  41. 41. INITIALIZE AN AKKA ACTOR Server: Accepting a EventSource connection // Client A connects to the server and is assigned connectionIdA public Result listen() { } 41
  42. 42. INITIALIZE AN AKKA ACTOR Server: Accepting a EventSource connection // Client A connects to the server and is assigned connectionIdA public Result listen() { return ok(EventSource.whenConnected(eventSource -> { String connectionId = UUID.randomUUID().toString(); })); } 42
  43. 43. INITIALIZE AN AKKA ACTOR Server: Accepting a EventSource connection // Client A connects to the server and is assigned connectionIdA public Result listen() { return ok(EventSource.whenConnected(eventSource -> { String connectionId = UUID.randomUUID().toString(); // construct an Akka Actor to manage connection _actorSystem.actorOf( ClientConnectionActor.props(connectionId, eventSource), connectionId ); })); } 43
  44. 44. Publish Events Using Akka Actors Akka Actor Real-time Delivery System Connection ID cid1 44
  45. 45. EVENT PUBLISH FROM PUBLISHER Publish Events Using Akka Actors Actor 1 Real-time Delivery System Actor 2 Actor 3 Actor 4 Actor 5 Akka Supervisor Actor cid1 cid2 cid3 cid4 cid5 45
  46. 46. EVENT PUBLISH FROM SUPERVISOR TO CHILD ACTORS Publish Events Using Akka Actors Actor 1 Real-time Delivery System Actor 2 Actor 3 Actor 4 Actor 5 Akka Supervisor Actor cid1 cid2 cid3 cid4 cid5 46
  47. 47. EVENT PUBLISH FROM CHILD ACTORS TO CLIENT CONNECTIONS Publish Events Using Akka Actors Actor 1 Real-time Delivery System Actor 2 Actor 3 Actor 4 Actor 5 Akka Supervisor Actor cid1 cid2 cid3 cid4 cid5 47
  48. 48. EVENT PUBLISH FROM CHILD ACTORS TO CLIENT CONNECTIONS Publishing on EventSource eventSource.send(<like object>); 48
  49. 49. EVENT PUBLISH FROM CHILD ACTORS TO CLIENT CONNECTIONS Publish Events Using Akka Actors Actor 1 Real-time Delivery System Actor 2 Actor 3 Actor 4 Actor 5 Akka Supervisor Actor cid1 cid2 cid3 cid4 cid5 49
  50. 50. EVENT PUBLISH FROM CHILD ACTORS TO CLIENT CONNECTIONS Publish Events Using Akka Actors Actor 1 Real-time Delivery System Actor 2 Actor 3 Actor 4 Actor 5 Akka Supervisor Actor cid1 cid2 cid3 cid4 cid5 50
  51. 51. EVENTSOURCE INTERFACE Received Events on the Clients HTTP/1.1 200 OK - RESPONSE HEADERS - Content-Type: text/event-stream - RESPONSE BODY - data: {“User A liked the video…” object} 51
  52. 52. Next Challenge? Real-time Delivery 52
  53. 53. Next Challenge: Multiple Live Videos Real-time Delivery 53
  54. 54. Challenge 3:Multiple Live Videos 54
  55. 55. START WATCHING LIVE VIDEO Multiple Live Videos cid3 cid5 Live Video 1 Live Video 2 Real-time Delivery System 55
  56. 56. HTTP SUBSCRIPTION REQUEST Viewers subscribe to Live Videos Topic ConnectionId In-Memory Subscriptions Table cid3 cid5 Live Video 1 Live Video 2 /subscribe 56
  57. 57. IN-MEMORY SUBSCRIPTION ENTRY Viewers subscribe to Live Videos Topic ConnectionId In-Memory Subscriptions Table cid5 Live Video 2 /subscribe /subscribe cid3 Live Video 1 57
  58. 58. IN-MEMORY SUBSCRIPTION ENTRY Viewers subscribe to Live Videos Topic ConnectionId In-Memory Subscriptions Table cid3 cid5 Live Video 1 Live Video 2 /subscribe /subscribe 58
  59. 59. Publish using Subscription Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 Live Video 2 cid5 Live Video 2 cid1 Live Video 1 cid4 Live Video 2 cid2 59
  60. 60. PUBLISHER PUBLISH TO SUPERVISOR ACTOR Publish using Subscription Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 Live Video 2 cid5 Live Video 2 cid1 Live Video 1 cid4 Live Video 2 cid2 60
  61. 61. SUBSCRIBER LOOKUP FROM IN MEMORY TABLE Publish using Subscription Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 Live Video 2 cid5 Live Video 2 cid1 Live Video 1 cid4 Live Video 2 cid2 61
  62. 62. PUBLISH TO CLIENT CONNECTIONS Publish using Subscription Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 Live Video 2 cid5 Live Video 2 cid1 Live Video 1 cid4 Live Video 2 cid2 62
  63. 63. PUBLISHER PUBLISH TO SUPERVISOR ACTOR Publish using Subscription Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 Live Video 2 cid5 Live Video 2 cid1 Live Video 1 cid4 Live Video 2 cid2 63
  64. 64. SUBSCRIBER LOOKUP FROM IN MEMORY TABLE Publish using Subscription Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 Live Video 2 cid5 Live Video 2 cid1 Live Video 1 cid4 Live Video 2 cid2 64
  65. 65. PUBLISH TO CLIENT CONNECTIONS Publish using Subscription Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 Live Video 2 cid5 Live Video 2 cid1 Live Video 1 cid4 Live Video 2 cid2 65
  66. 66. Next Challenge? Real-time Delivery 66
  67. 67. Next Challenge: 10K Concurrent Viewers Real-time Delivery 67
  68. 68. Challenge 4:10K Concurrent Viewers 68
  69. 69. 69
  70. 70. 10K Concurrent Viewers Real-time Delivery 70
  71. 71. S C A L I N G B A C K E N D S Y S T E M S When in doubt, add a machine! 71
  72. 72. 10K Concurrent Viewers Real-time Delivery 72
  73. 73. 10K Concurrent Viewers Frontend Frontend 73
  74. 74. 10K Concurrent Viewers Frontend Frontend 74
  75. 75. DISPATCHES BETWEEN FRONTEND NODES Real-time Dispatcher Frontend Frontend Real-time Dispatcher 75
  76. 76. DISPATCHES BETWEEN FRONTEND NODES Real-time Dispatcher Frontend Frontend Real-time Dispatcher 76
  77. 77. DISPATCHES BETWEEN FRONTEND NODES Real-time Dispatcher Frontend Frontend Real-time Dispatcher 77
  78. 78. Frontend Node Subscriptions node1 node2 Live Video 1 Live Video 2 Frontend Frontend Real-time Dispatcher 78 Live Video 1
  79. 79. HTTP SUBSCRIPTION REQUEST Frontend Node Subscriptions node1 node2 Live Video 1 Live Video 2 Frontend Frontend Topic Frontend Host Real-time Dispatcher Subscriptions Table /subscribe 79 Live Video 1
  80. 80. SUBSCRIPTIONS TABLE ENTRY Frontend Node Subscriptions node1 node2 Live Video 1 Live Video 2 Frontend Frontend Topic Frontend Host Real-time Dispatcher Subscriptions Table /subscribe /subscribe 80 Live Video 1
  81. 81. SUBSCRIPTIONS TABLE ENTRY Frontend Node Subscriptions node1 node2 Live Video 1 Live Video 2 Frontend Frontend Topic Frontend Host Real-time Dispatcher Subscriptions Table /subscribe /subscribe 81 node2 Live Video 1
  82. 82. SUBSCRIPTIONS TABLE ENTRY Frontend Node Subscriptions node1 node2 Live Video 1 Live Video 2 Frontend Frontend Topic Frontend Host Real-time Dispatcher Subscriptions Table /subscribe /subscribe 82 node2 Live Video 1
  83. 83. Publish to Frontend Nodes node1 Real-time Dispatcher Frontend node2 Frontend node3 Frontend node4 Frontend node5 Frontend Live Video 1 node1 Live Video 2 node5 Live Video 2 node3 Live Video 1 node2 Live Video 2 node2 83
  84. 84. PUBLISHER PUBLISH TO DISPATCHER Publish to Frontend Nodes node1 Real-time Dispatcher Frontend node2 Frontend node3 Frontend node4 Frontend node5 Frontend Dispatcher 84 Live Video 1 node1 Live Video 2 node5 Live Video 2 node3 Live Video 1 node2 Live Video 2 node2
  85. 85. SUBSCRIBER LOOKUP Publish to Frontend Nodes node1 Real-time Dispatcher Frontend node2 Frontend node3 Frontend node4 Frontend node5 Frontend Dispatcher 85 Live Video 1 node1 Live Video 2 node5 Live Video 2 node3 Live Video 1 node2 Live Video 2 node2
  86. 86. PUBLISH TO SUBSCRIBED FRONTEND NODES Publish to Frontend Nodes node1 Real-time Dispatcher Frontend node2 Frontend node3 Frontend node4 Frontend node5 Frontend Dispatcher 86 Live Video 1 node1 Live Video 2 node5 Live Video 2 node3 Live Video 1 node2 Live Video 2 node2
  87. 87. Frontend Node to Client Publish Frontend 87
  88. 88. Frontend Node to Client Publish Frontend 88
  89. 89. What’s the next challenge? Frontend Frontend Frontend Frontend Frontend Dispatcher Live Video 1 node3 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 89
  90. 90. Challenge 5:100 Likes/second 90
  91. 91. 100 Likes/second Frontend Frontend Frontend Frontend Frontend Dispatcher Live Video 1 node3 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 91
  92. 92. S C A L I N G B A C K E N D S Y S T E M S When in doubt, add a machine! 92
  93. 93. 100 Likes/second Frontend Frontend Frontend Frontend Frontend Dispatcher Live Video 1 node3 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 93
  94. 94. 100 Likes/second Frontend Frontend Frontend Frontend Frontend Dispatcher Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 Dispatcher Frontend Frontend 94
  95. 95. 100 Likes/second Frontend Frontend Frontend Frontend Frontend Dispatcher Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 Dispatcher Frontend Frontend 95
  96. 96. 100 Likes/second Frontend Frontend Frontend Frontend Frontend Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 Dispatcher Frontend Frontend 96
  97. 97. COUCHBASE/REDIS ETC. Key-Value Store for Subscriptions Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 Key-Value Store Dispatcher 97
  98. 98. Publish to Dispatcher Nodes Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value Store Dispatcher Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 98
  99. 99. Subscriber lookup from KV Store Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value Store Dispatcher Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 node1 node2 node3 node4 node5 node6 node7 99
  100. 100. Publish to Frontend Nodes Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value Store Dispatcher Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 node1 node2 node3 node4 node5 node6 node7 100
  101. 101. Publish to Frontend Nodes Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value Store Dispatcher node1 node2 node3 node4 node5 node6 node7 Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 101
  102. 102. Publish to Frontend Nodes Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value Store Dispatcher node1 node2 node3 node4 node5 node6 node7 Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 102
  103. 103. Challenge 6: 100 Likes/s, 10K Viewers Distribution of 1M Likes/s 103
  104. 104. Mobile LINKEDIN LIVE Likes 104
  105. 105. The Subscription Flow Frontend Dispatcher Key-Value Store 1 105
  106. 106. START WATCHING LIVE VIDEO Viewers subscribing to Live Video Likes cid3 cid5 Live Video 1 Live Video 2 Frontend 106
  107. 107. HTTP SUBSCRIPTION REQUEST Viewers subscribing to Live Video Likes Topic ConnectionId cid3 cid5 Live Video 1 Live Video 2 /subscribe Frontend In-Memory Subscriptions Table 107
  108. 108. FRONTEND IN-MEMORY SUBSCRIPTION ENTRY Viewers subscribing to Live Video Likes Topic ConnectionId cid5 Live Video 2 /subscribe /subscribe cid3 Live Video 1 Frontend In-Memory Subscriptions Table 108
  109. 109. FRONTEND IN-MEMORY SUBSCRIPTION ENTRY Viewers subscribing to Live Video Likes Topic ConnectionId cid3 cid5 Live Video 1 Live Video 2 /subscribe /subscribe Frontend In-Memory Subscriptions Table 109
  110. 110. FRONTEND IN-MEMORY SUBSCRIPTION ENTRY Viewers subscribing to Live Video Likes Topic ConnectionId cid3 cid5 Live Video 1 Live Video 2 /subscribe /subscribe Frontend In-Memory Subscriptions Table 110
  111. 111. The Subscription Flow Frontend Dispatcher Key-Value Store 1 2 3 111
  112. 112. FRONTEND RECEIVES SUBSCRIPTION REQUEST Frontend Node Subscriptions in Dispatcher node1 node2 Live Video 1 Live Video 2 Frontend Frontend Key-Value Store Dispatcher Dispatcher 112
  113. 113. FRONTEND HTTP SUBSCRIPTION REQUEST TO DISPATCHER Frontend Node Subscriptions in Dispatcher node1 node2 Live Video 1 Live Video 2 Frontend Frontend Topic Frontend Host /subscribe Key-Value Store Dispatcher Dispatcher 113
  114. 114. DISPATCHER SUBSCRIPTION ENTRY IN KV STORE Frontend Node Subscriptions in Dispatcher node1 node2 Live Video 1 Live Video 2 Frontend Frontend Topic Frontend Host /subscribe /subscribe Dispatcher Dispatcher Key-Value Store 114
  115. 115. DISPATCHER SUBSCRIPTION ENTRY IN KV STORE Frontend Node Subscriptions in Dispatcher node1 node2 Live Video 1 Live Video 2 Frontend Frontend Topic Frontend Host /subscribe /subscribe Dispatcher Dispatcher Key-Value Store 115
  116. 116. The Publish Flow Frontend Dispatcher Key-Value Store 1 116
  117. 117. Viewers liking Live Videos Likes Backend (Publisher) 117
  118. 118. Likes Published to Backend Likes Backend (Publisher) 118
  119. 119. The Publish Flow Frontend Dispatcher Key-Value Store 1 2 3 4 5 119
  120. 120. The Publish Flow Frontend Dispatcher Key-Value Store 2 3 4 5 120
  121. 121. BACKEND PUBLISH TO DISPATCHER NODES Dispatcher Publish to Frontend Nodes Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value Store Dispatcher Live Video 1 node3 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 121
  122. 122. SUBSCRIBER LOOKUP FROM KV STORE Dispatcher Publish to Frontend Nodes Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value Store Dispatcher Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 node1 node2 node3 node4 node5 node6 node7 122
  123. 123. SUBSCRIBER LOOKUP FROM KV STORE Dispatcher Publish to Frontend Nodes Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value Store Dispatcher Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 node1 node2 node3 node4 node5 node6 node7 123
  124. 124. PUBLISH TO FRONTEND NODES Dispatcher Publish to Frontend Nodes Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value Store Dispatcher node1 node2 node3 node4 node5 node6 node7 Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 124
  125. 125. The Publish Flow Frontend Dispatcher Key-Value Store 5 125
  126. 126. DISPATCHER PUBLISH TO FRONTEND NODE SUPERVISOR ACTOR Frontend Publish to Mobile Clients Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 Live Video 2 cid5 Live Video 2 cid1 Live Video 1 cid4 Live Video 2 cid2 Dispatcher Frontend In-Memory Subscriptions Table 126
  127. 127. SUBSCRIBED CLIENT CONNECTION LOOKUP FROM IN-MEMORY TABLE Frontend Publish to Mobile Clients Actor 1 Frontend Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 Live Video 2 cid5 Live Video 2 cid1 Live Video 1 cid4 Live Video 2 cid2 Dispatcher In-Memory Subscriptions Table 127
  128. 128. PUBLISH TO MOBILE CLIENTS Frontend Publish to Mobile Clients Actor 1 Frontend Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 Live Video 2 cid5 Live Video 2 cid1 Live Video 1 cid4 Live Video 2 cid2 Dispatcher In-Memory Subscriptions Table 128
  129. 129. Bonus Challenge: Another data center 129
  130. 130. 130
  131. 131. D i s t r i b u t e d S y s t e m s Start small and add simple layers to your architecture 131
  132. 132. Frontend Dispatcher DC-1 132
  133. 133. Frontend Dispatcher Frontend Dispatcher DC-2 DC-1 133
  134. 134. Frontend Dispatcher Frontend Dispatcher Frontend Dispatcher DC-3 DC-2 DC-1 134
  135. 135. Frontend Dispatcher Frontend Dispatcher Frontend Dispatcher DC-3 DC-2 DC-1 135
  136. 136. Frontend Dispatcher Frontend Dispatcher Frontend Dispatcher DC-3 DC-2 DC-1 136
  137. 137. Frontend Dispatcher Frontend Dispatcher Frontend Dispatcher DC-3 DC-2 DC-1 137
  138. 138. Frontend Dispatcher Frontend Dispatcher Frontend Dispatcher DC-3 DC-2 DC-1 Cross Data Center Publish 138
  139. 139. Frontend Dispatcher Frontend Dispatcher Frontend Dispatcher DC-3 DC-2 DC-1 139
  140. 140. Frontend Dispatcher Frontend Dispatcher Frontend Dispatcher DC-3 DC-2 DC-1 140
  141. 141. Performance & Scale 141
  142. 142. How many persistent connections can a single machine hold? Frontend Server ? 142
  143. 143. > 100,000 143
  144. 144. ROYAL WEDDING Second Largest Stream >18M viewers © time.com 144
  145. 145. > 100,000 We can hold those 18M connections with just 180 machines 145
  146. 146. LINKEDIN ENGINEERING BLOG tiny.cc/ linkedinscaling >100K 146
  147. 147. LinkedIn Realtime Performance Persistent Connections/machine 5K/s Dispatcher Publish QPS/machine 75ms E2E Publish Latency >100K 147
  148. 148. Dispatcher Performance Frontend Frontend Frontend Frontend Frontend Dispatcher How many events per second can be published to a single dispatcher machine? ? events/second 148
  149. 149. Dispatcher Performance Number of frontend nodes to publish to is limited per event Frontend Frontend Frontend Frontend Frontend Dispatcher ? events/second 149
  150. 150. Dispatcher Performance Frontend Frontend Frontend Frontend Frontend Dispatcher ? events/second 150
  151. 151. 5K We can publish 50K likes per second to frontend nodes with just 10 dispatcher machines 151
  152. 152. LinkedIn Realtime Performance >100K Persistent Connections/machine Dispatcher Publish QPS/machine 75ms E2E Publish Latency 5K 152
  153. 153. E2E Publish Latency Frontend Dispatcher t1 Key-Value Store 153
  154. 154. E2E Publish Latency Frontend Dispatcher t1 Key-Value Store t2 t2-t1 = 75ms p90 154
  155. 155. LinkedIn Realtime Performance >100K Persistent Connections/machine Dispatcher Publish QPS/machine 75ms E2E Publish Latency 5K 155
  156. 156. LINKEDIN ENGINEERING BLOG tiny.cc/ linkedinlatency Measurement System E2E Publish Latency 156
  157. 157. Why does this system scale? Horizontally Scalable System Frontend Dispatcher Frontend Frontend Frontend Dispatcher Dispatcher Dispatcher Distributed K/V Storage System 157
  158. 158. Active Status/Presence Online/Offline indicators LINKEDIN REALTIME TECHNOLOGY 158
  159. 159. LINKEDIN ENGINEERING BLOG tiny.cc/ linkedinpresence Active Status/Presence 159
  160. 160. Key takeaways 160
  161. 161. R E A LT I M E I N T E R A C T I O N S Enable dynamic instant experiences on your apps 161
  162. 162. E V E N T S O U R C E / S S E Built-in support in most server frameworks Readily available client libraries HTTP/1.1 200 OK - RESPONSE HEADERS - Content-Type: text/event-stream 162
  163. 163. P l a y F r a m e w o r k a n d A k k a A c t o r s Powerful frameworks to manage millions of connections State Behavior Mailbox 163
  164. 164. D i s t r i b u t e d S y s t e m s Start small and add simple layers to your architecture 164
  165. 165. S C A L I N G B A C K E N D S Y S T E M S When in doubt, add a machine! 165
  166. 166. R E A LT I M E I N T E R A C T I O N P L A T F O R M Can be built on any server/storage technology Horizontally scalable Frontend Dispatcher Key- 166
  167. 167. R E A LT I M E I N T E R A C T I O N P L A T F O R M You can do it for your app! 167
  168. 168. R E A LT I M E E N G I N E E R , L I N K E D I N @agupta03 t i n y . c c / q c o n 2 0 2 0 168 AMA at 1.40pm at Guild, Mezzanine

×