1.
Streaming a million likes/second
Real-time interactions on Live Video
Akhilesh Gupta
Realtime Engineer, LinkedIn
1

2.
tiny.cc/qconlive
2
Login with LinkedIn
credentials

3.
Which was the largest live stream in the world?
(largest number of concurrent viewers)
?
3

4.
INDIA VS NEW ZEALAND WORLD CUP
Largest Live Stream
>25.3M
viewers
© The Indian Express https://images.indianexpress.com/2019/06/13th-june_india-vs-new-zealand_759.jpg
4

5.
5

6.
D i s t r i b u t e d S y s t e m s
Start small and add simple layers to your
architecture
6

7.
7

8.
What is a Live Video?
8

9.
What are real-time interactions
on Live Video?
9

10.
Mobile
LINKEDIN LIVE
Likes/Comments
Concurrent Viewer Counts
10

11.
LINKEDIN LIVE
Desktop
Likes/Comments
Concurrent Viewer Counts
11

12.
Distribution of Likes
A
S
12

13.
Distribution of Likes
A
S
Simple HTTP Request
13

14.
Distribution of Likes
A
S
Simple HTTP Request
?
14

15.
Challenge 1:The delivery pipe
15

16.
The delivery pipe
Real-time
Delivery
A
S
Likes Backend (Publisher) 16

17.
The delivery pipe
Real-time
Delivery
A
S
Likes Backend (Publisher)
Simple HTTP Request
17

18.
The delivery pipe
Real-time
Delivery
A
S
Likes Backend (Publisher)
Persistent Connection
18

19.
The delivery pipe
Real-time
Delivery
A
S
Likes Backend (Publisher)
Persistent Connection
19

20.
Persistent Connection
Real-time
Delivery
20

21.
TRY IT!
Login at linkedin.com
Persistent Connection
tiny.cc/realtime
21

22.
22

23.
EVENTSOURCE INTERFACE CLIENT REQUEST
HTTP Long Poll with Server Sent Events
GET /connect HTTP/1.1
Accept: text/event-stream
23

24.
EVENTSOURCE INTERFACE SERVER RESPONSE
HTTP Long Poll with Server Sent Events
HTTP/1.1 200 OK
- RESPONSE HEADERS -
Content-Type: text/event-stream
24

25.
EVENTSOURCE INTERFACE SERVER RESPONSE
HTTP Long Poll with Server Sent Events
HTTP/1.1 200 OK
- RESPONSE HEADERS -
Content-Type: text/event-stream
- RESPONSE BODY -
data: {“like” object}
25

26.
EVENTSOURCE INTERFACE SERVER RESPONSE
HTTP Long Poll with Server Sent Events
HTTP/1.1 200 OK
- RESPONSE HEADERS -
Content-Type: text/event-stream
- RESPONSE BODY -
data: {“like” object}
..
data: {“comment” object}
26

27.
EVENTSOURCE INTERFACE SERVER RESPONSE
HTTP Long Poll with Server Sent Events
HTTP/1.1 200 OK
- RESPONSE HEADERS -
Content-Type: text/event-stream
- RESPONSE BODY -
data: {“like” object}
..
data: {“comment” object}
..
data: {“like” object}
27

28.
WEB
EventSource Client Libraries
var evtSource =
new EventSource("https://www.linkedin.com/realtime/connect");
evtSource.onmessage = function(e) {
var likeObject = JSON.parse(e.data);
}
28

29.
ANDROID & IOS SUPPORT
EventSource Client Libraries
Android: https://github.com/tylerjroach/eventsource-android
iOS Swift: https://github.com/inaka/EventSource
29

30.
Next Challenge?
Real-time
Delivery
30

31.
Next Challenge: Multiple Connections
Real-time
Delivery
31

32.
Challenge 2: Connection Management
32

33.
PLAY & AKKA
Connection Management
Akka is a toolkit for building highly concurrent,
distributed, and resilient message-driven applications
33

34.
© Disney, All Rights Reserved, Pirates of the Caribbean
34

35.
AN AKKA ACTOR
Connection Management
State
Behavior
35

36.
AN AKKA ACTOR
Connection Management
State
Behavior
A
B
C
Mailbox
One per Actor
36

37.
AN AKKA ACTOR
Connection Management
State
Behavior
A
B
C
Mailbox
One per Actor
37

38.
AN AKKA ACTOR
Connection Management
State
Behavior
A
B
C
Mailbox
One per Actor
Millions of Actors
38

39.
AN AKKA ACTOR
Connection Management
State
Behavior
A
B
C
Mailbox
One per Actor
No. of threads proportional to
no. of cores
Millions of Actors
39

40.
AN AKKA ACTOR
Connection Management
Connection
Publish Event
A
B
C
Events to be published
Millions of Connections
40

41.
INITIALIZE AN AKKA ACTOR
Server: Accepting a EventSource connection
// Client A connects to the server and is assigned connectionIdA
public Result listen() {
} 41

42.
INITIALIZE AN AKKA ACTOR
Server: Accepting a EventSource connection
// Client A connects to the server and is assigned connectionIdA
public Result listen() {
return ok(EventSource.whenConnected(eventSource -> {
String connectionId = UUID.randomUUID().toString();
}));
} 42

43.
INITIALIZE AN AKKA ACTOR
Server: Accepting a EventSource connection
// Client A connects to the server and is assigned connectionIdA
public Result listen() {
return ok(EventSource.whenConnected(eventSource -> {
String connectionId = UUID.randomUUID().toString();
// construct an Akka Actor to manage connection
_actorSystem.actorOf(
ClientConnectionActor.props(connectionId, eventSource),
connectionId
);
}));
} 43

44.
Publish Events Using Akka Actors
Akka
Actor
Real-time
Delivery
System
Connection ID
cid1
44

45.
EVENT PUBLISH FROM PUBLISHER
Publish Events Using Akka Actors
Actor 1
Real-time
Delivery
System
Actor 2
Actor 3
Actor 4
Actor 5
Akka
Supervisor
Actor
cid1
cid2
cid3
cid4
cid5
45

46.
EVENT PUBLISH FROM SUPERVISOR TO CHILD ACTORS
Publish Events Using Akka Actors
Actor 1
Real-time
Delivery
System
Actor 2
Actor 3
Actor 4
Actor 5
Akka
Supervisor
Actor
cid1
cid2
cid3
cid4
cid5
46

47.
EVENT PUBLISH FROM CHILD ACTORS TO CLIENT CONNECTIONS
Publish Events Using Akka Actors
Actor 1
Real-time
Delivery
System
Actor 2
Actor 3
Actor 4
Actor 5
Akka
Supervisor
Actor
cid1
cid2
cid3
cid4
cid5
47

48.
EVENT PUBLISH FROM CHILD ACTORS TO CLIENT CONNECTIONS
Publishing on EventSource
eventSource.send(<like object>);
48

49.
EVENT PUBLISH FROM CHILD ACTORS TO CLIENT CONNECTIONS
Publish Events Using Akka Actors
Actor 1
Real-time
Delivery
System
Actor 2
Actor 3
Actor 4
Actor 5
Akka
Supervisor
Actor
cid1
cid2
cid3
cid4
cid5
49

50.
EVENT PUBLISH FROM CHILD ACTORS TO CLIENT CONNECTIONS
Publish Events Using Akka Actors
Actor 1
Real-time
Delivery
System
Actor 2
Actor 3
Actor 4
Actor 5
Akka
Supervisor
Actor
cid1
cid2
cid3
cid4
cid5
50

51.
EVENTSOURCE INTERFACE
Received Events on the Clients
HTTP/1.1 200 OK
- RESPONSE HEADERS -
Content-Type: text/event-stream
- RESPONSE BODY -
data: {“User A liked the video…” object}
51

52.
Next Challenge?
Real-time
Delivery
52

53.
Next Challenge: Multiple Live Videos
Real-time
Delivery
53

54.
Challenge 3:Multiple Live Videos
54

55.
START WATCHING LIVE VIDEO
Multiple Live Videos
cid3
cid5
Live Video 1
Live Video 2
Real-time
Delivery
System
55

56.
HTTP SUBSCRIPTION REQUEST
Viewers subscribe to Live Videos
Topic ConnectionId
In-Memory
Subscriptions
Table
cid3
cid5
Live Video 1
Live Video 2
/subscribe
56

57.
IN-MEMORY SUBSCRIPTION ENTRY
Viewers subscribe to Live Videos
Topic ConnectionId
In-Memory
Subscriptions
Table
cid5
Live Video 2
/subscribe
/subscribe
cid3
Live Video 1
57

58.
IN-MEMORY SUBSCRIPTION ENTRY
Viewers subscribe to Live Videos
Topic ConnectionId
In-Memory
Subscriptions
Table
cid3
cid5
Live Video 1
Live Video 2
/subscribe
/subscribe
58

59.
Publish using Subscription
Actor 1
Actor 2
Actor 3
Actor 4
Actor 5
Supervisor
cid1
cid2
cid3
cid4
cid5
Live Video 1 cid3
Live Video 2 cid5
Live Video 2 cid1
Live Video 1 cid4
Live Video 2 cid2
59

60.
PUBLISHER PUBLISH TO SUPERVISOR ACTOR
Publish using Subscription
Actor 1
Actor 2
Actor 3
Actor 4
Actor 5
Supervisor
cid1
cid2
cid3
cid4
cid5
Live Video 1 cid3
Live Video 2 cid5
Live Video 2 cid1
Live Video 1 cid4
Live Video 2 cid2
60

61.
SUBSCRIBER LOOKUP FROM IN MEMORY TABLE
Publish using Subscription
Actor 1
Actor 2
Actor 3
Actor 4
Actor 5
Supervisor
cid1
cid2
cid3
cid4
cid5
Live Video 1 cid3
Live Video 2 cid5
Live Video 2 cid1
Live Video 1 cid4
Live Video 2 cid2
61

62.
PUBLISH TO CLIENT CONNECTIONS
Publish using Subscription
Actor 1
Actor 2
Actor 3
Actor 4
Actor 5
Supervisor
cid1
cid2
cid3
cid4
cid5
Live Video 1 cid3
Live Video 2 cid5
Live Video 2 cid1
Live Video 1 cid4
Live Video 2 cid2
62

63.
PUBLISHER PUBLISH TO SUPERVISOR ACTOR
Publish using Subscription
Actor 1
Actor 2
Actor 3
Actor 4
Actor 5
Supervisor
cid1
cid2
cid3
cid4
cid5
Live Video 1 cid3
Live Video 2 cid5
Live Video 2 cid1
Live Video 1 cid4
Live Video 2 cid2
63

64.
SUBSCRIBER LOOKUP FROM IN MEMORY TABLE
Publish using Subscription
Actor 1
Actor 2
Actor 3
Actor 4
Actor 5
Supervisor
cid1
cid2
cid3
cid4
cid5
Live Video 1 cid3
Live Video 2 cid5
Live Video 2 cid1
Live Video 1 cid4
Live Video 2 cid2
64

65.
PUBLISH TO CLIENT CONNECTIONS
Publish using Subscription
Actor 1
Actor 2
Actor 3
Actor 4
Actor 5
Supervisor
cid1
cid2
cid3
cid4
cid5
Live Video 1 cid3
Live Video 2 cid5
Live Video 2 cid1
Live Video 1 cid4
Live Video 2 cid2
65

66.
Next Challenge?
Real-time
Delivery
66

67.
Next Challenge: 10K Concurrent Viewers
Real-time
Delivery
67

68.
Challenge 4:10K Concurrent Viewers
68

69.
69

70.
10K Concurrent Viewers
Real-time
Delivery
70

71.
S C A L I N G B A C K E N D S Y S T E M S
When in doubt, add a machine!
71

72.
10K Concurrent Viewers
Real-time
Delivery
72

73.
10K Concurrent Viewers
Frontend
Frontend
73

74.
10K Concurrent Viewers
Frontend
Frontend
74

75.
DISPATCHES BETWEEN FRONTEND NODES
Real-time Dispatcher
Frontend
Frontend
Real-time Dispatcher
75

76.
DISPATCHES BETWEEN FRONTEND NODES
Real-time Dispatcher
Frontend
Frontend
Real-time Dispatcher
76

77.
DISPATCHES BETWEEN FRONTEND NODES
Real-time Dispatcher
Frontend
Frontend
Real-time Dispatcher
77

78.
Frontend Node Subscriptions
node1
node2
Live Video 1
Live Video 2
Frontend
Frontend
Real-time Dispatcher
78
Live Video 1

79.
HTTP SUBSCRIPTION REQUEST
Frontend Node Subscriptions
node1
node2
Live Video 1
Live Video 2
Frontend
Frontend
Topic
Frontend
Host
Real-time Dispatcher
Subscriptions
Table
/subscribe
79
Live Video 1

80.
SUBSCRIPTIONS TABLE ENTRY
Frontend Node Subscriptions
node1
node2
Live Video 1
Live Video 2
Frontend
Frontend
Topic
Frontend
Host
Real-time Dispatcher
Subscriptions
Table
/subscribe
/subscribe
80
Live Video 1

81.
SUBSCRIPTIONS TABLE ENTRY
Frontend Node Subscriptions
node1
node2
Live Video 1
Live Video 2
Frontend
Frontend
Topic
Frontend
Host
Real-time Dispatcher
Subscriptions
Table
/subscribe
/subscribe
81
node2
Live Video 1

82.
SUBSCRIPTIONS TABLE ENTRY
Frontend Node Subscriptions
node1
node2
Live Video 1
Live Video 2
Frontend
Frontend
Topic
Frontend
Host
Real-time Dispatcher
Subscriptions
Table
/subscribe
/subscribe
82
node2
Live Video 1

83.
Publish to Frontend Nodes
node1
Real-time Dispatcher
Frontend
node2 Frontend
node3 Frontend
node4 Frontend
node5 Frontend
Live Video 1 node1
Live Video 2 node5
Live Video 2 node3
Live Video 1 node2
Live Video 2 node2
83

84.
PUBLISHER PUBLISH TO DISPATCHER
Publish to Frontend Nodes
node1
Real-time Dispatcher
Frontend
node2 Frontend
node3 Frontend
node4 Frontend
node5 Frontend
Dispatcher
84
Live Video 1 node1
Live Video 2 node5
Live Video 2 node3
Live Video 1 node2
Live Video 2 node2

85.
SUBSCRIBER LOOKUP
Publish to Frontend Nodes
node1
Real-time Dispatcher
Frontend
node2 Frontend
node3 Frontend
node4 Frontend
node5 Frontend
Dispatcher
85
Live Video 1 node1
Live Video 2 node5
Live Video 2 node3
Live Video 1 node2
Live Video 2 node2

86.
PUBLISH TO SUBSCRIBED FRONTEND NODES
Publish to Frontend Nodes
node1
Real-time Dispatcher
Frontend
node2 Frontend
node3 Frontend
node4 Frontend
node5 Frontend
Dispatcher
86
Live Video 1 node1
Live Video 2 node5
Live Video 2 node3
Live Video 1 node2
Live Video 2 node2

87.
Frontend Node to Client Publish
Frontend
87

88.
Frontend Node to Client Publish
Frontend
88

89.
What’s the next challenge?
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
Live Video
1
node3
Live Video
2
node5
Live Video
2
node1
Live Video
1
node4
Live Video
2
node2
89

90.
Challenge 5:100 Likes/second
90

91.
100 Likes/second
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
Live Video
1
node3
Live Video
2
node5
Live Video
2
node1
Live Video
1
node4
Live Video
2
node2
91

92.
S C A L I N G B A C K E N D S Y S T E M S
When in doubt, add a machine!
92

93.
100 Likes/second
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
Live Video
1
node3
Live Video
2
node5
Live Video
2
node1
Live Video
1
node4
Live Video
2
node2
93

94.
100 Likes/second
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
Live Video
1
node7
Live Video
2
node5
Live Video
2
node1
Live Video
1
node4
Live Video
2
node2
Dispatcher
Frontend
Frontend
94

95.
100 Likes/second
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
Live Video
1
node7
Live Video
2
node5
Live Video
2
node1
Live Video
1
node4
Live Video
2
node2
Dispatcher
Frontend
Frontend
95

96.
100 Likes/second
Frontend
Frontend
Frontend
Frontend
Frontend
Live Video
1
node7
Live Video
2
node5
Live Video
2
node1
Live Video
1
node4
Live Video
2
node2
Dispatcher
Frontend
Frontend
96

97.
COUCHBASE/REDIS ETC.
Key-Value Store for Subscriptions
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
Dispatcher
Frontend
Frontend
Live Video 1 node7
Live Video 2 node5
Live Video 2 node1
Live Video 1 node4
Live Video 2 node2
Key-Value Store
Dispatcher
97

98.
Publish to Dispatcher Nodes
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
Dispatcher
Frontend
Frontend
Key-Value Store
Dispatcher
Live Video 1 node7
Live Video 2 node5
Live Video 2 node1
Live Video 1 node4
Live Video 2 node2
98

99.
Subscriber lookup from KV Store
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
Dispatcher
Frontend
Frontend
Key-Value Store
Dispatcher
Live Video 1 node7
Live Video 2 node5
Live Video 2 node1
Live Video 1 node4
Live Video 2 node2
node1
node2
node3
node4
node5
node6
node7
99

100.
Publish to Frontend Nodes
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
Dispatcher
Frontend
Frontend
Key-Value Store
Dispatcher
Live Video 1 node7
Live Video 2 node5
Live Video 2 node1
Live Video 1 node4
Live Video 2 node2
node1
node2
node3
node4
node5
node6
node7
100

101.
Publish to Frontend Nodes
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
Dispatcher
Frontend
Frontend
Key-Value Store
Dispatcher
node1
node2
node3
node4
node5
node6
node7
Live Video 1 node7
Live Video 2 node5
Live Video 2 node1
Live Video 1 node4
Live Video 2 node2
101

102.
Publish to Frontend Nodes
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
Dispatcher
Frontend
Frontend
Key-Value Store
Dispatcher
node1
node2
node3
node4
node5
node6
node7
Live Video 1 node7
Live Video 2 node5
Live Video 2 node1
Live Video 1 node4
Live Video 2 node2
102

103.
Challenge 6: 100 Likes/s, 10K Viewers
Distribution of 1M Likes/s
103

104.
Mobile
LINKEDIN LIVE
Likes
104

105.
The Subscription Flow
Frontend Dispatcher
Key-Value Store
1
105

106.
START WATCHING LIVE VIDEO
Viewers subscribing to Live Video Likes
cid3
cid5
Live Video 1
Live Video 2
Frontend
106

107.
HTTP SUBSCRIPTION REQUEST
Viewers subscribing to Live Video Likes
Topic ConnectionId
cid3
cid5
Live Video 1
Live Video 2
/subscribe
Frontend
In-Memory
Subscriptions Table
107

108.
FRONTEND IN-MEMORY SUBSCRIPTION ENTRY
Viewers subscribing to Live Video Likes
Topic ConnectionId
cid5
Live Video 2
/subscribe
/subscribe
cid3
Live Video 1
Frontend
In-Memory
Subscriptions Table
108

109.
FRONTEND IN-MEMORY SUBSCRIPTION ENTRY
Viewers subscribing to Live Video Likes
Topic ConnectionId
cid3
cid5
Live Video 1
Live Video 2
/subscribe
/subscribe
Frontend
In-Memory
Subscriptions Table
109

110.
FRONTEND IN-MEMORY SUBSCRIPTION ENTRY
Viewers subscribing to Live Video Likes
Topic ConnectionId
cid3
cid5
Live Video 1
Live Video 2
/subscribe
/subscribe
Frontend
In-Memory
Subscriptions Table
110

111.
The Subscription Flow
Frontend Dispatcher
Key-Value Store
1 2 3
111

112.
FRONTEND RECEIVES SUBSCRIPTION REQUEST
Frontend Node Subscriptions in Dispatcher
node1
node2
Live Video 1
Live Video 2
Frontend
Frontend
Key-Value Store
Dispatcher
Dispatcher
112

113.
FRONTEND HTTP SUBSCRIPTION REQUEST TO DISPATCHER
Frontend Node Subscriptions in Dispatcher
node1
node2
Live Video 1
Live Video 2
Frontend
Frontend
Topic
Frontend
Host
/subscribe
Key-Value Store
Dispatcher
Dispatcher
113

114.
DISPATCHER SUBSCRIPTION ENTRY IN KV STORE
Frontend Node Subscriptions in Dispatcher
node1
node2
Live Video 1
Live Video 2
Frontend
Frontend
Topic
Frontend
Host
/subscribe
/subscribe
Dispatcher
Dispatcher
Key-Value Store
114

115.
DISPATCHER SUBSCRIPTION ENTRY IN KV STORE
Frontend Node Subscriptions in Dispatcher
node1
node2
Live Video 1
Live Video 2
Frontend
Frontend
Topic
Frontend
Host
/subscribe
/subscribe
Dispatcher
Dispatcher
Key-Value Store
115

116.
The Publish Flow
Frontend Dispatcher
Key-Value Store
1
116

117.
Viewers liking Live Videos
Likes Backend (Publisher)
117

118.
Likes Published to Backend
Likes Backend (Publisher)
118

119.
The Publish Flow
Frontend Dispatcher
Key-Value Store
1
2
3
4
5
119

120.
The Publish Flow
Frontend Dispatcher
Key-Value Store
2
3
4
5
120

121.
BACKEND PUBLISH TO DISPATCHER NODES
Dispatcher Publish to Frontend Nodes
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
Dispatcher
Frontend
Frontend
Key-Value Store
Dispatcher
Live Video 1 node3
Live Video 2 node5
Live Video 2 node1
Live Video 1 node4
Live Video 2 node2
121

122.
SUBSCRIBER LOOKUP FROM KV STORE
Dispatcher Publish to Frontend Nodes
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
Dispatcher
Frontend
Frontend
Key-Value Store
Dispatcher
Live Video 1 node7
Live Video 2 node5
Live Video 2 node1
Live Video 1 node4
Live Video 2 node2
node1
node2
node3
node4
node5
node6
node7
122

123.
SUBSCRIBER LOOKUP FROM KV STORE
Dispatcher Publish to Frontend Nodes
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
Dispatcher
Frontend
Frontend
Key-Value Store
Dispatcher
Live Video 1 node7
Live Video 2 node5
Live Video 2 node1
Live Video 1 node4
Live Video 2 node2
node1
node2
node3
node4
node5
node6
node7
123

124.
PUBLISH TO FRONTEND NODES
Dispatcher Publish to Frontend Nodes
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
Dispatcher
Frontend
Frontend
Key-Value Store
Dispatcher
node1
node2
node3
node4
node5
node6
node7
Live Video 1 node7
Live Video 2 node5
Live Video 2 node1
Live Video 1 node4
Live Video 2 node2
124

125.
The Publish Flow
Frontend Dispatcher
Key-Value Store
5
125

126.
DISPATCHER PUBLISH TO FRONTEND NODE SUPERVISOR ACTOR
Frontend Publish to Mobile Clients
Actor 1
Actor 2
Actor 3
Actor 4
Actor 5
Supervisor
cid1
cid2
cid3
cid4
cid5
Live Video 1 cid3
Live Video 2 cid5
Live Video 2 cid1
Live Video 1 cid4
Live Video 2 cid2
Dispatcher
Frontend In-Memory
Subscriptions
Table
126

127.
SUBSCRIBED CLIENT CONNECTION LOOKUP FROM IN-MEMORY TABLE
Frontend Publish to Mobile Clients
Actor 1
Frontend
Actor 2
Actor 3
Actor 4
Actor 5
Supervisor
cid1
cid2
cid3
cid4
cid5
Live Video 1 cid3
Live Video 2 cid5
Live Video 2 cid1
Live Video 1 cid4
Live Video 2 cid2
Dispatcher
In-Memory
Subscriptions
Table
127

128.
PUBLISH TO MOBILE CLIENTS
Frontend Publish to Mobile Clients
Actor 1
Frontend
Actor 2
Actor 3
Actor 4
Actor 5
Supervisor
cid1
cid2
cid3
cid4
cid5
Live Video 1 cid3
Live Video 2 cid5
Live Video 2 cid1
Live Video 1 cid4
Live Video 2 cid2
Dispatcher
In-Memory
Subscriptions
Table
128

129.
Bonus Challenge:
Another data center
129

130.
130

131.
D i s t r i b u t e d S y s t e m s
Start small and add simple layers to your
architecture
131

132.
Frontend Dispatcher
DC-1
132

133.
Frontend Dispatcher
Frontend Dispatcher
DC-2
DC-1
133

134.
Frontend Dispatcher
Frontend Dispatcher
Frontend Dispatcher
DC-3
DC-2
DC-1
134

135.
Frontend Dispatcher
Frontend Dispatcher
Frontend Dispatcher
DC-3
DC-2
DC-1
135

136.
Frontend Dispatcher
Frontend Dispatcher
Frontend Dispatcher
DC-3
DC-2
DC-1
136

137.
Frontend Dispatcher
Frontend Dispatcher
Frontend Dispatcher
DC-3
DC-2
DC-1
137

138.
Frontend Dispatcher
Frontend Dispatcher
Frontend Dispatcher
DC-3
DC-2
DC-1
Cross
Data
Center
Publish
138

139.
Frontend Dispatcher
Frontend Dispatcher
Frontend Dispatcher
DC-3
DC-2
DC-1
139

140.
Frontend Dispatcher
Frontend Dispatcher
Frontend Dispatcher
DC-3
DC-2
DC-1
140

141.
Performance & Scale
141

142.
How many persistent connections can a single
machine hold?
Frontend
Server
?
142

143.
> 100,000
143

144.
ROYAL WEDDING
Second Largest Stream
>18M
viewers
© time.com
144

145.
> 100,000
We can hold those 18M connections with just
180 machines
145

146.
LINKEDIN ENGINEERING BLOG
tiny.cc/
linkedinscaling
>100K
146

147.
LinkedIn Realtime Performance
Persistent Connections/machine
5K/s
Dispatcher Publish QPS/machine
75ms
E2E Publish Latency
>100K
147

148.
Dispatcher Performance
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
How many events per second can be published to a
single dispatcher machine?
?
events/second
148

149.
Dispatcher Performance
Number of frontend nodes to publish to is limited
per event
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
?
events/second
149

150.
Dispatcher Performance
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
?
events/second
150

151.
5K
We can publish 50K likes per second to frontend
nodes with just 10 dispatcher machines
151

152.
LinkedIn Realtime Performance
>100K
Persistent Connections/machine Dispatcher Publish QPS/machine
75ms
E2E Publish Latency
5K
152

153.
E2E Publish Latency
Frontend Dispatcher
t1
Key-Value Store
153

154.
E2E Publish Latency
Frontend Dispatcher
t1
Key-Value Store
t2
t2-t1 = 75ms p90
154

155.
LinkedIn Realtime Performance
>100K
Persistent Connections/machine Dispatcher Publish QPS/machine
75ms
E2E Publish Latency
5K
155

156.
LINKEDIN ENGINEERING BLOG
tiny.cc/
linkedinlatency
Measurement System
E2E Publish Latency
156

157.
Why does this system scale?
Horizontally Scalable System
Frontend Dispatcher
Frontend
Frontend
Frontend
Dispatcher
Dispatcher
Dispatcher
Distributed K/V Storage System
157

158.
Active Status/Presence
Online/Offline indicators
LINKEDIN REALTIME TECHNOLOGY
158

159.
LINKEDIN ENGINEERING BLOG
tiny.cc/
linkedinpresence
Active Status/Presence
159

160.
Key takeaways
160

161.
R E A LT I M E I N T E R A C T I O N S
Enable dynamic instant experiences on your apps
161

162.
E V E N T S O U R C E / S S E
Built-in support in most server frameworks
Readily available client libraries
HTTP/1.1 200 OK
- RESPONSE HEADERS -
Content-Type: text/event-stream
162

163.
P l a y F r a m e w o r k a n d A k k a A c t o r s
Powerful frameworks to manage millions of
connections
State
Behavior
Mailbox
163

164.
D i s t r i b u t e d S y s t e m s
Start small and add simple layers to your
architecture
164

165.
S C A L I N G B A C K E N D S Y S T E M S
When in doubt, add a machine!
165

166.
R E A LT I M E I N T E R A C T I O N P L A T F O R M
Can be built on any server/storage technology
Horizontally scalable
Frontend Dispatcher
Key-
166

167.
R E A LT I M E I N T E R A C T I O N P L A T F O R M
You can do it for your app!
167

168.
R E A LT I M E E N G I N E E R , L I N K E D I N
@agupta03
t i n y . c c / q c o n 2 0 2 0
168
AMA at 1.40pm at Guild, Mezzanine