Presented at Netflix HQ, March 21st, 2017 The time it takes you to debug your systems is critical, both in production and development. In this talk we'll reveal how we harness the power of Reactive Programming with RxJS and RxJava to provide real-time querying and transformation of extremely high volume logs.

1. Real-time Insights
powered by Reactive Programming
Jay Phelps | @_jayphelps

2. Jay Phelps | @_jayphelps
Let's talk why

4. Jay Phelps | @_jayphelps
Time is money

5. Jay Phelps | @_jayphelps
127+ million
person-hours lost
per year
$500k — $1+ million
per hour
average hourly cost of
critical failure
https://devops.com/real-cost-downtime/
Fortune 500

6. Jay Phelps | @_jayphelps
Every second counts

7. Jay Phelps | @_jayphelps
Not just outages...

8. Jay Phelps | @_jayphelps
debugging, testing, and even
Information Security too

9. Jay Phelps | @_jayphelps
improving Developer Experience

10. Senior Software Engineer |
@_jayphelps
Jay Phelps

11. Jay Phelps | @_jayphelps
debugging, testing, and InfoSec

12. Jay Phelps | @_jayphelps
InfoSec
“preventing unauthorized access, use, or
disruption of information”

13. Jay Phelps | @_jayphelps
stopping hackers

14. Jay Phelps | @_jayphelps
InfoSec

15. Jay Phelps | @_jayphelps

16. Jay Phelps | @_jayphelps
I'm on it.

17. Jay Phelps | @_jayphelps
We can block exploits
using our gateway proxy

18. Jay Phelps | @_jayphelps
...we need to know it's working...

19. Jay Phelps | @_jayphelps
...we want to watch attackers try...

20. Jay Phelps | @_jayphelps
...in real-time

21. Jay Phelps | @_jayphelps
We need real-time insights
For debugging, testing, and InfoSec

22. Jay Phelps | @_jayphelps
Logging is the answer

23. Jay Phelps | @_jayphelps
LOG ALL THE THINGSLOG ALL THE THINGSLOG ALL THE THINGSLOG ALL THE THINGSLOG ALL THE THINGS

24. Jay Phelps | @_jayphelps
One little problem...

25. Jay Phelps | @_jayphelps
We have millions of devices...

26. Jay Phelps | @_jayphelps
We have thousands of servers...

27. Jay Phelps | @_jayphelps
We need to process them in real-time

28. Jay Phelps | @_jayphelps
“Netflix is a log generating company that
happens to stream movies” - Adrian Cockroft

29. Jay Phelps | @_jayphelps
Massiveamount of streaming logs

30. Jay Phelps | @_jayphelps
Reactive Extensions

31. Jay Phelps | @_jayphelps
Rx

32. Jay Phelps | @_jayphelps
The best ideas from the Observer pattern, the
Iterator pattern, and functional programming

33. Jay Phelps | @_jayphelps
“lodash for events”

34. Jay Phelps | @_jayphelps

35. Jay Phelps | @_jayphelps

36. Jay Phelps | @_jayphelps
High-level intro

37. Jay Phelps | @_jayphelps
.map(value => value * 10)
.forEach(value => {
console.log(value);
});
[1, 2, 3]

38. Jay Phelps | @_jayphelps
.map(value => value * 10)
.forEach(value => {
console.log(value);
});
[1, 2, 3]

39. Jay Phelps | @_jayphelps
.map(value => value * 10)
.forEach(value => {
console.log(value);
});
[1, 2, 3]

40. Jay Phelps | @_jayphelps
.map(value => value * 10)
.forEach(value => {
console.log(value);
});
[1, 2, 3]

41. Jay Phelps | @_jayphelps
.map(value => value * 10)
.forEach(value => {
console.log(value);
});
[1, 2, 3]

42. Jay Phelps | @_jayphelps
.map(value => value * 10)
.forEach(value => {
console.log(value);
});
[1, 2, 3]
// 10

43. Jay Phelps | @_jayphelps
.map(value => value * 10)
.forEach(value => {
console.log(value);
});
[1, 2, 3]
// 10
// 20

44. Jay Phelps | @_jayphelps
.map(value => value * 10)
.forEach(value => {
console.log(value);
});
[1, 2, 3]
// 10
// 20
// 30

45. Jay Phelps | @_jayphelps
[1, 2, 3]
// 10
// 20
// 30
.map(value => value * 10)
.forEach(value => {
console.log(value);
});

46. Jay Phelps | @_jayphelps
[1, 2, 3]
// 10
// 20
// 30
.map(value => value * 10)
.forEach(value => {
console.log(value);
});

47. Jay Phelps | @_jayphelps
Observable.of(1, 2, 3)
.map(value => value * 10)
.forEach(value => {
console.log(value);
});

48. Jay Phelps | @_jayphelps
Observable.of(1, 2, 3)
.map(value => value * 10)
.forEach(value => {
console.log(value);
});

49. Jay Phelps | @_jayphelps
Observable.of(1, 2, 3)
.map(value => value * 10)
.subscribe(value => {
console.log(value);
});

50. Jay Phelps | @_jayphelps
Observable.of(1, 2, 3)
.map(value => value * 10)
.subscribe(value => {
console.log(value);
});

51. Jay Phelps | @_jayphelps
Observable.of(1, 2, 3)
// 10
.map(value => value * 10)
.subscribe(value => {
console.log(value);
});

52. Jay Phelps | @_jayphelps
Observable.of(1, 2, 3)
// 10
// 20
.map(value => value * 10)
.subscribe(value => {
console.log(value);
});

53. Jay Phelps | @_jayphelps
Observable.of(1, 2, 3)
// 10
// 20
// 30
.map(value => value * 10)
.subscribe(value => {
console.log(value);
});

54. Jay Phelps | @_jayphelps
Array is a collection

55. Jay Phelps | @_jayphelps
Observable is a collection that arrives over time

56. Jay Phelps | @_jayphelps
Represents a stream

57. Jay Phelps | @_jayphelps
button.addEventListener('click', event => {
console.log('you clicked!');
});

58. Jay Phelps | @_jayphelps
Observable.fromEvent(button, 'click')
.subscribe(event => {
console.log('you clicked!');
});

59. .subscribe(event => {
console.log('you clicked!');
});
Jay Phelps | @_jayphelps
Observable.fromEvent(button, 'click')
.debounceTime(500)

60. .subscribe(event => {
console.log('you clicked!');
});
Jay Phelps | @_jayphelps
Observable.fromEvent(button, 'click')
.debounceTime(500)

61. Jay Phelps | @_jayphelps
Observable.fromEvent(button, 'click')
.debounceTime(500)

62. Jay Phelps | @_jayphelps
Observable.fromEvent(button, 'click')
.debounceTime(500)

63. Jay Phelps | @_jayphelps
Observable.fromEvent(button, 'click')
.debounceTime(500)

64. Jay Phelps | @_jayphelps
Observable.fromEvent(button, 'click')
.debounceTime(500)

65. Jay Phelps | @_jayphelps
Observable.fromEvent(button, 'click')
.debounceTime(500)

66. Observable.fromEvent(button, 'click')
.debounceTime(500)
Jay Phelps | @_jayphelps

67. Observable.fromEvent(button, 'click')
.debounceTime(500)
Jay Phelps | @_jayphelps

68. Observable.fromEvent(button, 'click')
.debounceTime(500)
Jay Phelps | @_jayphelps

69. Observable.fromEvent(button, 'click')
.debounceTime(500)
Jay Phelps | @_jayphelps

70. Observable.fromEvent(button, 'click')
.debounceTime(500)
Jay Phelps | @_jayphelps

71. Observable.fromEvent(button, 'click')
.debounceTime(500)
Jay Phelps | @_jayphelps

72. Observable.fromEvent(button, 'click')
.debounceTime(500)
Jay Phelps | @_jayphelps
500 ms

73. Observable.fromEvent(button, 'click')
.debounceTime(500)
Jay Phelps | @_jayphelps
500 ms

74. Observable.fromEvent(button, 'click')
.debounceTime(500)
Jay Phelps | @_jayphelps
500 ms

75. Observable.fromEvent(button, 'click')
.debounceTime(500)
Jay Phelps | @_jayphelps
500 ms

76. Observable.fromEvent(button, 'click')
.debounceTime(500)
Jay Phelps | @_jayphelps
500 ms

77. Observable.fromEvent(button, 'click')
.debounceTime(500)
Jay Phelps | @_jayphelps
500 ms

78. Observable.fromEvent(button, 'click')
.debounceTime(500)
Jay Phelps | @_jayphelps
500 ms

79. Observable.fromEvent(button, 'click')
.debounceTime(500)
Jay Phelps | @_jayphelps
500 ms

80. Observable.fromEvent(button, 'click')
.debounceTime(500)
Jay Phelps | @_jayphelps
500 ms 500 ms

81. Observable.fromEvent(button, 'click')
.debounceTime(500)
Jay Phelps | @_jayphelps
500 ms 500 ms

82. Observable.fromEvent(button, 'click')
.debounceTime(500)
Jay Phelps | @_jayphelps
500 ms 500 ms

83. Observable.fromEvent(button, 'click')
.debounceTime(500)
Jay Phelps | @_jayphelps
500 ms 500 ms

84. Observable.fromEvent(button, 'click')
.debounceTime(500)
Jay Phelps | @_jayphelps
500 ms 500 ms500 ms

85. Observable.fromEvent(button, 'click')
.debounceTime(500)
Jay Phelps | @_jayphelps
500 ms 500 ms500 ms

86. Observable.fromEvent(button, 'click')
.debounceTime(500)
Jay Phelps | @_jayphelps
500 ms 500 ms500 ms

87. Jay Phelps | @_jayphelps
Observable.fromEvent(button, 'click')
.debounceTime(500)

88. Jay Phelps | @_jayphelps
Stream of logs

89. Jay Phelps | @_jayphelps
{
"path": "/something",
"status": 500
}
Log message in JSON

90. Jay Phelps | @_jayphelps
import { webSocket } from "rxjs/observable/webSocket";
let streamOfLogs = webSocket("ws://logs.netflix.com")

91. .subscribe(msg => {
console.log(msg);
});
Jay Phelps | @_jayphelps
streamOfLogs
.filter(msg => msg.status !== 200)

92. .subscribe(msg => {
console.log(msg);
});
Jay Phelps | @_jayphelps
streamOfLogs
.filter(msg => msg.status !== 200)

93. 500 200 404 301
Jay Phelps | @_jayphelps
500
streamOfLogs
.filter(msg => msg.status !== 200)
404

94. Jay Phelps | @_jayphelps
Learning Rx isn't easy

95. Jay Phelps | @_jayphelps
It's worth it!

96. Jay Phelps | @_jayphelps
It's worth it!
Jay!!

97. Jay Phelps | @_jayphelps
Case Study
Scaling high volume logs in real-time

98. Jay Phelps | @_jayphelps
We're going to log JSON

99. Jay Phelps | @_jayphelps
{
"path": "/something",
"status": 200
}
Device
{
"event": "StartPlay",
"device": "XBox 360"
}
Server

100. Jay Phelps | @_jayphelps
Stream them all through a single pipeline

101. Jay Phelps | @_jayphelps
How do you find and process the logs you want

102. Jay Phelps | @_jayphelps
Write a custom job in Java?

103. Jay Phelps | @_jayphelps
Job: a unit of work

104. Jay Phelps | @_jayphelps
{
"path": "/something",
"status": 500,
"duration": 135,
// etc ...
}
Server log message

105. Jay Phelps | @_jayphelps
streamOfLogs
.filter(event -> event.get("status") != 200)
.map(event ->
new JSONObject()
.put("path", event.get("path"))
.put("status", event.get("status"))
);

106. Jay Phelps | @_jayphelps
Powerful, but... not ideal

107. Jay Phelps | @_jayphelps
Query Language

108. Jay Phelps | @_jayphelps
SELECT path,status WHERE status != 200

109. Jay Phelps | @_jayphelps
Just-in-time (JIT) Compilation

110. Jay Phelps | @_jayphelps
SELECT path,status,duration WHERE status != 200

111. Jay Phelps | @_jayphelps
SELECT path,status,duration WHERE status != 200

112. Jay Phelps | @_jayphelps
streamOfLogs
.filter(event -> event.get("status") != 200)
.map(event ->
new JSONObject()
.put("path", event.get("path"))
.put("status", event.get("status"))
);
SELECT path,status,duration WHERE status != 200

113. Jay Phelps | @_jayphelps
What's next?

114. Jay Phelps | @_jayphelps
8+ million messages per second, peakmessagespersecond
Time

115. Jay Phelps | @_jayphelps
How do we scale this?

116. Jay Phelps | @_jayphelps
Distribute work via autoscaling

117. Jay Phelps | @_jayphelps
A
Source
B
Server
B
Server
50%
50%
Load balancing a job

118. Jay Phelps | @_jayphelps
Chain jobs together

119. Jay Phelps | @_jayphelps
SELECT path,status WHERE source == "API"
Chain jobs together

120. Jay Phelps | @_jayphelps
A
Job
B C
Job Job
Chain jobs together

121. Jay Phelps | @_jayphelps
High-volume, distributed systems have a problem...

122. Jay Phelps | @_jayphelps
Backpressure

123. Jay Phelps | @_jayphelps
“pressure opposed to the desired flow of gases in
confined places such as a pipe” - wikipedia.org

124. Jay Phelps | @_jayphelps
A B C
Server Server Server
100 rps 75 rps

125. Jay Phelps | @_jayphelps
A B C
Server Server Server
60 sec * 25 rps = 1,500 rpm!
100 rps 75 rps

126. Jay Phelps | @_jayphelps
A B C
Server Server Server
100 rps 75 rps
60 sec * 60 min * 25 rps = 90,000 rph!

127. Jay Phelps | @_jayphelps
Buffer or Drop

128. Jay Phelps | @_jayphelps
onBackpressureBuffer

129. .onBackpressureBuffer()
Jay Phelps | @_jayphelps
streamOfLogs
.subscribe(value -> {
// something expensive
Thread.sleep(100);
});

130. .onBackpressureBuffer()
Jay Phelps | @_jayphelps
streamOfLogs
.subscribe(value -> {
// something expensive
Thread.sleep(100);
});

131. .onBackpressureBuffer()
Jay Phelps | @_jayphelps
streamOfLogs
.subscribe(value -> {
// something expensive
Thread.sleep(100);
});

132. Jay Phelps | @_jayphelps
1 2 3 4 5
.onBackpressureBuffer()
[3, 4, 5]
1 2

133. Jay Phelps | @_jayphelps
onBackpressureDrop

134. .onBackpressureDrop()
streamOfLogs
.subscribe(value -> {
// something expensive
Thread.sleep(100);
});
Jay Phelps | @_jayphelps

135. .onBackpressureDrop()
streamOfLogs
.subscribe(value -> {
// something expensive
Thread.sleep(100);
});
Jay Phelps | @_jayphelps

136. .onBackpressureDrop()
streamOfLogs
.subscribe(value -> {
// something expensive
Thread.sleep(100);
});
Jay Phelps | @_jayphelps

137. Jay Phelps | @_jayphelps
1 2 3 4 5
.onBackpressureDrop()
1 4

138. Jay Phelps | @_jayphelps
Job authors choose which

139. Jay Phelps | @_jayphelps
700+ jobs running 24/7
8+ million events/second
Autoscaling
Mantis
Low-latency, high throughput stream-processing job platform
http://techblog.netflix.com

140. Jay Phelps | @_jayphelps
We need somewhere to submit and view query results

141. Jay Phelps | @_jayphelps
Query Builder UI

142. Jay Phelps | @_jayphelps
SELECT path,status WHERE status != 200

143. Jay Phelps | @_jayphelps

144. Jay Phelps | @_jayphelps

145. Jay Phelps | @_jayphelps

146. Jay Phelps | @_jayphelps

147. Jay Phelps | @_jayphelps

148. Jay Phelps | @_jayphelps

153. Jay Phelps | @_jayphelps
Can be extremely high-volume
100k+ rps

154. Jay Phelps | @_jayphelps
We simply can’t render that fast!

155. Jay Phelps | @_jayphelps
Even if we could, it would be bad UX

156. Jay Phelps | @_jayphelps
Performance solutions are often driven by UX

157. Jay Phelps | @_jayphelps
So what do we do?

158. Jay Phelps | @_jayphelps
UI virtualization aka Virtual Table?

159. VirtualizedNOT Virtualized vs.

160. Jay Phelps | @_jayphelps
We still can’t update that virtual
table 100k per second

161. Jay Phelps | @_jayphelps
This is also backpressure

162. Jay Phelps | @_jayphelps
Buffer or Drop

163. Jay Phelps | @_jayphelps
Buffer for 1 second

164. Jay Phelps | @_jayphelps
getWebSocket()
.bufferTime(1000)

165. Jay Phelps | @_jayphelps
Buffer size is unbounded

166. Jay Phelps | @_jayphelps
See what your users actually do

167. Jay Phelps | @_jayphelps
Users want a sample

168. Jay Phelps | @_jayphelps
Batch sampling

169. Jay Phelps | @_jayphelps
Drop after reaching a certain threshold

170. Jay Phelps | @_jayphelps
Batch sampling

171. Jay Phelps | @_jayphelps
Batch sampling

172. Jay Phelps | @_jayphelps
Batch sampling

173. Jay Phelps | @_jayphelps
Batch sampling

174. Jay Phelps | @_jayphelps
let buffer = getWebSocket()
.bufferTime(1000);
let gate = new BehaviorSubject(true);
let batchSize = 50;
let batchSizeCounter = 0;
let results = gate$
.switchMap(enabled => enabled ? buffer : Observable.never())
.do(buffer => {
batchSizeCounter += buffer.length;
if (batchSizeCounter >= batchSize) {
// truncates the array, if it's over batchSize
buffer.length = batchSize;
// turns on the gate, pausing the stream
gate.next(false);
batchSizeCounter = 0;
}
});
https://goo.gl/DMOqBA

175. Jay Phelps | @_jayphelps
let buffer = getWebSocket()
.bufferTime(1000);
let gate = new BehaviorSubject(true);
let batchSize = 50;
let batchSizeCounter = 0;
let results = gate$
.switchMap(enabled => enabled ? buffer : Observable.never())
.do(buffer => {
batchSizeCounter += buffer.length;
if (batchSizeCounter >= batchSize) {
// truncates the array, if it's over batchSize
buffer.length = batchSize;
// turns on the gate, pausing the stream
gate.next(false);
batchSizeCounter = 0;
}
});
https://goo.gl/DMOqBA

176. Jay Phelps | @_jayphelps
let buffer = getWebSocket()
.bufferTime(1000);
let gate = new BehaviorSubject(true);
let batchSize = 50;
let batchSizeCounter = 0;
let results = gate$
.switchMap(enabled => enabled ? buffer : Observable.never())
.do(buffer => {
batchSizeCounter += buffer.length;
if (batchSizeCounter >= batchSize) {
// truncates the array, if it's over batchSize
buffer.length = batchSize;
// turns on the gate, pausing the stream
gate.next(false);
batchSizeCounter = 0;
}
});
https://goo.gl/DMOqBA

177. Jay Phelps | @_jayphelps
let buffer = getWebSocket()
.bufferTime(1000);
let gate = new BehaviorSubject(true);
let batchSize = 50;
let batchSizeCounter = 0;
let results = gate$
.switchMap(enabled => enabled ? buffer : Observable.never())
.do(buffer => {
batchSizeCounter += buffer.length;
if (batchSizeCounter >= batchSize) {
// truncates the array, if it's over batchSize
buffer.length = batchSize;
// turns on the gate, pausing the stream
gate.next(false);
batchSizeCounter = 0;
}
});
https://goo.gl/DMOqBA

178. Jay Phelps | @_jayphelps
let buffer = getWebSocket()
.bufferTime(1000);
let gate = new BehaviorSubject(true);
let batchSize = 50;
let batchSizeCounter = 0;
let results = gate$
.switchMap(enabled => enabled ? buffer : Observable.never())
.do(buffer => {
batchSizeCounter += buffer.length;
if (batchSizeCounter >= batchSize) {
// truncates the array, if it's over batchSize
buffer.length = batchSize;
// turns on the gate, pausing the stream
gate.next(false);
batchSizeCounter = 0;
}
});
https://goo.gl/DMOqBA

179. Jay Phelps | @_jayphelps
let buffer = getWebSocket()
.bufferTime(1000);
let gate = new BehaviorSubject(true);
let batchSize = 50;
let batchSizeCounter = 0;
let results = gate$
.switchMap(enabled => enabled ? buffer : Observable.never())
.do(buffer => {
batchSizeCounter += buffer.length;
if (batchSizeCounter >= batchSize) {
// truncates the array, if it's over batchSize
buffer.length = batchSize;
// turns on the gate, pausing the stream
gate.next(false);
batchSizeCounter = 0;
}
});
https://goo.gl/DMOqBA

180. Jay Phelps | @_jayphelps
let buffer = getWebSocket()
.bufferTime(1000);
let gate = new BehaviorSubject(true);
let batchSize = 50;
let batchSizeCounter = 0;
let results = gate$
.switchMap(enabled => enabled ? buffer : Observable.never())
.do(buffer => {
batchSizeCounter += buffer.length;
if (batchSizeCounter >= batchSize) {
// truncates the array, if it's over batchSize
buffer.length = batchSize;
// turns on the gate, pausing the stream
gate.next(false);
batchSizeCounter = 0;
}
});
https://goo.gl/DMOqBA

181. Jay Phelps | @_jayphelps
Works for low-volume queries too

182. Jay Phelps | @_jayphelps
Raven

183. Jay Phelps | @_jayphelps
RavenSweet
Christmas

184. Jay Phelps | @_jayphelps
improved debugging, testing, and InfoSec

185. Jay Phelps | @_jayphelps
Netflix loves Rx

186. Jay Phelps | @_jayphelps
Rx is powerful, cross-platform

187. Thanks!
@_jayphelps