Speaker: Joseph Rea, Engineer, Confluent Joseph will talk about how to visualize topics in Apache Kafka®, the difference between a stream and table in KSQL and his lessons learned on tackling this technical challenge with millions of Kafka messages consumed per second. With such functionalities, users can understand their data easier and in a highly performant and scalable way. This talk covers understanding web workers as they relate to webpack, web socket management, debugging browser performance and the future of the applications that can now be built. Joseph Rea started engineering with the LAMP stack building custom e-commerce checkouts, ERP systems and enterprise water/sewer billing software. He worked at Yahoo as a front end engineer in the media org before doing Android and iOS development for the video SDK. He also worked at LifeLock to build an application that updated PII on various service sites. He likes turtles. He currently works at Confluent building so much UI. He blogs at https://cnfl.io/blog-joseph-rea.

1. 1
Visualizing messages in Kafka®
Joseph Rea

2. 22
You get to
drink from
the firehose

3. 33
Who dis?! Software engineer Worked at LifeLock®
,
protected 0 identities
Ok guy Built the thing
Why I’m qualified to give
this talk (I’m not, but
going to anyway)

4. 4
Stream/Tables
Fast, maybe
Lots of them, maybe
Is it a stack, maybe
Call me, maybe

5. 5
Eye don’t know.
The human eye can identify an image in 13-80ms[1]
That’s like ~12 records/sec, if it’s cat videos.
One thread, no replication, fella named Jay
produced 821,557 100 byte records/sec[2]
.
Pictures are probably larger than 100 bytes, so for
a fair comparison, they are very small cats.
Visualizing
Something that can’t be
seen
[1] http://news.mit.edu/2014/in-the-blink-of-an-eye-0116
[2] https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines

6. 66
821,557 > 12

7. 77
Give the people what
they want
821,557 records/sec seems
impossible to deal with.
Do we even have options to keep
things up to date in records/sec?
What does HTTP give us?

8. 8
Ways to deal
with it
HTTP long poll
● variable rates
● load management
● $$$$
Use cURL
● How do I even?
● Best option
Web sockets
● rate can do what it wants
● more Kafkaesque
● cool
Don’t
● get a static amount
● Fast, but very 2000-late.
● boring
http://websocket.org/quantum.html

9. 99
Long polling
This isn’t even messages.
Case A: 1000 users, 1 request/sec -
80 GB/day
$225.61. At least data in is free.
Cloud’s actual header: $112.46
Case C:
It’s actually $16,097.21
http://websocket.org/quantum.html

10. 1010
Web sockets
24 Oct 2011 (LinkedIn went public in 2011)
Clients control the integration (more work)
Getting them into the build
Bad with proxies
Tell the server to do things (futurology)
Using a stream to render a stream
Can handle the load, maybe

11. 1111
Practical example
Message browser late-2018
Use case: people want to see
messages
Shows messages, can fail
spectacularly

12. 121212
1 message/sec
CPU: ~18%
No real system resource
problems

13. 13
20 messages/sec
Still renders, so that’s good
What is happening to the
browser

14. 14
100 message/sec
Huston, we have a problem.
At this point, you literally can’t do
anything in the browser. You
have to kill the tab. That’s bad.

15. 15
cpu load
idle
cpu load refresh messages/sec message bytes noticeable degradation failure*
16% 38% 1 350 no no
22% 68% 5 350 no no
31% 90% 10 350 yes no
39% >100% 12 350 yes no
72% >100% 50 350 yes no
99% >100% 100 350 yes yes

16. 1616
What now? Refactor
● Use web workers
● Seems obvious
● Shows messages
Do something different
● Use web workers
● Could solve other
problems
● Builds for future
use cases
● Shows messages
50 messages/sec is
pretty good for a UI?
Knowing that
821,557
is a possible number per
second, maybe it’s bad

17. 1717
If we’re going to take a look web workers, maybe
we can build it better/faster/stronger.
What are we trying to solve?

18. 1818
Idly staring at a highway for hours
Keeping track of the colors of cars

19. 1919
How long have I been staring at this highway?
In the last 15 minutes, how many red cars have I
seen?

20. 20
We need to show a user the information in their
stream as it relates to time
We need a way for a user to view a snapshot in
time that updates relevant data changes

21. 21
100 message/sec
Huston, we have a problem.
At this point, you literally can’t do
anything in the browser. You
have to kill the tab. That’s bad.

22. 2222
Main UI process
99.9999% of browsers only need 1
process.
Rendering is expensive
Making big XHR requests are
expensive
99.9999% of cases, you can do these
async and it is fine
We can split those processes and $$

23. 23
Threads in
JavaScript
They do exist
They are called workers

24. 2424
What I said before
was a lie
Does parallel things
No object reference
It’s really a separate process, but
effectively does the same job

25. 25
High
throughput
Remember sockets?
At the mercy of the BE
High/Low throughput delta is bad
Heavy Computation
Now with more DOM nodes!
New and improved messages!
Heavy
Computation

26. 262626
It’s a
stream
that’s a
table.

27. 27
It’s a
table
that’s a
table.

28. 28
Perf tools in the
browser to prove this
works
Memory management
(2GB max for V8)
What we didn’t cover
Stopping time

29. 29