(Viktor Gamov, Confluent) Kafka Summit SF 2018 All things change constantly! And dealing with constantly changing data at low latency is pretty hard. It doesn’t need to be that way. Apache Kafka, the de facto standard open source distributed stream processing system. Many of us know Kafka’s architectural and pub/sub API particulars. But that doesn’t mean we’re equipped to build the kind of real-time streaming data systems that the next generation of business requirements are going to demand. We need to get on board with streams! Viktor Gamov will introduce Kafka Streams and KSQL—an important recent addition to the Confluent Open Source platform that lets us build sophisticated stream processing systems with little to no code at all! They will talk about how to deploy stream processing applications and look at the actual working code that will bring your thinking about streaming data systems from the ancient history of batch processing into the current era of streaming data! P.S. No prior knowledge of Kafka Streams, KSQL or Ghostbusters needed!

1. KAFKA SUMMIT, SAN FRANCISCO, CA,
OCTOBER 2018
Crossing the Streams:
Rethinking Stream Processing with
Kafka Streams and KSQL

2. https://twitter.com/gAmUssA/status/1048258981595111424

3. @@gamussa #KafkaSummit @confluentinc

4. @@gamussa #KafkaSummit @confluentinc

5. @@gamussa #KafkaSummit @confluentinc
High Throughput
Streaming platform

6. @@gamussa #KafkaSummit @confluentinc
Java Apps / Kafka
Streams
Continuous
Computation
High Throughput
Streaming platform
API based
clustering

7. @@gamussa #KafkaSummit @confluentinc
Serving Layer
(Cassandra,
Elastic, etc.)
Java Apps / Kafka
Streams
Continuous
Computation
High Throughput
Streaming platform
API based
clustering

8. @gamussa #KafkaSummit @confluentinc
Streaming
is the toolset
for dealing
with events
as they move!

9. @gamussa #KafkaSummit @confluentinc
Stream Processing by Analogy
$ cat < in.txt | grep "ksql" | tr a-z A-Z > out.txt

10. @gamussa #KafkaSummit @confluentinc
Stream Processing by Analogy
Kafka Cluster
$ cat < in.txt | grep "ksql" | tr a-z A-Z > out.txt

11. @gamussa #KafkaSummit @confluentinc
Stream Processing by Analogy
Kafka Cluster
Connect API Connect API
$ cat < in.txt | grep "ksql" | tr a-z A-Z > out.txt

12. @gamussa #KafkaSummit @confluentinc
Stream Processing by Analogy
Kafka Cluster
Connect API Stream Processing Connect API
$ cat < in.txt | grep "ksql" | tr a-z A-Z > out.txt

13. @@gamussa #KafkaSummit @confluentinc
Streaming Platform Architecture

14. @@gamussa #KafkaSummit @confluentinc
Streaming Platform Architecture
Kafka Brokers

15. @@gamussa #KafkaSummit @confluentinc
Streaming Platform Architecture
Zookeeper NodesKafka Brokers

16. @@gamussa #KafkaSummit @confluentinc
Streaming Platform Architecture
Zookeeper Nodes
Application
Native Client
library
Kafka Brokers

17. @@gamussa #KafkaSummit @confluentinc
Streaming Platform Architecture
Zookeeper Nodes
Schema Registry
Application
Native Client
library
Kafka Brokers

18. @@gamussa #KafkaSummit @confluentinc
Streaming Platform Architecture
Zookeeper Nodes
Schema Registry
Application
Native Client
library
Application
Kafka Streams
Kafka Brokers

19. @@gamussa #KafkaSummit @confluentinc
Streaming Platform Architecture
Kafka Connect
Zookeeper Nodes
Schema Registry
Application
Native Client
library
Application
Kafka Streams
Kafka Brokers

20. @@gamussa #KafkaSummit @confluentinc
Streaming Platform Architecture
Kafka Connect
Zookeeper Nodes
Schema RegistryREST Proxy
Application
Load Balancer *
Application
Native Client
library
Application
Kafka Streams
Kafka Brokers

21. @@gamussa #KafkaSummit @confluentinc
Streaming Platform Architecture
Kafka Connect
Zookeeper Nodes
Schema RegistryREST Proxy
Application
Load Balancer *
Application
Native Client
library
Application
Kafka Streams
Kafka Brokers

22. https://twitter.com/monitoring_king/status/1048264580743479296

23. @@gamussa #KafkaSummit @confluentinc
LET’S TALK ABOUT THIS FRAMEWORK
OF YOURS.
I THINK ITS GOOD, EXCEPT IT SUCKS

24. @@gamussa #KafkaSummit @confluentinc
SO LET ME WRITE THE FRAMEWORK
THAT’S WHY IT MIGHT BE REALLY GOOD

25. @gamussa #KafkaSummit @confluentinc
Every framework Wants to be when it grows up

26. @gamussa #KafkaSummit @confluentinc
Every framework Wants to be when it grows up
Scalable

27. @gamussa #KafkaSummit @confluentinc
Every framework Wants to be when it grows up
Scalable Elastic

28. @gamussa #KafkaSummit @confluentinc
Every framework Wants to be when it grows up
Scalable Elastic Fault-tolerant

29. @gamussa #KafkaSummit @confluentinc
Every framework Wants to be when it grows up
Scalable Elastic Fault-tolerant
Stateful

30. @gamussa #KafkaSummit @confluentinc
Every framework Wants to be when it grows up
Scalable Elastic Fault-tolerant
Stateful Distributed

31. https://twitter.com/157rahul/status/1050505569746841600

32. @gamussa #KafkaSummit @confluentinc
The log is a simple idea
Messages are added at the end of the log
Old New

33. @gamussa #KafkaSummit @confluentinc
Consumers have a position all of their own
Sally
is here
George
is here
Fred
is here
Old New
Scan
Scan
Scan

34. @gamussa #KafkaSummit @confluentinc
Only Sequential Access
Old New
Read to offset & scan

35. @gamussa #KafkaSummit @confluentinc
Shard data to get scalability

36. @gamussa #KafkaSummit @confluentinc
Shard data to get scalability
Cluster of
machines

37. @gamussa #KafkaSummit @confluentinc
Shard data to get scalability
Producer (1) Producer (2) Producer (3)
Cluster of
machines
Partitions live on
different machines
Messages are sent to
different partitions

39. CONSUMER GROUP
COORDINATOR

40. CONSUMER GROUP
COORDINATORCONSUMERS

41. CONSUMER GROUP
COORDINATORCONSUMERS
CONSUMER GROUP

42. @gamussa #KafkaSummit @confluentinc
Linearly Scalable Architecture
Producers
Consumers

43. @gamussa #KafkaSummit @confluentinc
Linearly Scalable Architecture
Single topic:
- Many producers machines
- Many consumer machines
- Many Broker machines
Producers
Consumers

44. @gamussa #KafkaSummit @confluentinc
Linearly Scalable Architecture
Single topic:
- Many producers machines
- Many consumer machines
- Many Broker machines
No Bottleneck!!
Producers
Consumers

45. Talk is cheap! Show me code!
https://cnfl.io/streams-movie

46. Talk is cheap! Show me code!
https://cnfl.io/streams-movie

47. @gamussa #KafkaSummit @confluentinc
As developers,
we want to build APPS
not INFRASTRUCTURE

48. @

49. @gamussa #KafkaSummit @confluentinc
the KAFKA STREAMS API is a
JAVA API to
BUILD REAL-TIME APPLICATIONS

50. @gamussa #KafkaSummit @confluentinc
App
Streams
API

51. @gamussa #KafkaSummit @confluentinc
App
Streams
API
Not running
inside brokers!

52. @gamussa #KafkaSummit @confluentinc
Brokers?
Nope!
App
Streams
API
App
Streams
API
App
Streams
API
Same app, many instances

53. @gamussa #KafkaSummit @confluentinc
Brokers?
Nope!
App
Streams
API
App
Streams
API
App
Streams
API
Same app, many instances

54. @gamussa #KafkaSummit @confluentinc
Before
DashboardProcessing Cluster
Your Job
Shared Database

55. @gamussa #KafkaSummit @confluentinc
After
Dashboard
APP
Streams
API

56. this means you can
DEPLOYyour app ANYWHERE using
WHATEVER TECHNOLOGY YOU WANT

57. @gamussa #KafkaSummit @confluentinc
So many places to run you app!
...and many more...

58. @gamussa #KafkaSummit @confluentinc
Things Kafka Stream Does
Open Source Elastic, Scalable,
Fault-tolerant
Supports Streams
and Tables
Runs Everywhere
Exactly-Once
Processing
Event-Time
Processing
Kafka Security
Integration
Powerful Processing incl.
Filters, Transforms, Joins,
Aggregations, Windowing
Enterprise Support

59. @gamussa #KafkaSummit @confluentinc
Table-Stream Duality

60. @gamussa #KafkaSummit @confluentinc
Table-Stream Duality

62. Do you think that’s a table
you are querying ?

63. @gamussa #KafkaSummit @confluentinc
TABLE STREAM TABLE

64. @gamussa #KafkaSummit @confluentinc
Gwen 1
TABLE STREAM TABLE

65. @gamussa #KafkaSummit @confluentinc
Gwen 1
TABLE STREAM TABLE
(“Gwen”, 1)

66. @gamussa #KafkaSummit @confluentinc
Gwen 1
TABLE STREAM TABLE
(“Gwen”, 1) Gwen 1

67. @gamussa #KafkaSummit @confluentinc
Gwen 1
Gwen 1
Matthias 1
TABLE STREAM TABLE
(“Gwen”, 1) Gwen 1

68. @gamussa #KafkaSummit @confluentinc
Gwen 1
Gwen 1
Matthias 1
TABLE STREAM TABLE
(“Gwen”, 1)
(“Matthias”, 1)
Gwen 1
Gwen 1
Matthias 1

69. @gamussa #KafkaSummit @confluentinc
Gwen 1
Gwen 1
Matthias 1
Gwen 2
Matthias 1
TABLE STREAM TABLE
(“Gwen”, 1)
(“Matthias”, 1)
(“Gwen”, 2)
Gwen 1
Gwen 1
Matthias 1

70. @gamussa #KafkaSummit @confluentinc
Gwen 1
Gwen 1
Matthias 1
Gwen 2
Matthias 1
TABLE STREAM TABLE
(“Gwen”, 1)
(“Matthias”, 1)
(“Gwen”, 2)
Gwen 1
Gwen 1
Matthias 1
Gwen 2
Matthias 1

71. @gamussa #KafkaSummit @confluentinc
Gwen 1
Gwen 1
Matthias 1
Gwen 2
Matthias 1
Gwen 2
Matthias 1
Viktor 1
TABLE STREAM TABLE
(“Gwen”, 1)
(“Matthias”, 1)
(“Gwen”, 2)
(“Viktor”, 1)
Gwen 1
Gwen 1
Matthias 1
Gwen 2
Matthias 1

72. @gamussa #KafkaSummit @confluentinc
Gwen 1
Gwen 1
Matthias 1
Gwen 2
Matthias 1
Gwen 2
Matthias 1
Viktor 1
TABLE STREAM TABLE
(“Gwen”, 1)
(“Matthias”, 1)
(“Gwen”, 2)
(“Viktor”, 1)
Gwen 1
Gwen 1
Matthias 1
Gwen 2
Matthias 1
Gwen 2
Matthias 1
Viktor 1

73. Talk is cheap! Show me code!

74. Talk is cheap! Show me code!

75. What’s next?

76. https://twitter.com/IDispose/status/1048602857191170054

77. @gamussa #KafkaSummit @confluentinc
KSQL #FTW

78. @gamussa #KafkaSummit @confluentinc
KSQL #FTW
1 UI

79. @gamussa #KafkaSummit @confluentinc
KSQL #FTW
1 UI
ksql>
2 CLI

80. @gamussa #KafkaSummit @confluentinc
KSQL #FTW
1 UI
ksql>
2 CLI
POST /query
3 REST

81. @gamussa #KafkaSummit @confluentinc
KSQL #FTW
1 UI
ksql>
2 CLI
POST /query
3 REST 4 Headless

82. @gamussa #KafkaSummit @confluentinc
Interaction with Kafka
Kafka
(data)

83. @gamussa #KafkaSummit @confluentinc
Interaction with Kafka
Kafka
(data)
KSQL
(processing)
JVM application
with Kafka Streams (processing)
Does not run on
Kafka brokers
Does not run on
Kafka brokers

84. @gamussa #KafkaSummit @confluentinc
Interaction with Kafka
Kafka
(data)
KSQL
(processing)
JVM application
with Kafka Streams (processing)
Does not run on
Kafka brokers
Does not run on
Kafka brokers

85. @gamussa #KafkaSummit @confluentinc
Fault-Tolerance, powered by Kafka

86. @gamussa #KafkaSummit @confluentinc
Fault-Tolerance, powered by Kafka

87. @gamussa #KafkaSummit @confluentinc
Fault-Tolerance, powered by Kafka

88. @gamussa #KafkaSummit @confluentinc
Standing on the shoulders of Streaming Giants

89. @gamussa #KafkaSummit @confluentinc
Standing on the shoulders of Streaming Giants

90. @gamussa #KafkaSummit @confluentinc
Standing on the shoulders of Streaming Giants
Producer,
Consumer APIs

91. @gamussa #KafkaSummit @confluentinc
Standing on the shoulders of Streaming Giants
Producer,
Consumer APIs
Kafka Streams

92. @gamussa #KafkaSummit @confluentinc
Standing on the shoulders of Streaming Giants
Producer,
Consumer APIs
Kafka Streams
Powered by

93. @gamussa #KafkaSummit @confluentinc
Standing on the shoulders of Streaming Giants
Producer,
Consumer APIs
Kafka Streams
KSQL
KSQL UDFs
Powered by

94. @gamussa #KafkaSummit @confluentinc
Standing on the shoulders of Streaming Giants
Producer,
Consumer APIs
Kafka Streams
KSQL
KSQL UDFs
Powered by
Powered by

95. @gamussa #KafkaSummit @confluentinc
Standing on the shoulders of Streaming Giants
Producer,
Consumer APIs
Kafka Streams
KSQL
Ease of use
Flexibility
KSQL UDFs
Powered by
Powered by

96. @@gamussa #KafkaSummit @confluentinc
Thanks!
@gamussa
viktor@confluent.io
We are hiring!
https://www.confluent.io/careers/