The document discusses 3 realities of modern programming: 1) The rise of managed services where over half of Kafka users are using cloud versions. 2) Data is exploding in volume and streaming is needed. 3) Microservices have increased in popularity but communication between services can be complex; Kafka helps solve this as a backbone. Yelp moved to microservices and uses Kafka to connect over 70 services, saving $10M.

1. Stream it together
3 realities of modern programming
Neha Narkhede

2. 2
Kafka is core

3. 3
3 realities of
modern programming

4. 4
managed servicesmicroservicesdata explosion
1 1 1 10 0 0 10 1 10
1 1 1 11 0 0 10 1 01
0 1 1 00 0 0 10 1 00
0 0 1 11 1 0 11 1 10
1 1 0 1
1
1 0 0 10 1 01
1 1 1 00 0
0
0 1 1
0 0 11
0
0
0
0
0
01
1
0 11 1 00
0 1 1
1
1
1 1
1
10 0
0
01
0 10
0 1
1
1
1
0
0
1 1 0 0
1
0
1 11
1 0 0
11
1
0
00
0
0
1
1 0 10
0
0
0
0
0
0
0
1
1
1
1
1
0
1
0
0
0
1 10
1 0
0
11
1 0
0
0
0
0
1 0
0
1 11 11
0 1 1
11
1
1
1
0
0
01
1
1
1
1
0 0 0
0
1
1
11 1
1
1
1
0
0 0
0 1
1
0
0
0
0
0 0
0
1
0
0
11
11
1
1
11
1
1
1
1
1
0 0
0
1
01
1
1
1
0
0
1
0 0
1
0
1
0
0
11
1
0
00
0
1
1
0
0
0
0
0 0
0
1
1
1
1
0
1
0
0
0
10
0
1
1
1
0
0
0 0
0
0
0
1
1
11
1
1
1
1
1
0
0
01
1
1
1
1
0
0
0
111
1
00
1 1 0 11 0 0 10 1 11
1 0 1 10 0 1 01 1 10
0 1 0 10 1 0 10 1 11
1 0 0 11 0 1 10 0 10
1
1 1 0
0
0
0
1 10
0
0
0
0
0
01 0
0
11
1
1
1
1
1
1
1
1
0
0 01 0
1
1
1
1
0
0
1
0 0 1
0
1
0
0
11
1
0
0
0
0
1
1 0
0
0
0
1
1
1
1
0
1
0
0
0
1
0
0
1
11
00
0
0 0
0
0
00
11 1
1
11
1
1
1
1
0
0
0
1
1
1
1
1
1
0 0
0
111
1
1 1 1 10 0 0 10 1 10
1 1 1 11 0 0 10 1 01
0 1 1 00 0 0 10 1 00
0 0 1 11 1 0 11 1 10
1 1 0 11 0 0 10 1 01
1 1 1 00 0 0 10 0 11
0 0 0 01 1 0 11 1 00
0 1 1 10 0 0 01 0 10
0 1 1 01 1 0 10 1 11
1 0 1 00 0 0 11 0 10
0 0 0 11 1 0 10 1 10
1 0 1 01 0 0 11 1 11
0 1 1 10 0 0 11 1 00
1 1 0 11 0 0 10 1 11
1 0 1 10 0 1 01 1 10
0 1 0 10 1 0 10 1 11
1 0 0 11 0 1 10 0 10

5. 5

6. 6
c
ache
cache
cache

7. 7
all your data as a stream
Imagine…

8. 8

9. 9
Why microservices?

10. 10
The monolithic application

11. 11
Microservices are great…

12. 12
But communicating between services can get messy

13. 13
A stream fixes the madness

14. 14

15. 15
Over 1m lines of code in a single repo
Single monolithic app: yelp-main
2011
Move to micro services; Kafka is the backbone
70+ production services
2013
Saved $10m in R&D from 2013-today
2017 Reduced code complexity and downtime

16. 16
The rise of
Managed services

17. 17
of Kafka users are in the cloud
52%

18. Running distributed systems is really, really tough
18

19. 19
Coding
Can be pretty tricky

20. Twice as hard as writing the program
Debugging
20

21. 21
Where do you even start?
Operations

22. 22
sh!thappens

23. 23
Change is inevitable; systems evolve

24. 24
High throughput messaging broker

25. 25
Highly available replicated log

26. 26
Full streaming platform with Connect + Streams

27. 27
Exactly once streaming platform

28. 28
Exactly once streaming platform

29. 29
Exactly once streaming platform

30. 30
Exactly once streaming platform

31. 31
Exactly once streaming platform

32. 32
Exactly once streaming platform

33. 33
You can’t fix what you can’t see

34. 34
Observability is a dark art perfected
only with time and experience!

35. 35
Sh!t happens
Figure out observability
Keep up with change

36. 36

37. 37
Focus on building your ideas - not running Kafka

38. 38
Apache Kafka as a Service

39. 39
Launch
Signup in
Throughput
Retention 30 days
100 MB/s
3 seconds

40. 40
Not just Kafka, but its entire OSS ecosystem
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
...
Producer<String, String> producer = new KafkaProducer<>(props);
for(int i = 0; i < 100; i++)
producer.send(new ProducerRecord<String, String>("my-topic", Integer.toString(i), Integer.toString(i)));
producer.close();
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("foo", "bar"));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(100);
for (ConsumerRecord<String, String> record : records)
System.out.printf("offset = %d, key = %s, value = %s", record.offset(), record.key(), record.value());
}

41. 41
No lock-in; freedom to deploy Kafka in
any cloud

42. Kafka everywhere
On prem, in the cloud
42

43. 43
Have an amazing Kafka Summit!
http://confluent.io/confluent-cloud