- Kafka는 왜 만들어졌을까? - Kafka ecosystem

1. gshock94@gmail.com
Kafka,

2. Kafka?
source : LinkedIn, Quora
※ Kafka : WAL (write-ahead logging) system

3. Kafka?
source : Confluent
(2017 12 )

4. Part1. Kafka ?

5. Kafka LinkedIn data pipeline
source : Confluent

6. Kafka LinkedIn data pipeline
source : Confluent
.
?

7. …Our approach was very ad hoc: we built jerry-rigged piping between systems
and applications on an as needed basis and shoe-horned any asynchronous
processing into request-response web services. Over time this set-up got more
and more complex as we ended up building pipelines between all kinds of
different systems:
source : Confluent

8. …Our approach was very ad hoc: we built jerry-rigged piping between systems
and applications on an as needed basis and shoe-horned any asynchronous
processing into request-response web services. Over time this set-up got more
and more complex as we ended up building pipelines between all kinds of
different systems:
source : Confluent

9. 1. pipeline
2. pipeline scaling
3.
4. user activity
⋮

10. 1. pipeline
2. pipeline scaling
3.
4. user activity
⋮
( )
streaming data

11. source : Confluent
“Kafka stuff” “the global commit log thingy .

12. Kafka
LinkedIn system architecture

13. source : Confluent

14. source : Confluent
Kafka
central hub of data streams

15. Part2. Kafka ecosystem

16. Kafka+@ ,
services
ecosystem

17. • storage : Kafka broker
• publish & subscribe : Kafka producer, Kafka consumer, Kafka connect
• stream processing : Kafka streams, KSQL
• else : Confluent schema registry, Confluent REST proxy

18. • storage : Kafka broker
• publish & subscribe : Kafka producer, Kafka consumer, Kafka connect
• stream processing : Kafka streams, KSQL
• else : Confluent schema registry, Confluent REST proxy

19. source : Confluent

20. source : Confluent
service

21. • Kafka application server/node/machine
• cluster Kafka cluster
• Kafka disk
storage : Kafka broker

22. • Kafka Producer : Kafka topic application
• Kafka Consumer : Kafka topic application
• Kafka Connect : database Kafka framework
publish & subscribe : Kafka Producer, Kafka Consumer, Kafka Connect

23. • Kafka Streams : Kafka client library (DSL Processor API )
• KSQL : Kafka streaming SQL engine (SQL )
stream processing : Kafka Streams, KSQL

24. • REST Proxy : Kafka client Kafka cluster RESTful
• Schema Registry : Kafka topic Avro schema / RESTful
Confluent schema registry, Confluent REST proxy

26. source : Confluent

27. https://www.facebook.com/groups/kafkakorea
https://github.com/kafka-kru