Apresentação sobre os conceitos básicos envolvendo o sistema de stream distribuído apache kafka.

2. About.me
 Desenvolvedor de software +10 anos
 Ilegra desde novembro / 2016
 Arquitetura Unicred
 Projeto core banking
 Estudado:
 Microservices
 Reactive services

3. About
 History
 Motivation
 What is?
 Design concepts
 Use case

4. History

5. Motivation
 Complex infrastructure
 Different tools
 Performance issues
 Governance
 Persistent

6. Motivation

7. Motivation

8. Motivation

9. Motivation

10. What is?

11. What is?

12. What is?
 Distributed streaming platform
 You can produce and consume messages similar to conventional message systems.
 Store data - fault tolerant
 Process events in order

13. High level architecture

14. Log
 Persistent sequence messages
 Newest messages are append to the log
 Similar to database transaction log
 State recovery
 Sequence of actions / events
 Debug
 Audit

15. Topic
 Logical division
 Subject
 Unit of coupling
 Can be split into partitions

16. Partition

17. Partition
 Fisic unit
 Each partition are a log
 Producers and consumers connect
 Split into brokers

18. Partition

19. Producer
 Write messages to partitions
 Round Robbin - Balanced
 Semantic Strategy - Hot Spot attention
 Messages can be compressed
 Sync
 Send and wait for response.
 Exception will be treat and resend manually
 Async
 Register a callback

20. Producer
 Several configs
 acks = 0
■ Very fast.
■ Without consistent guarantee
 acks = 1
■ Wait the response of the leader
■ Sync can increase the latency
 acks = all
■ Wait for all replication process

21. Consumer
 Consume messages
 Control the offset
 Commit the offset
 Have a consumer group

22. Consumer

23. Consumer

24. Consumer

25. Consumer

26. Consumer

27. Summary
 Topic can have partitions
 Each partition is a log
 Producers send messages
 Consumers pull messages and control the offset
 Each consumer group receive just one message per partition
 Number of partitions is the unit of parallel processing

28.  Use to connect data systems to kafka
 Can override common ETL systems
 Available connectors:
 Amazon S3
 JDBC, MYSQL
 HANA
 Cassandra
 Elastic Search
 FTP
Connector

29. Who are using?

30. Who are using?
 ⅓ Fortune 500
 https://kafka.apache.org/powered-by
 Linkedin
 Netflix
 Yahoo
 Twitter
 Goldman Sachs

31. Other features
 Stream
 Aggregation
 Generate new topics
 Complex data pipelines
 Dynamic data processor
 KSQL
 Real time streaming with sql language

32. Other features - Stream

33. Other features - Stream

34. Other features - Stream

35. Use cases - Data replication
App1 App2

36. Use cases - Messaging
App1 App2

37. Use cases - Messaging
App1 App2
App3

38. Use cases - Messaging
App1 App2
App3

39. User activity
Analysis / Dashboard

40. ETL
BI

41. Events, Stream, Client Join
MS1
MS2

42. Events, Stream, Client Join

43. Questions