introduction-to-apache-kafka
•
6 likes•3,308 views
This document provides an introduction to Apache Kafka. It begins with an overview of Kafka as a distributed messaging system that is real-time, scalable, low latency, and fault tolerant. It then covers key concepts such as topics, partitions, producers, consumers, and replication. The document explains how Kafka achieves fast reads and writes through its design and use of disk flushing and replication for durability. It also discusses how Kafka can be used to build real-time systems and provides examples like connected cars. Finally, it introduces Apache Metron as an example of a cyber security solution built on Kafka.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to introduction-to-apache-kafka
Similar to introduction-to-apache-kafka (20)
More from Yifeng Jiang
More from Yifeng Jiang (20)
Recently uploaded
Recently uploaded (20)
introduction-to-apache-kafka
- 1. 1 © Hortonworks Inc. 2011 – 2017 All Rights Reserved1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved はじめよう！ Apache Kafkaでリアルタイムデータ処理 Yifeng Jiang Solutions Engineering Lead September 6, 2017
- 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved About Me à 蒋 逸峰 (しょう いつほう / Yifeng Jiang) à Solutions Engineering Lead, NAPAC, Hortonworks – Hadooper since 2009 – HBase book author – Software engineer, cloud, PaaS, DevOps à Jogger, hiker à Twitter: @uprush
- 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved DATA AT REST DATA IN MOTION ACTIONABLE INTELLIGENCE Modern IoT Data Applications PERISHABLE INSIGHTS HISTORICAL INSIGHTS INTERNET OF ANYTHING Hortonworks DataFlow Hortonworks Data Platform Hortonworks Delivers Connected Data Platforms
- 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache Kafka à Distributed messaging systems – Real-time – Scalable to handle large data volume – Low Latency – Fault tolerant à Originated at LinkedIn – Aimed at solving data movement across systems – Scala and Java – Open Source (Apache 2.0) – Adapted at many companies
- 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Kafka: Anatomy of a Topic Partition 0 Partition 1 Partition 2 0 0 0 1 1 1 2 2 2 3 3 3 4 4 4 5 5 5 6 6 6 7 7 7 8 8 8 9 9 9 10 10 11 11 12 Writes Old New à Messages (logs) are stored on broker’s local disk à Messages are appended to log file à Log Retention – time and size based
- 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Kafka Replication à Partition has replicas – Leader replica, Follower replicas à Replicas are distributed to multiple brokers à Leader maintains in-sync-replicas (ISR) https://www.slideshare.net/junrao/kafka-replication-apachecon2013
- 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Kafka Producer • Create a new message and publish to a Topic and Partition • Original messages are partitioned and then split into batches • Each split batch is sent to leader broker (and then replicated to ISR) • Each send is acknowledged by either leader broker and/or all ISR p3 p2 p1 p2 p1m5 m4 m3 m2 m1 Broker-0 P0.R0 (L) P1.R0 Broker-1 P0.R1 P2.R1 (L) Broker-2 P1.R2 (L) P2.R2 Topic with 3 partition and Replica factor 2 App Producer Lib partitioner Split batch
- 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Kafka Consumer à Consumers pull data from brokers à Consumer apps have to keep track of the topic-partition offset read à Consumer Groups: Allow multiple hosts to form a group to access a topic – Max parallelism – determined by topic partitions Broker-0 P3 Broker-1 P1 P2 C1 C2 Consumer Group - 1 C3 C4 Consumer Group - 2 C5 C6 P0
- 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Kafka – Why Kafka is fast Fast Writes Writes are appends to file system Partitions improve performance and throughput Uses OS buffer cache Lots of memory on the machine helps Fast Reads Hot data sits in memory, most time data is served without disk I/O File descriptor to socket descriptor efficient transfer Linux sendfile(), JVM transferTo() implementation Why Performance? Disk flushes are delayed Durability is guaranteed via replication When consumers are reading the latest data, it reads from page cache
- 16. 1 6 © Hortonworks Inc. 2011 – 2017 All Rights Reserved Real-time System Building Blocks • Streams – An unbounded sequence of messages, events, information packets or tuples (named list of values) • Data Pipe – Message/Information bus – Decouple publishers (providers) and consumers (subscribers) – Scalability, Centralized, Distributed • Stream Processing – Semantics (operations and processing primitives) – Stateless or with state • Low Latency Storage – NoSQL database
- 17. 1 7 © Hortonworks Inc. 2011 – 2017 All Rights Reserved Real-time System Building Blocks • Streams -- Nifi, Fluentd, etc. – An unbounded sequence of messages, events, information packets or tuples (named list of values) • Data Pipe – Message/Information bus -- Kafka – Decouple publishers (providers) and consumers (subscribers) – Scalability, Centralized, Distributed • Stream Processing -- Storm, Spark Streaming, etc. – Semantics (operations and processing primitives) – Stateless or with state • Low Latency Storage -- HBase, Redis, Druid, etc. – NoSQL database
- 18. 1 8 © Hortonworks Inc. 2011 – 2017 All Rights Reserved Apache Metron: Real-time Big Data Cyber Security powered By Kafka
- 20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Metron Architecture – Real-time System Built on Kafka Parsers Kafka enrichments topic Kafka indexing topic
- 21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Metron Architecture – Real-time System Built on Kafka Kafka indexing topic Metron UI and Dashbaords
- 22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Understood, but that looks difficult…
- 28. 28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Build Your First Streaming Analytics App in Under 30 Minutes
- 29. 2 9 © Hortonworks Inc. 2011 – 2017 All Rights Reserved Trucking company w/ large fleet of international trucks A truck generates millions of events for a given route; an event could be: § 'Normal' events: starting / stopping of the vehicle § ‘Violation’ events: speeding, excessive acceleration and breaking, unsafe tail distance § ‘Speed’ Events: The speed of a driver that comes in every minute. Company uses an application that monitors truck locations and violations from the truck/driver in real- time Route? Truck? Driver? Analysts query a broad history to understand if today’s violations are part of a larger problem with specific routes, trucks, or drivers
- 32. 32 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Question?
- 33. Data PlatformConference Tokyo 2017ビッグデータ x IoT / クラウド / AI（人工知能）を利用した データ駆動型ビジネスの本格的な実現に向けて 2017年10月10日開催 主催：株式会社インプレス 共催：ホートンワークスジャパン株式会社 申し込み・詳細 dataplatform.jp
- 34. 3 4 © Hortonworks Inc. 2011 – 2017 All Rights Reserved THANK YOU Yifeng Jiang @uprush