Kafka & Hadoop - for NYC Kafka Meetup
•Download as PPTX, PDF•
18 likes•11,954 views
The document discusses various options for getting data from Kafka into Hadoop, including Camus, Flume, Spark Streaming, and Storm. It provides information on how each works and their advantages and disadvantages. The presenter has 15 years of experience moving data and is now a Cloudera engineer working on projects like Flume, Sqoop, and Kafka.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Kafka & Hadoop - for NYC Kafka Meetup
Similar to Kafka & Hadoop - for NYC Kafka Meetup (20)
More from Gwen (Chen) Shapira
More from Gwen (Chen) Shapira (20)
Recently uploaded
Recently uploaded (20)
Kafka & Hadoop - for NYC Kafka Meetup
- 1. Kafka & Hadoop Gwen Shapira / Software Engineer
- 2. About Me • 15 years of moving data around • Formerly consultant • Now Cloudera Engineer: ©2014 Cloudera, Inc. All rights reserved. 2 – Flume – Sqoop – Kafka
- 3. There’s a book on that! ©2014 Cloudera, Inc. All rights reserved. 3
- 4. We are also blogging ©2014 Cloudera, Inc. All rights reserved. 4
- 5. 5 Getting Data from Kafka to Hadoop There are only bad options. It's about finding the best one. ©2014 Cloudera, Inc. All rights reserved.
- 6. ©2014 Cloudera, Inc. All rights reserved. 6 Camus
- 7. ©2014 Cloudera, Inc. All rights reserved. 7 Camus Setup ZooKeeper Topic Offsets Other Systems HDFS Processes Task Task Task In process Avro Files In process Avro Files Audit Counts Clean Up Kakfa B A C D F G H I E
- 8. Missing in Action • Kafka has no MR layer – InputFormat, OutputFormat, Utils… • Sqoop is a generic batch ingest framework ©2014 Cloudera, Inc. All rights reserved. 8 – Why no Kafka?
- 9. Flume + Kafka = Flafka ©2014 Cloudera, Inc. All rights reserved. 9
- 10. 10 How does work? Sources Interceptors Selectors Channels Sinks Flume Agent Twitter, logs, webserver, Kafka… Mask, re-format, validate… DR, critical Memory, file HDFS, Hbase, Solr, Kafka
- 11. 11 But I just want to get data from Kafka to Hbase / HDFS ©2014 Cloudera, Inc. All rights reserved.
- 12. 12 Channels Sinks Kafka Channel Flume Agent Kafka! HDFS, Hbase, Solr
- 13. SparkStreaming Single Pass ©2014 Cloudera, Inc. All rights reserved. 13 Source RawInput DStream RDD Source RawInput DStream RDD RDD Filter Count Print Source RawInput DStream RDD RDD RDD Single Pass Filter Count Print Pre-first Batch First Batch Second Batch
- 14. ©2014 Cloudera, Inc. All rights reserved. 14 Storm Spout Source Split words bolts Split words bolts Spout Split words bolts Split words bolts Count Count Count Spout Layer Fan out Layer 1 Shuffle Layer 2
- 15. Retro Thoughts ©2014 Cloudera, Inc. All rights reserved. 15
- 16. • Data often has schema • At least it should • Kafka is unaware – which is good • Need capability to figure out schema for events • Without including it in every event ©2014 Cloudera, Inc. All rights reserved. 16 Schema
- 17. Kafka in Cloudera Manager ©2014 Cloudera, Inc. All rights reserved. 17
- 18. 18 Visit us at Booth #305 BOOK SIGNINGS THEATER SESSIONS TECHNICAL DEMOS GIVEAWAYS
Editor's Notes
- This gives me a lot of perspective regarding the use of Hadoop
- https://gist.github.com/gwenshap/9699072
- Batch MapReduce job. Exactly once semantics. Run once every X minutes.
- A - The setup stage fetches broker urls and topic information from ZooKeeper. B - The setup stage persists information about topics and offsets in HDFS for the tasks to read. C - The tasks read the persisted information from the setup stage. D - The tasks get events from Kakfa. E - The tasks write data to a temp location in HDFS in the format defined by the user defined decoder, in this case Avro formatted files. F - The tasks move the data in the temp location to a final location when the task is cleaning up. G - The task writes out audit counts on its activities. H - A clean up stage reads all the audit counts from all the tasks. I - The clean up stage reports back to Kakfa what has been persisted.
- Kafka source + sink for Flume
- Does not require programming.
- Does not require programming.
- MicroBatch stream processing framework. Basically Spark code executed in a slightly different context and some windowing functions.
- Stream processing framework. Quite popular. Can be event-based or micro-batching (with Trident). Requires low level awareness of API.