Downloaded 60 times
Uploaded byAshish Singh
Real time analytics with Kafka and SparkStreaming
The document discusses the significance of real-time analytics using Kafka and Spark Streaming, highlighting its emergence due to the rapid increase in continuous data streams and the availability of open-source tools. It provides examples from various industries, demonstrating applications like fraud detection, real-time traffic management, and personalized digital marketing. It also outlines the canonical stream processing architecture and introduces an example application called 'Pankh' to showcase real-time analytics capabilities.
More Related Content
Similar to Real time analytics with Kafka and SparkStreaming
![ANPARA THERMAL POWER STATION[1] sangam.pdf](https://cdn.slidesharecdn.com/ss_thumbnails/anparathermalpowerstation1sangam-251121115219-9261cde4-thumbnail.jpg?width=640&height=640&fit=bounds)
Real time analytics with Kafka and SparkStreaming
- 1. 1© Cloudera, Inc.All rights reserved. Real-time analytics with Kafka and Spark Streaming Ashish Singh | Software Engineer, Cloudera
- 2. 2© Cloudera, Inc.All rights reserved. It’s Real-Time time Why now? Complex Event Processing (CEP) is not a new concept.
- 3. 3© Cloudera, Inc.All rights reserved. It’s Real-Time time Emergence of Real-Time Stream Processing Exponential growth in continuous data streams Open Source tools for reliable high-throughput low latency event queuing and processing Tools run on “Commodity” Hardware Why now? Complex Event Processing (CEP) is not a new concept.
- 4. 4© Cloudera, Inc.All rights reserved. It’s happening! …Across Industries Credit Card & Monetary Transactions Identify fraudulent transactions as soon as they occur. Transportation & Logistics • Real-time traffic conditions • Tracking fleet and cargo locations and dynamic re-routing to meet SLAs Retail • Real-time in-store Offers and Recommendations. • Email and marketing campaigns based on real- time social trends Consumer Internet, Mobile & E-Commerce Optimize user engagement based on user’s current behavior. Deliver recommendations relevant “in the moment” Healthcare Continuously monitor patient vital stats and proactively identify at-risk patients. Manufacturing • Identify equipment failures and react instantly • Perform proactive maintenance. • Identify product quality defects immediately to prevent resource wastage. Security & Surveillance Identify threats and intrusions, both digital and physical, in real- time. Digital Advertising & Marketing Optimize and personalize digital ads based on real-time information.
- 5. 5© Cloudera, Inc.All rights reserved. Canonical Stream Processing Architecture Data Sources
- 6. 6© Cloudera, Inc.All rights reserved. Canonical Stream Processing Architecture Data Sources Kafka Flume
- 7. 7© Cloudera, Inc.All rights reserved. Canonical Stream Processing Architecture Data Sources Kafka Flume Filter Enrich Transform Stats on Sliding Windows Stream Joins Feature Engineering Predictive Analytics Active Model Training . . . . And combinations of the above
- 8. 8© Cloudera, Inc.All rights reserved. Canonical Stream Processing Architecture Data Sources Kafka Flume HDFS
- 9. 9© Cloudera, Inc.All rights reserved. Canonical Stream Processing Architecture Data Sources Kafka Flume NoSql HDFS
- 10. 10© Cloudera, Inc.All rights reserved. Canonical Stream Processing Architecture Data Sources Kafka Flume NoSql HDFS
- 11. 11© Cloudera, Inc.All rights reserved. Canonical Stream Processing Architecture Data Sources Kafka Flume NoSql HDFS
- 12. 12© Cloudera, Inc.All rights reserved. Canonical Stream Processing Architecture Data Sources Kafka Flume NoSql HDFS
- 13. 13© Cloudera, Inc.All rights reserved. Canonical Stream Processing Architecture Data Sources Kafka Flume NoSql HDFS Kafka
- 14. 14© Cloudera, Inc.All rights reserved. Canonical Stream Processing Architecture Data Sources Kafka Flume NoSql HDFS Kafka . . .
- 15. 15© Cloudera, Inc.All rights reserved. Too much? 15© Cloudera, Inc. All rights reserved.
- 16. 16© Cloudera, Inc.All rights reserved. Example application to demonstrate how real time analytics can be done using Kafka and Spark Streaming Pankh https://github.com/SinghAsDev/pankh
- 17. 17© Cloudera, Inc.All rights reserved. Pankh – Building Pieces Data Sources
- 18. 18© Cloudera, Inc.All rights reserved. Pankh – Building Pieces Data Sources Kafka
- 19. 19© Cloudera, Inc.All rights reserved. Pankh – Building Pieces Data Sources Kafka
- 20. 20© Cloudera, Inc.All rights reserved. Pankh – Building Pieces Data Sources Kafka NoSql
- 21. 21© Cloudera, Inc.All rights reserved. Pankh – Building Pieces Data Sources Kafka NoSql
- 22. 22© Cloudera, Inc.All rights reserved. Demo Time 22© Cloudera, Inc. All rights reserved.
- 23. 25© Cloudera, Inc.All rights reserved. Kappa Architecture
- 24. 26© Cloudera, Inc.All rights reserved. Demo Time 26© Cloudera, Inc. All rights reserved.
- 25. 27© Cloudera, Inc.All rights reserved. Thank you Ashish Singh asingh@cloudera.com @singhasdev
Editor's Notes
- #2 Canonical architecture of real-time stream processing
- #3 It is easy to get carried away, because real-time sounds cool. But it is totally fair if you are skeptical about the business value of real time stream processing. After all, this is not a new concept. There have been tools provided by traditional enterprise software vendors to do real-time data processing. They were formerly known as “Complex Event Processing” or CEP systems. In fact, even now the traditional vendors refer to their real-time stream processing tools as CEP systems. If CEP systems have been around, but never really took off, are we just seeing the
- #6 Lets look at an end to end architecture of putting together open source tools to do real time stream processing. Lets start with the sources of data.
- #7 You want to write this data to a reliable high-throughput low latency messaging system, Kafka and Flume are popular choices, but there are many options out there, like ActiveMQ, RabbitMQ,etc. Kafka is the system that is gaining the most popularity right now. ====== With this architecture, the real-time processed data only gets leveraged when the next application query comes in. But often you want to take some action based on the real-time analysis of your data. For proactive actions, write relevant events out to Kafka. Again, based on yoru stream processign engine you will find libraries that make this easy. You can have an application that is continusouly listeing on your event queue, and can issues alerts, emails, etc
- #8 A stream processing system like Spark Streaming can then read your data streams from the messaging system. Filter Enrich or embellish your data with relevant metadata Transform Compute statistics based on moving windows of time Feature Engineering + Predictive Analytics … and much more
- #9 Almost always, you want to take your full fidelity raw data, and put it in HDFS, or an object store if your are running in the cloud. The raw data can then be used in batch jobs where you may want to do deep complex processing that can not be done in a streaming fashion. Or you may have a team of data scientists who may want to explore the data and uncover new insights. Why the dotted line: how you dump your data to HDFS depends on your messaging system. Almost all messaging systems will provide a way to transfer your data to HDFS
- #10 All this real-time processing is great, but not very useful if you can not serve the processed data to your application in real-time. Your need a system that can enable a lot of fast reads and writes. That is where NoSql stores come in. There are many choices here. Hbase, Cassnadra and MongoDb are popular choices. All those end applications Also, for most stream procsssing engine and NoSql store pairs, there are libraries available that make it easy to read from or write to your NoSql store from the stream processing engine: for example, the SparkOnHbase library makes it easy to write to Hbase from spark streamign jobs.
- #11 Another common scenario is indexing your data, in real-time, into a search system. This is great if the data your are dealing with is textual data. There are libararies that enable real-time indexing of your data in your stream proocessing engine, and writing it to a Search Engine.
- #12 Now the data is ready to be queried by your application. This is a very common and popular architecture, and I am guessing this is in keeping with what most of you would have expected.
- #13 Again, write your processed output to HDFS. Again, why the dotter arrow. Weather or not you need to dump data to HDFS depends upon your serving system of choice. If you write it to Hbase, you may not need to duplicate it in HDFS. But if you are indexing the data in search or writing to a system like Redis, you may want to also write the processed otuptut to HDFS. Why? If nothing else, for auditing purposes. Errors will happen. And you may need to go back and audit what was done in your stream processing engine. Hence, put the data in hdfs and keep it there are some amount of time.
- #14 With this architecture, the real-time processed data only gets leveraged when the next application query comes in. But often you want to take some action based on the real-time analysis of your data. For proactive actions, write relevant events out to Kafka. Again, based on yoru stream processign engine you will find libraries that make this easy. You can have an application that is continusouly listeing on your event queue, and can issues alerts, emails, etc
- #15 By writing it to a message queue, you enable multiple downstream applications to consume the data as its produced, including enabling furthur processing of your data with a stream processing engine. Such multi-stage architectures, where you cosnume from say Kafka, process the data, produce a new stream in Kafka, and process
- #26 For moment like these, streaming systems provide the capability to rewind, at least they should.