Interactive real time dashboards on data streams using Kafka, Druid, and Superset
•Download as PPTX, PDF•
15 likes•9,658 views
When interacting with analytics dashboards, in order to achieve a smooth user experience, two major key requirements are quick response time and data freshness. To meet the requirements of creating fast interactive BI dashboards over streaming data, organizations often struggle with selecting a proper serving layer. Cluster computing frameworks such as Hadoop or Spark work well for storing large volumes of data, although they are not optimized for making it available for queries in real time. Long query latencies also make these systems suboptimal choices for powering interactive dashboards and BI use cases. This talk presents an open source real time data analytics stack using Apache Kafka, Druid, and Superset. The stack combines the low-latency streaming and processing capabilities of Kafka with Druid, which enables immediate exploration and provides low-latency queries over the ingested data streams. Superset provides the visualization and dashboarding that integrates nicely with Druid. In this talk we will discuss why this architecture is well suited to interactive applications over streaming data, present an end-to-end demo of complete stack, discuss its key features, and discuss performance characteristics from real-world use cases. Speaker Nishant Bangarwa, Software Engineer, Hortonworks
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Interactive real time dashboards on data streams using Kafka, Druid, and Superset
Similar to Interactive real time dashboards on data streams using Kafka, Druid, and Superset (20)
More from DataWorks Summit
More from DataWorks Summit (20)
Recently uploaded
Recently uploaded (20)
Interactive real time dashboards on data streams using Kafka, Druid, and Superset
- 1. 1 © Hortonworks Inc. 2011–2018. All rights reserved Interactive Realtime Dashboards on Data Streams Nishant Bangarwa Hortonworks Druid Committer, PMC
- 2. 2 © Hortonworks Inc. 2011–2018. All rights reserved Sample Data Stream : Wikipedia Edits
- 3. 3 © Hortonworks Inc. 2011–2018. All rights reserved Step by Step Breakdown Consume Events Enrich / Transform (Add Geolocation from IP Address) Store Events Visualize Events Sample Event : [[Eoghan Harris]] https://en.wikipedia.org/w/index.php?diff=792474242&oldid=787592607 * 7.114.169.238 * (+167) Added fact
- 4. 4 © Hortonworks Inc. 2011–2018. All rights reserved Required Components Event Flow Event Processing Data Store Visualization Layer
- 5. 5 © Hortonworks Inc. 2011–2018. All rights reserved Event Flow
- 6. 6 © Hortonworks Inc. 2011–2018. All rights reserved Event Flow : Requirements Event Producers Queue Event Consumers Low latency High Throughput Failure Handling Message delivery guarantees – Atleast Once, Exactly once, Atmost Once Varies based on use-case, Exactly once being the holy grail and the most difficult to achieve Scalability
- 7. 7 © Hortonworks Inc. 2011–2018. All rights reserved Apache Kafka
- 8. 8 © Hortonworks Inc. 2011–2018. All rights reserved Apache Kafka Low Latency High Throughput Message Delivery guarantees At-least once Exactly Once (Fully introduced in apache kafka v0.11.0 June 2017) Reliable design to Handle Failures Message Acks between producers and brokers Data Replication on brokers Consumers can Read from any desired offset Handle multiple producers/consumers Scalable
- 9. 9 © Hortonworks Inc. 2011–2018. All rights reserved Event Processing
- 10. 10 © Hortonworks Inc. 2011–2018. All rights reserved Event Processing : Requirements Consume-Process-Produce Pattern Enrich and Transform event streams Windowing Apply business logic Consume and Join multiple streams into single Failure Handling Scalability Consume Process Produce
- 11. 11 © Hortonworks Inc. 2011–2018. All rights reserved Kafka Streams Rich Lightweight Stream processing library Event-at-a-time Stateful processing : windowing, joining, aggregation operators Local state using RocksDb Backed by changelog in kafka Highly scalable, distributed, fault tolerant Compared to a standard Kafka consumer: Higher level: faster to build a sophisticated app Less control for very fine-grained consumption
- 12. 12 © Hortonworks Inc. 2011–2018. All rights reserved Kafka Streams : Wikipedia Data Enrichment
- 13. © Hortonworks Inc. 2011 – 2016. All Rights Reserved 1 Data Store
- 14. 14 © Hortonworks Inc. 2011–2018. All rights reserved Data Store
- 15. 15 © Hortonworks Inc. 2011–2018. All rights reserved Data Store : Requirements Processed Events Data Store Queries Ability to ingest Streaming data Power Interactive dashboards Sub-Second Query Response time Ad-hoc arbitrary slicing and dicing of data Data Freshness Summarized/aggregated data is queried Scalability High Availability
- 16. 16 © Hortonworks Inc. 2011–2018. All rights reserved 1 Druid • Column-oriented distributed datastore • Sub-Second query times • Realtime streaming ingestion • Arbitrary slicing and dicing of data • Automatic Data Summarization • Approximate algorithms (hyperLogLog, theta) • Scalable to petabytes of data • Highly available
- 17. 17 © Hortonworks Inc. 2011–2018. All rights reserved Suitable Use Cases • Powering Interactive user facing applications • Arbitrary slicing and dicing of large datasets • User behavior analysis • measuring distinct counts • retention analysis • funnel analysis • A/B testing • Exploratory analytics/root cause analysis • Not interested in dumping entire dataset
- 18. 18 © Hortonworks Inc. 2011–2018. All rights reserved Druid: Segments • Data in Druid is stored in Segment Files. • Partitioned by time • Ideally, segment files are each smaller than 1GB. • If files are large, smaller time partitions are needed. Time Segment 1: Monday Segment 2: Tuesday Segment 3: Wednesday Segment 4: Thursday Segment 5_2: Friday Segment 5_1: Friday
- 19. 19 © Hortonworks Inc. 2011–2018. All rights reserved 1 Example Wikipedia Edit Dataset timestamp page language city country … added deleted 2011-01-01T00:01:35Z Justin Bieber en SF USA 10 65 2011-01-01T00:03:63Z Justin Bieber en SF USA 15 62 2011-01-01T00:04:51Z Justin Bieber en SF USA 32 45 2011-01-01T00:05:35Z Ke$ha en Calgary CA 17 87 2011-01-01T00:06:41Z Ke$ha en Calgary CA 43 99 2011-01-02T00:08:35Z Selena Gomes en Calgary CA 12 53 Timestamp Dimensions Metrics
- 20. 20 © Hortonworks Inc. 2011–2018. All rights reserved 2 Data Rollup timestamp page language city country … added deleted 2011-01-01T00:01:35Z Justin Bieber en SF USA 10 65 2011-01-01T00:03:63Z Justin Bieber en SF USA 15 62 2011-01-01T00:04:51Z Justin Bieber en SF USA 32 45 2011-01-01T00:05:35Z Ke$ha en Calgary CA 17 87 2011-01-01T00:06:41Z Ke$ha en Calgary CA 43 99 2011-01-02T00:08:35Z Selena Gomes en Calgary CA 12 53 timestamp page language city country count sum_added sum_deleted min_added max_added …. 2011-01-01T00:00:00Z Justin Bieber en SF USA 3 57 172 10 32 2011-01-01T00:00:00Z Ke$ha en Calgary CA 2 60 186 17 43 2011-01-02T00:00:00Z Selena Gomes en Calgary CA 1 12 53 12 12 Rollup by hour
- 21. 21 © Hortonworks Inc. 2011–2018. All rights reserved 2 Dictionary Encoding • Create and store Ids for each value • e.g. page column ⬢ Values - Justin Bieber, Ke$ha, Selena Gomes ⬢ Encoding - Justin Bieber : 0, Ke$ha: 1, Selena Gomes: 2 ⬢ Column Data - [0 0 0 1 1 2] • city column - [0 0 0 1 1 1] timestamp page language city country … added deleted 2011-01-01T00:01:35Z Justin Bieber en SF USA 10 65 2011-01-01T00:03:63Z Justin Bieber en SF USA 15 62 2011-01-01T00:04:51Z Justin Bieber en SF USA 32 45 2011-01-01T00:05:35Z Ke$ha en Calgary CA 17 87 2011-01-01T00:06:41Z Ke$ha en Calgary CA 43 99 2011-01-02T00:08:35Z Selena Gomes en Calgary CA 12 53
- 22. 22 © Hortonworks Inc. 2011–2018. All rights reserved 2 Bitmap Indices • Store Bitmap Indices for each value ⬢ Justin Bieber -> [0, 1, 2] -> [1 1 1 0 0 0] ⬢ Ke$ha -> [3, 4] -> [0 0 0 1 1 0] ⬢ Selena Gomes -> [5] -> [0 0 0 0 0 1] • Queries ⬢ Justin Bieber or Ke$ha -> [1 1 1 0 0 0] OR [0 0 0 1 1 0] -> [1 1 1 1 1 0] ⬢ language = en and country = CA -> [1 1 1 1 1 1] AND [0 0 0 1 1 1] -> [0 0 0 1 1 1] • Indexes compressed with Concise or Roaring encoding timestamp page language city country … added deleted 2011-01-01T00:01:35Z Justin Bieber en SF USA 10 65 2011-01-01T00:03:63Z Justin Bieber en SF USA 15 62 2011-01-01T00:04:51Z Justin Bieber en SF USA 32 45 2011-01-01T00:01:35Z Ke$ha en Calgary CA 17 87 2011-01-01T00:01:35Z Ke$ha en Calgary CA 43 99 2011-01-01T00:01:35Z Selena Gomes en Calgary CA 12 53
- 23. 23 © Hortonworks Inc. 2011–2018. All rights reserved 2 Approximate Sketch Columns timestamp page userid language city country … added deleted 2011-01-01T00:01:35Z Justin Bieber user1111111 en SF USA 10 65 2011-01-01T00:03:63Z Justin Bieber user1111111 en SF USA 15 62 2011-01-01T00:04:51Z Justin Bieber user2222222 en SF USA 32 45 2011-01-01T00:05:35Z Ke$ha user3333333 en Calgary CA 17 87 2011-01-01T00:06:41Z Ke$ha user4444444 en Calgary CA 43 99 2011-01-02T00:08:35Z Selena Gomes user1111111 en Calgary CA 12 53 timestamp page language city country count sum_added sum_delete d min_added Userid_sket ch …. 2011-01-01T00:00:00Z Justin Bieber en SF USA 3 57 172 10 {sketch} 2011-01-01T00:00:00Z Ke$ha en Calgary CA 2 60 186 17 {sketch} 2011-01-02T00:00:00Z Selena Gomes en Calgary CA 1 12 53 12 {sketch} Rollup by hour
- 24. 24 © Hortonworks Inc. 2011–2018. All rights reserved Approximate Algorithms • Store Sketch objects, instead of raw column values • Better rollup for high cardinality columns e.g userid • Reduced storage size • Use Cases • Fast approximate distinct counts • Approximate histograms • Funnel/retention analysis • Limitation • Not possible to do exact counts • filter on individual row values
- 25. 25 © Hortonworks Inc. 2011–2018. All rights reserved Realtime Nodes Historical Nodes 2 Druid Architecture Batch Data Event Historical Nodes Broker Nodes Realtime Index Tasks Streaming Data Historical Nodes Handoff
- 26. 26 © Hortonworks Inc. 2011–2018. All rights reserved Performance and Scalability : Fast Facts Most Events per Day 300 Billion Events / Day (Metamarkets) Most Computed Metrics 1 Billion Metrics / Min (Jolata) Largest Cluster 200 Nodes (Metamarkets) Largest Hourly Ingestion 2TB per Hour (Netflix)
- 27. 27 © Hortonworks Inc. 2011–2018. All rights reserved 2 Companies Using Druid
- 28. 28 © Hortonworks Inc. 2011–2018. All rights reserved Visualization Layer
- 29. 29 © Hortonworks Inc. 2011–2018. All rights reserved Visualization Layer : Requirements • Rich dashboarding capabilities • Work with multiple datasoucres • Security/Access control • Allow for extension • Add custom visualizations Data Store Visualization Layer User Dashboards
- 30. 30 © Hortonworks Inc. 2011–2018. All rights reserved Superset • Python backend • Flask app builder • Authentication • Pandas for rich analytics • SqlAlchemy for SQL toolkit • Javascript frontend • React, NVD3 • Deep integration with Druid
- 31. 31 © Hortonworks Inc. 2011–2018. All rights reserved Superset Rich Dashboarding Capabilities: Treemaps
- 32. 32 © Hortonworks Inc. 2011–2018. All rights reserved Superset Rich Dashboarding Capabilities: Sunburst
- 33. 33 © Hortonworks Inc. 2011–2018. All rights reserved Superset UI Provides Powerful Visualizations Rich library of dashboard visualizations: Basic: • Bar Charts • Pie Charts • Line Charts Advanced: • Sankey Diagrams • Treemaps • Sunburst • Heatmaps And More!
- 34. 34 © Hortonworks Inc. 2011–2018. All rights reserved Wikipedia Real-Time Dashboard Kafka Connect IP-to- Geolocation Processor wikipedia-raw topic wikipedia-raw topic wikipedia-enriched topic wikipedia-enriched topic
- 35. 35 © Hortonworks Inc. 2011–2018. All rights reserved Demo: Wikipedia Real-Time Dashboard (Accelerated 30x)
- 36. 36 © Hortonworks Inc. 2011–2018. All rights reserved Project Websites Kafka - http://kafka.apache.org Druid - http://druid.io Superset - http://superset.incubator.apache.org
- 37. 37 © Hortonworks Inc. 2011–2018. All rights reserved Thank you Twitter - @NishantBangarwa Email - nbangarwa@hortonworks.com Linkedin - https://www.linkedin.com/in/nishant-bangarwa
- 38. 38 © Hortonworks Inc. 2011–2018. All rights reserved Questions?
- 39. © Hortonworks Inc. 2011 – 2016. All Rights Reserved 3 Thank you ! Questions ? • Twitter - @NishantBangarwa • Email - nbangarwa@hortonworks.com • Linkedin - https://www.linkedin.com/in/nishant-bangarwa
Editor's Notes
- In this talk we will discuss an end-to-end stack using open-source technologies and build a dashboard on top of streaming data. We will discuss the challenges involved and how each component in the stack addresses those challenges. As a sample problem, we will look at wikipedia editstream provided by wikipedia. Whenever any page is edited on wikipedia an edit event is generated which contains details about which page was edited,
- Lets try to break down this problem further A sample event from wikipedia editstream is formatted as follows – Title , URL of page edited, IP address of user, number of characters added/deleted First we would like to consume the events as coming from wikipedia Second we would like to enrich the event by doing an IP lookup and add the geolocation info about the user, and add more fields liks city, country from where the edit is being made. Third we would like to store these streaming events in a data store from where they can be queried and finally visualized on a dashboard.
- So to solve the wikipedia problem we need to have four components – First, A solution that can move events from one place to another in a reliable and guaranteed. Second, Event Processing layer which can process events and transform/enrich them. (Also termed as ETL) Third, Data Storage layer that can provide an sub-second queries on incoming data streams. Finally, A Visualization layer that allows creating of dashboards on top the data store, users can interact with the dashboards to gain insights out of the data.
- Producers produce the events in some message queue from where consumers fetch those events.
- In Apache Kafka, Each topic is divided into set of partitions, Ordering of events are guaranteed within one partition. Each producer can produce to multiple partitions, Each message in the queue is identifiable by an offset. Consumers consume messages from partitions sequentially and are also responsible for keeping track of their offsets. This also helps in minimizing the overhead.
- Local State – data is locally stored in RocksDb, for recovery each change to the local state is also propagated as an event in kafka. In case of failure or topology restart, Local State is restored from the changelog topic in kafka. The changelog topic is periodically compacted to reduce size.
- Druid Architecture
- Column-oriented distributed datastore – data is stored in columnar format, in general many datasets have a large number of dimensions e.g 100s or 1000s , but most of the time queries only need 5-10s of columns, the column oriented format helps druid in only scanning the required columns. Sub-Second query times – It utilizes various techniques like bitmap indexes to do fast filtering of data, uses memory mapped files to serve data from memory, data summarization and compression, query caching to do fast filtering of data and have very optimized algorithms for different query types. And is able to achievesub second query times Realtime streaming ingestion from almost any ETL pipeline. Arbitrary slicing and dicing of data – no need to create pre-canned drill downs Automatic Data Summarization – during ingestion it can summarize your data based, e.g If my dashboard only shows events aggregated by HOUR, we can optionally configure druid to do pre-aggregation at ingestion time. Approximate algorithms (hyperLogLog, theta) – for fast approximate answers Scalable to petabytes of data Highly available
- Retention analysis
- Druid: Segments Data in Druid is stored in Segment Files. Partitioned by time Ideally, segment files are each smaller than 1GB. If files are large, smaller time partitions are needed.
- Druid has concept of different nodes, where each node is designed and optimized to perform specific set of tasks. Realtime Index Tasks / Realtime Nodes- Handle Real-Time Ingestion, Support both pull & push based ingestion. Handle Queries - Ability to serve queries as soon as data is ingested. Store data in write optimized data structure on heap, periodically convert it to write optimized time partitioned immutable segments and persist it to deep storage. In case you need to do any ETL like data enrichment or joining multiple streams of data, you can do it in a separate ETL and send your massaged data to druid. Deep storage can be any distributed FS and acts as a permanent backup of data Historical Nodes - Main workhorses of druid cluster Use Memory Mapped files to load immutable segments Respond to User queries Now Lets see the how data can be queried. Broker Nodes - Keeps track of the data chunks being loaded by each node in the cluster Ability to scatter query across multiple Historical and Realtime nodes Caching Layer Now Lets discuss another case, when you are not having streaming data, but want to Ingest Batch data into druid Batch ingestion can be done using either Hadoop MR or spark job, which converts your data into time partitioned segments and persist it to deep storage.