Downloaded 200 times
![3 © Hortonworks Inc. 2011–2018. All rights reserved
Step by Step Breakdown
Consume Events
Enrich / Transform
(Add Geolocation
from IP Address)
Store Events
Visualize Events
Sample Event : [[Eoghan Harris]] https://en.wikipedia.org/w/index.php?diff=792474242&oldid=787592607 * 7.114.169.238 * (+167) Added fact](https://image.slidesharecdn.com/interactiverealtimedashboards-180426224043/85/Interactive-real-time-dashboards-on-data-streams-using-Kafka-Druid-and-Superset-3-320.jpg)
![21 © Hortonworks Inc. 2011–2018. All rights reserved
2
Dictionary Encoding
• Create and store Ids for each value
• e.g. page column
⬢ Values - Justin Bieber, Ke$ha, Selena Gomes
⬢ Encoding - Justin Bieber : 0, Ke$ha: 1, Selena Gomes: 2
⬢ Column Data - [0 0 0 1 1 2]
• city column - [0 0 0 1 1 1]
timestamp page language city country … added deleted
2011-01-01T00:01:35Z Justin Bieber en SF USA 10 65
2011-01-01T00:03:63Z Justin Bieber en SF USA 15 62
2011-01-01T00:04:51Z Justin Bieber en SF USA 32 45
2011-01-01T00:05:35Z Ke$ha en Calgary CA 17 87
2011-01-01T00:06:41Z Ke$ha en Calgary CA 43 99
2011-01-02T00:08:35Z Selena Gomes en Calgary CA 12 53](https://image.slidesharecdn.com/interactiverealtimedashboards-180426224043/85/Interactive-real-time-dashboards-on-data-streams-using-Kafka-Druid-and-Superset-21-320.jpg)
![22 © Hortonworks Inc. 2011–2018. All rights reserved
2
Bitmap Indices
• Store Bitmap Indices for each value
⬢ Justin Bieber -> [0, 1, 2] -> [1 1 1 0 0 0]
⬢ Ke$ha -> [3, 4] -> [0 0 0 1 1 0]
⬢ Selena Gomes -> [5] -> [0 0 0 0 0 1]
• Queries
⬢ Justin Bieber or Ke$ha -> [1 1 1 0 0 0] OR [0 0 0 1 1 0] -> [1 1 1 1 1 0]
⬢ language = en and country = CA -> [1 1 1 1 1 1] AND [0 0 0 1 1 1] -> [0 0 0 1 1 1]
• Indexes compressed with Concise or Roaring encoding
timestamp page language city country … added deleted
2011-01-01T00:01:35Z Justin Bieber en SF USA 10 65
2011-01-01T00:03:63Z Justin Bieber en SF USA 15 62
2011-01-01T00:04:51Z Justin Bieber en SF USA 32 45
2011-01-01T00:01:35Z Ke$ha en Calgary CA 17 87
2011-01-01T00:01:35Z Ke$ha en Calgary CA 43 99
2011-01-01T00:01:35Z Selena Gomes en Calgary CA 12 53](https://image.slidesharecdn.com/interactiverealtimedashboards-180426224043/85/Interactive-real-time-dashboards-on-data-streams-using-Kafka-Druid-and-Superset-22-320.jpg)
Uploaded byDataWorks Summit
Interactive real time dashboards on data streams using Kafka, Druid, and Superset
The document discusses the architecture and requirements for building interactive real-time dashboards using data streams, specifically focusing on components like Apache Kafka for event processing and Druid for data storage and visualization. Key features include low latency, high throughput, scalability, and the ability to ingest streaming data for creating actionable insights. It also emphasizes the importance of tools like Superset for visualization, enhancing user interaction with data analytics.
More Related Content
![[236] 카카오의데이터파이프라인 윤도영](https://cdn.slidesharecdn.com/ss_thumbnails/236-161025031702-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Interactive real time dashboards on data streams using Kafka, Druid, and Superset
![Building streaming data applications using Kafka*[Connect + Core + Streams] b...](https://cdn.slidesharecdn.com/ss_thumbnails/buildingstreamingdataapplicationsusingapachekafka-171011211455-thumbnail.jpg?width=640&height=640&fit=bounds)
More from DataWorks Summit
Interactive real time dashboards on data streams using Kafka, Druid, and Superset
- 1. 1 © HortonworksInc. 2011–2018. All rights reserved Interactive Realtime Dashboards on Data Streams Nishant Bangarwa Hortonworks Druid Committer, PMC
- 2. 2 © HortonworksInc. 2011–2018. All rights reserved Sample Data Stream : Wikipedia Edits
- 3. 3 © HortonworksInc. 2011–2018. All rights reserved Step by Step Breakdown Consume Events Enrich / Transform (Add Geolocation from IP Address) Store Events Visualize Events Sample Event : [[Eoghan Harris]] https://en.wikipedia.org/w/index.php?diff=792474242&oldid=787592607 * 7.114.169.238 * (+167) Added fact
- 4. 4 © HortonworksInc. 2011–2018. All rights reserved Required Components Event Flow Event Processing Data Store Visualization Layer
- 5. 5 © HortonworksInc. 2011–2018. All rights reserved Event Flow
- 6. 6 © HortonworksInc. 2011–2018. All rights reserved Event Flow : Requirements Event Producers Queue Event Consumers Low latency High Throughput Failure Handling Message delivery guarantees – Atleast Once, Exactly once, Atmost Once Varies based on use-case, Exactly once being the holy grail and the most difficult to achieve Scalability
- 7. 7 © HortonworksInc. 2011–2018. All rights reserved Apache Kafka
- 8. 8 © HortonworksInc. 2011–2018. All rights reserved Apache Kafka Low Latency High Throughput Message Delivery guarantees At-least once Exactly Once (Fully introduced in apache kafka v0.11.0 June 2017) Reliable design to Handle Failures Message Acks between producers and brokers Data Replication on brokers Consumers can Read from any desired offset Handle multiple producers/consumers Scalable
- 9. 9 © HortonworksInc. 2011–2018. All rights reserved Event Processing
- 10. 10 © HortonworksInc. 2011–2018. All rights reserved Event Processing : Requirements Consume-Process-Produce Pattern Enrich and Transform event streams Windowing Apply business logic Consume and Join multiple streams into single Failure Handling Scalability Consume Process Produce
- 11. 11 © HortonworksInc. 2011–2018. All rights reserved Kafka Streams Rich Lightweight Stream processing library Event-at-a-time Stateful processing : windowing, joining, aggregation operators Local state using RocksDb Backed by changelog in kafka Highly scalable, distributed, fault tolerant Compared to a standard Kafka consumer: Higher level: faster to build a sophisticated app Less control for very fine-grained consumption
- 12. 12 © HortonworksInc. 2011–2018. All rights reserved Kafka Streams : Wikipedia Data Enrichment
- 13. © Hortonworks Inc.2011 – 2016. All Rights Reserved 1 Data Store
- 14. 14 © HortonworksInc. 2011–2018. All rights reserved Data Store
- 15. 15 © HortonworksInc. 2011–2018. All rights reserved Data Store : Requirements Processed Events Data Store Queries Ability to ingest Streaming data Power Interactive dashboards Sub-Second Query Response time Ad-hoc arbitrary slicing and dicing of data Data Freshness Summarized/aggregated data is queried Scalability High Availability
- 16. 16 © HortonworksInc. 2011–2018. All rights reserved 1 Druid • Column-oriented distributed datastore • Sub-Second query times • Realtime streaming ingestion • Arbitrary slicing and dicing of data • Automatic Data Summarization • Approximate algorithms (hyperLogLog, theta) • Scalable to petabytes of data • Highly available
- 17. 17 © HortonworksInc. 2011–2018. All rights reserved Suitable Use Cases • Powering Interactive user facing applications • Arbitrary slicing and dicing of large datasets • User behavior analysis • measuring distinct counts • retention analysis • funnel analysis • A/B testing • Exploratory analytics/root cause analysis • Not interested in dumping entire dataset
- 18. 18 © HortonworksInc. 2011–2018. All rights reserved Druid: Segments • Data in Druid is stored in Segment Files. • Partitioned by time • Ideally, segment files are each smaller than 1GB. • If files are large, smaller time partitions are needed. Time Segment 1: Monday Segment 2: Tuesday Segment 3: Wednesday Segment 4: Thursday Segment 5_2: Friday Segment 5_1: Friday
- 19. 19 © HortonworksInc. 2011–2018. All rights reserved 1 Example Wikipedia Edit Dataset timestamp page language city country … added deleted 2011-01-01T00:01:35Z Justin Bieber en SF USA 10 65 2011-01-01T00:03:63Z Justin Bieber en SF USA 15 62 2011-01-01T00:04:51Z Justin Bieber en SF USA 32 45 2011-01-01T00:05:35Z Ke$ha en Calgary CA 17 87 2011-01-01T00:06:41Z Ke$ha en Calgary CA 43 99 2011-01-02T00:08:35Z Selena Gomes en Calgary CA 12 53 Timestamp Dimensions Metrics
- 20. 20 © HortonworksInc. 2011–2018. All rights reserved 2 Data Rollup timestamp page language city country … added deleted 2011-01-01T00:01:35Z Justin Bieber en SF USA 10 65 2011-01-01T00:03:63Z Justin Bieber en SF USA 15 62 2011-01-01T00:04:51Z Justin Bieber en SF USA 32 45 2011-01-01T00:05:35Z Ke$ha en Calgary CA 17 87 2011-01-01T00:06:41Z Ke$ha en Calgary CA 43 99 2011-01-02T00:08:35Z Selena Gomes en Calgary CA 12 53 timestamp page language city country count sum_added sum_deleted min_added max_added …. 2011-01-01T00:00:00Z Justin Bieber en SF USA 3 57 172 10 32 2011-01-01T00:00:00Z Ke$ha en Calgary CA 2 60 186 17 43 2011-01-02T00:00:00Z Selena Gomes en Calgary CA 1 12 53 12 12 Rollup by hour
- 21. 21 © HortonworksInc. 2011–2018. All rights reserved 2 Dictionary Encoding • Create and store Ids for each value • e.g. page column ⬢ Values - Justin Bieber, Ke$ha, Selena Gomes ⬢ Encoding - Justin Bieber : 0, Ke$ha: 1, Selena Gomes: 2 ⬢ Column Data - [0 0 0 1 1 2] • city column - [0 0 0 1 1 1] timestamp page language city country … added deleted 2011-01-01T00:01:35Z Justin Bieber en SF USA 10 65 2011-01-01T00:03:63Z Justin Bieber en SF USA 15 62 2011-01-01T00:04:51Z Justin Bieber en SF USA 32 45 2011-01-01T00:05:35Z Ke$ha en Calgary CA 17 87 2011-01-01T00:06:41Z Ke$ha en Calgary CA 43 99 2011-01-02T00:08:35Z Selena Gomes en Calgary CA 12 53
- 22. 22 © HortonworksInc. 2011–2018. All rights reserved 2 Bitmap Indices • Store Bitmap Indices for each value ⬢ Justin Bieber -> [0, 1, 2] -> [1 1 1 0 0 0] ⬢ Ke$ha -> [3, 4] -> [0 0 0 1 1 0] ⬢ Selena Gomes -> [5] -> [0 0 0 0 0 1] • Queries ⬢ Justin Bieber or Ke$ha -> [1 1 1 0 0 0] OR [0 0 0 1 1 0] -> [1 1 1 1 1 0] ⬢ language = en and country = CA -> [1 1 1 1 1 1] AND [0 0 0 1 1 1] -> [0 0 0 1 1 1] • Indexes compressed with Concise or Roaring encoding timestamp page language city country … added deleted 2011-01-01T00:01:35Z Justin Bieber en SF USA 10 65 2011-01-01T00:03:63Z Justin Bieber en SF USA 15 62 2011-01-01T00:04:51Z Justin Bieber en SF USA 32 45 2011-01-01T00:01:35Z Ke$ha en Calgary CA 17 87 2011-01-01T00:01:35Z Ke$ha en Calgary CA 43 99 2011-01-01T00:01:35Z Selena Gomes en Calgary CA 12 53
- 23. 23 © HortonworksInc. 2011–2018. All rights reserved 2 Approximate Sketch Columns timestamp page userid language city country … added deleted 2011-01-01T00:01:35Z Justin Bieber user1111111 en SF USA 10 65 2011-01-01T00:03:63Z Justin Bieber user1111111 en SF USA 15 62 2011-01-01T00:04:51Z Justin Bieber user2222222 en SF USA 32 45 2011-01-01T00:05:35Z Ke$ha user3333333 en Calgary CA 17 87 2011-01-01T00:06:41Z Ke$ha user4444444 en Calgary CA 43 99 2011-01-02T00:08:35Z Selena Gomes user1111111 en Calgary CA 12 53 timestamp page language city country count sum_added sum_delete d min_added Userid_sket ch …. 2011-01-01T00:00:00Z Justin Bieber en SF USA 3 57 172 10 {sketch} 2011-01-01T00:00:00Z Ke$ha en Calgary CA 2 60 186 17 {sketch} 2011-01-02T00:00:00Z Selena Gomes en Calgary CA 1 12 53 12 {sketch} Rollup by hour
- 24. 24 © HortonworksInc. 2011–2018. All rights reserved Approximate Algorithms • Store Sketch objects, instead of raw column values • Better rollup for high cardinality columns e.g userid • Reduced storage size • Use Cases • Fast approximate distinct counts • Approximate histograms • Funnel/retention analysis • Limitation • Not possible to do exact counts • filter on individual row values
- 25. 25 © HortonworksInc. 2011–2018. All rights reserved Realtime Nodes Historical Nodes 2 Druid Architecture Batch Data Event Historical Nodes Broker Nodes Realtime Index Tasks Streaming Data Historical Nodes Handoff
- 26. 26 © HortonworksInc. 2011–2018. All rights reserved Performance and Scalability : Fast Facts Most Events per Day 300 Billion Events / Day (Metamarkets) Most Computed Metrics 1 Billion Metrics / Min (Jolata) Largest Cluster 200 Nodes (Metamarkets) Largest Hourly Ingestion 2TB per Hour (Netflix)
- 27. 27 © HortonworksInc. 2011–2018. All rights reserved 2 Companies Using Druid
- 28. 28 © HortonworksInc. 2011–2018. All rights reserved Visualization Layer
- 29. 29 © HortonworksInc. 2011–2018. All rights reserved Visualization Layer : Requirements • Rich dashboarding capabilities • Work with multiple datasoucres • Security/Access control • Allow for extension • Add custom visualizations Data Store Visualization Layer User Dashboards
- 30. 30 © HortonworksInc. 2011–2018. All rights reserved Superset • Python backend • Flask app builder • Authentication • Pandas for rich analytics • SqlAlchemy for SQL toolkit • Javascript frontend • React, NVD3 • Deep integration with Druid
- 31. 31 © HortonworksInc. 2011–2018. All rights reserved Superset Rich Dashboarding Capabilities: Treemaps
- 32. 32 © HortonworksInc. 2011–2018. All rights reserved Superset Rich Dashboarding Capabilities: Sunburst
- 33. 33 © HortonworksInc. 2011–2018. All rights reserved Superset UI Provides Powerful Visualizations Rich library of dashboard visualizations: Basic: • Bar Charts • Pie Charts • Line Charts Advanced: • Sankey Diagrams • Treemaps • Sunburst • Heatmaps And More!
- 34. 34 © HortonworksInc. 2011–2018. All rights reserved Wikipedia Real-Time Dashboard Kafka Connect IP-to- Geolocation Processor wikipedia-raw topic wikipedia-raw topic wikipedia-enriched topic wikipedia-enriched topic
- 35. 35 © HortonworksInc. 2011–2018. All rights reserved Demo: Wikipedia Real-Time Dashboard (Accelerated 30x)
- 36. 36 © HortonworksInc. 2011–2018. All rights reserved Project Websites Kafka - http://kafka.apache.org Druid - http://druid.io Superset - http://superset.incubator.apache.org
- 37. 37 © HortonworksInc. 2011–2018. All rights reserved Thank you Twitter - @NishantBangarwa Email - nbangarwa@hortonworks.com Linkedin - https://www.linkedin.com/in/nishant-bangarwa
- 38. 38 © HortonworksInc. 2011–2018. All rights reserved Questions?
- 39. © Hortonworks Inc.2011 – 2016. All Rights Reserved 3 Thank you ! Questions ? • Twitter - @NishantBangarwa • Email - nbangarwa@hortonworks.com • Linkedin - https://www.linkedin.com/in/nishant-bangarwa
Editor's Notes
- #3 In this talk we will discuss an end-to-end stack using open-source technologies and build a dashboard on top of streaming data. We will discuss the challenges involved and how each component in the stack addresses those challenges. As a sample problem, we will look at wikipedia editstream provided by wikipedia. Whenever any page is edited on wikipedia an edit event is generated which contains details about which page was edited,
- #4 Lets try to break down this problem further A sample event from wikipedia editstream is formatted as follows – Title , URL of page edited, IP address of user, number of characters added/deleted First we would like to consume the events as coming from wikipedia Second we would like to enrich the event by doing an IP lookup and add the geolocation info about the user, and add more fields liks city, country from where the edit is being made. Third we would like to store these streaming events in a data store from where they can be queried and finally visualized on a dashboard.
- #5 So to solve the wikipedia problem we need to have four components – First, A solution that can move events from one place to another in a reliable and guaranteed. Second, Event Processing layer which can process events and transform/enrich them. (Also termed as ETL) Third, Data Storage layer that can provide an sub-second queries on incoming data streams. Finally, A Visualization layer that allows creating of dashboards on top the data store, users can interact with the dashboards to gain insights out of the data.
- #7 Producers produce the events in some message queue from where consumers fetch those events.
- #8 In Apache Kafka, Each topic is divided into set of partitions, Ordering of events are guaranteed within one partition. Each producer can produce to multiple partitions, Each message in the queue is identifiable by an offset. Consumers consume messages from partitions sequentially and are also responsible for keeping track of their offsets. This also helps in minimizing the overhead.
- #12 Local State – data is locally stored in RocksDb, for recovery each change to the local state is also propagated as an event in kafka. In case of failure or topology restart, Local State is restored from the changelog topic in kafka. The changelog topic is periodically compacted to reduce size.
- #14 Druid Architecture
- #17 Column-oriented distributed datastore – data is stored in columnar format, in general many datasets have a large number of dimensions e.g 100s or 1000s , but most of the time queries only need 5-10s of columns, the column oriented format helps druid in only scanning the required columns. Sub-Second query times – It utilizes various techniques like bitmap indexes to do fast filtering of data, uses memory mapped files to serve data from memory, data summarization and compression, query caching to do fast filtering of data and have very optimized algorithms for different query types. And is able to achievesub second query times Realtime streaming ingestion from almost any ETL pipeline. Arbitrary slicing and dicing of data – no need to create pre-canned drill downs Automatic Data Summarization – during ingestion it can summarize your data based, e.g If my dashboard only shows events aggregated by HOUR, we can optionally configure druid to do pre-aggregation at ingestion time. Approximate algorithms (hyperLogLog, theta) – for fast approximate answers Scalable to petabytes of data Highly available
- #18 Retention analysis
- #19 Druid: Segments Data in Druid is stored in Segment Files. Partitioned by time Ideally, segment files are each smaller than 1GB. If files are large, smaller time partitions are needed.
- #26 Druid has concept of different nodes, where each node is designed and optimized to perform specific set of tasks. Realtime Index Tasks / Realtime Nodes- Handle Real-Time Ingestion, Support both pull & push based ingestion. Handle Queries - Ability to serve queries as soon as data is ingested. Store data in write optimized data structure on heap, periodically convert it to write optimized time partitioned immutable segments and persist it to deep storage. In case you need to do any ETL like data enrichment or joining multiple streams of data, you can do it in a separate ETL and send your massaged data to druid. Deep storage can be any distributed FS and acts as a permanent backup of data Historical Nodes - Main workhorses of druid cluster Use Memory Mapped files to load immutable segments Respond to User queries Now Lets see the how data can be queried. Broker Nodes - Keeps track of the data chunks being loaded by each node in the cluster Ability to scatter query across multiple Historical and Realtime nodes Caching Layer Now Lets discuss another case, when you are not having streaming data, but want to Ingest Batch data into druid Batch ingestion can be done using either Hadoop MR or spark job, which converts your data into time partitioned segments and persist it to deep storage.