Realtime Analytics with Druid
•Download as PPTX, PDF•
2 likes•370 views
1. The document discusses realtime analytics with Druid and LeanStream. 2. It describes the history of technologies used including NiFi, Kafka, Druid and LeanStream and provides an overview of the realtime analytics flow. 3. Examples of realtime analytics queries are shown, including using dimensions like time and metrics like counts to analyze streaming data.
Report
Share
Report
Share
Recommended
Aggregated queries with Druid on terrabytes and petabytes of data
Overview of the Druid.io, features, pros/cons, querying and etc.
DataStax and Esri: Geotemporal IoT Search and Analytics
Internet of Things (IoT) data frequently has a location and time component. Getting value out of this "geotemporal" data can be tricky. We'll explore when and how to leverage Cassandra, DSE Search and DSE Analytics to surface meaningful information from your geotemporal data.
Web analytics at scale with Druid at naver.com
The slides of Strata 2018 London
https://conferences.oreilly.com/strata/strata-eu/public/schedule/detail/65329
Building a Complex, Real-Time Data Management Application
Congratulations: you've been selected to build an application that will manage whether or not the rooms for PGConf.EU are being occupied by a session!
On the surface, this sounds simple, but we will be managing the rooms of PGConf.EU, so we know that a lot of people will be accessing the system. Therefore, we need to ensure that the system can handle all of the eager users that will be flooding the PGConf.EU website checking to see what availability each of the PGConf.EU rooms has.
To do this, we will explore the following PGConf.EU features:
* Data types and their functionality, such as:
* Data/Time types
* Ranges
Indexes such as:
* GiST
* SP-Gist
* Common Table Expressions and Recursion
* Set generating functions and LATERAL queries
* Functions and the PL/PGSQL
* Triggers
* Logical decoding and streaming
We will be writing our application primary with SQL, though we will sneak in a little bit of Python and using Kafka to demonstrate the power of logical decoding.
At the end of the presentation, we will have a working application, and you will be happy knowing that you provided a wonderful user experience for all PGConf.EU attendees made possible by the innovation of PGConf.EU!
HBaseCon 2012 | HBase for the Worlds Libraries - OCLC
WorldCat is the world’s largest network of library content and services. Over 25,000 libraries in 170 countries have cooperated for 40 years to build WorldCat. OCLC is currently in the process of transitioning Worldcat from Oracle to Apache HBase. This session will discuss our data design for representing the constantly changing ownership information for thousands of libraries (billions of data points, millions of daily updates) and our plans for how we’re managing HBase in an environment that is equal parts end user facing and batch.
Building Data Product Based on Apache Spark at Airbnb with Jingwei Lu and Liy...
Building data product requires having Lambda Architecture to bridge the batch and streaming processing. AirStream is a framework built on top of Apache Spark to allow users to easily build data products at Airbnb. It proved Spark is impactful and useful in the production for mission-critical data products.
On the streaming side, hear how AirStream integrates multiple ecosystems with Spark Streaming, such as HBase, Elasticsearch, MySQL, DynamoDB, Memcache and Redis. On the batch side, learn how to apply the same computation logic in Spark over large data sets from Hive and S3. The speakers will also go through a few production use cases, and share several best practices on how to manage Spark jobs in production.
Apache Spark Overview part2 (20161117)
Apache Spark Overview part2(2016.11.17)
- Spark Streaming
- Spark GraphX
- Spark ML
Recommended
Aggregated queries with Druid on terrabytes and petabytes of data
Overview of the Druid.io, features, pros/cons, querying and etc.
DataStax and Esri: Geotemporal IoT Search and Analytics
Internet of Things (IoT) data frequently has a location and time component. Getting value out of this "geotemporal" data can be tricky. We'll explore when and how to leverage Cassandra, DSE Search and DSE Analytics to surface meaningful information from your geotemporal data.
Web analytics at scale with Druid at naver.com
The slides of Strata 2018 London
https://conferences.oreilly.com/strata/strata-eu/public/schedule/detail/65329
Building a Complex, Real-Time Data Management Application
Congratulations: you've been selected to build an application that will manage whether or not the rooms for PGConf.EU are being occupied by a session!
On the surface, this sounds simple, but we will be managing the rooms of PGConf.EU, so we know that a lot of people will be accessing the system. Therefore, we need to ensure that the system can handle all of the eager users that will be flooding the PGConf.EU website checking to see what availability each of the PGConf.EU rooms has.
To do this, we will explore the following PGConf.EU features:
* Data types and their functionality, such as:
* Data/Time types
* Ranges
Indexes such as:
* GiST
* SP-Gist
* Common Table Expressions and Recursion
* Set generating functions and LATERAL queries
* Functions and the PL/PGSQL
* Triggers
* Logical decoding and streaming
We will be writing our application primary with SQL, though we will sneak in a little bit of Python and using Kafka to demonstrate the power of logical decoding.
At the end of the presentation, we will have a working application, and you will be happy knowing that you provided a wonderful user experience for all PGConf.EU attendees made possible by the innovation of PGConf.EU!
HBaseCon 2012 | HBase for the Worlds Libraries - OCLC
WorldCat is the world’s largest network of library content and services. Over 25,000 libraries in 170 countries have cooperated for 40 years to build WorldCat. OCLC is currently in the process of transitioning Worldcat from Oracle to Apache HBase. This session will discuss our data design for representing the constantly changing ownership information for thousands of libraries (billions of data points, millions of daily updates) and our plans for how we’re managing HBase in an environment that is equal parts end user facing and batch.
Building Data Product Based on Apache Spark at Airbnb with Jingwei Lu and Liy...
Building data product requires having Lambda Architecture to bridge the batch and streaming processing. AirStream is a framework built on top of Apache Spark to allow users to easily build data products at Airbnb. It proved Spark is impactful and useful in the production for mission-critical data products.
On the streaming side, hear how AirStream integrates multiple ecosystems with Spark Streaming, such as HBase, Elasticsearch, MySQL, DynamoDB, Memcache and Redis. On the batch side, learn how to apply the same computation logic in Spark over large data sets from Hive and S3. The speakers will also go through a few production use cases, and share several best practices on how to manage Spark jobs in production.
Apache Spark Overview part2 (20161117)
Apache Spark Overview part2(2016.11.17)
- Spark Streaming
- Spark GraphX
- Spark ML
Cassandra Tools and Distributed Administration (Jeffrey Berger, Knewton) | C*...
At Knewton we operate across five different VPCs a total of 29 clusters, each ranging from 3 nodes to 24 nodes. For a team of three to maintain this is not herculean, however good tools to diagnose issues and gather information in a distributed manner are vital to moving quickly and minimizing engineering time spent.
The database team at Knewton has been successfully using a combination of Ansible and custom open sourced tools to maintain and improve the Cassandra deployment at Knewton. I will be talking about several of these tools and giving examples of how we are using them. Specifically I will discuss the cassandra-tracing tool, which analyzes the contents of the system_traces keyspace, and the cassandra-stat tool, which gives real-time output of the operations of a cassandra cluster. Distributed administration with ad-hoc Ansible will also be covered and I will walk through examples of using these commands to identify and remediate clusterwide issues.
About the Speaker
Jeffrey Berger Lead Database Engineer, Knewton
Dr. Jeffrey Berger is currently the lead database engineer at Knewton, an education tech startup in NYC. He joined the tech scene in NYC in 2013 and spent two years working with MongoDB, becoming a certified MongoDB administrator and a MongoDB Master. He received his Cassandra Administrator certification at Cassandra Summit 2015. He holds a Ph.D. in Theoretical Physics from Penn State and spent several years working on high energy nuclear interactions.
Running Apache Spark on a High-Performance Cluster Using RDMA and NVMe Flash ...
Effectively leveraging fast networking and storage hardware (e.g., RDMA, NVMe, etc.) in Apache Spark remains challenging. Current ways to integrate the hardware at the operating system level fall short, as the hardware performance advantages are shadowed by higher layer software overheads. This session will show how to integrate RDMA and NVMe hardware in Spark in a way that allows applications to bypass both the operating system and the Java virtual machine during I/O operations. With such an approach, the hardware performance advantages become visible at the application level, and eventually translate into workload runtime improvements. Stuedi will demonstrate how to run various Spark workloads (e.g, SQL, Graph, etc.) effectively on 100Gbit/s networks and NVMe flash.
Lambda Architecture with Cassandra (Vaibhav Puranik, GumGum) | C* Summit 2016
Over the past year data at GumGum has quadrupled. Now a days we process 20 TB of new data every day. New data/reporting requirements pour in every week. Usage of the real time data is growing with the daily data. In this talk we are tying to answer the following questions: How do we serve real time data with daily batched data to our consumers together? What is Lambda Architecture and how does it help? What role does Cassandra play in Lambda Architecture at GumGum? How did we solve few bottlenecks in the architecture using Cassandra? How Cassandra can help you avoid microbatching and give you a true realtime data?
About the Speaker
Vaibhav Puranik VP of Engineering, Big Data & Platform, GumGum
Vaibhav has 15 years of experience in Software. He began his career with Johnson Space Center in Houston and has a masters degree in computer science. For past 5 years Vaibhav has been responsible for architecting multiple big data systems at GumGum. He manages Data Science, Data Engineering and DevOps teams at GumGum.
Stsg17 speaker yousunjeong
Strata Singapore 2017 business use case section
"Big Telco Real-Time Network Analytics"
https://conferences.oreilly.com/strata/strata-sg/public/schedule/detail/62797
WEBINAR - Introducing Scylla Open Source 3.0: Materialized Views, Secondary I...
The industry’s most performant NoSQL database just got better. Scylla Open Source 3.0 introduces much-anticipated new features for more efficient querying, reduced storage requirements, lower repair times, and better overall database performance. It includes production-ready capabilities beyond those available with Apache Cassandra or any other NoSQL database.
Join ScyllaDB CEO and co-founder Dor Laor and vice president of field engineering Glauber Costa for a technical overview of the new features and capabilities in Scylla Open Source 3.0, including:
- Materialized Views
- Global Secondary Indexes
- New storage format: SSTable 3.0
- Hinted Handoff
- Streaming Improvements
- Full scan improvements
Experience of Running Spark on Kubernetes on OpenStack for High Energy Physic...
The physicists at CERN are increasingly turning to Spark to process large physics datasets in a distributed fashion with the aim of reducing time-to-physics with increased interactivity. The physics data itself is stored in CERN’s mass storage system: EOS and CERN’s IT department runs on-premise private cloud based on OpenStack as a way to provide on-demand compute resources to physicists. This provides both opportunity and challenges to Big Data team at CERN to provide elastic, scalable, reliable spark-as-a-service on OpenStack.
The talk focuses on the design choices made and challenges faced while developing spark-as-a-service over kubernetes on openstack to simplify provisioning, automate management, and minimize the operating burden of managing Spark Clusters. In addition, the service tooling simplifies submitting applications on the behalf of the users, mounting user-specified ConfigMaps, copying application logs to s3 buckets for troubleshooting, performance analysis and accounting of spark applications and support for stateful spark streaming applications. We will also share results from running large scale sustained workloads over terabytes of physics data.
Getting Started Running Apache Spark on Apache Mesos
O'Reilly Media webcast 2014-01-24 http://www.oreillynet.com/pub/e/2986
Numberly on Joining Billions of Rows in Seconds: Replacing MongoDB and Hive w...
Many organizations struggle to balance traditional big data infrastructure with NoSQL databases. Other organizations do the smart thing and consolidate the two. This presentation explores Numberly’s experience migrating an intensive and join-hungry production workload from MongoDB and Hive to Scylla.
Join Alexys Jacob, CTO of Numberly, to learn how they joined billions of rows in seconds and dramatically reduced operational and development complexity by using a single database for their hybrid analytical use case.
As a bonus, Alexys will also cover benchmarks for Dask (a flexible parallel computing library for analytic computing) and Spark, highlighting their differences and lessons learned along the way.
Spark on Mesos-A Deep Dive-(Dean Wampler and Tim Chen, Typesafe and Mesosphere)
Presentation at Spark Summit 2015
Real-time Cassandra
Talk given at Denormalised London, 2012-09-20. Discussion of what a real-time system needs to do and why Cassandra is a good fit.
Monitoring Cassandra at Scale (Jason Cacciatore, Netflix) | C* Summit 2016
Cassandra is the dominant data store used at Netflix and it's health is critical to many of its services. In this talk we will share details of the recent redesign of our health monitoring system and how we leveraged a reactive stream processing system to give us a real-time view our entire fleet while dramatically improving accuracy and reducing false alarms in our alerting.
About the Speaker
Jason Cacciatore Senior Software Engineer, Netflix
Jason Cacciatore is a Senior Software Engineer at Netflix, where he's been working for the past several years. He's interested in stateful distributed systems and has a diverse background in technology. In his spare time he enjoys spending time with his wife and two sons, reading non-fiction, and watching Netflix documentaries.
Webinar: Does it Still Make Sense to do Big Data with Small Nodes?
In the world of Big Data, scaling out is the norm. However, many Big Data deployments are trapped in a sea of small box clusters.
With the advent of scalable platforms like Scylla, node performance is no longer an issue and doubling the size of the nodes can double the available storage, memory, and processing power. So what stops people from going big in the Cloud Native world?
Watch this webinar to learn the pros and cons of large nodes, and explore why people resist using big machines, including:
- Is the cost of recovering from failures higher in larger nodes?
- Does performance increase linearly as machines get bigger?
- Does cluster performance suffer for the entire time of recovery from failures?
C* for Deep Learning (Andrew Jefferson, Tracktable) | Cassandra Summit 2016
A deep learning startup has a requirement for a robust and scalable data architecture. Training a Deep Neural Network requires 10s-100s of millions of examples consisting of data and metadata. In addition to training it is necessary to support test/validation, data exploration and more traditional data science analytics workloads. As a startup we have minimal resources and an engineering team of 1.
Cassandra, Spark and Kafka running on Mesos in AWS is a scalable architecture that is fast and easy to set up and maintain to deliver a data architecture for Deep Learning.
About the Speaker
Andrew Jefferson VP Engineering, Tractable
A software engineer specialising in realtime data systems. I've worked at companies from Startups to Apple on applications ranging from Ticketing to Genetics. Currently building data systems for training and exploiting Deep Neural Networks.
2011 06-30-hadoop-summit v5
Slides from presentation at Hadoop Summit 2011 on Facebook's Data Freeway system
Engineering fast indexes
Contemporary computing hardware offers massive new performance opportunities. Yet high-performance programming remains a daunting challenge.
We present some of the lessons learned while designing faster indexes, with a particular emphasis on compressed bitmap indexes. Compressed bitmap indexes accelerate queries in popular systems such as Apache Spark, Git, Elastic, Druid and Apache Kylin.
Apache Cassandra and Python for Analyzing Streaming Big Data
This presentation was made at the Open Source India Conference Nov 2015. It explains how Apache Spark, pySpark, Cassandra, Node.js and D3.js can be used for creating a platform for visualizing and analyzing streaming big data
Unconference Round Table Notes
Unconference Round Table Notes
The future of real-time stream processing
WASM (Web Assembly)
Petabyte, 5000 Node Clusters, Smart Hyper Scaling
Multi-language support (Python, Rust, Kotlin, Golang, Carbon, JVM)
Machine Learning, Deep Learning, AI and Advanced Math
Low Code Development like Apache NiFi, DataFlow Designer, SQL
Dynamic Hybrid Deployment
Citizen Stream Engineer
IoT, Edge Streaming and Hybrid Edge Streaming
Java 20, 21; Java Loom Virtual Threading
Ultra low latency, trillions of events per second, massive RAM/network
Current challenges of real-time stream processing and proposed solutions
Deployment, Automation and Scaling
Choosing right project/sizing for use case
Simple Event Processing vs Complex Event Processing
Leveraging existing applications
Developer Skills
Self management and monitoring
Cost issues -> autoscaling, optimizing, performance, hybrid deployment
Performance / Benchmarking real-time stream processing
Kafka/Pulsar: https://openmessaging.cloud/docs/benchmarks/
NiFi: https://blog.cloudera.com/benchmarking-nifi-performance-and-scalability/
Flink: https://github.com/ververica/flink-sql-benchmark
Hazelcast: https://hazelcast.com/press-release/hazelcast-demonstrates-cloud-efficiency-real-time-stream-processing-of-one-billion-events-per-second/
Current trends of real-time stream processing in 2023
Current challenges of real-time stream processing and proposed solutions
Performance / Benchmarking real-time stream processing
The future of real-time stream processing
Current trends of real-time stream processing in 2023
Lightweight serverless
Hazelcast
SQL
Flink
Kafka or Pulsar as Messaging Hub
Java 17+
Managed Clusters, Containers and Environments
Real-Time Analytics
Fast Storage Options
Overview of Apache Flink: the 4G of Big Data Analytics Frameworks
Overview of Apache Flink: the 4G of Big Data Analytics Frameworks
More Related Content
What's hot
Cassandra Tools and Distributed Administration (Jeffrey Berger, Knewton) | C*...
At Knewton we operate across five different VPCs a total of 29 clusters, each ranging from 3 nodes to 24 nodes. For a team of three to maintain this is not herculean, however good tools to diagnose issues and gather information in a distributed manner are vital to moving quickly and minimizing engineering time spent.
The database team at Knewton has been successfully using a combination of Ansible and custom open sourced tools to maintain and improve the Cassandra deployment at Knewton. I will be talking about several of these tools and giving examples of how we are using them. Specifically I will discuss the cassandra-tracing tool, which analyzes the contents of the system_traces keyspace, and the cassandra-stat tool, which gives real-time output of the operations of a cassandra cluster. Distributed administration with ad-hoc Ansible will also be covered and I will walk through examples of using these commands to identify and remediate clusterwide issues.
About the Speaker
Jeffrey Berger Lead Database Engineer, Knewton
Dr. Jeffrey Berger is currently the lead database engineer at Knewton, an education tech startup in NYC. He joined the tech scene in NYC in 2013 and spent two years working with MongoDB, becoming a certified MongoDB administrator and a MongoDB Master. He received his Cassandra Administrator certification at Cassandra Summit 2015. He holds a Ph.D. in Theoretical Physics from Penn State and spent several years working on high energy nuclear interactions.
Running Apache Spark on a High-Performance Cluster Using RDMA and NVMe Flash ...
Effectively leveraging fast networking and storage hardware (e.g., RDMA, NVMe, etc.) in Apache Spark remains challenging. Current ways to integrate the hardware at the operating system level fall short, as the hardware performance advantages are shadowed by higher layer software overheads. This session will show how to integrate RDMA and NVMe hardware in Spark in a way that allows applications to bypass both the operating system and the Java virtual machine during I/O operations. With such an approach, the hardware performance advantages become visible at the application level, and eventually translate into workload runtime improvements. Stuedi will demonstrate how to run various Spark workloads (e.g, SQL, Graph, etc.) effectively on 100Gbit/s networks and NVMe flash.
Lambda Architecture with Cassandra (Vaibhav Puranik, GumGum) | C* Summit 2016
Over the past year data at GumGum has quadrupled. Now a days we process 20 TB of new data every day. New data/reporting requirements pour in every week. Usage of the real time data is growing with the daily data. In this talk we are tying to answer the following questions: How do we serve real time data with daily batched data to our consumers together? What is Lambda Architecture and how does it help? What role does Cassandra play in Lambda Architecture at GumGum? How did we solve few bottlenecks in the architecture using Cassandra? How Cassandra can help you avoid microbatching and give you a true realtime data?
About the Speaker
Vaibhav Puranik VP of Engineering, Big Data & Platform, GumGum
Vaibhav has 15 years of experience in Software. He began his career with Johnson Space Center in Houston and has a masters degree in computer science. For past 5 years Vaibhav has been responsible for architecting multiple big data systems at GumGum. He manages Data Science, Data Engineering and DevOps teams at GumGum.
Stsg17 speaker yousunjeong
Strata Singapore 2017 business use case section
"Big Telco Real-Time Network Analytics"
https://conferences.oreilly.com/strata/strata-sg/public/schedule/detail/62797
WEBINAR - Introducing Scylla Open Source 3.0: Materialized Views, Secondary I...
The industry’s most performant NoSQL database just got better. Scylla Open Source 3.0 introduces much-anticipated new features for more efficient querying, reduced storage requirements, lower repair times, and better overall database performance. It includes production-ready capabilities beyond those available with Apache Cassandra or any other NoSQL database.
Join ScyllaDB CEO and co-founder Dor Laor and vice president of field engineering Glauber Costa for a technical overview of the new features and capabilities in Scylla Open Source 3.0, including:
- Materialized Views
- Global Secondary Indexes
- New storage format: SSTable 3.0
- Hinted Handoff
- Streaming Improvements
- Full scan improvements
Experience of Running Spark on Kubernetes on OpenStack for High Energy Physic...
The physicists at CERN are increasingly turning to Spark to process large physics datasets in a distributed fashion with the aim of reducing time-to-physics with increased interactivity. The physics data itself is stored in CERN’s mass storage system: EOS and CERN’s IT department runs on-premise private cloud based on OpenStack as a way to provide on-demand compute resources to physicists. This provides both opportunity and challenges to Big Data team at CERN to provide elastic, scalable, reliable spark-as-a-service on OpenStack.
The talk focuses on the design choices made and challenges faced while developing spark-as-a-service over kubernetes on openstack to simplify provisioning, automate management, and minimize the operating burden of managing Spark Clusters. In addition, the service tooling simplifies submitting applications on the behalf of the users, mounting user-specified ConfigMaps, copying application logs to s3 buckets for troubleshooting, performance analysis and accounting of spark applications and support for stateful spark streaming applications. We will also share results from running large scale sustained workloads over terabytes of physics data.
Getting Started Running Apache Spark on Apache Mesos
O'Reilly Media webcast 2014-01-24 http://www.oreillynet.com/pub/e/2986
Numberly on Joining Billions of Rows in Seconds: Replacing MongoDB and Hive w...
Many organizations struggle to balance traditional big data infrastructure with NoSQL databases. Other organizations do the smart thing and consolidate the two. This presentation explores Numberly’s experience migrating an intensive and join-hungry production workload from MongoDB and Hive to Scylla.
Join Alexys Jacob, CTO of Numberly, to learn how they joined billions of rows in seconds and dramatically reduced operational and development complexity by using a single database for their hybrid analytical use case.
As a bonus, Alexys will also cover benchmarks for Dask (a flexible parallel computing library for analytic computing) and Spark, highlighting their differences and lessons learned along the way.
Spark on Mesos-A Deep Dive-(Dean Wampler and Tim Chen, Typesafe and Mesosphere)
Presentation at Spark Summit 2015
Real-time Cassandra
Talk given at Denormalised London, 2012-09-20. Discussion of what a real-time system needs to do and why Cassandra is a good fit.
Monitoring Cassandra at Scale (Jason Cacciatore, Netflix) | C* Summit 2016
Cassandra is the dominant data store used at Netflix and it's health is critical to many of its services. In this talk we will share details of the recent redesign of our health monitoring system and how we leveraged a reactive stream processing system to give us a real-time view our entire fleet while dramatically improving accuracy and reducing false alarms in our alerting.
About the Speaker
Jason Cacciatore Senior Software Engineer, Netflix
Jason Cacciatore is a Senior Software Engineer at Netflix, where he's been working for the past several years. He's interested in stateful distributed systems and has a diverse background in technology. In his spare time he enjoys spending time with his wife and two sons, reading non-fiction, and watching Netflix documentaries.
Webinar: Does it Still Make Sense to do Big Data with Small Nodes?
In the world of Big Data, scaling out is the norm. However, many Big Data deployments are trapped in a sea of small box clusters.
With the advent of scalable platforms like Scylla, node performance is no longer an issue and doubling the size of the nodes can double the available storage, memory, and processing power. So what stops people from going big in the Cloud Native world?
Watch this webinar to learn the pros and cons of large nodes, and explore why people resist using big machines, including:
- Is the cost of recovering from failures higher in larger nodes?
- Does performance increase linearly as machines get bigger?
- Does cluster performance suffer for the entire time of recovery from failures?
C* for Deep Learning (Andrew Jefferson, Tracktable) | Cassandra Summit 2016
A deep learning startup has a requirement for a robust and scalable data architecture. Training a Deep Neural Network requires 10s-100s of millions of examples consisting of data and metadata. In addition to training it is necessary to support test/validation, data exploration and more traditional data science analytics workloads. As a startup we have minimal resources and an engineering team of 1.
Cassandra, Spark and Kafka running on Mesos in AWS is a scalable architecture that is fast and easy to set up and maintain to deliver a data architecture for Deep Learning.
About the Speaker
Andrew Jefferson VP Engineering, Tractable
A software engineer specialising in realtime data systems. I've worked at companies from Startups to Apple on applications ranging from Ticketing to Genetics. Currently building data systems for training and exploiting Deep Neural Networks.
2011 06-30-hadoop-summit v5
Slides from presentation at Hadoop Summit 2011 on Facebook's Data Freeway system
Engineering fast indexes
Contemporary computing hardware offers massive new performance opportunities. Yet high-performance programming remains a daunting challenge.
We present some of the lessons learned while designing faster indexes, with a particular emphasis on compressed bitmap indexes. Compressed bitmap indexes accelerate queries in popular systems such as Apache Spark, Git, Elastic, Druid and Apache Kylin.
Apache Cassandra and Python for Analyzing Streaming Big Data
This presentation was made at the Open Source India Conference Nov 2015. It explains how Apache Spark, pySpark, Cassandra, Node.js and D3.js can be used for creating a platform for visualizing and analyzing streaming big data
What's hot (20)
Cassandra Tools and Distributed Administration (Jeffrey Berger, Knewton) | C*...
Cassandra Tools and Distributed Administration (Jeffrey Berger, Knewton) | C*...
Running Apache Spark on a High-Performance Cluster Using RDMA and NVMe Flash ...
Running Apache Spark on a High-Performance Cluster Using RDMA and NVMe Flash ...
Lambda Architecture with Cassandra (Vaibhav Puranik, GumGum) | C* Summit 2016
Lambda Architecture with Cassandra (Vaibhav Puranik, GumGum) | C* Summit 2016
WEBINAR - Introducing Scylla Open Source 3.0: Materialized Views, Secondary I...
WEBINAR - Introducing Scylla Open Source 3.0: Materialized Views, Secondary I...
Experience of Running Spark on Kubernetes on OpenStack for High Energy Physic...
Experience of Running Spark on Kubernetes on OpenStack for High Energy Physic...
Getting Started Running Apache Spark on Apache Mesos
Getting Started Running Apache Spark on Apache Mesos
Numberly on Joining Billions of Rows in Seconds: Replacing MongoDB and Hive w...
Numberly on Joining Billions of Rows in Seconds: Replacing MongoDB and Hive w...
Spark on Mesos-A Deep Dive-(Dean Wampler and Tim Chen, Typesafe and Mesosphere)
Spark on Mesos-A Deep Dive-(Dean Wampler and Tim Chen, Typesafe and Mesosphere)
Monitoring Cassandra at Scale (Jason Cacciatore, Netflix) | C* Summit 2016
Monitoring Cassandra at Scale (Jason Cacciatore, Netflix) | C* Summit 2016
Webinar: Does it Still Make Sense to do Big Data with Small Nodes?
Webinar: Does it Still Make Sense to do Big Data with Small Nodes?
C* for Deep Learning (Andrew Jefferson, Tracktable) | Cassandra Summit 2016
C* for Deep Learning (Andrew Jefferson, Tracktable) | Cassandra Summit 2016
Apache Cassandra and Python for Analyzing Streaming Big Data
Apache Cassandra and Python for Analyzing Streaming Big Data
IEEE International Conference on Data Engineering 2015
IEEE International Conference on Data Engineering 2015
Similar to Realtime Analytics with Druid
Unconference Round Table Notes
Unconference Round Table Notes
The future of real-time stream processing
WASM (Web Assembly)
Petabyte, 5000 Node Clusters, Smart Hyper Scaling
Multi-language support (Python, Rust, Kotlin, Golang, Carbon, JVM)
Machine Learning, Deep Learning, AI and Advanced Math
Low Code Development like Apache NiFi, DataFlow Designer, SQL
Dynamic Hybrid Deployment
Citizen Stream Engineer
IoT, Edge Streaming and Hybrid Edge Streaming
Java 20, 21; Java Loom Virtual Threading
Ultra low latency, trillions of events per second, massive RAM/network
Current challenges of real-time stream processing and proposed solutions
Deployment, Automation and Scaling
Choosing right project/sizing for use case
Simple Event Processing vs Complex Event Processing
Leveraging existing applications
Developer Skills
Self management and monitoring
Cost issues -> autoscaling, optimizing, performance, hybrid deployment
Performance / Benchmarking real-time stream processing
Kafka/Pulsar: https://openmessaging.cloud/docs/benchmarks/
NiFi: https://blog.cloudera.com/benchmarking-nifi-performance-and-scalability/
Flink: https://github.com/ververica/flink-sql-benchmark
Hazelcast: https://hazelcast.com/press-release/hazelcast-demonstrates-cloud-efficiency-real-time-stream-processing-of-one-billion-events-per-second/
Current trends of real-time stream processing in 2023
Current challenges of real-time stream processing and proposed solutions
Performance / Benchmarking real-time stream processing
The future of real-time stream processing
Current trends of real-time stream processing in 2023
Lightweight serverless
Hazelcast
SQL
Flink
Kafka or Pulsar as Messaging Hub
Java 17+
Managed Clusters, Containers and Environments
Real-Time Analytics
Fast Storage Options
Overview of Apache Flink: the 4G of Big Data Analytics Frameworks
Overview of Apache Flink: the 4G of Big Data Analytics Frameworks
Overview of Apache Fink: the 4 G of Big Data Analytics Frameworks
Slides of my talk at the Hadoop Summit Europe in Dublin, Ireland on April 13th, 2016. The talk introduces Apache Flink as both a multi-purpose Big Data analytics framework and real-world streaming analytics framework. It is focusing on Flink's key differentiators and suitability for streaming analytics use cases. It also shows how Flink enables novel use cases such as distributed CEP (Complex Event Processing) and querying the state by behaving like a key value data store.
Overview of Apache Fink: The 4G of Big Data Analytics Frameworks
Slides of my talk at the Hadoop Summit Europe in Dublin, Ireland on April 13th, 2016. The talk introduces Apache Flink as both a multi-purpose Big Data analytics framework and real-world streaming analytics framework. It is focusing on Flink's key differentiators and suitability for streaming analytics use cases. It also shows how Flink enables novel use cases such as distributed CEP (Complex Event Processing) and querying the state by behaving like a key value data store.
Flink September 2015 Community Update
The Sept 2015 Community Update presented at the Berlin Flink Meetup group.
2024 XTREMEJ_ Building Real-time Pipelines with FLaNK_ A Case Study with Tra...
2024 XTREMEJ_ Building Real-time Pipelines with FLaNK_ A Case Study with Transit Data
https://xtremej.dev/2023/schedule/
Building Real-time Pipelines with FLaNK: A Case Study with Transit Data
Overview of the problem, the application (code walkthru and running), overview of FLaNK, introduction to NiFi, introduction to Kafka, and introduction to Flink.
JConWorld_ Continuous SQL with Kafka and Flink
JConWorld: Continuous SQL with Kafka and Flink
In this talk, I will walk through how someone can setup and run continous SQL queries against Kafka topics utilizing Apache Flink. We will walk through creating Kafka topics, schemas and publishing data.
We will then cover consuming Kafka data, joining Kafka topics and inserting new events into Kafka topics as they arrive. This basic over view will show hands-on techniques, tips and examples of how to do this.
Tim Spann is the Principal Developer Advocate for Data in Motion @ Cloudera where he works with Apache Kafka, Apache Flink, Apache NiFi, Apache Iceberg, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal Field Engineer at Cloudera, a Senior Solutions Architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science. https://www.datainmotion.dev/p/about-me.html https://dzone.com/users/297029/bunkertor.html
https://www.youtube.com/channel/UCDIDMDfje6jAvNE8DGkJ3_w?view_as=subscriber
Recap of OpenStack Tokyo Summit
Some highlights of topics from the OpenStack Summit, as presented to the OpenStack St. Louis Meetup in November 2015. Most slides sourced from the summit videos (https://www.openstack.org/summit/tokyo-2015/videos/)
Neo4j Graph Platform Overview, Kurt Freytag, Neo4j
Neo4j GraphTour 2019:
Neo4j Graph Platform Overview, Kurt Freytag, Neo4j, Director of Product Management Cloud, Neo4j
Five Fabulous Sinks for Your Kafka Data. #3 will surprise you! (Rachel Pedres...
Apache Kafka has come the modern central point for a fast and scalable streaming platform. Now, thanks to the open source explosion over the last decade, there are now numerous data stores available as sinks for Kafka-brokered data, from search to document stores, columnular DBs, time series DBs and more. While many claim they are the swiss army knife, in reality each is designed for specific types of data and analytics approaches. In this talk, we will cover the taxonomy of various data sinks, delve into each categories pros, cons and ideal use cases, so you can select the right ones and tie them together with Kafka into a well-considered architecture.
Graph Day 2017 Spring Boot
A slide deck on an example implementation of the technology we used at ten-x to enable business users to find data in a graph.
[Hackersuli] Élő szövet a fémvázon: Python és gépi tanulás a Zeek platformon
Élő szövet a fémvázon: Python és gépi tanulás a Zeek platformon
Streaming Data Analytics with ksqlDB and Superset | Robert Stolz, Preset
Streaming data systems have been growing rapidly in importance to the modern data stack. Kafka’s kSQL provides an interface for analytic tools that speak SQL. Apache Superset, the most popular modern open-source visualization and analytics solution, plugs into nearly any data source that speaks SQL, including Kafka. Here, we review and compare methods for connecting Kafka to Superset to enable streaming analytics use cases including anomaly detection, operational monitoring, and online data integration.
Microservices and Integration: what's next with Istio service mesh
Red Hat Canada day presentation about service mesh, Istio and integration
KubeCon USA 2017 brief Overview - from Kubernetes meetup Bangalore
Brief overview of the KubeCon USA 2017 from the participants perspective. Provides summary of the events and some glance of the event activities.
Patterns of Streaming Applications
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2UkZRIC.
Monal Daxini presents a blueprint for streaming data architectures and a review of desirable features of a streaming engine. He also talks about streaming application patterns and anti-patterns, and use cases and concrete examples using Apache Flink. Filmed at qconsf.com.
Monal Daxini is the Tech Lead for Stream Processing platform for business insights at Netflix. He helped build the petabyte scale Keystone pipeline running on the Flink powered platform. He introduced Flink to Netflix, and also helped define the vision for this platform. He has over 17 years of experience building scalable distributed systems.
Realtime streaming architecture in INFINARIO
About our experience with realtime analyses on never-ending stream of user events. Discuss Lambda architecture, Kappa, Apache Kafka and our own approach.
Similar to Realtime Analytics with Druid (20)
Overview of Apache Flink: the 4G of Big Data Analytics Frameworks
Overview of Apache Flink: the 4G of Big Data Analytics Frameworks
Overview of Apache Fink: the 4 G of Big Data Analytics Frameworks
Overview of Apache Fink: the 4 G of Big Data Analytics Frameworks
Overview of Apache Fink: The 4G of Big Data Analytics Frameworks
Overview of Apache Fink: The 4G of Big Data Analytics Frameworks
2024 XTREMEJ_ Building Real-time Pipelines with FLaNK_ A Case Study with Tra...
2024 XTREMEJ_ Building Real-time Pipelines with FLaNK_ A Case Study with Tra...
Neo4j Graph Platform Overview, Kurt Freytag, Neo4j
Neo4j Graph Platform Overview, Kurt Freytag, Neo4j
Five Fabulous Sinks for Your Kafka Data. #3 will surprise you! (Rachel Pedres...
Five Fabulous Sinks for Your Kafka Data. #3 will surprise you! (Rachel Pedres...
[Hackersuli] Élő szövet a fémvázon: Python és gépi tanulás a Zeek platformon
[Hackersuli] Élő szövet a fémvázon: Python és gépi tanulás a Zeek platformon
Streaming Data Analytics with ksqlDB and Superset | Robert Stolz, Preset
Streaming Data Analytics with ksqlDB and Superset | Robert Stolz, Preset
Microservices and Integration: what's next with Istio service mesh
Microservices and Integration: what's next with Istio service mesh
KubeCon USA 2017 brief Overview - from Kubernetes meetup Bangalore
KubeCon USA 2017 brief Overview - from Kubernetes meetup Bangalore
Recently uploaded
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...
Large Language Models (LLMs) are currently the center of attention in the tech world, particularly for their potential to advance research. In this presentation, we'll explore a straightforward and effective method for quickly initiating inference runs on supercomputers using the vLLM tool with Globus Compute, specifically on the Polaris system at ALCF. We'll begin by briefly discussing the popularity and applications of LLMs in various fields. Following this, we will introduce the vLLM tool, and explain how it integrates with Globus Compute to efficiently manage LLM operations on Polaris. Attendees will learn the practical aspects of setting up and remotely triggering LLMs from local machines, focusing on ease of use and efficiency. This talk is ideal for researchers and practitioners looking to leverage the power of LLMs in their work, offering a clear guide to harnessing supercomputing resources for quick and effective LLM inference.
A Sighting of filterA in Typelevel Rite of Passage
Slide deck home: https://fpilluminated.com/deck/220
Globus Compute Introduction - GlobusWorld 2024
We describe the deployment and use of Globus Compute for remote computation. This content is aimed at researchers who wish to compute on remote resources using a unified programming interface, as well as system administrators who will deploy and operate Globus Compute services on their research computing infrastructure.
First Steps with Globus Compute Multi-User Endpoints
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
👉👉 Click Here To Get More Info 👇👇
https://sumonreview.com/ai-pilot-review/
AI Pilot Review: Key Features
✅Deploy AI expert bots in Any Niche With Just A Click
✅With one keyword, generate complete funnels, websites, landing pages, and more.
✅More than 85 AI features are included in the AI pilot.
✅No setup or configuration; use your voice (like Siri) to do whatever you want.
✅You Can Use AI Pilot To Create your version of AI Pilot And Charge People For It…
✅ZERO Manual Work With AI Pilot. Never write, Design, Or Code Again.
✅ZERO Limits On Features Or Usages
✅Use Our AI-powered Traffic To Get Hundreds Of Customers
✅No Complicated Setup: Get Up And Running In 2 Minutes
✅99.99% Up-Time Guaranteed
✅30 Days Money-Back Guarantee
✅ZERO Upfront Cost
See My Other Reviews Article:
(1) TubeTrivia AI Review: https://sumonreview.com/tubetrivia-ai-review
(2) SocioWave Review: https://sumonreview.com/sociowave-review
(3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review
(4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review
APIs for Browser Automation (MoT Meetup 2024)
APIs for Browser Automation:
Selenium, Cypress, Puppeteer, and Playwright. Ministry of Testing Athens. Meetup on the Beach. 30 May 2024.
Essentials of Automations: The Art of Triggers and Actions in FME
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I didn't get rich from it but it did have 63K downloads (powered possible tens of thousands of websites).
Enterprise Software Development with No Code Solutions.pptx
In the ever-evolving landscape of technology, enterprise software development is undergoing a significant transformation. Traditional coding methods are being challenged by innovative no-code solutions, which promise to streamline and democratize the software development process.
This shift is particularly impactful for enterprises, which require robust, scalable, and efficient software to manage their operations. In this article, we will explore the various facets of enterprise software development with no-code solutions, examining their benefits, challenges, and the future potential they hold.
GraphSummit Paris - The art of the possible with Graph Technology
Sudhir Hasbe, Chief Product Officer, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Prosigns: Transforming Business with Tailored Technology Solutions
Unlocking Business Potential: Tailored Technology Solutions by Prosigns
Discover how Prosigns, a leading technology solutions provider, partners with businesses to drive innovation and success. Our presentation showcases our comprehensive range of services, including custom software development, web and mobile app development, AI & ML solutions, blockchain integration, DevOps services, and Microsoft Dynamics 365 support.
Custom Software Development: Prosigns specializes in creating bespoke software solutions that cater to your unique business needs. Our team of experts works closely with you to understand your requirements and deliver tailor-made software that enhances efficiency and drives growth.
Web and Mobile App Development: From responsive websites to intuitive mobile applications, Prosigns develops cutting-edge solutions that engage users and deliver seamless experiences across devices.
AI & ML Solutions: Harnessing the power of Artificial Intelligence and Machine Learning, Prosigns provides smart solutions that automate processes, provide valuable insights, and drive informed decision-making.
Blockchain Integration: Prosigns offers comprehensive blockchain solutions, including development, integration, and consulting services, enabling businesses to leverage blockchain technology for enhanced security, transparency, and efficiency.
DevOps Services: Prosigns' DevOps services streamline development and operations processes, ensuring faster and more reliable software delivery through automation and continuous integration.
Microsoft Dynamics 365 Support: Prosigns provides comprehensive support and maintenance services for Microsoft Dynamics 365, ensuring your system is always up-to-date, secure, and running smoothly.
Learn how our collaborative approach and dedication to excellence help businesses achieve their goals and stay ahead in today's digital landscape. From concept to deployment, Prosigns is your trusted partner for transforming ideas into reality and unlocking the full potential of your business.
Join us on a journey of innovation and growth. Let's partner for success with Prosigns.
Understanding Globus Data Transfers with NetSage
NetSage is an open privacy-aware network measurement, analysis, and visualization service designed to help end-users visualize and reason about large data transfers. NetSage traditionally has used a combination of passive measurements, including SNMP and flow data, as well as active measurements, mainly perfSONAR, to provide longitudinal network performance data visualization. It has been deployed by dozens of networks world wide, and is supported domestically by the Engagement and Performance Operations Center (EPOC), NSF #2328479. We have recently expanded the NetSage data sources to include logs for Globus data transfers, following the same privacy-preserving approach as for Flow data. Using the logs for the Texas Advanced Computing Center (TACC) as an example, this talk will walk through several different example use cases that NetSage can answer, including: Who is using Globus to share data with my institution, and what kind of performance are they able to achieve? How many transfers has Globus supported for us? Which sites are we sharing the most data with, and how is that changing over time? How is my site using Globus to move data internally, and what kind of performance do we see for those transfers? What percentage of data transfers at my institution used Globus, and how did the overall data transfer performance compare to the Globus users?
BoxLang: Review our Visionary Licenses of 2024
Unleash Unlimited Potential with One-Time Purchase
BoxLang is more than just a language; it's a community. By choosing a Visionary License, you're not just investing in your success, you're actively contributing to the ongoing development and support of BoxLang.
Lecture 1 Introduction to games development
Software engineering
Game Development
unity Game Development
Navigating the Metaverse: A Journey into Virtual Evolution"
Join us for an exploration of the Metaverse's evolution, where innovation meets imagination. Discover new dimensions of virtual events, engage with thought-provoking discussions, and witness the transformative power of digital realms."
May Marketo Masterclass, London MUG May 22 2024.pdf
Can't make Adobe Summit in Vegas? No sweat because the EMEA Marketo Engage Champions are coming to London to share their Summit sessions, insights and more!
This is a MUG with a twist you don't want to miss.
Quarkus Hidden and Forbidden Extensions
Quarkus has a vast extension ecosystem and is known for its subsonic and subatomic feature set. Some of these features are not as well known, and some extensions are less talked about, but that does not make them less interesting - quite the opposite.
Come join this talk to see some tips and tricks for using Quarkus and some of the lesser known features, extensions and development techniques.
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...informapgpstrackings
Keep tabs on your field staff effortlessly with Informap Technology Centre LLC. Real-time tracking, task assignment, and smart features for efficient management. Request a live demo today!
For more details, visit us : https://informapuae.com/field-staff-tracking/Introduction to Pygame (Lecture 7 Python Game Development)
Introduction to Pygame (Lecture 7 Python Game Development)
Recently uploaded (20)
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...
A Sighting of filterA in Typelevel Rite of Passage
A Sighting of filterA in Typelevel Rite of Passage
First Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User Endpoints
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...
Enterprise Software Development with No Code Solutions.pptx
Enterprise Software Development with No Code Solutions.pptx
GraphSummit Paris - The art of the possible with Graph Technology
GraphSummit Paris - The art of the possible with Graph Technology
Prosigns: Transforming Business with Tailored Technology Solutions
Prosigns: Transforming Business with Tailored Technology Solutions
Navigating the Metaverse: A Journey into Virtual Evolution"
Navigating the Metaverse: A Journey into Virtual Evolution"
May Marketo Masterclass, London MUG May 22 2024.pdf
May Marketo Masterclass, London MUG May 22 2024.pdf
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...
Introduction to Pygame (Lecture 7 Python Game Development)
Introduction to Pygame (Lecture 7 Python Game Development)
Realtime Analytics with Druid
- 1. 2017.06.08 Realtime Analytics (aka. LeanStream) with Druid KT NexR - Seungwoo Han / bucky.han@nexr.com
- 2. WHO AM I • Startup Build Fail x 2 • KT NexR (feat. 믹서기) • LeanStream 개발
- 3. Introduction • History • Realtime Analytics Flow • Query Spec • Used Query • Realtime Analytics in Lean Stream Demo
- 4. History
- 5. History NiFi1.0 Kafka0.9.0 SchemaRegistry1.2 Phoenix4.9.0/Hbase1.2.3 Druid0.9.0/Pivot Lean Stream Web 1.4 MySql/MariaDB/etc Spark 1.6.2 Yarn 2.7.3 HDFS 2.7.3 Zookeeper 3.4.6 Lean App 1.4
- 6. History
- 7. History
- 8. History NiFi1.0 Kafka0.9.0 SchemaRegistry1.2 Phoenix4.9.0/Hbase1.2.3 Druid0.9.0/Pivot Lean Stream Web 1.4 MySql/MariaDB/etc Spark 1.6.2 Yarn 2.7.3 HDFS 2.7.3 Zookeeper 3.4.6 Lean App 1.4 Druid0.9.0/RealtimeAnalytics
- 9. Realtime Analytics Flow Knockout JQuery request response Serialize as Data Generate JSON Query Queries Front Backend Echart
- 10. Front Backend Realtime Analytics Flow
- 11. Aggregation Query Spec • Data Source • Interval • Dimensions • Filters • Aggregations • Granularity • Limit • 데이터가 있는 저장소에서 • 원하는 시간에 • 원하는 컬럼값을 • 필터링할건 하고 • Dimension의 Metric값을 연산하여 • 어떤 시간(sec,min,hour,day,etc)으로 쪼개서 • 몇개를 조회할것인가!
- 12. Used Query
- 13. Used Query •Time(__timestamp) only grouping •One string dimension grouping Generally Timeseries , TopN > GroupBy
- 14. Used Query
- 15. Realtime Analytics in Lean Stream Dimension : USER_ID, USER_NAME …. Metric : count 가상의 실시간 사용자 통화데이터
- 16. Realtime Analytics in Lean Stream Play Realtime Analytics
- 17. Chart Type
- 18. Chart Type
- 19. Chart Type
- 20. Chart Type
- 21. Chart Type
- 22. Chart Type
- 23. Q&A
- 24. One More Thing
Editor's Notes
- 1
- 4
- 5
- 6
- 7
- 8