SlideShare a Scribd company logo
@apachepinot | @KishoreBytes
Look how easy it is to go from events,
to User-facing Realtime Analytics!
Using Apache Kafka and Apache Pinot
@apachepinot | @KishoreBytes
About Us
Neha Pawar
Engineer, Stealth Startup
Apache Pinot Committer
Tim Berglund
Developer Advocate
Confluent
@apachepinot | @KishoreBytes
User-facing Realtime
Analytics
Analytics for the ALL
Analytics of the NOW
@apachepinot | @KishoreBytes
User-facing Realtime
Analytics
Analytics for ALL
end-users
Queries triggered
by Apps
Personalized
analytics
@apachepinot | @KishoreBytes Time
Value
Of
Data
User-facing Realtime
Analytics
Analytics of the
NOW
@apachepinot | @KishoreBytes
Who Viewed My Profile
Seunghyun Lee
Senior Software Engineer
LinkedIn
Chinmay Soman
Founding Engineer
Total users 700 Million+
QPS 1000s
Latency SLA < 100 ms p99th
Freshness Seconds
@apachepinot | @KishoreBytes
UberEats Restaurant Manager
● Identify surge in realtime
● Detect missed/inaccurate
orders in realtime
Total users 500000+
QPS 100s
Latency SLA < 100 ms p99th
Freshness Seconds - Minutes
@apachepinot | @KishoreBytes
Challenges for the underlying system
User-facing
Realtime
Analytics System
Large Volume &
Velocity of Data
Realtime
Ingestion
1000s of QPS
Milliseconds
Latency
Seconds
Freshness
High
Dimensionality
Scalable
@apachepinot | @KishoreBytes
Kafka: The perfect solution for the events capturing part
Velocity of
ingestion
Realtime
Ingestion
Seconds
Freshness
High
Dimensionality
Scalable
@apachepinot | @KishoreBytes
Apache Kafka
@apachepinot | @KishoreBytes
Topics
@apachepinot | @KishoreBytes
Partitioning
@apachepinot | @KishoreBytes
Partitions Assignment
@apachepinot | @KishoreBytes
Producers
@apachepinot | @KishoreBytes
Consumers
@apachepinot | @KishoreBytes
How to solve low-latency high-throughput analytics part?
Large Volume &
Velocity of Data
Realtime
Ingestion
1000s of QPS
Milliseconds
Latency
Seconds
Freshness
High
Dimensionality
Scalable
@apachepinot | @KishoreBytes
Need a specialized analytics database that can..
Ingest from
Kafka & serve
real-time data
Handle high
event rate from
Kafka and
scale with
Kafka
Provide ultra
low latency at
high queries
per second
Handle dynamic query
patterns on highly
dimensional data w/o
exploding storage
1 2 3 4
@apachepinot | @KishoreBytes
Options?
Spark SQL
Presto
Big Query
Druid
Elastic Search
Kylin
KV Store
Latency
Flexibility
low
high
low
high
@apachepinot | @KishoreBytes
Introducing Apache Pinot
@apachepinot | @KishoreBytes
Apache Pinot Community
Slack Users
1100+
Companies
50+
Join our growing community
https://communityinviter.com/apps/apache-pinot/apache-pinot
Events/sec
1M+
Peak QPS
170K+
Query latency
ms
@apachepinot | @KishoreBytes
Need a specialized analytics database that can..
Ingest from
Kafka & serve
realtime data
Handle high
event rate from
Kafka and
scale with
Kafka
Provide ultra
low latency at
high queries
per second
Handle dynamic query
patterns on highly
dimensional data w/o
exploding storage
1 2 3 4
@apachepinot | @KishoreBytes
Apache Pinot Architecture
Pinot
Controller Zookeeper
Server 2
Server 1
Pinot Servers
Server 3
Pinot
Broker
Pinot
Brokers
Queries
Scatter - gather
Consuming,
indexing, serving
@apachepinot | @KishoreBytes
Server 3
Server 2
Server 1
p0 -> Server 1
p1 -> Server 2
p2 -> Server 3
p3 -> Server 1
Pinot
Broker
Pinot
Brokers
Pinot Servers
Pinot
Controller Zookeeper
Pinot Realtime Ingestion
Queries
Consuming,
indexing, serving
Partition -> Pinot Server
@apachepinot | @KishoreBytes
Server 3
Server 2
Server 1
p0 -> Server 1, CONSUMING
p1 -> Server 2, CONSUMING
p2 -> Server 3, CONSUMING
p3 -> Server 1, CONSUMING
Pinot
Broker
Pinot
Brokers
Pinot Servers
Pinot
Controller Zookeeper
Pinot Realtime Ingestion
Queries
Consuming,
indexing, serving
State
@apachepinot | @KishoreBytes
Server 3
Server 2
Server 1
p0 -> Server 1, CONSUMING, 102
p1 -> Server 2, CONSUMING, 120
p2 -> Server 3, CONSUMING, 105
p3 -> Server 1, CONSUMING, 100
Pinot
Broker
Pinot
Brokers
Pinot Servers
Pinot
Controller Zookeeper
Pinot Realtime Ingestion
Queries
Consuming,
indexing, serving
Start Offset
Kafka
Consumers
@apachepinot | @KishoreBytes
Server 3
Server 2
Server 1
p0 -> Server 1, DONE, 102, 300
p1 -> Server 2, CONSUMING, 120
p2 -> Server 3, CONSUMING, 105
p3 -> Server 1, CONSUMING, 100
Pinot
Broker
Pinot
Brokers
Pinot Servers
Pinot
Controller Zookeeper
Pinot Realtime Ingestion
Queries
Consuming,
indexing, serving
@apachepinot | @KishoreBytes
Server 3
Server 2
Server 1
p0 -> Server 1, DONE, 102, 300
p1 -> Server 2, CONSUMING, 120
p2 -> Server 3, CONSUMING, 105
p3 -> Server 1, CONSUMING, 100
p0 -> Server 1, CONSUMING, 300
Pinot
Broker
Pinot
Brokers
Pinot Servers
Pinot
Controller Zookeeper
Pinot Realtime Ingestion
Queries
Consuming,
indexing, serving
@apachepinot | @KishoreBytes
Demo
● Emit events into a Kafka topic
● Create Pinot Schema and Table
● Query!
@apachepinot | @KishoreBytes
Need a specialized analytics database that can..
Ingest from
Kafka & serve
realtime data
Handle high
event rate from
Kafka and
scale with
Kafka
Provide ultra
low latency at
high queries
per second
Handle dynamic query
patterns on highly
dimensional data w/o
exploding storage
1 2 3 4
@apachepinot | @KishoreBytes
Indexing
User-facing
Realtime Analytics
Events
Indexes
Sorted
Inverted
Range
Star-tree
JSON
Geospatial
Text
@apachepinot | @KishoreBytes
Need a specialized analytics database that can..
Ingest from
Kafka & serve
realtime data
Handle high
event rate from
Kafka and
scale with
Kafka
Provide ultra
low latency at
high queries
per second
Handle dynamic query
patterns on highly
dimensional data w/o
exploding storage
1 2 3 4
@apachepinot | @KishoreBytes
Star-tree Indexing
country browser device os clicks
us chrome ... ... ...
ca firefox ... ... ...
jp ie ... ... ...
us firefox ... ... ...
ca ie ... ... ...
… … ... ... ...
select count(*) from X
where country = us and
browser = chrome
Country
Browser
Star-Tree Index
US CA
IE C IE C
@apachepinot | @KishoreBytes
Need a specialized analytics database that can..
Ingest from
Kafka & serve
realtime data
Handle high
event rate from
Kafka and
scale with
Kafka
Provide ultra
low latency at
high queries
per second
Handle dynamic query
patterns on highly
dimensional data w/o
exploding storage
1 2 3 4
@apachepinot | @KishoreBytes
Horizontal scaling by adding new Kafka brokers + Pinot servers
Apps
Kafka
Cluster
Pinot
Cluster
Server
Brokers
Producers
Kafka Consumers
Events
@apachepinot | @KishoreBytes
Kafka Producer - Reprise
@apachepinot | @KishoreBytes
Partition aware routing
Realtime Analytics
Events
Partitioned
stream
murmur3(memberId)
% numPartitions
Partition-aware
segments
Seg0 - partition #0
Seg1 - partition #1
Seg2 - partition #2
Seg3 - partition #0
Seg4 - partition #1
Seg5 - partition #2
Partition aware
query routing
SELECT SUM(metric)
WHERE memberId = m289
Choose a column as
partitioning key
Person
{
"memberId" : "m809",
"name" : "adam",
"age" : 30,
"addresses" : [ {
"number" : 1,
"street" : "main st",
"country" : "us"
}]
}
murmur3(m289) % numPartitions
@apachepinot | @KishoreBytes
User-facing Realtime Analytics System
Large Volume &
Velocity of Data
Realtime
Ingestion
1000s of QPS
Milliseconds
Latency
Seconds
Freshness
High
Dimensionality
Scalable
@apachepinot | @KishoreBytes
Takeaway
● Kafka is the key data infrastructure for event-streamed systems
● Kafka has good analytics solutions for operationalized streaming queries
● Pinot is purpose-built for ultra-low latency analytics, at high-throughput
● Pinot is a great solution for user-facing real-time analytics
● It is very easy to go from events in Kafka to analytics in Pinot
● Kafka + Pinot is the perfect combination for user-facing real-time analytics

More Related Content

What's hot

Kafka for Real-Time Event Processing in Serverless Environments
Kafka for Real-Time Event Processing in Serverless EnvironmentsKafka for Real-Time Event Processing in Serverless Environments
Kafka for Real-Time Event Processing in Serverless Environments
confluent
 
Everything you ever needed to know about Kafka on Kubernetes but were afraid ...
Everything you ever needed to know about Kafka on Kubernetes but were afraid ...Everything you ever needed to know about Kafka on Kubernetes but were afraid ...
Everything you ever needed to know about Kafka on Kubernetes but were afraid ...
HostedbyConfluent
 
Deploying Confluent Platform for Production
Deploying Confluent Platform for ProductionDeploying Confluent Platform for Production
Deploying Confluent Platform for Production
confluent
 
Flexible Authentication Strategies with SASL/OAUTHBEARER (Michael Kaminski, T...
Flexible Authentication Strategies with SASL/OAUTHBEARER (Michael Kaminski, T...Flexible Authentication Strategies with SASL/OAUTHBEARER (Michael Kaminski, T...
Flexible Authentication Strategies with SASL/OAUTHBEARER (Michael Kaminski, T...
confluent
 
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
HostedbyConfluent
 
Common issues with Apache Kafka® Producer
Common issues with Apache Kafka® ProducerCommon issues with Apache Kafka® Producer
Common issues with Apache Kafka® Producer
confluent
 
Stream Processing with Apache Kafka and .NET
Stream Processing with Apache Kafka and .NETStream Processing with Apache Kafka and .NET
Stream Processing with Apache Kafka and .NET
confluent
 
Securing the Message Bus with Kafka Streams | Paul Otto and Ryan Salcido, Raf...
Securing the Message Bus with Kafka Streams | Paul Otto and Ryan Salcido, Raf...Securing the Message Bus with Kafka Streams | Paul Otto and Ryan Salcido, Raf...
Securing the Message Bus with Kafka Streams | Paul Otto and Ryan Salcido, Raf...
HostedbyConfluent
 
Kafka Summit SF 2017 - Database Streaming at WePay
Kafka Summit SF 2017 - Database Streaming at WePayKafka Summit SF 2017 - Database Streaming at WePay
Kafka Summit SF 2017 - Database Streaming at WePay
confluent
 
Tradeoffs in Distributed Systems Design: Is Kafka The Best? (Ben Stopford and...
Tradeoffs in Distributed Systems Design: Is Kafka The Best? (Ben Stopford and...Tradeoffs in Distributed Systems Design: Is Kafka The Best? (Ben Stopford and...
Tradeoffs in Distributed Systems Design: Is Kafka The Best? (Ben Stopford and...
HostedbyConfluent
 
Tips & Tricks for Apache Kafka®
Tips & Tricks for Apache Kafka®Tips & Tricks for Apache Kafka®
Tips & Tricks for Apache Kafka®
confluent
 
Can Kafka Handle a Lyft Ride? (Andrey Falko & Can Cecen, Lyft) Kafka Summit 2020
Can Kafka Handle a Lyft Ride? (Andrey Falko & Can Cecen, Lyft) Kafka Summit 2020Can Kafka Handle a Lyft Ride? (Andrey Falko & Can Cecen, Lyft) Kafka Summit 2020
Can Kafka Handle a Lyft Ride? (Andrey Falko & Can Cecen, Lyft) Kafka Summit 2020
HostedbyConfluent
 
Kafka Summit SF 2017 - Real-Time Document Rankings with Kafka Streams
Kafka Summit SF 2017 - Real-Time Document Rankings with Kafka StreamsKafka Summit SF 2017 - Real-Time Document Rankings with Kafka Streams
Kafka Summit SF 2017 - Real-Time Document Rankings with Kafka Streams
confluent
 
Confluent On Azure: Why you should add Confluent to your Azure toolkit | Alic...
Confluent On Azure: Why you should add Confluent to your Azure toolkit | Alic...Confluent On Azure: Why you should add Confluent to your Azure toolkit | Alic...
Confluent On Azure: Why you should add Confluent to your Azure toolkit | Alic...
HostedbyConfluent
 
Building an Event-oriented Data Platform with Kafka, Eric Sammer
Building an Event-oriented Data Platform with Kafka, Eric Sammer Building an Event-oriented Data Platform with Kafka, Eric Sammer
Building an Event-oriented Data Platform with Kafka, Eric Sammer
confluent
 
Introducing Kafka's Streams API
Introducing Kafka's Streams APIIntroducing Kafka's Streams API
Introducing Kafka's Streams API
confluent
 
Building Stream Processing Applications with Apache Kafka Using KSQL (Robin M...
Building Stream Processing Applications with Apache Kafka Using KSQL (Robin M...Building Stream Processing Applications with Apache Kafka Using KSQL (Robin M...
Building Stream Processing Applications with Apache Kafka Using KSQL (Robin M...
confluent
 
The New Way of Configuring Grace Periods for Windowed Operations in Kafka Str...
The New Way of Configuring Grace Periods for Windowed Operations in Kafka Str...The New Way of Configuring Grace Periods for Windowed Operations in Kafka Str...
The New Way of Configuring Grace Periods for Windowed Operations in Kafka Str...
HostedbyConfluent
 
Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019
Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019 Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019
Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019
confluent
 
URP? Excuse You! The Three Metrics You Have to Know
URP? Excuse You! The Three Metrics You Have to Know URP? Excuse You! The Three Metrics You Have to Know
URP? Excuse You! The Three Metrics You Have to Know
confluent
 

What's hot (20)

Kafka for Real-Time Event Processing in Serverless Environments
Kafka for Real-Time Event Processing in Serverless EnvironmentsKafka for Real-Time Event Processing in Serverless Environments
Kafka for Real-Time Event Processing in Serverless Environments
 
Everything you ever needed to know about Kafka on Kubernetes but were afraid ...
Everything you ever needed to know about Kafka on Kubernetes but were afraid ...Everything you ever needed to know about Kafka on Kubernetes but were afraid ...
Everything you ever needed to know about Kafka on Kubernetes but were afraid ...
 
Deploying Confluent Platform for Production
Deploying Confluent Platform for ProductionDeploying Confluent Platform for Production
Deploying Confluent Platform for Production
 
Flexible Authentication Strategies with SASL/OAUTHBEARER (Michael Kaminski, T...
Flexible Authentication Strategies with SASL/OAUTHBEARER (Michael Kaminski, T...Flexible Authentication Strategies with SASL/OAUTHBEARER (Michael Kaminski, T...
Flexible Authentication Strategies with SASL/OAUTHBEARER (Michael Kaminski, T...
 
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
 
Common issues with Apache Kafka® Producer
Common issues with Apache Kafka® ProducerCommon issues with Apache Kafka® Producer
Common issues with Apache Kafka® Producer
 
Stream Processing with Apache Kafka and .NET
Stream Processing with Apache Kafka and .NETStream Processing with Apache Kafka and .NET
Stream Processing with Apache Kafka and .NET
 
Securing the Message Bus with Kafka Streams | Paul Otto and Ryan Salcido, Raf...
Securing the Message Bus with Kafka Streams | Paul Otto and Ryan Salcido, Raf...Securing the Message Bus with Kafka Streams | Paul Otto and Ryan Salcido, Raf...
Securing the Message Bus with Kafka Streams | Paul Otto and Ryan Salcido, Raf...
 
Kafka Summit SF 2017 - Database Streaming at WePay
Kafka Summit SF 2017 - Database Streaming at WePayKafka Summit SF 2017 - Database Streaming at WePay
Kafka Summit SF 2017 - Database Streaming at WePay
 
Tradeoffs in Distributed Systems Design: Is Kafka The Best? (Ben Stopford and...
Tradeoffs in Distributed Systems Design: Is Kafka The Best? (Ben Stopford and...Tradeoffs in Distributed Systems Design: Is Kafka The Best? (Ben Stopford and...
Tradeoffs in Distributed Systems Design: Is Kafka The Best? (Ben Stopford and...
 
Tips & Tricks for Apache Kafka®
Tips & Tricks for Apache Kafka®Tips & Tricks for Apache Kafka®
Tips & Tricks for Apache Kafka®
 
Can Kafka Handle a Lyft Ride? (Andrey Falko & Can Cecen, Lyft) Kafka Summit 2020
Can Kafka Handle a Lyft Ride? (Andrey Falko & Can Cecen, Lyft) Kafka Summit 2020Can Kafka Handle a Lyft Ride? (Andrey Falko & Can Cecen, Lyft) Kafka Summit 2020
Can Kafka Handle a Lyft Ride? (Andrey Falko & Can Cecen, Lyft) Kafka Summit 2020
 
Kafka Summit SF 2017 - Real-Time Document Rankings with Kafka Streams
Kafka Summit SF 2017 - Real-Time Document Rankings with Kafka StreamsKafka Summit SF 2017 - Real-Time Document Rankings with Kafka Streams
Kafka Summit SF 2017 - Real-Time Document Rankings with Kafka Streams
 
Confluent On Azure: Why you should add Confluent to your Azure toolkit | Alic...
Confluent On Azure: Why you should add Confluent to your Azure toolkit | Alic...Confluent On Azure: Why you should add Confluent to your Azure toolkit | Alic...
Confluent On Azure: Why you should add Confluent to your Azure toolkit | Alic...
 
Building an Event-oriented Data Platform with Kafka, Eric Sammer
Building an Event-oriented Data Platform with Kafka, Eric Sammer Building an Event-oriented Data Platform with Kafka, Eric Sammer
Building an Event-oriented Data Platform with Kafka, Eric Sammer
 
Introducing Kafka's Streams API
Introducing Kafka's Streams APIIntroducing Kafka's Streams API
Introducing Kafka's Streams API
 
Building Stream Processing Applications with Apache Kafka Using KSQL (Robin M...
Building Stream Processing Applications with Apache Kafka Using KSQL (Robin M...Building Stream Processing Applications with Apache Kafka Using KSQL (Robin M...
Building Stream Processing Applications with Apache Kafka Using KSQL (Robin M...
 
The New Way of Configuring Grace Periods for Windowed Operations in Kafka Str...
The New Way of Configuring Grace Periods for Windowed Operations in Kafka Str...The New Way of Configuring Grace Periods for Windowed Operations in Kafka Str...
The New Way of Configuring Grace Periods for Windowed Operations in Kafka Str...
 
Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019
Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019 Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019
Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019
 
URP? Excuse You! The Three Metrics You Have to Know
URP? Excuse You! The Three Metrics You Have to Know URP? Excuse You! The Three Metrics You Have to Know
URP? Excuse You! The Three Metrics You Have to Know
 

Similar to Look how easy it is to go from events to blazing-fast analytics! | Neha Pawar, Stealth Startup

Apache Pinot Case Study: Building Distributed Analytics Systems Using Apache ...
Apache Pinot Case Study: Building Distributed Analytics Systems Using Apache ...Apache Pinot Case Study: Building Distributed Analytics Systems Using Apache ...
Apache Pinot Case Study: Building Distributed Analytics Systems Using Apache ...
HostedbyConfluent
 
Netflix Keystone SPaaS: Real-time Stream Processing as a Service - ABD320 - r...
Netflix Keystone SPaaS: Real-time Stream Processing as a Service - ABD320 - r...Netflix Keystone SPaaS: Real-time Stream Processing as a Service - ABD320 - r...
Netflix Keystone SPaaS: Real-time Stream Processing as a Service - ABD320 - r...
Amazon Web Services
 
History of Apache Pinot
History of Apache Pinot History of Apache Pinot
History of Apache Pinot
Kishore Gopalakrishna
 
Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...
Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...
Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...
Natan Silnitsky
 
8 Lessons Learned from Using Kafka in 1500 microservices - confluent streamin...
8 Lessons Learned from Using Kafka in 1500 microservices - confluent streamin...8 Lessons Learned from Using Kafka in 1500 microservices - confluent streamin...
8 Lessons Learned from Using Kafka in 1500 microservices - confluent streamin...
Natan Silnitsky
 
Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...
Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...
Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...
Natan Silnitsky
 
8 Lessons Learned from Using Kafka in 1000 Scala microservices - Scale by the...
8 Lessons Learned from Using Kafka in 1000 Scala microservices - Scale by the...8 Lessons Learned from Using Kafka in 1000 Scala microservices - Scale by the...
8 Lessons Learned from Using Kafka in 1000 Scala microservices - Scale by the...
Natan Silnitsky
 
AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017
AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017
AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017
Monal Daxini
 
Resilient Event Driven Systems With Kafka
Resilient Event Driven Systems With KafkaResilient Event Driven Systems With Kafka
Resilient Event Driven Systems With Kafka
Iccha Sethi
 
Monitoring and tuning your chef server - chef conf talk
Monitoring and tuning your chef server - chef conf talk Monitoring and tuning your chef server - chef conf talk
Monitoring and tuning your chef server - chef conf talk
Andrew DuFour
 
Keystone - ApacheCon 2016
Keystone - ApacheCon 2016Keystone - ApacheCon 2016
Keystone - ApacheCon 2016
Peter Bakas
 
10 Lessons Learned from using Kafka with 1000 microservices - java global summit
10 Lessons Learned from using Kafka with 1000 microservices - java global summit10 Lessons Learned from using Kafka with 1000 microservices - java global summit
10 Lessons Learned from using Kafka with 1000 microservices - java global summit
Natan Silnitsky
 
Pinot: Enabling Real-time Analytics Applications @ LinkedIn's Scale
Pinot: Enabling Real-time Analytics Applications @ LinkedIn's ScalePinot: Enabling Real-time Analytics Applications @ LinkedIn's Scale
Pinot: Enabling Real-time Analytics Applications @ LinkedIn's Scale
Seunghyun Lee
 
10 Lessons Learned from using Kafka in 1000 microservices - ScalaUA
10 Lessons Learned from using Kafka in 1000 microservices - ScalaUA10 Lessons Learned from using Kafka in 1000 microservices - ScalaUA
10 Lessons Learned from using Kafka in 1000 microservices - ScalaUA
Natan Silnitsky
 
찾아가는 AWS 세미나(구로,가산,판교) - AWS 기반 빅데이터 활용 방법 (김일호 솔루션즈 아키텍트)
찾아가는 AWS 세미나(구로,가산,판교) - AWS 기반 빅데이터 활용 방법 (김일호 솔루션즈 아키텍트)찾아가는 AWS 세미나(구로,가산,판교) - AWS 기반 빅데이터 활용 방법 (김일호 솔루션즈 아키텍트)
찾아가는 AWS 세미나(구로,가산,판교) - AWS 기반 빅데이터 활용 방법 (김일호 솔루션즈 아키텍트)
Amazon Web Services Korea
 
Realtime Analytics on AWS
Realtime Analytics on AWSRealtime Analytics on AWS
Realtime Analytics on AWS
Sungmin Kim
 
Apache StreamPipes – Flexible Industrial IoT Management
Apache StreamPipes – Flexible Industrial IoT ManagementApache StreamPipes – Flexible Industrial IoT Management
Apache StreamPipes – Flexible Industrial IoT Management
Apache StreamPipes
 
Polyglot, Fault Tolerant Event-Driven Programming with Kafka, Kubernetes and ...
Polyglot, Fault Tolerant Event-Driven Programming with Kafka, Kubernetes and ...Polyglot, Fault Tolerant Event-Driven Programming with Kafka, Kubernetes and ...
Polyglot, Fault Tolerant Event-Driven Programming with Kafka, Kubernetes and ...
Natan Silnitsky
 
Pinot: Near Realtime Analytics @ Uber
Pinot: Near Realtime Analytics @ UberPinot: Near Realtime Analytics @ Uber
Pinot: Near Realtime Analytics @ Uber
Xiang Fu
 
Amazon Kinesis
Amazon KinesisAmazon Kinesis
Amazon Kinesis
Amazon Web Services
 

Similar to Look how easy it is to go from events to blazing-fast analytics! | Neha Pawar, Stealth Startup (20)

Apache Pinot Case Study: Building Distributed Analytics Systems Using Apache ...
Apache Pinot Case Study: Building Distributed Analytics Systems Using Apache ...Apache Pinot Case Study: Building Distributed Analytics Systems Using Apache ...
Apache Pinot Case Study: Building Distributed Analytics Systems Using Apache ...
 
Netflix Keystone SPaaS: Real-time Stream Processing as a Service - ABD320 - r...
Netflix Keystone SPaaS: Real-time Stream Processing as a Service - ABD320 - r...Netflix Keystone SPaaS: Real-time Stream Processing as a Service - ABD320 - r...
Netflix Keystone SPaaS: Real-time Stream Processing as a Service - ABD320 - r...
 
History of Apache Pinot
History of Apache Pinot History of Apache Pinot
History of Apache Pinot
 
Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...
Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...
Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...
 
8 Lessons Learned from Using Kafka in 1500 microservices - confluent streamin...
8 Lessons Learned from Using Kafka in 1500 microservices - confluent streamin...8 Lessons Learned from Using Kafka in 1500 microservices - confluent streamin...
8 Lessons Learned from Using Kafka in 1500 microservices - confluent streamin...
 
Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...
Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...
Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...
 
8 Lessons Learned from Using Kafka in 1000 Scala microservices - Scale by the...
8 Lessons Learned from Using Kafka in 1000 Scala microservices - Scale by the...8 Lessons Learned from Using Kafka in 1000 Scala microservices - Scale by the...
8 Lessons Learned from Using Kafka in 1000 Scala microservices - Scale by the...
 
AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017
AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017
AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017
 
Resilient Event Driven Systems With Kafka
Resilient Event Driven Systems With KafkaResilient Event Driven Systems With Kafka
Resilient Event Driven Systems With Kafka
 
Monitoring and tuning your chef server - chef conf talk
Monitoring and tuning your chef server - chef conf talk Monitoring and tuning your chef server - chef conf talk
Monitoring and tuning your chef server - chef conf talk
 
Keystone - ApacheCon 2016
Keystone - ApacheCon 2016Keystone - ApacheCon 2016
Keystone - ApacheCon 2016
 
10 Lessons Learned from using Kafka with 1000 microservices - java global summit
10 Lessons Learned from using Kafka with 1000 microservices - java global summit10 Lessons Learned from using Kafka with 1000 microservices - java global summit
10 Lessons Learned from using Kafka with 1000 microservices - java global summit
 
Pinot: Enabling Real-time Analytics Applications @ LinkedIn's Scale
Pinot: Enabling Real-time Analytics Applications @ LinkedIn's ScalePinot: Enabling Real-time Analytics Applications @ LinkedIn's Scale
Pinot: Enabling Real-time Analytics Applications @ LinkedIn's Scale
 
10 Lessons Learned from using Kafka in 1000 microservices - ScalaUA
10 Lessons Learned from using Kafka in 1000 microservices - ScalaUA10 Lessons Learned from using Kafka in 1000 microservices - ScalaUA
10 Lessons Learned from using Kafka in 1000 microservices - ScalaUA
 
찾아가는 AWS 세미나(구로,가산,판교) - AWS 기반 빅데이터 활용 방법 (김일호 솔루션즈 아키텍트)
찾아가는 AWS 세미나(구로,가산,판교) - AWS 기반 빅데이터 활용 방법 (김일호 솔루션즈 아키텍트)찾아가는 AWS 세미나(구로,가산,판교) - AWS 기반 빅데이터 활용 방법 (김일호 솔루션즈 아키텍트)
찾아가는 AWS 세미나(구로,가산,판교) - AWS 기반 빅데이터 활용 방법 (김일호 솔루션즈 아키텍트)
 
Realtime Analytics on AWS
Realtime Analytics on AWSRealtime Analytics on AWS
Realtime Analytics on AWS
 
Apache StreamPipes – Flexible Industrial IoT Management
Apache StreamPipes – Flexible Industrial IoT ManagementApache StreamPipes – Flexible Industrial IoT Management
Apache StreamPipes – Flexible Industrial IoT Management
 
Polyglot, Fault Tolerant Event-Driven Programming with Kafka, Kubernetes and ...
Polyglot, Fault Tolerant Event-Driven Programming with Kafka, Kubernetes and ...Polyglot, Fault Tolerant Event-Driven Programming with Kafka, Kubernetes and ...
Polyglot, Fault Tolerant Event-Driven Programming with Kafka, Kubernetes and ...
 
Pinot: Near Realtime Analytics @ Uber
Pinot: Near Realtime Analytics @ UberPinot: Near Realtime Analytics @ Uber
Pinot: Near Realtime Analytics @ Uber
 
Amazon Kinesis
Amazon KinesisAmazon Kinesis
Amazon Kinesis
 

More from HostedbyConfluent

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
HostedbyConfluent
 
Renaming a Kafka Topic | Kafka Summit London
Renaming a Kafka Topic | Kafka Summit LondonRenaming a Kafka Topic | Kafka Summit London
Renaming a Kafka Topic | Kafka Summit London
HostedbyConfluent
 
Evolution of NRT Data Ingestion Pipeline at Trendyol
Evolution of NRT Data Ingestion Pipeline at TrendyolEvolution of NRT Data Ingestion Pipeline at Trendyol
Evolution of NRT Data Ingestion Pipeline at Trendyol
HostedbyConfluent
 
Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques
Ensuring Kafka Service Resilience: A Dive into Health-Checking TechniquesEnsuring Kafka Service Resilience: A Dive into Health-Checking Techniques
Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques
HostedbyConfluent
 
Exactly-once Stream Processing with Arroyo and Kafka
Exactly-once Stream Processing with Arroyo and KafkaExactly-once Stream Processing with Arroyo and Kafka
Exactly-once Stream Processing with Arroyo and Kafka
HostedbyConfluent
 
Fish Plays Pokemon | Kafka Summit London
Fish Plays Pokemon | Kafka Summit LondonFish Plays Pokemon | Kafka Summit London
Fish Plays Pokemon | Kafka Summit London
HostedbyConfluent
 
Tiered Storage 101 | Kafla Summit London
Tiered Storage 101 | Kafla Summit LondonTiered Storage 101 | Kafla Summit London
Tiered Storage 101 | Kafla Summit London
HostedbyConfluent
 
Building a Self-Service Stream Processing Portal: How And Why
Building a Self-Service Stream Processing Portal: How And WhyBuilding a Self-Service Stream Processing Portal: How And Why
Building a Self-Service Stream Processing Portal: How And Why
HostedbyConfluent
 
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
HostedbyConfluent
 
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
HostedbyConfluent
 
Navigating Private Network Connectivity Options for Kafka Clusters
Navigating Private Network Connectivity Options for Kafka ClustersNavigating Private Network Connectivity Options for Kafka Clusters
Navigating Private Network Connectivity Options for Kafka Clusters
HostedbyConfluent
 
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
Apache Flink: Building a Company-wide Self-service Streaming Data PlatformApache Flink: Building a Company-wide Self-service Streaming Data Platform
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
HostedbyConfluent
 
Explaining How Real-Time GenAI Works in a Noisy Pub
Explaining How Real-Time GenAI Works in a Noisy PubExplaining How Real-Time GenAI Works in a Noisy Pub
Explaining How Real-Time GenAI Works in a Noisy Pub
HostedbyConfluent
 
TL;DR Kafka Metrics | Kafka Summit London
TL;DR Kafka Metrics | Kafka Summit LondonTL;DR Kafka Metrics | Kafka Summit London
TL;DR Kafka Metrics | Kafka Summit London
HostedbyConfluent
 
A Window Into Your Kafka Streams Tasks | KSL
A Window Into Your Kafka Streams Tasks | KSLA Window Into Your Kafka Streams Tasks | KSL
A Window Into Your Kafka Streams Tasks | KSL
HostedbyConfluent
 
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
Mastering Kafka Producer Configs: A Guide to Optimizing PerformanceMastering Kafka Producer Configs: A Guide to Optimizing Performance
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
HostedbyConfluent
 
Data Contracts Management: Schema Registry and Beyond
Data Contracts Management: Schema Registry and BeyondData Contracts Management: Schema Registry and Beyond
Data Contracts Management: Schema Registry and Beyond
HostedbyConfluent
 
Code-First Approach: Crafting Efficient Flink Apps
Code-First Approach: Crafting Efficient Flink AppsCode-First Approach: Crafting Efficient Flink Apps
Code-First Approach: Crafting Efficient Flink Apps
HostedbyConfluent
 
Debezium vs. the World: An Overview of the CDC Ecosystem
Debezium vs. the World: An Overview of the CDC EcosystemDebezium vs. the World: An Overview of the CDC Ecosystem
Debezium vs. the World: An Overview of the CDC Ecosystem
HostedbyConfluent
 
Beyond Tiered Storage: Serverless Kafka with No Local Disks
Beyond Tiered Storage: Serverless Kafka with No Local DisksBeyond Tiered Storage: Serverless Kafka with No Local Disks
Beyond Tiered Storage: Serverless Kafka with No Local Disks
HostedbyConfluent
 

More from HostedbyConfluent (20)

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
 
Renaming a Kafka Topic | Kafka Summit London
Renaming a Kafka Topic | Kafka Summit LondonRenaming a Kafka Topic | Kafka Summit London
Renaming a Kafka Topic | Kafka Summit London
 
Evolution of NRT Data Ingestion Pipeline at Trendyol
Evolution of NRT Data Ingestion Pipeline at TrendyolEvolution of NRT Data Ingestion Pipeline at Trendyol
Evolution of NRT Data Ingestion Pipeline at Trendyol
 
Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques
Ensuring Kafka Service Resilience: A Dive into Health-Checking TechniquesEnsuring Kafka Service Resilience: A Dive into Health-Checking Techniques
Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques
 
Exactly-once Stream Processing with Arroyo and Kafka
Exactly-once Stream Processing with Arroyo and KafkaExactly-once Stream Processing with Arroyo and Kafka
Exactly-once Stream Processing with Arroyo and Kafka
 
Fish Plays Pokemon | Kafka Summit London
Fish Plays Pokemon | Kafka Summit LondonFish Plays Pokemon | Kafka Summit London
Fish Plays Pokemon | Kafka Summit London
 
Tiered Storage 101 | Kafla Summit London
Tiered Storage 101 | Kafla Summit LondonTiered Storage 101 | Kafla Summit London
Tiered Storage 101 | Kafla Summit London
 
Building a Self-Service Stream Processing Portal: How And Why
Building a Self-Service Stream Processing Portal: How And WhyBuilding a Self-Service Stream Processing Portal: How And Why
Building a Self-Service Stream Processing Portal: How And Why
 
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
 
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
 
Navigating Private Network Connectivity Options for Kafka Clusters
Navigating Private Network Connectivity Options for Kafka ClustersNavigating Private Network Connectivity Options for Kafka Clusters
Navigating Private Network Connectivity Options for Kafka Clusters
 
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
Apache Flink: Building a Company-wide Self-service Streaming Data PlatformApache Flink: Building a Company-wide Self-service Streaming Data Platform
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
 
Explaining How Real-Time GenAI Works in a Noisy Pub
Explaining How Real-Time GenAI Works in a Noisy PubExplaining How Real-Time GenAI Works in a Noisy Pub
Explaining How Real-Time GenAI Works in a Noisy Pub
 
TL;DR Kafka Metrics | Kafka Summit London
TL;DR Kafka Metrics | Kafka Summit LondonTL;DR Kafka Metrics | Kafka Summit London
TL;DR Kafka Metrics | Kafka Summit London
 
A Window Into Your Kafka Streams Tasks | KSL
A Window Into Your Kafka Streams Tasks | KSLA Window Into Your Kafka Streams Tasks | KSL
A Window Into Your Kafka Streams Tasks | KSL
 
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
Mastering Kafka Producer Configs: A Guide to Optimizing PerformanceMastering Kafka Producer Configs: A Guide to Optimizing Performance
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
 
Data Contracts Management: Schema Registry and Beyond
Data Contracts Management: Schema Registry and BeyondData Contracts Management: Schema Registry and Beyond
Data Contracts Management: Schema Registry and Beyond
 
Code-First Approach: Crafting Efficient Flink Apps
Code-First Approach: Crafting Efficient Flink AppsCode-First Approach: Crafting Efficient Flink Apps
Code-First Approach: Crafting Efficient Flink Apps
 
Debezium vs. the World: An Overview of the CDC Ecosystem
Debezium vs. the World: An Overview of the CDC EcosystemDebezium vs. the World: An Overview of the CDC Ecosystem
Debezium vs. the World: An Overview of the CDC Ecosystem
 
Beyond Tiered Storage: Serverless Kafka with No Local Disks
Beyond Tiered Storage: Serverless Kafka with No Local DisksBeyond Tiered Storage: Serverless Kafka with No Local Disks
Beyond Tiered Storage: Serverless Kafka with No Local Disks
 

Recently uploaded

Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Ramesh Iyer
 
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
Product School
 
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
KatiaHIMEUR1
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Jeffrey Haguewood
 
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
BookNet Canada
 
Knowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and backKnowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and back
Elena Simperl
 
Elevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object CalisthenicsElevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object Calisthenics
Dorra BARTAGUIZ
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
Cheryl Hung
 
The Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and SalesThe Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and Sales
Laura Byrne
 
Accelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish CachingAccelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish Caching
Thijs Feryn
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Tobias Schneck
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Product School
 
GraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge GraphGraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge Graph
Guy Korland
 
The Future of Platform Engineering
The Future of Platform EngineeringThe Future of Platform Engineering
The Future of Platform Engineering
Jemma Hussein Allen
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
Product School
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Product School
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Inflectra
 
PCI PIN Basics Webinar from the Controlcase Team
PCI PIN Basics Webinar from the Controlcase TeamPCI PIN Basics Webinar from the Controlcase Team
PCI PIN Basics Webinar from the Controlcase Team
ControlCase
 
Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
Safe Software
 
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
Product School
 

Recently uploaded (20)

Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
 
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
 
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
 
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
 
Knowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and backKnowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and back
 
Elevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object CalisthenicsElevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object Calisthenics
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
 
The Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and SalesThe Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and Sales
 
Accelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish CachingAccelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish Caching
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
 
GraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge GraphGraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge Graph
 
The Future of Platform Engineering
The Future of Platform EngineeringThe Future of Platform Engineering
The Future of Platform Engineering
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
 
PCI PIN Basics Webinar from the Controlcase Team
PCI PIN Basics Webinar from the Controlcase TeamPCI PIN Basics Webinar from the Controlcase Team
PCI PIN Basics Webinar from the Controlcase Team
 
Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
 
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
 

Look how easy it is to go from events to blazing-fast analytics! | Neha Pawar, Stealth Startup

  • 1. @apachepinot | @KishoreBytes Look how easy it is to go from events, to User-facing Realtime Analytics! Using Apache Kafka and Apache Pinot
  • 2. @apachepinot | @KishoreBytes About Us Neha Pawar Engineer, Stealth Startup Apache Pinot Committer Tim Berglund Developer Advocate Confluent
  • 3. @apachepinot | @KishoreBytes User-facing Realtime Analytics Analytics for the ALL Analytics of the NOW
  • 4. @apachepinot | @KishoreBytes User-facing Realtime Analytics Analytics for ALL end-users Queries triggered by Apps Personalized analytics
  • 5. @apachepinot | @KishoreBytes Time Value Of Data User-facing Realtime Analytics Analytics of the NOW
  • 6. @apachepinot | @KishoreBytes Who Viewed My Profile Seunghyun Lee Senior Software Engineer LinkedIn Chinmay Soman Founding Engineer Total users 700 Million+ QPS 1000s Latency SLA < 100 ms p99th Freshness Seconds
  • 7. @apachepinot | @KishoreBytes UberEats Restaurant Manager ● Identify surge in realtime ● Detect missed/inaccurate orders in realtime Total users 500000+ QPS 100s Latency SLA < 100 ms p99th Freshness Seconds - Minutes
  • 8. @apachepinot | @KishoreBytes Challenges for the underlying system User-facing Realtime Analytics System Large Volume & Velocity of Data Realtime Ingestion 1000s of QPS Milliseconds Latency Seconds Freshness High Dimensionality Scalable
  • 9. @apachepinot | @KishoreBytes Kafka: The perfect solution for the events capturing part Velocity of ingestion Realtime Ingestion Seconds Freshness High Dimensionality Scalable
  • 16. @apachepinot | @KishoreBytes How to solve low-latency high-throughput analytics part? Large Volume & Velocity of Data Realtime Ingestion 1000s of QPS Milliseconds Latency Seconds Freshness High Dimensionality Scalable
  • 17. @apachepinot | @KishoreBytes Need a specialized analytics database that can.. Ingest from Kafka & serve real-time data Handle high event rate from Kafka and scale with Kafka Provide ultra low latency at high queries per second Handle dynamic query patterns on highly dimensional data w/o exploding storage 1 2 3 4
  • 18. @apachepinot | @KishoreBytes Options? Spark SQL Presto Big Query Druid Elastic Search Kylin KV Store Latency Flexibility low high low high
  • 20. @apachepinot | @KishoreBytes Apache Pinot Community Slack Users 1100+ Companies 50+ Join our growing community https://communityinviter.com/apps/apache-pinot/apache-pinot Events/sec 1M+ Peak QPS 170K+ Query latency ms
  • 21. @apachepinot | @KishoreBytes Need a specialized analytics database that can.. Ingest from Kafka & serve realtime data Handle high event rate from Kafka and scale with Kafka Provide ultra low latency at high queries per second Handle dynamic query patterns on highly dimensional data w/o exploding storage 1 2 3 4
  • 22. @apachepinot | @KishoreBytes Apache Pinot Architecture Pinot Controller Zookeeper Server 2 Server 1 Pinot Servers Server 3 Pinot Broker Pinot Brokers Queries Scatter - gather Consuming, indexing, serving
  • 23. @apachepinot | @KishoreBytes Server 3 Server 2 Server 1 p0 -> Server 1 p1 -> Server 2 p2 -> Server 3 p3 -> Server 1 Pinot Broker Pinot Brokers Pinot Servers Pinot Controller Zookeeper Pinot Realtime Ingestion Queries Consuming, indexing, serving Partition -> Pinot Server
  • 24. @apachepinot | @KishoreBytes Server 3 Server 2 Server 1 p0 -> Server 1, CONSUMING p1 -> Server 2, CONSUMING p2 -> Server 3, CONSUMING p3 -> Server 1, CONSUMING Pinot Broker Pinot Brokers Pinot Servers Pinot Controller Zookeeper Pinot Realtime Ingestion Queries Consuming, indexing, serving State
  • 25. @apachepinot | @KishoreBytes Server 3 Server 2 Server 1 p0 -> Server 1, CONSUMING, 102 p1 -> Server 2, CONSUMING, 120 p2 -> Server 3, CONSUMING, 105 p3 -> Server 1, CONSUMING, 100 Pinot Broker Pinot Brokers Pinot Servers Pinot Controller Zookeeper Pinot Realtime Ingestion Queries Consuming, indexing, serving Start Offset Kafka Consumers
  • 26. @apachepinot | @KishoreBytes Server 3 Server 2 Server 1 p0 -> Server 1, DONE, 102, 300 p1 -> Server 2, CONSUMING, 120 p2 -> Server 3, CONSUMING, 105 p3 -> Server 1, CONSUMING, 100 Pinot Broker Pinot Brokers Pinot Servers Pinot Controller Zookeeper Pinot Realtime Ingestion Queries Consuming, indexing, serving
  • 27. @apachepinot | @KishoreBytes Server 3 Server 2 Server 1 p0 -> Server 1, DONE, 102, 300 p1 -> Server 2, CONSUMING, 120 p2 -> Server 3, CONSUMING, 105 p3 -> Server 1, CONSUMING, 100 p0 -> Server 1, CONSUMING, 300 Pinot Broker Pinot Brokers Pinot Servers Pinot Controller Zookeeper Pinot Realtime Ingestion Queries Consuming, indexing, serving
  • 28. @apachepinot | @KishoreBytes Demo ● Emit events into a Kafka topic ● Create Pinot Schema and Table ● Query!
  • 29. @apachepinot | @KishoreBytes Need a specialized analytics database that can.. Ingest from Kafka & serve realtime data Handle high event rate from Kafka and scale with Kafka Provide ultra low latency at high queries per second Handle dynamic query patterns on highly dimensional data w/o exploding storage 1 2 3 4
  • 30. @apachepinot | @KishoreBytes Indexing User-facing Realtime Analytics Events Indexes Sorted Inverted Range Star-tree JSON Geospatial Text
  • 31. @apachepinot | @KishoreBytes Need a specialized analytics database that can.. Ingest from Kafka & serve realtime data Handle high event rate from Kafka and scale with Kafka Provide ultra low latency at high queries per second Handle dynamic query patterns on highly dimensional data w/o exploding storage 1 2 3 4
  • 32. @apachepinot | @KishoreBytes Star-tree Indexing country browser device os clicks us chrome ... ... ... ca firefox ... ... ... jp ie ... ... ... us firefox ... ... ... ca ie ... ... ... … … ... ... ... select count(*) from X where country = us and browser = chrome Country Browser Star-Tree Index US CA IE C IE C
  • 33. @apachepinot | @KishoreBytes Need a specialized analytics database that can.. Ingest from Kafka & serve realtime data Handle high event rate from Kafka and scale with Kafka Provide ultra low latency at high queries per second Handle dynamic query patterns on highly dimensional data w/o exploding storage 1 2 3 4
  • 34. @apachepinot | @KishoreBytes Horizontal scaling by adding new Kafka brokers + Pinot servers Apps Kafka Cluster Pinot Cluster Server Brokers Producers Kafka Consumers Events
  • 36. @apachepinot | @KishoreBytes Partition aware routing Realtime Analytics Events Partitioned stream murmur3(memberId) % numPartitions Partition-aware segments Seg0 - partition #0 Seg1 - partition #1 Seg2 - partition #2 Seg3 - partition #0 Seg4 - partition #1 Seg5 - partition #2 Partition aware query routing SELECT SUM(metric) WHERE memberId = m289 Choose a column as partitioning key Person { "memberId" : "m809", "name" : "adam", "age" : 30, "addresses" : [ { "number" : 1, "street" : "main st", "country" : "us" }] } murmur3(m289) % numPartitions
  • 37. @apachepinot | @KishoreBytes User-facing Realtime Analytics System Large Volume & Velocity of Data Realtime Ingestion 1000s of QPS Milliseconds Latency Seconds Freshness High Dimensionality Scalable
  • 38. @apachepinot | @KishoreBytes Takeaway ● Kafka is the key data infrastructure for event-streamed systems ● Kafka has good analytics solutions for operationalized streaming queries ● Pinot is purpose-built for ultra-low latency analytics, at high-throughput ● Pinot is a great solution for user-facing real-time analytics ● It is very easy to go from events in Kafka to analytics in Pinot ● Kafka + Pinot is the perfect combination for user-facing real-time analytics