The document discusses real-time analytics using Apache Pinot and how companies like LinkedIn, Uber Eats, and Stripe utilize it to create dashboards for millions of users. It highlights the evolution of analytics, the need for data freshness, and the architectural advantages of Pinot, including its indexing techniques and scalability. Additionally, it touches on the future trends in big data and how organizations can leverage real-time analytics for enhanced business intelligence.

1. Real-time
Analytics Using
Apache Pinot
How LinkedIn, Uber Eats and Stripe create
Real Time Dashboards for millions of users.

2. Agenda
Who is Barkha? (why would you want to listen to me?)
The evolution of Analytics
How LinkedIn Solved their Problem
Try some Pinot with me

3. Overheard @ Big Data Fest 2023
• 5 Year trends in Big data will see
• Streaming APIs
• Will Data Warehouse Survive?
• Integration with LLM/AI/ML
• Thiago de Faria
• 5 Year trends in Big data will see
• Democratization of Data Warehousing
• Commoditized Data Warehousing
• Most companies are barely doing BI let alone AI.
• Joe Reis

4. About Barkha
• Founder South Florida Women in
Technology
• Developer Advocate @StarTree
• Linkedin.com/in/BarkhaHerman
• Twitter @BarkhaH

5. Analytics?
Real Time?
Scale?
OH WHY?
Why do we need Real-time
Analytics? Or Analytics? Or at
Scale?

6. Historic
Analytics
Batch
Shared Data
No Scale Concerns

7. Modern
Analytics
Data Freshness
Daily reports vs.
How late is my food
delivery?
Query Performance
Reports < 2 minutes vs.
Dashboards take < 10
millisecond to load
Scale
All division managers
worldwide access report
(> 1000) vs.
Millions of users access
dashboard

8. How LinkedIn
solved Analytics @
Scale
By inventing Pinot

9. LinkedIn: Who Viewed
your Profile? • Capture profile view information
and its deduplication
• Compute view sources (e.g.,
search, profile page, etc.)
• View relevance (e.g., a senior
leader viewed your profile)
• View obfuscations based on the
viewing member’s privacy settings

10. Before Pinot
• Elastic Search based solution
• 1000 Nodes
• 1500 queries / sec
• 20+ million users

11. After Pinot
• 75 Nodes
• 5000 queries / sec
• 70+ million users

12. Pinot
Building
Blocks • Segment is the physical
store.
• Table are conceptual and
accept both real-time and
batch data.
• Tenants provide
functional segregation.
• Cluster allow for scale
based on use.

13. Pinot
Building
Blocks

14. Indexes
Pinot
supports
the
following
indexing
techniques
Inverted index - Used for exact lookups
Range index - Used for range queries.
Text index - Used for phrase, term, Boolean, prefix, or regex queries.
Geospatial index - Based on H3, a hexagon-based hierarchical gridding.
Used for finding points that exist within a certain distance from another point.
JSON index - Used for querying columns in JSON documents.
Star-Tree index - Pre-aggregates results across multiple columns.

15. StarTree Index
Don’t pre cube everything…

16. Apache Pinot Architecture

17. Demo
Pizza Shop Demo
https://github.com/startreedata/pizza-shop-demo

18. Overheard @ Big Data Fest 2023
• 5 Year trends in Big data will see
• Streaming APIs  Apache Pinot is built to solve Streaming First Problems
• Will Data Warehouse Survive?  Apache Pinot builds Customer Facing Analytics which is on the rise
• Integration with LLM/AI/ML  Apps built on top of Pinot such as ThirdEye use Statistics and allow for AI/ML Add Ons.
• Thiago de Faria
• 5 Year trends in Big data will see
• Democratization of Data Warehousing  Apache Pinot builds Customer Facing Analytics which is on the rise
• Commoditized Data Warehousing  Apache Pinot builds Customer Facing Analytics which is on the rise
• Most companies are barely doing BI let alone AI.  Easy Analytics + Apps built on top of Pinot such as ThirdEye.
• Joe Reis

19. Using Real-
time Analytics
@ Scale
What can you do with it?

20. Who Uses
Apache
Pinot?

21. What’s Next?
Please Connect!!!!! I need brownie points.

22. Thank you for listening!