Optimizing Tiered Storage for Low-Latency Real-Time Analytics by Ne...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

A ScyllaDB Community
Optimizing Tiered Storage for
Low-latency Real-time Analytics
Neha Pawar
Founding Engineer, StarTree
...

Apache Pinot’s Tiered Storage Journey
■ What is Apache Pinot
■ How we built tiered storage
■ How we optimized it for low-l...

Apache Pinot

User-facing real-time analytics
4
1M+
250k+
ms
Events/sec
Peak QPS
Query Latency

Apache Pinot Architecture
Zookeeper
Pinot
Broker
Pinot
Brokers
Server 2
Server 1
Pinot Servers
Server 3
Segments
Host the ...

Tightly coupled storage & compute
Server 1 Server 2
Disk/SSD Disk/SSD
Disk / SSD
Access speed micro to milliseconds
Access...

Decoupled storage-compute
Disk / SSD
Access speed micro to milliseconds
Access method POSIX APIs
Access
availability
Singl...

Need a best of both worlds
Of Real-time Analytics
Databases
Of Cloud Object Storage Of Using a Single System
$$
SPEED COST...

Pinot
Broker
Brokers
Server 1 Server 2
Disk/SSD Disk/SSD
Server 3 Server 4
Disk/SSD Disk/SSD
Cloud Object Storage
Recent d...

Pinot
Broker
Brokers
Fully decoupled
Server 3 Server 4
Cloud Object Storage
Server 1 Server 2
Disk/SSD Disk/SSD
Fully tigh...

Disk / SSD
Access speed micro to milliseconds
Access method POSIX APIs
Access
availability
Single instance
Cost $
Cloud Ob...

What data to
read?
Two key questions
When to read?

Why lazy-loading won’t work
Server 1 Server 2
Cloud Object Storage
1st query slow
2nd query fast
Pinot segment
Q: What to ...

Pinot segment format
Columns:
browser, region, country,
impressions, cost,
timestamp
browser.fwd_idx
browser.inv_idx
Brows...

select sum(impressions)
where region=”rivendell”
columns.psf
browser.fwd_idx
browser.inv_idx
Browser.dict
region.inv_idx
r...

When to read - Fetch during segment execution?
Planning phase Make segment execution plan
Pinot Server
Pinot server proces...

Planning phase Make segment execution plan
Pinot Server
Columnar Fetch from S3
Segment
Execution
Segment
Execution
Segment...

40 segments, 8 parallelism
Fetch for batch 1: 200ms
Fetch for batch 2: 200ms
Fetch for batch 3: 200ms
Fetch for batch 4: 2...

Planning phase Make segment execution plan &
prefetch all segment columns
Pinot Server
Prefetch
Wait for segment
columns t...

# of segments: 40 segments; parallelism: 8; Each S3 access: ~200ms
Fetch for batch 1: 200ms
Fetch for batch 2: 200ms
Fetch...

Pinot
Broker
Query
Server 1 Server 2 Server 3
1 2
3 4
5 6
7 8
9 10
11 12
1 2 3 4 5 6 7 8 9 10 11 12
Total segments to proc...

Pruning technique
Min/max value based pruning
How to access
Computed by default & cached locally
Segment Pruning
Partition...

Pinot
Broker
Query
Server 1 Server 2 Server 3
1 2
3 4
5 6
7 8
9 10
11 12
1 2 3 4 5 6 7 8 9 10 11 12
Total segments to proc...

Inverted Index
Sorted Index
Range Index
JSON Index
What makes Pinot fast - Indexes
Text Index
Geo Index
Filtering Optimiza...

Server 1 Server 2
Pinot segment
1 2
5 6
3 4
7
Pin any column index locally
preload.index.keys :
account_id.bloom_ﬁlter,
tr...

Query
Decoupled
System
SELECT COUNT (*)
FROM GithubEventsTier
WHERE DAY >= 20200701 AND DAY <= 20200714
5340ms
Pinot
Tiere...

Read less data
Increase parallelism
■ Network bandwidth could be saturated easily
● Fetching the full column from requeste...

0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
impressions.fwd_idx
region.inv_idx
gondor -> 3, 4, 5, ...

Selectivity
Columnar fetch
Latency
High
Medium
Low
Block fetch
Block reads
Data read
330 ms 131MB
3500 ms 3.3GB
50000 ms 5...

browser.fwd_idx
browser.inv_idx
browser.dict
region.inv_idx
region.fwd_idx
region.dict
impressions.fwd_idx
impressions.dic...

Query Column read
SELECT payload_pull_request // <- large raw forward
index
FROM github_events_bc_big
WHERE repo_name = ‘....

Apache Pinot’s Tiered Storage Journey
■ What is Apache Pinot
■ How we built tiered storage
■ How we optimized it for low-l...

Thank You!

User-facing real-time analytics - Uber
Missed
orders
Inaccurate
orders
Downtime
Top selling
items
Menu item
Feedback

Apache Pinot Community

Cost
Flexibility
Performance
Flexibility of using
a single system
for historical +
real-time data
Without worrying
about t...

All data is not equal

All data is not equal
Missed
orders
Inaccurate
orders
Downtime
Top selling
items
Menu item
Feedback
Real-time, user-facing...

What analytics infra should I choose?
Cost
Flexibility
Performance
Service the different
workloads, while
maintaining the
...

Real-time analytics (OLAP) databases

Tightly coupled storage & compute
Server 1
Disk/SSD Disk/SSD Disk/SSD Disk/SSD Disk/SSD
Server 2 Server 3 Server 4 Server ...

Tightly Coupled Systems - High Cost to Serve
Cost
Flexibility
Performance
Retention in months
Query
Frequency
Latency (ms)...

Data warehouses, data lakes, lake houses
Server 1 Server 2
Cloud Object Storage

Decoupled Systems - Tradeoff cost for latency
Cost
Flexibility
Performance
Retention in months
Latency (seconds)
Cost to S...

Cost
Flexibility
Performance
Use both systems together
Real-time Data Historical data
Data warehouses,
data lakes
Realtime...

Data Retention
Latency (seconds)
Cost to Serve
$
Query
Frequency
Takeaways
Data Retention
Latency (ms)
Cost to Serve
$$$$$...

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1045526 views
Storytelling For The Web: Integrate... by Chiara Aliotta 958884 views
Artificial Intelligence, Data and C... by OECD Directorate ... 886438 views
How to Leverage AI to Boost Employe... by SocialHRCamp 373696 views
2024 State of Marketing Report – by... by Marius Sescu 212106 views
Everything You Need To Know About C... by Expeed Software 231238 views

View on Slideshare

1 of 46 Ad

View on Slideshare

1 of 46 Ad