Scaling Apache Pulsar to 10 PB/day

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

© 2019 SPLUNK INC.
Scaling Apache Pulsar to 10 PB/day
June 2021
Karthik Ramasamy
Splunk

© 2020 SPLUNK INC.
Karthik
Ramasamy
Senior Director of Engineering
@karthikz
streaming @splunk | ex-CEO of @streamlio | co...

During the course of this presentation, we may make forward‐looking statements
regarding future events or plans of the com...

© 2019 SPLUNK INC.
Agenda 1) Introduction to Splunk & DSP

2) Requirements, Use Cases & Deployment

3) Initial Cluster S...

© 2020 SPLUNK INC.
Data
 
Lakes
Master Data
Management
ETL
Point Data
Management
 
Solutions
Data
 
Silos
Business
Process...

© 2019 SPLUNK INC.
Splunk DSP
A real time stream processing solution that collects, processes and delivers data to Splunk ...

© 2019 SPLUNK INC.
DSP - Bird’s Eye View
HEC
S2S
Batch
Apache Pulsar
Stream Processing
Engine
External
Systems
REST Client...

© 2019 SPLUNK INC.
Customer Requirements & Deployment

© 2019 SPLUNK INC.
Customers
✦ DSP is deployed at several customer
s

✦ Some of those customers are marquee customers with...

© 2019 SPLUNK INC.
Use Cases
✦ Microservices and applications emit log
s

✦ Logs contain rich informatio
n

✦ Process thes...

© 2019 SPLUNK INC.
Data Requirements
✦ Environment - Google Cloud Platform
✦ Use of n1-standard-32 VM
s

✦ Raw data ingest...

© 2019 SPLUNK INC.
DSP Ingest Cluster
DSP Compute Cluster
DSP Compute Cluster
DSP Compute Cluster
DSP Deployment
Log
Publi...

© 2019 SPLUNK INC.
DSP Deployment
✦ Separation of ingestion and computatio
n

✦ Pipeline isolation and no noisy neighbor i...

© 2019 SPLUNK INC.
VM Configuration - n1-standard-32
✦ 32 vCPU
s

✦ 120 GB of memor
y

✦ Max number of PDs (EBS equivalent...

© 2020 SPLUNK INC.
Storage Options in GCP
P-SSD
P-HDD L-SSD

© 2019 SPLUNK INC.
Initial Estimation

© 2019 SPLUNK INC.
Apache Pulsar Requirements
✦ Replica factor of
3

✦ Need to handle 120 GBps of raw traf
fi
c

✦ Need to...

© 2019 SPLUNK INC.
Pulsar Cluster Size Estimation
✦ Size of a Pulsar Cluster for a given workload depends on three paramet...

© 2019 SPLUNK INC.
Estimating Storage Bandwidth
Bookie
Bookie
Bookie
Broker
Producer
Journal
Data
Journal
Data
Journal
Dat...

© 2019 SPLUNK INC.
Estimating Network Bandwidth
Bookie
Bookie
Bookie
Broker
Producer
Journal
egress
ingress
egress
ingress...

© 2020 SPLUNK INC.
Estimating VMs using P-HDD
0
1000
2000
3000
4000
Storage Bandwidth Storage Density Network Bandwidth
30...

© 2020 SPLUNK INC.
Estimating VMs using P-SDD
0
500
1000
1500
2000
Storage Bandwidth Storage Density Network Bandwidth
300...

© 2019 SPLUNK INC.
Estimating Network Bandwidth using L-HDD
Bookie
Bookie
Bookie
Broker
Producer
Journal
ingress
egress
in...

© 2020 SPLUNK INC.
Estimating VMs using L-SDD
0
225
450
675
900
Storage Bandwidth Storage Density Network Bandwidth
120
44...

© 2020 SPLUNK INC.
Estimation of VMs - Comparison
0
1000
2000
3000
4000
P-HDD P-SSD L-SSD
868
1843
3686
VMs w/journal

© 2019 SPLUNK INC.
Optimizations

© 2019 SPLUNK INC.
Optimization #1 - Eliminating Journal
✦ Different types of durabilit
y

✦ Persistent Durability - No da...

© 2019 SPLUNK INC.
Replicated Durability
Bookie
Bookie
Bookie
Broker
Producer
Data
Data
Data

© 2020 SPLUNK INC.
Estimating VMs
0 ms
1000 ms
2000 ms
3000 ms
4000 ms
P-HDD P-SSD L-SSD
444
922
1843
868
1843
3686
VMs w/...

© 2019 SPLUNK INC.
Optimization #2 - Direct I/O
✦ Overhead of page cache in container environment is pretty hig
h

✦ Kerne...

© 2020 SPLUNK INC.
Performance of Direct I/O
0 MB/s
400 MB/s
800 MB/s
1200 MB/s
1600 MB/s
P-HDD P-SSD L-SSD
1600
600
300
8...

© 2020 SPLUNK INC.
Estimating VMs
0 ms
1000 ms
2000 ms
3000 ms
4000 ms
P-HDD P-SSD L-SSD
444
614
1228
444
922
1843
868
184...

© 2019 SPLUNK INC.
Optimization #3 - Use of Compression
Bookie
Bookie
Bookie
Broker
Producer
Data
Data
Data
Compressed dat...

© 2019 SPLUNK INC.
Employing compression
✦ Compression ratio of 4
x

✦ Need to handle 360 GBps —> 90 GBps of storage write...

© 2020 SPLUNK INC.
Estimating VMs
0 ms
1000 ms
2000 ms
3000 ms
4000 ms
P-HDD P-SSD L-SSD
111
154
308
444
614
1228
444
922
...

© 2019 SPLUNK INC.
Surviving Zone Failure
Segment 1
Segment 2
Segment n
. 
. 
.
Segment 2
Segment 3
Segment n
. 
. 
.
Segm...

© 2020 SPLUNK INC.
Estimating VMs
0 ms
1000 ms
2000 ms
3000 ms
4000 ms
P-HDD P-SSD L-SSD
222
308
616
111
154
308
444
614
1...

© 2019 SPLUNK INC.
Optimization #4 - C++ Client CPU & Memory Usage
✦ Better round robin across partitions - maximizing the...

© 2019 SPLUNK INC.
Finally …
Running 200 n1-standard-32 VMs for Pulsar cluster with 24 L-SSDs per VM

© 2019 SPLUNK INC.
OSS Contributions
✦ Github Pull Request 8283 - C++ Client is allocating buffer bigger than neede
d

✦ G...

© 2019 SPLUNK INC.
Future Work
✦ Typical operations involve

✦ Upgrade to new version of Pulsa
r

✦ Upgrade to new OS ve...

Thank You
© 2019 SPLUNK INC.

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1066413 views
Storytelling For The Web: Integrate... by Chiara Aliotta 986196 views
Artificial Intelligence, Data and C... by OECD Directorate ... 930965 views
How to Leverage AI to Boost Employe... by SocialHRCamp 396116 views
2024 State of Marketing Report – by... by Marius Sescu 223818 views
Everything You Need To Know About C... by Expeed Software 238978 views

View on Slideshare

1 of 42 Ad

View on Slideshare

1 of 42 Ad