Scaling Apache Pulsar to 10 Petabytes/Day

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Brought to you by
Scaling Apache
Pulsar to 10 PB/day
Karthik Ramasamy
Senior Director of Engineering at

Karthik Ramasamy
Senior Director of Engineering
@karthikz
streaming @splunk | ex-CEO of @streamlio | co-creator of @herons...

During the course of this presentation, we may make forward‐looking statements
regarding future events or plans of the com...

1) Introduction to Splunk & DSP
2) Requirements, Use Cases & Deployment
3) Initial Cluster Size Estimation
4) Optimization...

Splunk Data Stream Processor
Detect Data Patterns or Conditions
Mask Sensitive Data
Aggregate Format
Normalize Transform
F...

DSP - Bird’s Eye View
HEC
S2S
Batch
Apache Pulsar
Stream Processing
Engine
External
Systems
REST Client
Forwarders
Data So...

Customer Requirements & Deployment

■ Marquee customer is in ﬁnance and payments
■ Microservices and applications emit logs
■ Logs contain rich information
■ ...

■ Environment - Google Cloud Platform
■ Use of n1-standard-32 VMs
■ Raw data ingestion of 10 PB/day that translates ~120 G...

DSP Ingest Cluster
DSP Compute Cluster
DSP Compute Cluster
DSP Compute Cluster
Log
Publisher
Log
Publisher
Log
Publisher
A...

DSP Deployment
■ Separation of ingestion and computation
■ Pipeline isolation and no noisy neighbor issues
■ Troubleshooti...

■ 32 vCPUs
■ 120 GB of memory
■ Max number of PDs (EBS equivalent) - 128
■ Max total PD size - 257 TB
■ Max egress network...

P-SSD
P-HDD L-SSD
Storage Options in GCP

Initial Estimation

■ Replica factor of 3
■ Need to handle 120 GBps of raw traﬃc
■ Need to handle 360 GBps of storage write bandwidth
■ With j...

■ Size of a Pulsar Cluster for a given workload depends on three
parameters -
■ Storage Density - Aggregate storage capaci...

Max of 200 MB/sec write
throughput per VM
Max of 9 TB
per instance
Max of 4 GBps egress
and ingress bandwidth
Dominated by...

Max of 400 MB/sec write
throughput per VM
Max of 9 TB
per instance
Max of 4 GBps egress
and ingress bandwidth
Dominated by...

Max of 850 MB/sec write
throughput per VM
Max of 9 TB
per instance
Max of 4 GBps egress
and ingress bandwidth
Dominated by...

Estimation of VMs - Comparison

Optimizations

Optimization #1 - Eliminating Journal
Since all the data is machine logs, we implemented replicated durability
■ Different...

Estimating VMs
Dominated by
Storage Bandwidth
Dominated by
Storage Bandwidth
Dominated by
Storage Density

Optimization #2 - Direct I/O
■ Overhead of page cache in container environment is pretty high
■ Kernel needs to keep track...

Performance of Direct I/O

Estimating VMs
Dominated by
Storage Bandwidth
Dominated by
Storage Bandwidth Dominated by
Storage Density

Optimization #3 - Use of Compression
Bookie
Bookie
Bookie
Broker
Producer
Data
Data
Data
Compressed data
Consumer
Compress...

Employing compression
■ Compression ratio of 4x
■ Need to handle 360 GBps —> 90 GBps of storage write bandwidth
■ Total st...

Estimating VMs
Dominated by
Storage Bandwidth Dominated by
Storage Bandwidth Dominated by
Storage Density

Surviving Zone Failure
Segment 1
Segment 2
Segment n
.
.
.
Segment 2
Segment 3
Segment n
.
.
.
Segment 3
Segment 1
Segment...

Estimating VMs

Optimization #4 - C++ Client CPU & Memory Usage
■ Better round robin across partitions - maximizing the batch
size per par...

Finally …
Running 200 n1-standard-32 VMs for Pulsar cluster with 24 L-SSDs per VM

Brought to you by
@Karthikz
Karthik Ramasamy

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1045526 views
Storytelling For The Web: Integrate... by Chiara Aliotta 958884 views
Artificial Intelligence, Data and C... by OECD Directorate ... 886438 views
How to Leverage AI to Boost Employe... by SocialHRCamp 373696 views
2024 State of Marketing Report – by... by Marius Sescu 212106 views
Everything You Need To Know About C... by Expeed Software 231238 views

View on Slideshare

1 of 34 Ad

View on Slideshare

1 of 34 Ad