SlideShare a Scribd company logo
1 of 42
Download to read offline
© 2019 SPLUNK INC.
Scaling Apache Pulsar to 10 PB/day
June 2021
Karthik Ramasamy
Splunk
© 2020 SPLUNK INC.
Karthik
Ramasamy
Senior Director of Engineering
@karthikz
streaming @splunk | ex-CEO of @streamlio | co-creator of @heronstreaming | ex @Twitter | Ph.D
During the course of this presentation, we may make forward‐looking statements
regarding future events or plans of the company. We caution you that such statements
reflect our current expectations and estimates based on factors currently known to us
and that actual events or results may differ materially. The forward-looking statements
made in the this presentation are being made as of the time and date of its live
presentation. If reviewed after its live presentation, it may not contain current or
accurate information. We do not assume any obligation to update any forward‐
looking statements made herein.


In addition, any information about our roadmap outlines our general product direction
and is subject to change at any time without notice. It is for informational purposes only,
and shall not be incorporated into any contract or other commitment. Splunk undertakes
no obligation either to develop the features or functionalities described or to include any
such feature or functionality in a future release.


Splunk, Splunk>, Data-to-Everything, D2E and Turn Data Into Doing are trademarks and registered trademarks of Splunk Inc. in the
United States and other countries. All other brand names, product names or trademarks belong to their respective owners. © 2020
Splunk Inc. All rights reserved
Forward-


Looking


Statements
© 2020 SPLUNK INC.
© 2019 SPLUNK INC.
Agenda 1) Introduction to Splunk & DSP


2) Requirements, Use Cases & Deployment


3) Initial Cluster Size Estimation


4) Optimizations


5) Conclusion
© 2020 SPLUNK INC.
Data


Lakes
Master Data
Management
ETL
Point Data
Management


Solutions
Data


Silos
Business
Processes
The


Data-to-Everything
Platform
IT
Security
DevOps
© 2019 SPLUNK INC.
Splunk DSP
A real time stream processing solution that collects, processes and delivers data to Splunk and other
destinations in milliseconds
Splunk Data Stream Processor
Detect Data Patterns or Conditions
Mask Sensitive Data
Aggregate Format
Normalize Transform
Filter Enhance
Turn Raw Data Into


High-value Information


Protect Sensitive Data
Distribute Data To Splunk


Or Other Destinations
Data


Warehouse
Public


Cloud
Message


Bus
© 2019 SPLUNK INC.
DSP - Bird’s Eye View
HEC
S2S
Batch
Apache Pulsar
Stream Processing
Engine
External
Systems
REST Client
Forwarders
Data Source
Splunk


Indexer
Apache Pulsar is at the core of DSP
© 2019 SPLUNK INC.
Customer Requirements & Deployment
© 2019 SPLUNK INC.
Customers
✦ DSP is deployed at several customer
s

✦ Some of those customers are marquee customers with large volume of dat
a

✦ One such marquee customer is in
fi
nance and payments
© 2019 SPLUNK INC.
Use Cases
✦ Microservices and applications emit log
s

✦ Logs contain rich informatio
n

✦ Process these logs and extract monitoring & tracing informatio
n

✦ Filter these logs depending on log volume and if there is high value - justifying retentio
n

✦ Compute real time business metrics
© 2019 SPLUNK INC.
Data Requirements
✦ Environment - Google Cloud Platform
✦ Use of n1-standard-32 VM
s

✦ Raw data ingestion of 10 PB/day that translates ~120 GB/se
c

✦ Data retention of 3 hour
s

✦ Need to handle the entire traf
fi
c load when a zone fails
© 2019 SPLUNK INC.
DSP Ingest Cluster
DSP Compute Cluster
DSP Compute Cluster
DSP Compute Cluster
DSP Deployment
Log
Publisher
Log
Publisher
Log
Publisher
Apache Pulsar


Cluster
Pipeline 1
Pipeline 2
Pipeline 3
Splunk
Enterprise
Splunk
Observability
© 2019 SPLUNK INC.
DSP Deployment
✦ Separation of ingestion and computatio
n

✦ Pipeline isolation and no noisy neighbor issue
s

✦ Troubleshooting single pipeline gets easie
r

✦ Might not need over provisioning except for peak load + fudge factor (as compared to deploying a single
cluster)
© 2019 SPLUNK INC.
VM Configuration - n1-standard-32
✦ 32 vCPU
s

✦ 120 GB of memor
y

✦ Max number of PDs (EBS equivalent) - 12
8

✦ Max total PD size - 257 T
B

✦ Max egress network bandwidth - 32 Gbps (4 GBps
)

✦ Max 24 L-SSDs for a total of 9 TB
© 2020 SPLUNK INC.
Storage Options in GCP
P-SSD
P-HDD L-SSD
© 2019 SPLUNK INC.
Initial Estimation
© 2019 SPLUNK INC.
Apache Pulsar Requirements
✦ Replica factor of
3

✦ Need to handle 120 GBps of raw traf
fi
c

✦ Need to handle 360 GBps of storage write bandwidt
h

✦ With journal required write bandwidth 720 GBp
s

✦ Total storage required for retention - 3.9 P
B

✦ Total ingress network bandwidth - 480 GBp
s

✦ Total egress network bandwidth - 1200 GBps
© 2019 SPLUNK INC.
Pulsar Cluster Size Estimation
✦ Size of a Pulsar Cluster for a given workload depends on three parameters -
 

✦ Storage Density - Aggregate storage capacity needed in the cluster and proportional to retention of dat
a

✦ Storage Bandwidth - Aggregate write throughput and read throughput needed for data ingestion and
consumption. Heavily depends on storage medi
a

✦ Network Bandwidth - Aggregate network bandwidth available in the cluster for input traf
fi
c and output
traf
fi
c.
© 2019 SPLUNK INC.
Estimating Storage Bandwidth
Bookie
Bookie
Bookie
Broker
Producer
Journal
Data
Journal
Data
Journal
Data
Consumer 1X data
2X data
3X data
2X data
2X data
6X data
© 2019 SPLUNK INC.
Estimating Network Bandwidth
Bookie
Bookie
Bookie
Broker
Producer
Journal
egress
ingress
egress
ingress
Data
Journal
Data
Journal
Data
ingress
ingress
egress
egress
egress
egress
egress
Consumer
egress
egress
egress
1X ingress


4X egress
1X ingress


2X egress
4X ingress


10X egress
1X ingress


2X egress
1X ingress


2X egress
© 2020 SPLUNK INC.
Estimating VMs using P-HDD
0
1000
2000
3000
4000
Storage Bandwidth Storage Density Network Bandwidth
300
444
3686
VMs w/journal
Max of 200 MB/sec write
throughput per VM
Max of 9 TB


per instance
Max of 4 GBps egress
and ingress bandwidth
Dominated by
Storage Bandwidth
© 2020 SPLUNK INC.
Estimating VMs using P-SDD
0
500
1000
1500
2000
Storage Bandwidth Storage Density Network Bandwidth
300
444
1843
VMs w/journal
Max of 400 MB/sec write
throughput per VM
Max of 9 TB


per instance
Max of 4 GBps egress
and ingress bandwidth
Dominated by
Storage Bandwidth
© 2019 SPLUNK INC.
Estimating Network Bandwidth using L-HDD
Bookie
Bookie
Bookie
Broker
Producer
Journal
ingress
egress
ingress
Data
Journal
Data
Journal
Data
ingress
ingress
Consumer
egress
egress
egress
1X ingress


4X egress
1X ingress
1X ingress
1X ingress
4X ingress


5X egress
© 2020 SPLUNK INC.
Estimating VMs using L-SDD
0
225
450
675
900
Storage Bandwidth Storage Density Network Bandwidth
120
444
868
VMs w/journal
Max of 850 MB/sec write
throughput per VM
Max of 9 TB


per instance
Max of 4 GBps egress
and ingress bandwidth
Dominated by
Storage Bandwidth
© 2020 SPLUNK INC.
Estimation of VMs - Comparison
0
1000
2000
3000
4000
P-HDD P-SSD L-SSD
868
1843
3686
VMs w/journal
© 2019 SPLUNK INC.
Optimizations
© 2019 SPLUNK INC.
Optimization #1 - Eliminating Journal
✦ Different types of durabilit
y

✦ Persistent Durability - No data loss in the presence of nodes failures or entire cluster failur
e

✦ Replicated Durability - No data loss in the presence of limited nodes failure
s

✦ Transient Durability - Data loss in the presence of failure
s

Since all the data is machine logs, we implemented replicated durability
© 2019 SPLUNK INC.
Replicated Durability
Bookie
Bookie
Bookie
Broker
Producer
Data
Data
Data
© 2020 SPLUNK INC.
Estimating VMs
0 ms
1000 ms
2000 ms
3000 ms
4000 ms
P-HDD P-SSD L-SSD
444
922
1843
868
1843
3686
VMs w/journal VMs w/o journal
Dominated by
Storage Bandwidth
Dominated by
Storage Bandwidth
Dominated by
Storage Density
© 2019 SPLUNK INC.
Optimization #2 - Direct I/O
✦ Overhead of page cache in container environment is pretty hig
h

✦ Kernel needs to keep track of the usage quota per container for the page cach
e

✦ These translate into maintaining additional data structures and lookups (older kernel had n^2 lookup time for
getting pages in & out
)

✦ Bypassed page cache for BookKeeper entry log, using JNI
:

✦ We already have in memory caches (write and read-ahead
)

✦ We have better control on what to cache and when to evic
t

✦ Avoid double buffering
© 2020 SPLUNK INC.
Performance of Direct I/O
0 MB/s
400 MB/s
800 MB/s
1200 MB/s
1600 MB/s
P-HDD P-SSD L-SSD
1600
600
300
850
400
200
Before Direct I/O After Direct I/O
© 2020 SPLUNK INC.
Estimating VMs
0 ms
1000 ms
2000 ms
3000 ms
4000 ms
P-HDD P-SSD L-SSD
444
614
1228
444
922
1843
868
1843
3686
VMs w/journal VMs w/o journal VMs with direct i/o
Dominated by
Storage Bandwidth
Dominated by
Storage Bandwidth Dominated by
Storage Density
© 2019 SPLUNK INC.
Optimization #3 - Use of Compression
Bookie
Bookie
Bookie
Broker
Producer
Data
Data
Data
Compressed data
Consumer Compressed data
C
U
© 2019 SPLUNK INC.
Employing compression
✦ Compression ratio of 4
x

✦ Need to handle 360 GBps —> 90 GBps of storage write bandwidt
h

✦ Total storage required for retention - 3.9 PB —> 975 T
B

✦ Total ingress network bandwidth - 480 GBps —> 120 GBps
 

✦ Total egress network bandwidth - 1200 GBps —> 240 GBps
© 2020 SPLUNK INC.
Estimating VMs
0 ms
1000 ms
2000 ms
3000 ms
4000 ms
P-HDD P-SSD L-SSD
111
154
308
444
614
1228
444
922
1843
868
1843
3686
VMs w/journal VMs w/o journal VMs with direct i/o VMs with compression
Dominated by
Storage Bandwidth Dominated by
Storage Bandwidth Dominated by
Storage Density
© 2019 SPLUNK INC.
Surviving Zone Failure
Segment 1
Segment 2
Segment n
.

.

.
Segment 2
Segment 3
Segment n
.

.

.
Segment 3
Segment 1
Segment n
.

.

.
Storage
Broker
Serving
Broker Broker
✦ Zone/Rack Failure
s

✦ Bookies provide rack awarenes
s

✦ Broker replicate data to different racks/zone
s

✦ In the presence of zone/rack failure, data is available
in other zone
s

✦ One zone failure means two zones should be capable of
handling the entire traf
fi
c

✦ Requires 50% additional VM
s

Zone A Zone B Zone C
© 2020 SPLUNK INC.
Estimating VMs
0 ms
1000 ms
2000 ms
3000 ms
4000 ms
P-HDD P-SSD L-SSD
222
308
616
111
154
308
444
614
1228
444
922
1843
868
1843
3686
VMs w/journal VMs w/o journal VMs with direct i/o VMs with compression VMs to survive zone failure
© 2019 SPLUNK INC.
Optimization #4 - C++ Client CPU & Memory Usage
✦ Better round robin across partitions - maximizing the batch size per partitio
n

✦ Having bigger batches reduces the cpu usage for client, broker and bookie
s

✦ Increases the compression facto
r

✦ Reduced client memory usage
s

✦ Optimizations to minimize memory allocation overhead
 

✦ Implemented memory limit in C++ produce
r

✦ Simpli
fi
es the user con
fi
guration — One single setting instead of multiple queue sizes and complex
math
© 2019 SPLUNK INC.
Finally …
Running 200 n1-standard-32 VMs for Pulsar cluster with 24 L-SSDs per VM
© 2019 SPLUNK INC.
OSS Contributions
✦ Github Pull Request 8283 - C++ Client is allocating buffer bigger than neede
d

✦ Github Pull Request 8331 - C++ Client back-pressure is done on batches rather than number of message
s

✦ Github Pull Request 8395 - C++ Implement batch aware producer router
 

✦ Github Pull Request 9679 - C++ Implemented memory limit in C++ producer
© 2019 SPLUNK INC.
Future Work
✦ Typical operations involve
 

✦ Upgrade to new version of Pulsa
r

✦ Upgrade to new OS versio
n

✦ Apply new security patches to O
S

✦ New Pulsar version - In-place upgrade
s

✦ OS & security patches - Applying one VM at a time is too slow for large cluster
Thank You
© 2019 SPLUNK INC.

More Related Content

What's hot

Stream Processing with Flink and Stream Sharing
Stream Processing with Flink and Stream SharingStream Processing with Flink and Stream Sharing
Stream Processing with Flink and Stream Sharingconfluent
 
Building Reliable Lakehouses with Apache Flink and Delta Lake
Building Reliable Lakehouses with Apache Flink and Delta LakeBuilding Reliable Lakehouses with Apache Flink and Delta Lake
Building Reliable Lakehouses with Apache Flink and Delta LakeFlink Forward
 
Uber: Kafka Consumer Proxy
Uber: Kafka Consumer ProxyUber: Kafka Consumer Proxy
Uber: Kafka Consumer Proxyconfluent
 
Kafka Streams: What it is, and how to use it?
Kafka Streams: What it is, and how to use it?Kafka Streams: What it is, and how to use it?
Kafka Streams: What it is, and how to use it?confluent
 
When apache pulsar meets apache flink
When apache pulsar meets apache flinkWhen apache pulsar meets apache flink
When apache pulsar meets apache flinkStreamNative
 
Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...
Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...
Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...Databricks
 
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...Databricks
 
MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summi...
MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summi...MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summi...
MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summi...StreamNative
 
Autoscaling Flink with Reactive Mode
Autoscaling Flink with Reactive ModeAutoscaling Flink with Reactive Mode
Autoscaling Flink with Reactive ModeFlink Forward
 
Apache Spark Core – Practical Optimization
Apache Spark Core – Practical OptimizationApache Spark Core – Practical Optimization
Apache Spark Core – Practical OptimizationDatabricks
 
Securing your Pulsar Cluster with Vault_Chris Kellogg
Securing your Pulsar Cluster with Vault_Chris KelloggSecuring your Pulsar Cluster with Vault_Chris Kellogg
Securing your Pulsar Cluster with Vault_Chris KelloggStreamNative
 
Security and Multi-Tenancy with Apache Pulsar in Yahoo! (Verizon Media) - Pul...
Security and Multi-Tenancy with Apache Pulsar in Yahoo! (Verizon Media) - Pul...Security and Multi-Tenancy with Apache Pulsar in Yahoo! (Verizon Media) - Pul...
Security and Multi-Tenancy with Apache Pulsar in Yahoo! (Verizon Media) - Pul...StreamNative
 
Real-time Analytics with Trino and Apache Pinot
Real-time Analytics with Trino and Apache PinotReal-time Analytics with Trino and Apache Pinot
Real-time Analytics with Trino and Apache PinotXiang Fu
 
Getting Started with Confluent Schema Registry
Getting Started with Confluent Schema RegistryGetting Started with Confluent Schema Registry
Getting Started with Confluent Schema Registryconfluent
 
patroni-based citrus high availability environment deployment
patroni-based citrus high availability environment deploymentpatroni-based citrus high availability environment deployment
patroni-based citrus high availability environment deploymenthyeongchae lee
 
Apache Flink internals
Apache Flink internalsApache Flink internals
Apache Flink internalsKostas Tzoumas
 

What's hot (20)

Apache kafka
Apache kafkaApache kafka
Apache kafka
 
Stream Processing with Flink and Stream Sharing
Stream Processing with Flink and Stream SharingStream Processing with Flink and Stream Sharing
Stream Processing with Flink and Stream Sharing
 
Building Reliable Lakehouses with Apache Flink and Delta Lake
Building Reliable Lakehouses with Apache Flink and Delta LakeBuilding Reliable Lakehouses with Apache Flink and Delta Lake
Building Reliable Lakehouses with Apache Flink and Delta Lake
 
Uber: Kafka Consumer Proxy
Uber: Kafka Consumer ProxyUber: Kafka Consumer Proxy
Uber: Kafka Consumer Proxy
 
Kafka Streams: What it is, and how to use it?
Kafka Streams: What it is, and how to use it?Kafka Streams: What it is, and how to use it?
Kafka Streams: What it is, and how to use it?
 
When apache pulsar meets apache flink
When apache pulsar meets apache flinkWhen apache pulsar meets apache flink
When apache pulsar meets apache flink
 
Introduction to Apache Kafka
Introduction to Apache KafkaIntroduction to Apache Kafka
Introduction to Apache Kafka
 
Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...
Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...
Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...
 
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
 
MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summi...
MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summi...MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summi...
MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summi...
 
Autoscaling Flink with Reactive Mode
Autoscaling Flink with Reactive ModeAutoscaling Flink with Reactive Mode
Autoscaling Flink with Reactive Mode
 
Apache Spark Core – Practical Optimization
Apache Spark Core – Practical OptimizationApache Spark Core – Practical Optimization
Apache Spark Core – Practical Optimization
 
Apache Kafka at LinkedIn
Apache Kafka at LinkedInApache Kafka at LinkedIn
Apache Kafka at LinkedIn
 
Securing your Pulsar Cluster with Vault_Chris Kellogg
Securing your Pulsar Cluster with Vault_Chris KelloggSecuring your Pulsar Cluster with Vault_Chris Kellogg
Securing your Pulsar Cluster with Vault_Chris Kellogg
 
Security and Multi-Tenancy with Apache Pulsar in Yahoo! (Verizon Media) - Pul...
Security and Multi-Tenancy with Apache Pulsar in Yahoo! (Verizon Media) - Pul...Security and Multi-Tenancy with Apache Pulsar in Yahoo! (Verizon Media) - Pul...
Security and Multi-Tenancy with Apache Pulsar in Yahoo! (Verizon Media) - Pul...
 
Real-time Analytics with Trino and Apache Pinot
Real-time Analytics with Trino and Apache PinotReal-time Analytics with Trino and Apache Pinot
Real-time Analytics with Trino and Apache Pinot
 
Getting Started with Confluent Schema Registry
Getting Started with Confluent Schema RegistryGetting Started with Confluent Schema Registry
Getting Started with Confluent Schema Registry
 
patroni-based citrus high availability environment deployment
patroni-based citrus high availability environment deploymentpatroni-based citrus high availability environment deployment
patroni-based citrus high availability environment deployment
 
Presto
PrestoPresto
Presto
 
Apache Flink internals
Apache Flink internalsApache Flink internals
Apache Flink internals
 

Similar to Scaling Apache Pulsar to 10 PB/day

Scaling Apache Pulsar to 10 Petabytes/Day
Scaling Apache Pulsar to 10 Petabytes/DayScaling Apache Pulsar to 10 Petabytes/Day
Scaling Apache Pulsar to 10 Petabytes/DayScyllaDB
 
Pulsar summit-keynote-final
Pulsar summit-keynote-finalPulsar summit-keynote-final
Pulsar summit-keynote-finalKarthik Ramasamy
 
Toward Scalable and Powerful CloudStack
Toward Scalable and Powerful CloudStackToward Scalable and Powerful CloudStack
Toward Scalable and Powerful CloudStackTakashi Kanai
 
DDN: Protecting Your Data, Protecting Your Hardware
DDN: Protecting Your Data, Protecting Your HardwareDDN: Protecting Your Data, Protecting Your Hardware
DDN: Protecting Your Data, Protecting Your Hardwareinside-BigData.com
 
Sizing Splunk SmartStore - Spend Less and Get More Out of Splunk
Sizing Splunk SmartStore - Spend Less and Get More Out of SplunkSizing Splunk SmartStore - Spend Less and Get More Out of Splunk
Sizing Splunk SmartStore - Spend Less and Get More Out of SplunkPaula Koziol
 
IBM Power Systems - enabling cloud solutions
IBM Power Systems - enabling cloud solutionsIBM Power Systems - enabling cloud solutions
IBM Power Systems - enabling cloud solutionsDavid Spurway
 
MongoDB World 2019: Implementation and Operationalization of MongoDB Sharding...
MongoDB World 2019: Implementation and Operationalization of MongoDB Sharding...MongoDB World 2019: Implementation and Operationalization of MongoDB Sharding...
MongoDB World 2019: Implementation and Operationalization of MongoDB Sharding...MongoDB
 
Open power topics20191023
Open power topics20191023Open power topics20191023
Open power topics20191023Yutaka Kawai
 
HP flash optimized storage - webcast
HP flash optimized storage - webcastHP flash optimized storage - webcast
HP flash optimized storage - webcastCalvin Zito
 
Delivering Apache Hadoop for the Modern Data Architecture
Delivering Apache Hadoop for the Modern Data Architecture Delivering Apache Hadoop for the Modern Data Architecture
Delivering Apache Hadoop for the Modern Data Architecture Hortonworks
 
IBM System Storage DS8000 with SSDs An In-Depth Look at SSD Performance in th...
IBM System Storage DS8000 with SSDs An In-Depth Look at SSD Performance in th...IBM System Storage DS8000 with SSDs An In-Depth Look at SSD Performance in th...
IBM System Storage DS8000 with SSDs An In-Depth Look at SSD Performance in th...IBM India Smarter Computing
 
Accelerate and Scale Big Data Analytics with Disaggregated Compute and Storage
Accelerate and Scale Big Data Analytics with Disaggregated Compute and StorageAccelerate and Scale Big Data Analytics with Disaggregated Compute and Storage
Accelerate and Scale Big Data Analytics with Disaggregated Compute and StorageAlluxio, Inc.
 
#IBMEdge: Flash Storage Session
#IBMEdge: Flash Storage Session#IBMEdge: Flash Storage Session
#IBMEdge: Flash Storage SessionBrocade
 
Aerospike Roadmap Overview - Meetup Dec 2019
Aerospike Roadmap Overview - Meetup Dec 2019Aerospike Roadmap Overview - Meetup Dec 2019
Aerospike Roadmap Overview - Meetup Dec 2019Aerospike
 
MongoDB World 2018: Managing a Mission Critical eCommerce Application on Mong...
MongoDB World 2018: Managing a Mission Critical eCommerce Application on Mong...MongoDB World 2018: Managing a Mission Critical eCommerce Application on Mong...
MongoDB World 2018: Managing a Mission Critical eCommerce Application on Mong...MongoDB
 
Model-driven Telemetry: The Foundation of Big Data Analytics
Model-driven Telemetry: The Foundation of Big Data AnalyticsModel-driven Telemetry: The Foundation of Big Data Analytics
Model-driven Telemetry: The Foundation of Big Data AnalyticsCisco Canada
 
Splunk und Multi-Cloud
Splunk und Multi-CloudSplunk und Multi-Cloud
Splunk und Multi-CloudSplunk
 
Splunk and Multicloud
Splunk and MulticloudSplunk and Multicloud
Splunk and MulticloudSplunk
 

Similar to Scaling Apache Pulsar to 10 PB/day (20)

Scaling Apache Pulsar to 10 Petabytes/Day
Scaling Apache Pulsar to 10 Petabytes/DayScaling Apache Pulsar to 10 Petabytes/Day
Scaling Apache Pulsar to 10 Petabytes/Day
 
Pulsar summit-keynote-final
Pulsar summit-keynote-finalPulsar summit-keynote-final
Pulsar summit-keynote-final
 
Toward Scalable and Powerful CloudStack
Toward Scalable and Powerful CloudStackToward Scalable and Powerful CloudStack
Toward Scalable and Powerful CloudStack
 
DDN: Protecting Your Data, Protecting Your Hardware
DDN: Protecting Your Data, Protecting Your HardwareDDN: Protecting Your Data, Protecting Your Hardware
DDN: Protecting Your Data, Protecting Your Hardware
 
Apache Pulsar @Splunk
Apache Pulsar @SplunkApache Pulsar @Splunk
Apache Pulsar @Splunk
 
Sizing Splunk SmartStore - Spend Less and Get More Out of Splunk
Sizing Splunk SmartStore - Spend Less and Get More Out of SplunkSizing Splunk SmartStore - Spend Less and Get More Out of Splunk
Sizing Splunk SmartStore - Spend Less and Get More Out of Splunk
 
IBM Power Systems - enabling cloud solutions
IBM Power Systems - enabling cloud solutionsIBM Power Systems - enabling cloud solutions
IBM Power Systems - enabling cloud solutions
 
MongoDB World 2019: Implementation and Operationalization of MongoDB Sharding...
MongoDB World 2019: Implementation and Operationalization of MongoDB Sharding...MongoDB World 2019: Implementation and Operationalization of MongoDB Sharding...
MongoDB World 2019: Implementation and Operationalization of MongoDB Sharding...
 
Open power topics20191023
Open power topics20191023Open power topics20191023
Open power topics20191023
 
HP flash optimized storage - webcast
HP flash optimized storage - webcastHP flash optimized storage - webcast
HP flash optimized storage - webcast
 
Delivering Apache Hadoop for the Modern Data Architecture
Delivering Apache Hadoop for the Modern Data Architecture Delivering Apache Hadoop for the Modern Data Architecture
Delivering Apache Hadoop for the Modern Data Architecture
 
IBM System Storage DS8000 with SSDs An In-Depth Look at SSD Performance in th...
IBM System Storage DS8000 with SSDs An In-Depth Look at SSD Performance in th...IBM System Storage DS8000 with SSDs An In-Depth Look at SSD Performance in th...
IBM System Storage DS8000 with SSDs An In-Depth Look at SSD Performance in th...
 
Accelerate and Scale Big Data Analytics with Disaggregated Compute and Storage
Accelerate and Scale Big Data Analytics with Disaggregated Compute and StorageAccelerate and Scale Big Data Analytics with Disaggregated Compute and Storage
Accelerate and Scale Big Data Analytics with Disaggregated Compute and Storage
 
#IBMEdge: Flash Storage Session
#IBMEdge: Flash Storage Session#IBMEdge: Flash Storage Session
#IBMEdge: Flash Storage Session
 
Aerospike Roadmap Overview - Meetup Dec 2019
Aerospike Roadmap Overview - Meetup Dec 2019Aerospike Roadmap Overview - Meetup Dec 2019
Aerospike Roadmap Overview - Meetup Dec 2019
 
IBM PureSystems
IBM PureSystemsIBM PureSystems
IBM PureSystems
 
MongoDB World 2018: Managing a Mission Critical eCommerce Application on Mong...
MongoDB World 2018: Managing a Mission Critical eCommerce Application on Mong...MongoDB World 2018: Managing a Mission Critical eCommerce Application on Mong...
MongoDB World 2018: Managing a Mission Critical eCommerce Application on Mong...
 
Model-driven Telemetry: The Foundation of Big Data Analytics
Model-driven Telemetry: The Foundation of Big Data AnalyticsModel-driven Telemetry: The Foundation of Big Data Analytics
Model-driven Telemetry: The Foundation of Big Data Analytics
 
Splunk und Multi-Cloud
Splunk und Multi-CloudSplunk und Multi-Cloud
Splunk und Multi-Cloud
 
Splunk and Multicloud
Splunk and MulticloudSplunk and Multicloud
Splunk and Multicloud
 

More from Karthik Ramasamy

Apache Pulsar Seattle - Meetup
Apache Pulsar Seattle - MeetupApache Pulsar Seattle - Meetup
Apache Pulsar Seattle - MeetupKarthik Ramasamy
 
Unifying Messaging, Queueing & Light Weight Compute Using Apache Pulsar
Unifying Messaging, Queueing & Light Weight Compute Using Apache PulsarUnifying Messaging, Queueing & Light Weight Compute Using Apache Pulsar
Unifying Messaging, Queueing & Light Weight Compute Using Apache PulsarKarthik Ramasamy
 
Creating Data Fabric for #IOT with Apache Pulsar
Creating Data Fabric for #IOT with Apache PulsarCreating Data Fabric for #IOT with Apache Pulsar
Creating Data Fabric for #IOT with Apache PulsarKarthik Ramasamy
 
Linked In Stream Processing Meetup - Apache Pulsar
Linked In Stream Processing Meetup - Apache PulsarLinked In Stream Processing Meetup - Apache Pulsar
Linked In Stream Processing Meetup - Apache PulsarKarthik Ramasamy
 
Exactly once in Apache Heron
Exactly once in Apache HeronExactly once in Apache Heron
Exactly once in Apache HeronKarthik Ramasamy
 
Tutorial - Modern Real Time Streaming Architectures
Tutorial - Modern Real Time Streaming ArchitecturesTutorial - Modern Real Time Streaming Architectures
Tutorial - Modern Real Time Streaming ArchitecturesKarthik Ramasamy
 
Streaming Pipelines in Kubernetes Using Apache Pulsar, Heron and BookKeeper
Streaming Pipelines in Kubernetes Using Apache Pulsar, Heron and BookKeeperStreaming Pipelines in Kubernetes Using Apache Pulsar, Heron and BookKeeper
Streaming Pipelines in Kubernetes Using Apache Pulsar, Heron and BookKeeperKarthik Ramasamy
 
Twitter's Real Time Stack - Processing Billions of Events Using Distributed L...
Twitter's Real Time Stack - Processing Billions of Events Using Distributed L...Twitter's Real Time Stack - Processing Billions of Events Using Distributed L...
Twitter's Real Time Stack - Processing Billions of Events Using Distributed L...Karthik Ramasamy
 
Storm@Twitter, SIGMOD 2014 paper
Storm@Twitter, SIGMOD 2014 paperStorm@Twitter, SIGMOD 2014 paper
Storm@Twitter, SIGMOD 2014 paperKarthik Ramasamy
 
Storm@Twitter, SIGMOD 2014
Storm@Twitter, SIGMOD 2014Storm@Twitter, SIGMOD 2014
Storm@Twitter, SIGMOD 2014Karthik Ramasamy
 

More from Karthik Ramasamy (11)

Apache Pulsar Seattle - Meetup
Apache Pulsar Seattle - MeetupApache Pulsar Seattle - Meetup
Apache Pulsar Seattle - Meetup
 
Unifying Messaging, Queueing & Light Weight Compute Using Apache Pulsar
Unifying Messaging, Queueing & Light Weight Compute Using Apache PulsarUnifying Messaging, Queueing & Light Weight Compute Using Apache Pulsar
Unifying Messaging, Queueing & Light Weight Compute Using Apache Pulsar
 
Creating Data Fabric for #IOT with Apache Pulsar
Creating Data Fabric for #IOT with Apache PulsarCreating Data Fabric for #IOT with Apache Pulsar
Creating Data Fabric for #IOT with Apache Pulsar
 
Linked In Stream Processing Meetup - Apache Pulsar
Linked In Stream Processing Meetup - Apache PulsarLinked In Stream Processing Meetup - Apache Pulsar
Linked In Stream Processing Meetup - Apache Pulsar
 
Exactly once in Apache Heron
Exactly once in Apache HeronExactly once in Apache Heron
Exactly once in Apache Heron
 
Tutorial - Modern Real Time Streaming Architectures
Tutorial - Modern Real Time Streaming ArchitecturesTutorial - Modern Real Time Streaming Architectures
Tutorial - Modern Real Time Streaming Architectures
 
Streaming Pipelines in Kubernetes Using Apache Pulsar, Heron and BookKeeper
Streaming Pipelines in Kubernetes Using Apache Pulsar, Heron and BookKeeperStreaming Pipelines in Kubernetes Using Apache Pulsar, Heron and BookKeeper
Streaming Pipelines in Kubernetes Using Apache Pulsar, Heron and BookKeeper
 
Modern Data Pipelines
Modern Data PipelinesModern Data Pipelines
Modern Data Pipelines
 
Twitter's Real Time Stack - Processing Billions of Events Using Distributed L...
Twitter's Real Time Stack - Processing Billions of Events Using Distributed L...Twitter's Real Time Stack - Processing Billions of Events Using Distributed L...
Twitter's Real Time Stack - Processing Billions of Events Using Distributed L...
 
Storm@Twitter, SIGMOD 2014 paper
Storm@Twitter, SIGMOD 2014 paperStorm@Twitter, SIGMOD 2014 paper
Storm@Twitter, SIGMOD 2014 paper
 
Storm@Twitter, SIGMOD 2014
Storm@Twitter, SIGMOD 2014Storm@Twitter, SIGMOD 2014
Storm@Twitter, SIGMOD 2014
 

Recently uploaded

Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdfMicrosoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdfQ-Advise
 
Workforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdfWorkforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdfDeskTrack
 
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)Gáspár Nagy
 
INGKA DIGITAL: Linked Metadata by Design
INGKA DIGITAL: Linked Metadata by DesignINGKA DIGITAL: Linked Metadata by Design
INGKA DIGITAL: Linked Metadata by DesignNeo4j
 
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
KLARNA -  Language Models and Knowledge Graphs: A Systems ApproachKLARNA -  Language Models and Knowledge Graphs: A Systems Approach
KLARNA - Language Models and Knowledge Graphs: A Systems ApproachNeo4j
 
The Impact of PLM Software on Fashion Production
The Impact of PLM Software on Fashion ProductionThe Impact of PLM Software on Fashion Production
The Impact of PLM Software on Fashion ProductionWave PLM
 
SQL Injection Introduction and Prevention
SQL Injection Introduction and PreventionSQL Injection Introduction and Prevention
SQL Injection Introduction and PreventionMohammed Fazuluddin
 
how-to-download-files-safely-from-the-internet.pdf
how-to-download-files-safely-from-the-internet.pdfhow-to-download-files-safely-from-the-internet.pdf
how-to-download-files-safely-from-the-internet.pdfMehmet Akar
 
CompTIA Security+ (Study Notes) for cs.pdf
CompTIA Security+ (Study Notes) for cs.pdfCompTIA Security+ (Study Notes) for cs.pdf
CompTIA Security+ (Study Notes) for cs.pdfFurqanuddin10
 
AI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning FrameworkAI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning FrameworkAlluxio, Inc.
 
IT Software Development Resume, Vaibhav jha 2024
IT Software Development Resume, Vaibhav jha 2024IT Software Development Resume, Vaibhav jha 2024
IT Software Development Resume, Vaibhav jha 2024vaibhav130304
 
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdfA Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdfkalichargn70th171
 
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...rajkumar669520
 
AI/ML Infra Meetup | ML explainability in Michelangelo
AI/ML Infra Meetup | ML explainability in MichelangeloAI/ML Infra Meetup | ML explainability in Michelangelo
AI/ML Infra Meetup | ML explainability in MichelangeloAlluxio, Inc.
 
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdfImplementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdfVictor Lopez
 
Mastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdfMastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdfmbmh111980
 
A Guideline to Zendesk to Re:amaze Data Migration
A Guideline to Zendesk to Re:amaze Data MigrationA Guideline to Zendesk to Re:amaze Data Migration
A Guideline to Zendesk to Re:amaze Data MigrationHelp Desk Migration
 

Recently uploaded (20)

Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdfMicrosoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
 
Workforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdfWorkforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdf
 
AI Hackathon.pptx
AI                        Hackathon.pptxAI                        Hackathon.pptx
AI Hackathon.pptx
 
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
 
INGKA DIGITAL: Linked Metadata by Design
INGKA DIGITAL: Linked Metadata by DesignINGKA DIGITAL: Linked Metadata by Design
INGKA DIGITAL: Linked Metadata by Design
 
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
KLARNA -  Language Models and Knowledge Graphs: A Systems ApproachKLARNA -  Language Models and Knowledge Graphs: A Systems Approach
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
 
5 Reasons Driving Warehouse Management Systems Demand
5 Reasons Driving Warehouse Management Systems Demand5 Reasons Driving Warehouse Management Systems Demand
5 Reasons Driving Warehouse Management Systems Demand
 
The Impact of PLM Software on Fashion Production
The Impact of PLM Software on Fashion ProductionThe Impact of PLM Software on Fashion Production
The Impact of PLM Software on Fashion Production
 
SQL Injection Introduction and Prevention
SQL Injection Introduction and PreventionSQL Injection Introduction and Prevention
SQL Injection Introduction and Prevention
 
Top Mobile App Development Companies 2024
Top Mobile App Development Companies 2024Top Mobile App Development Companies 2024
Top Mobile App Development Companies 2024
 
how-to-download-files-safely-from-the-internet.pdf
how-to-download-files-safely-from-the-internet.pdfhow-to-download-files-safely-from-the-internet.pdf
how-to-download-files-safely-from-the-internet.pdf
 
CompTIA Security+ (Study Notes) for cs.pdf
CompTIA Security+ (Study Notes) for cs.pdfCompTIA Security+ (Study Notes) for cs.pdf
CompTIA Security+ (Study Notes) for cs.pdf
 
AI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning FrameworkAI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning Framework
 
IT Software Development Resume, Vaibhav jha 2024
IT Software Development Resume, Vaibhav jha 2024IT Software Development Resume, Vaibhav jha 2024
IT Software Development Resume, Vaibhav jha 2024
 
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdfA Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
 
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
 
AI/ML Infra Meetup | ML explainability in Michelangelo
AI/ML Infra Meetup | ML explainability in MichelangeloAI/ML Infra Meetup | ML explainability in Michelangelo
AI/ML Infra Meetup | ML explainability in Michelangelo
 
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdfImplementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdf
 
Mastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdfMastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdf
 
A Guideline to Zendesk to Re:amaze Data Migration
A Guideline to Zendesk to Re:amaze Data MigrationA Guideline to Zendesk to Re:amaze Data Migration
A Guideline to Zendesk to Re:amaze Data Migration
 

Scaling Apache Pulsar to 10 PB/day

  • 1. © 2019 SPLUNK INC. Scaling Apache Pulsar to 10 PB/day June 2021 Karthik Ramasamy Splunk
  • 2. © 2020 SPLUNK INC. Karthik Ramasamy Senior Director of Engineering @karthikz streaming @splunk | ex-CEO of @streamlio | co-creator of @heronstreaming | ex @Twitter | Ph.D
  • 3. During the course of this presentation, we may make forward‐looking statements regarding future events or plans of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results may differ materially. The forward-looking statements made in the this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, it may not contain current or accurate information. We do not assume any obligation to update any forward‐ looking statements made herein. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only, and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionalities described or to include any such feature or functionality in a future release. Splunk, Splunk>, Data-to-Everything, D2E and Turn Data Into Doing are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names or trademarks belong to their respective owners. © 2020 Splunk Inc. All rights reserved Forward- Looking Statements © 2020 SPLUNK INC.
  • 4. © 2019 SPLUNK INC. Agenda 1) Introduction to Splunk & DSP 2) Requirements, Use Cases & Deployment 3) Initial Cluster Size Estimation 4) Optimizations 5) Conclusion
  • 5. © 2020 SPLUNK INC. Data 
 Lakes Master Data Management ETL Point Data Management 
 Solutions Data 
 Silos Business Processes The 
 Data-to-Everything Platform IT Security DevOps
  • 6. © 2019 SPLUNK INC. Splunk DSP A real time stream processing solution that collects, processes and delivers data to Splunk and other destinations in milliseconds Splunk Data Stream Processor Detect Data Patterns or Conditions Mask Sensitive Data Aggregate Format Normalize Transform Filter Enhance Turn Raw Data Into 
 High-value Information Protect Sensitive Data Distribute Data To Splunk 
 Or Other Destinations Data 
 Warehouse Public 
 Cloud Message 
 Bus
  • 7. © 2019 SPLUNK INC. DSP - Bird’s Eye View HEC S2S Batch Apache Pulsar Stream Processing Engine External Systems REST Client Forwarders Data Source Splunk Indexer Apache Pulsar is at the core of DSP
  • 8. © 2019 SPLUNK INC. Customer Requirements & Deployment
  • 9. © 2019 SPLUNK INC. Customers ✦ DSP is deployed at several customer s ✦ Some of those customers are marquee customers with large volume of dat a ✦ One such marquee customer is in fi nance and payments
  • 10. © 2019 SPLUNK INC. Use Cases ✦ Microservices and applications emit log s ✦ Logs contain rich informatio n ✦ Process these logs and extract monitoring & tracing informatio n ✦ Filter these logs depending on log volume and if there is high value - justifying retentio n ✦ Compute real time business metrics
  • 11. © 2019 SPLUNK INC. Data Requirements ✦ Environment - Google Cloud Platform ✦ Use of n1-standard-32 VM s ✦ Raw data ingestion of 10 PB/day that translates ~120 GB/se c ✦ Data retention of 3 hour s ✦ Need to handle the entire traf fi c load when a zone fails
  • 12. © 2019 SPLUNK INC. DSP Ingest Cluster DSP Compute Cluster DSP Compute Cluster DSP Compute Cluster DSP Deployment Log Publisher Log Publisher Log Publisher Apache Pulsar Cluster Pipeline 1 Pipeline 2 Pipeline 3 Splunk Enterprise Splunk Observability
  • 13. © 2019 SPLUNK INC. DSP Deployment ✦ Separation of ingestion and computatio n ✦ Pipeline isolation and no noisy neighbor issue s ✦ Troubleshooting single pipeline gets easie r ✦ Might not need over provisioning except for peak load + fudge factor (as compared to deploying a single cluster)
  • 14. © 2019 SPLUNK INC. VM Configuration - n1-standard-32 ✦ 32 vCPU s ✦ 120 GB of memor y ✦ Max number of PDs (EBS equivalent) - 12 8 ✦ Max total PD size - 257 T B ✦ Max egress network bandwidth - 32 Gbps (4 GBps ) ✦ Max 24 L-SSDs for a total of 9 TB
  • 15. © 2020 SPLUNK INC. Storage Options in GCP P-SSD P-HDD L-SSD
  • 16. © 2019 SPLUNK INC. Initial Estimation
  • 17. © 2019 SPLUNK INC. Apache Pulsar Requirements ✦ Replica factor of 3 ✦ Need to handle 120 GBps of raw traf fi c ✦ Need to handle 360 GBps of storage write bandwidt h ✦ With journal required write bandwidth 720 GBp s ✦ Total storage required for retention - 3.9 P B ✦ Total ingress network bandwidth - 480 GBp s ✦ Total egress network bandwidth - 1200 GBps
  • 18. © 2019 SPLUNK INC. Pulsar Cluster Size Estimation ✦ Size of a Pulsar Cluster for a given workload depends on three parameters - ✦ Storage Density - Aggregate storage capacity needed in the cluster and proportional to retention of dat a ✦ Storage Bandwidth - Aggregate write throughput and read throughput needed for data ingestion and consumption. Heavily depends on storage medi a ✦ Network Bandwidth - Aggregate network bandwidth available in the cluster for input traf fi c and output traf fi c.
  • 19. © 2019 SPLUNK INC. Estimating Storage Bandwidth Bookie Bookie Bookie Broker Producer Journal Data Journal Data Journal Data Consumer 1X data 2X data 3X data 2X data 2X data 6X data
  • 20. © 2019 SPLUNK INC. Estimating Network Bandwidth Bookie Bookie Bookie Broker Producer Journal egress ingress egress ingress Data Journal Data Journal Data ingress ingress egress egress egress egress egress Consumer egress egress egress 1X ingress 4X egress 1X ingress 2X egress 4X ingress 10X egress 1X ingress 2X egress 1X ingress 2X egress
  • 21. © 2020 SPLUNK INC. Estimating VMs using P-HDD 0 1000 2000 3000 4000 Storage Bandwidth Storage Density Network Bandwidth 300 444 3686 VMs w/journal Max of 200 MB/sec write throughput per VM Max of 9 TB per instance Max of 4 GBps egress and ingress bandwidth Dominated by Storage Bandwidth
  • 22. © 2020 SPLUNK INC. Estimating VMs using P-SDD 0 500 1000 1500 2000 Storage Bandwidth Storage Density Network Bandwidth 300 444 1843 VMs w/journal Max of 400 MB/sec write throughput per VM Max of 9 TB per instance Max of 4 GBps egress and ingress bandwidth Dominated by Storage Bandwidth
  • 23. © 2019 SPLUNK INC. Estimating Network Bandwidth using L-HDD Bookie Bookie Bookie Broker Producer Journal ingress egress ingress Data Journal Data Journal Data ingress ingress Consumer egress egress egress 1X ingress 4X egress 1X ingress 1X ingress 1X ingress 4X ingress 5X egress
  • 24. © 2020 SPLUNK INC. Estimating VMs using L-SDD 0 225 450 675 900 Storage Bandwidth Storage Density Network Bandwidth 120 444 868 VMs w/journal Max of 850 MB/sec write throughput per VM Max of 9 TB per instance Max of 4 GBps egress and ingress bandwidth Dominated by Storage Bandwidth
  • 25. © 2020 SPLUNK INC. Estimation of VMs - Comparison 0 1000 2000 3000 4000 P-HDD P-SSD L-SSD 868 1843 3686 VMs w/journal
  • 26. © 2019 SPLUNK INC. Optimizations
  • 27. © 2019 SPLUNK INC. Optimization #1 - Eliminating Journal ✦ Different types of durabilit y ✦ Persistent Durability - No data loss in the presence of nodes failures or entire cluster failur e ✦ Replicated Durability - No data loss in the presence of limited nodes failure s ✦ Transient Durability - Data loss in the presence of failure s Since all the data is machine logs, we implemented replicated durability
  • 28. © 2019 SPLUNK INC. Replicated Durability Bookie Bookie Bookie Broker Producer Data Data Data
  • 29. © 2020 SPLUNK INC. Estimating VMs 0 ms 1000 ms 2000 ms 3000 ms 4000 ms P-HDD P-SSD L-SSD 444 922 1843 868 1843 3686 VMs w/journal VMs w/o journal Dominated by Storage Bandwidth Dominated by Storage Bandwidth Dominated by Storage Density
  • 30. © 2019 SPLUNK INC. Optimization #2 - Direct I/O ✦ Overhead of page cache in container environment is pretty hig h ✦ Kernel needs to keep track of the usage quota per container for the page cach e ✦ These translate into maintaining additional data structures and lookups (older kernel had n^2 lookup time for getting pages in & out ) ✦ Bypassed page cache for BookKeeper entry log, using JNI : ✦ We already have in memory caches (write and read-ahead ) ✦ We have better control on what to cache and when to evic t ✦ Avoid double buffering
  • 31. © 2020 SPLUNK INC. Performance of Direct I/O 0 MB/s 400 MB/s 800 MB/s 1200 MB/s 1600 MB/s P-HDD P-SSD L-SSD 1600 600 300 850 400 200 Before Direct I/O After Direct I/O
  • 32. © 2020 SPLUNK INC. Estimating VMs 0 ms 1000 ms 2000 ms 3000 ms 4000 ms P-HDD P-SSD L-SSD 444 614 1228 444 922 1843 868 1843 3686 VMs w/journal VMs w/o journal VMs with direct i/o Dominated by Storage Bandwidth Dominated by Storage Bandwidth Dominated by Storage Density
  • 33. © 2019 SPLUNK INC. Optimization #3 - Use of Compression Bookie Bookie Bookie Broker Producer Data Data Data Compressed data Consumer Compressed data C U
  • 34. © 2019 SPLUNK INC. Employing compression ✦ Compression ratio of 4 x ✦ Need to handle 360 GBps —> 90 GBps of storage write bandwidt h ✦ Total storage required for retention - 3.9 PB —> 975 T B ✦ Total ingress network bandwidth - 480 GBps —> 120 GBps ✦ Total egress network bandwidth - 1200 GBps —> 240 GBps
  • 35. © 2020 SPLUNK INC. Estimating VMs 0 ms 1000 ms 2000 ms 3000 ms 4000 ms P-HDD P-SSD L-SSD 111 154 308 444 614 1228 444 922 1843 868 1843 3686 VMs w/journal VMs w/o journal VMs with direct i/o VMs with compression Dominated by Storage Bandwidth Dominated by Storage Bandwidth Dominated by Storage Density
  • 36. © 2019 SPLUNK INC. Surviving Zone Failure Segment 1 Segment 2 Segment n .
 .
 . Segment 2 Segment 3 Segment n .
 .
 . Segment 3 Segment 1 Segment n .
 .
 . Storage Broker Serving Broker Broker ✦ Zone/Rack Failure s ✦ Bookies provide rack awarenes s ✦ Broker replicate data to different racks/zone s ✦ In the presence of zone/rack failure, data is available in other zone s ✦ One zone failure means two zones should be capable of handling the entire traf fi c ✦ Requires 50% additional VM s Zone A Zone B Zone C
  • 37. © 2020 SPLUNK INC. Estimating VMs 0 ms 1000 ms 2000 ms 3000 ms 4000 ms P-HDD P-SSD L-SSD 222 308 616 111 154 308 444 614 1228 444 922 1843 868 1843 3686 VMs w/journal VMs w/o journal VMs with direct i/o VMs with compression VMs to survive zone failure
  • 38. © 2019 SPLUNK INC. Optimization #4 - C++ Client CPU & Memory Usage ✦ Better round robin across partitions - maximizing the batch size per partitio n ✦ Having bigger batches reduces the cpu usage for client, broker and bookie s ✦ Increases the compression facto r ✦ Reduced client memory usage s ✦ Optimizations to minimize memory allocation overhead ✦ Implemented memory limit in C++ produce r ✦ Simpli fi es the user con fi guration — One single setting instead of multiple queue sizes and complex math
  • 39. © 2019 SPLUNK INC. Finally … Running 200 n1-standard-32 VMs for Pulsar cluster with 24 L-SSDs per VM
  • 40. © 2019 SPLUNK INC. OSS Contributions ✦ Github Pull Request 8283 - C++ Client is allocating buffer bigger than neede d ✦ Github Pull Request 8331 - C++ Client back-pressure is done on batches rather than number of message s ✦ Github Pull Request 8395 - C++ Implement batch aware producer router ✦ Github Pull Request 9679 - C++ Implemented memory limit in C++ producer
  • 41. © 2019 SPLUNK INC. Future Work ✦ Typical operations involve ✦ Upgrade to new version of Pulsa r ✦ Upgrade to new OS versio n ✦ Apply new security patches to O S ✦ New Pulsar version - In-place upgrade s ✦ OS & security patches - Applying one VM at a time is too slow for large cluster
  • 42. Thank You © 2019 SPLUNK INC.