SlideShare a Scribd company logo
1 of 42
Download to read offline
© 2019 SPLUNK INC.
Scaling Apache Pulsar to 10 PB/day
June 2021
Karthik Ramasamy
Splunk
© 2020 SPLUNK INC.
Karthik
Ramasamy
Senior Director of Engineering
@karthikz
streaming @splunk | ex-CEO of @streamlio | co-creator of @heronstreaming | ex @Twitter | Ph.D
During the course of this presentation, we may make forward‐looking statements
regarding future events or plans of the company. We caution you that such statements
reflect our current expectations and estimates based on factors currently known to us
and that actual events or results may differ materially. The forward-looking statements
made in the this presentation are being made as of the time and date of its live
presentation. If reviewed after its live presentation, it may not contain current or
accurate information. We do not assume any obligation to update any forward‐
looking statements made herein.


In addition, any information about our roadmap outlines our general product direction
and is subject to change at any time without notice. It is for informational purposes only,
and shall not be incorporated into any contract or other commitment. Splunk undertakes
no obligation either to develop the features or functionalities described or to include any
such feature or functionality in a future release.


Splunk, Splunk>, Data-to-Everything, D2E and Turn Data Into Doing are trademarks and registered trademarks of Splunk Inc. in the
United States and other countries. All other brand names, product names or trademarks belong to their respective owners. © 2020
Splunk Inc. All rights reserved
Forward-


Looking


Statements
© 2020 SPLUNK INC.
© 2019 SPLUNK INC.
Agenda 1) Introduction to Splunk & DSP


2) Requirements, Use Cases & Deployment


3) Initial Cluster Size Estimation


4) Optimizations


5) Conclusion
© 2020 SPLUNK INC.
Data


Lakes
Master Data
Management
ETL
Point Data
Management


Solutions
Data


Silos
Business
Processes
The


Data-to-Everything
Platform
IT
Security
DevOps
© 2019 SPLUNK INC.
Splunk DSP
A real time stream processing solution that collects, processes and delivers data to Splunk and other
destinations in milliseconds
Splunk Data Stream Processor
Detect Data Patterns or Conditions
Mask Sensitive Data
Aggregate Format
Normalize Transform
Filter Enhance
Turn Raw Data Into


High-value Information


Protect Sensitive Data
Distribute Data To Splunk


Or Other Destinations
Data


Warehouse
Public


Cloud
Message


Bus
© 2019 SPLUNK INC.
DSP - Bird’s Eye View
HEC
S2S
Batch
Apache Pulsar
Stream Processing
Engine
External
Systems
REST Client
Forwarders
Data Source
Splunk


Indexer
Apache Pulsar is at the core of DSP
© 2019 SPLUNK INC.
Customer Requirements & Deployment
© 2019 SPLUNK INC.
Customers
✦ DSP is deployed at several customer
s

✦ Some of those customers are marquee customers with large volume of dat
a

✦ One such marquee customer is in
fi
nance and payments
© 2019 SPLUNK INC.
Use Cases
✦ Microservices and applications emit log
s

✦ Logs contain rich informatio
n

✦ Process these logs and extract monitoring & tracing informatio
n

✦ Filter these logs depending on log volume and if there is high value - justifying retentio
n

✦ Compute real time business metrics
© 2019 SPLUNK INC.
Data Requirements
✦ Environment - Google Cloud Platform
✦ Use of n1-standard-32 VM
s

✦ Raw data ingestion of 10 PB/day that translates ~120 GB/se
c

✦ Data retention of 3 hour
s

✦ Need to handle the entire traf
fi
c load when a zone fails
© 2019 SPLUNK INC.
DSP Ingest Cluster
DSP Compute Cluster
DSP Compute Cluster
DSP Compute Cluster
DSP Deployment
Log
Publisher
Log
Publisher
Log
Publisher
Apache Pulsar


Cluster
Pipeline 1
Pipeline 2
Pipeline 3
Splunk
Enterprise
Splunk
Observability
© 2019 SPLUNK INC.
DSP Deployment
✦ Separation of ingestion and computatio
n

✦ Pipeline isolation and no noisy neighbor issue
s

✦ Troubleshooting single pipeline gets easie
r

✦ Might not need over provisioning except for peak load + fudge factor (as compared to deploying a single
cluster)
© 2019 SPLUNK INC.
VM Configuration - n1-standard-32
✦ 32 vCPU
s

✦ 120 GB of memor
y

✦ Max number of PDs (EBS equivalent) - 12
8

✦ Max total PD size - 257 T
B

✦ Max egress network bandwidth - 32 Gbps (4 GBps
)

✦ Max 24 L-SSDs for a total of 9 TB
© 2020 SPLUNK INC.
Storage Options in GCP
P-SSD
P-HDD L-SSD
© 2019 SPLUNK INC.
Initial Estimation
© 2019 SPLUNK INC.
Apache Pulsar Requirements
✦ Replica factor of
3

✦ Need to handle 120 GBps of raw traf
fi
c

✦ Need to handle 360 GBps of storage write bandwidt
h

✦ With journal required write bandwidth 720 GBp
s

✦ Total storage required for retention - 3.9 P
B

✦ Total ingress network bandwidth - 480 GBp
s

✦ Total egress network bandwidth - 1200 GBps
© 2019 SPLUNK INC.
Pulsar Cluster Size Estimation
✦ Size of a Pulsar Cluster for a given workload depends on three parameters -
 

✦ Storage Density - Aggregate storage capacity needed in the cluster and proportional to retention of dat
a

✦ Storage Bandwidth - Aggregate write throughput and read throughput needed for data ingestion and
consumption. Heavily depends on storage medi
a

✦ Network Bandwidth - Aggregate network bandwidth available in the cluster for input traf
fi
c and output
traf
fi
c.
© 2019 SPLUNK INC.
Estimating Storage Bandwidth
Bookie
Bookie
Bookie
Broker
Producer
Journal
Data
Journal
Data
Journal
Data
Consumer 1X data
2X data
3X data
2X data
2X data
6X data
© 2019 SPLUNK INC.
Estimating Network Bandwidth
Bookie
Bookie
Bookie
Broker
Producer
Journal
egress
ingress
egress
ingress
Data
Journal
Data
Journal
Data
ingress
ingress
egress
egress
egress
egress
egress
Consumer
egress
egress
egress
1X ingress


4X egress
1X ingress


2X egress
4X ingress


10X egress
1X ingress


2X egress
1X ingress


2X egress
© 2020 SPLUNK INC.
Estimating VMs using P-HDD
0
1000
2000
3000
4000
Storage Bandwidth Storage Density Network Bandwidth
300
444
3686
VMs w/journal
Max of 200 MB/sec write
throughput per VM
Max of 9 TB


per instance
Max of 4 GBps egress
and ingress bandwidth
Dominated by
Storage Bandwidth
© 2020 SPLUNK INC.
Estimating VMs using P-SDD
0
500
1000
1500
2000
Storage Bandwidth Storage Density Network Bandwidth
300
444
1843
VMs w/journal
Max of 400 MB/sec write
throughput per VM
Max of 9 TB


per instance
Max of 4 GBps egress
and ingress bandwidth
Dominated by
Storage Bandwidth
© 2019 SPLUNK INC.
Estimating Network Bandwidth using L-HDD
Bookie
Bookie
Bookie
Broker
Producer
Journal
ingress
egress
ingress
Data
Journal
Data
Journal
Data
ingress
ingress
Consumer
egress
egress
egress
1X ingress


4X egress
1X ingress
1X ingress
1X ingress
4X ingress


5X egress
© 2020 SPLUNK INC.
Estimating VMs using L-SDD
0
225
450
675
900
Storage Bandwidth Storage Density Network Bandwidth
120
444
868
VMs w/journal
Max of 850 MB/sec write
throughput per VM
Max of 9 TB


per instance
Max of 4 GBps egress
and ingress bandwidth
Dominated by
Storage Bandwidth
© 2020 SPLUNK INC.
Estimation of VMs - Comparison
0
1000
2000
3000
4000
P-HDD P-SSD L-SSD
868
1843
3686
VMs w/journal
© 2019 SPLUNK INC.
Optimizations
© 2019 SPLUNK INC.
Optimization #1 - Eliminating Journal
✦ Different types of durabilit
y

✦ Persistent Durability - No data loss in the presence of nodes failures or entire cluster failur
e

✦ Replicated Durability - No data loss in the presence of limited nodes failure
s

✦ Transient Durability - Data loss in the presence of failure
s

Since all the data is machine logs, we implemented replicated durability
© 2019 SPLUNK INC.
Replicated Durability
Bookie
Bookie
Bookie
Broker
Producer
Data
Data
Data
© 2020 SPLUNK INC.
Estimating VMs
0 ms
1000 ms
2000 ms
3000 ms
4000 ms
P-HDD P-SSD L-SSD
444
922
1843
868
1843
3686
VMs w/journal VMs w/o journal
Dominated by
Storage Bandwidth
Dominated by
Storage Bandwidth
Dominated by
Storage Density
© 2019 SPLUNK INC.
Optimization #2 - Direct I/O
✦ Overhead of page cache in container environment is pretty hig
h

✦ Kernel needs to keep track of the usage quota per container for the page cach
e

✦ These translate into maintaining additional data structures and lookups (older kernel had n^2 lookup time for
getting pages in & out
)

✦ Bypassed page cache for BookKeeper entry log, using JNI
:

✦ We already have in memory caches (write and read-ahead
)

✦ We have better control on what to cache and when to evic
t

✦ Avoid double buffering
© 2020 SPLUNK INC.
Performance of Direct I/O
0 MB/s
400 MB/s
800 MB/s
1200 MB/s
1600 MB/s
P-HDD P-SSD L-SSD
1600
600
300
850
400
200
Before Direct I/O After Direct I/O
© 2020 SPLUNK INC.
Estimating VMs
0 ms
1000 ms
2000 ms
3000 ms
4000 ms
P-HDD P-SSD L-SSD
444
614
1228
444
922
1843
868
1843
3686
VMs w/journal VMs w/o journal VMs with direct i/o
Dominated by
Storage Bandwidth
Dominated by
Storage Bandwidth Dominated by
Storage Density
© 2019 SPLUNK INC.
Optimization #3 - Use of Compression
Bookie
Bookie
Bookie
Broker
Producer
Data
Data
Data
Compressed data
Consumer Compressed data
C
U
© 2019 SPLUNK INC.
Employing compression
✦ Compression ratio of 4
x

✦ Need to handle 360 GBps —> 90 GBps of storage write bandwidt
h

✦ Total storage required for retention - 3.9 PB —> 975 T
B

✦ Total ingress network bandwidth - 480 GBps —> 120 GBps
 

✦ Total egress network bandwidth - 1200 GBps —> 240 GBps
© 2020 SPLUNK INC.
Estimating VMs
0 ms
1000 ms
2000 ms
3000 ms
4000 ms
P-HDD P-SSD L-SSD
111
154
308
444
614
1228
444
922
1843
868
1843
3686
VMs w/journal VMs w/o journal VMs with direct i/o VMs with compression
Dominated by
Storage Bandwidth Dominated by
Storage Bandwidth Dominated by
Storage Density
© 2019 SPLUNK INC.
Surviving Zone Failure
Segment 1
Segment 2
Segment n
.

.

.
Segment 2
Segment 3
Segment n
.

.

.
Segment 3
Segment 1
Segment n
.

.

.
Storage
Broker
Serving
Broker Broker
✦ Zone/Rack Failure
s

✦ Bookies provide rack awarenes
s

✦ Broker replicate data to different racks/zone
s

✦ In the presence of zone/rack failure, data is available
in other zone
s

✦ One zone failure means two zones should be capable of
handling the entire traf
fi
c

✦ Requires 50% additional VM
s

Zone A Zone B Zone C
© 2020 SPLUNK INC.
Estimating VMs
0 ms
1000 ms
2000 ms
3000 ms
4000 ms
P-HDD P-SSD L-SSD
222
308
616
111
154
308
444
614
1228
444
922
1843
868
1843
3686
VMs w/journal VMs w/o journal VMs with direct i/o VMs with compression VMs to survive zone failure
© 2019 SPLUNK INC.
Optimization #4 - C++ Client CPU & Memory Usage
✦ Better round robin across partitions - maximizing the batch size per partitio
n

✦ Having bigger batches reduces the cpu usage for client, broker and bookie
s

✦ Increases the compression facto
r

✦ Reduced client memory usage
s

✦ Optimizations to minimize memory allocation overhead
 

✦ Implemented memory limit in C++ produce
r

✦ Simpli
fi
es the user con
fi
guration — One single setting instead of multiple queue sizes and complex
math
© 2019 SPLUNK INC.
Finally …
Running 200 n1-standard-32 VMs for Pulsar cluster with 24 L-SSDs per VM
© 2019 SPLUNK INC.
OSS Contributions
✦ Github Pull Request 8283 - C++ Client is allocating buffer bigger than neede
d

✦ Github Pull Request 8331 - C++ Client back-pressure is done on batches rather than number of message
s

✦ Github Pull Request 8395 - C++ Implement batch aware producer router
 

✦ Github Pull Request 9679 - C++ Implemented memory limit in C++ producer
© 2019 SPLUNK INC.
Future Work
✦ Typical operations involve
 

✦ Upgrade to new version of Pulsa
r

✦ Upgrade to new OS versio
n

✦ Apply new security patches to O
S

✦ New Pulsar version - In-place upgrade
s

✦ OS & security patches - Applying one VM at a time is too slow for large cluster
Thank You
© 2019 SPLUNK INC.

More Related Content

What's hot

RocksDB detail
RocksDB detailRocksDB detail
RocksDB detailMIJIN AN
 
Oracle 12c PDB insights
Oracle 12c PDB insightsOracle 12c PDB insights
Oracle 12c PDB insightsKirill Loifman
 
Getting started with postgresql
Getting started with postgresqlGetting started with postgresql
Getting started with postgresqlbotsplash.com
 
Pulsar in the Lakehouse: Apache Pulsar™ with Apache Spark™ and Delta Lake - P...
Pulsar in the Lakehouse: Apache Pulsar™ with Apache Spark™ and Delta Lake - P...Pulsar in the Lakehouse: Apache Pulsar™ with Apache Spark™ and Delta Lake - P...
Pulsar in the Lakehouse: Apache Pulsar™ with Apache Spark™ and Delta Lake - P...StreamNative
 
Apache BookKeeper State Store: A Durable Key-Value Store - Pulsar Summit NA 2021
Apache BookKeeper State Store: A Durable Key-Value Store - Pulsar Summit NA 2021Apache BookKeeper State Store: A Durable Key-Value Store - Pulsar Summit NA 2021
Apache BookKeeper State Store: A Durable Key-Value Store - Pulsar Summit NA 2021StreamNative
 
GoldenGateテクニカルセミナー4「テクニカルコンサルタントが語るOracle GoldenGate現場で使える極意」(2016/5/11)
GoldenGateテクニカルセミナー4「テクニカルコンサルタントが語るOracle GoldenGate現場で使える極意」(2016/5/11)GoldenGateテクニカルセミナー4「テクニカルコンサルタントが語るOracle GoldenGate現場で使える極意」(2016/5/11)
GoldenGateテクニカルセミナー4「テクニカルコンサルタントが語るOracle GoldenGate現場で使える極意」(2016/5/11)オラクルエンジニア通信
 
Oracle RAC 19c: Best Practices and Secret Internals
Oracle RAC 19c: Best Practices and Secret InternalsOracle RAC 19c: Best Practices and Secret Internals
Oracle RAC 19c: Best Practices and Secret InternalsAnil Nair
 
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013mumrah
 
Batch Processing at Scale with Flink & Iceberg
Batch Processing at Scale with Flink & IcebergBatch Processing at Scale with Flink & Iceberg
Batch Processing at Scale with Flink & IcebergFlink Forward
 
RocksDB compaction
RocksDB compactionRocksDB compaction
RocksDB compactionMIJIN AN
 
Oracle_Multitenant_19c_-_All_About_Pluggable_D.pdf
Oracle_Multitenant_19c_-_All_About_Pluggable_D.pdfOracle_Multitenant_19c_-_All_About_Pluggable_D.pdf
Oracle_Multitenant_19c_-_All_About_Pluggable_D.pdfSrirakshaSrinivasan2
 
Percona XtraDB Cluster ( Ensure high Availability )
Percona XtraDB Cluster ( Ensure high Availability )Percona XtraDB Cluster ( Ensure high Availability )
Percona XtraDB Cluster ( Ensure high Availability )Mydbops
 
Deep Dive: Memory Management in Apache Spark
Deep Dive: Memory Management in Apache SparkDeep Dive: Memory Management in Apache Spark
Deep Dive: Memory Management in Apache SparkDatabricks
 
Premier Inside-Out: Apache Druid
Premier Inside-Out: Apache DruidPremier Inside-Out: Apache Druid
Premier Inside-Out: Apache DruidHortonworks
 
PostgreSQL HA
PostgreSQL   HAPostgreSQL   HA
PostgreSQL HAharoonm
 
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...Databricks
 
GoldenGateテクニカルセミナー3「Oracle GoldenGate Technical Deep Dive」(2016/5/11)
GoldenGateテクニカルセミナー3「Oracle GoldenGate Technical Deep Dive」(2016/5/11)GoldenGateテクニカルセミナー3「Oracle GoldenGate Technical Deep Dive」(2016/5/11)
GoldenGateテクニカルセミナー3「Oracle GoldenGate Technical Deep Dive」(2016/5/11)オラクルエンジニア通信
 
Understanding oracle rac internals part 1 - slides
Understanding oracle rac internals   part 1 - slidesUnderstanding oracle rac internals   part 1 - slides
Understanding oracle rac internals part 1 - slidesMohamed Farouk
 
[Oracle DBA & Developer Day 2016] しばちょう先生の特別講義!!ストレージ管理のベストプラクティス ~ASMからExada...
[Oracle DBA & Developer Day 2016] しばちょう先生の特別講義!!ストレージ管理のベストプラクティス ~ASMからExada...[Oracle DBA & Developer Day 2016] しばちょう先生の特別講義!!ストレージ管理のベストプラクティス ~ASMからExada...
[Oracle DBA & Developer Day 2016] しばちょう先生の特別講義!!ストレージ管理のベストプラクティス ~ASMからExada...オラクルエンジニア通信
 

What's hot (20)

RocksDB detail
RocksDB detailRocksDB detail
RocksDB detail
 
Oracle 12c PDB insights
Oracle 12c PDB insightsOracle 12c PDB insights
Oracle 12c PDB insights
 
Getting started with postgresql
Getting started with postgresqlGetting started with postgresql
Getting started with postgresql
 
Treinamento Oracle GoldenGate 19c
Treinamento Oracle GoldenGate 19cTreinamento Oracle GoldenGate 19c
Treinamento Oracle GoldenGate 19c
 
Pulsar in the Lakehouse: Apache Pulsar™ with Apache Spark™ and Delta Lake - P...
Pulsar in the Lakehouse: Apache Pulsar™ with Apache Spark™ and Delta Lake - P...Pulsar in the Lakehouse: Apache Pulsar™ with Apache Spark™ and Delta Lake - P...
Pulsar in the Lakehouse: Apache Pulsar™ with Apache Spark™ and Delta Lake - P...
 
Apache BookKeeper State Store: A Durable Key-Value Store - Pulsar Summit NA 2021
Apache BookKeeper State Store: A Durable Key-Value Store - Pulsar Summit NA 2021Apache BookKeeper State Store: A Durable Key-Value Store - Pulsar Summit NA 2021
Apache BookKeeper State Store: A Durable Key-Value Store - Pulsar Summit NA 2021
 
GoldenGateテクニカルセミナー4「テクニカルコンサルタントが語るOracle GoldenGate現場で使える極意」(2016/5/11)
GoldenGateテクニカルセミナー4「テクニカルコンサルタントが語るOracle GoldenGate現場で使える極意」(2016/5/11)GoldenGateテクニカルセミナー4「テクニカルコンサルタントが語るOracle GoldenGate現場で使える極意」(2016/5/11)
GoldenGateテクニカルセミナー4「テクニカルコンサルタントが語るOracle GoldenGate現場で使える極意」(2016/5/11)
 
Oracle RAC 19c: Best Practices and Secret Internals
Oracle RAC 19c: Best Practices and Secret InternalsOracle RAC 19c: Best Practices and Secret Internals
Oracle RAC 19c: Best Practices and Secret Internals
 
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
 
Batch Processing at Scale with Flink & Iceberg
Batch Processing at Scale with Flink & IcebergBatch Processing at Scale with Flink & Iceberg
Batch Processing at Scale with Flink & Iceberg
 
RocksDB compaction
RocksDB compactionRocksDB compaction
RocksDB compaction
 
Oracle_Multitenant_19c_-_All_About_Pluggable_D.pdf
Oracle_Multitenant_19c_-_All_About_Pluggable_D.pdfOracle_Multitenant_19c_-_All_About_Pluggable_D.pdf
Oracle_Multitenant_19c_-_All_About_Pluggable_D.pdf
 
Percona XtraDB Cluster ( Ensure high Availability )
Percona XtraDB Cluster ( Ensure high Availability )Percona XtraDB Cluster ( Ensure high Availability )
Percona XtraDB Cluster ( Ensure high Availability )
 
Deep Dive: Memory Management in Apache Spark
Deep Dive: Memory Management in Apache SparkDeep Dive: Memory Management in Apache Spark
Deep Dive: Memory Management in Apache Spark
 
Premier Inside-Out: Apache Druid
Premier Inside-Out: Apache DruidPremier Inside-Out: Apache Druid
Premier Inside-Out: Apache Druid
 
PostgreSQL HA
PostgreSQL   HAPostgreSQL   HA
PostgreSQL HA
 
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
 
GoldenGateテクニカルセミナー3「Oracle GoldenGate Technical Deep Dive」(2016/5/11)
GoldenGateテクニカルセミナー3「Oracle GoldenGate Technical Deep Dive」(2016/5/11)GoldenGateテクニカルセミナー3「Oracle GoldenGate Technical Deep Dive」(2016/5/11)
GoldenGateテクニカルセミナー3「Oracle GoldenGate Technical Deep Dive」(2016/5/11)
 
Understanding oracle rac internals part 1 - slides
Understanding oracle rac internals   part 1 - slidesUnderstanding oracle rac internals   part 1 - slides
Understanding oracle rac internals part 1 - slides
 
[Oracle DBA & Developer Day 2016] しばちょう先生の特別講義!!ストレージ管理のベストプラクティス ~ASMからExada...
[Oracle DBA & Developer Day 2016] しばちょう先生の特別講義!!ストレージ管理のベストプラクティス ~ASMからExada...[Oracle DBA & Developer Day 2016] しばちょう先生の特別講義!!ストレージ管理のベストプラクティス ~ASMからExada...
[Oracle DBA & Developer Day 2016] しばちょう先生の特別講義!!ストレージ管理のベストプラクティス ~ASMからExada...
 

Similar to Scaling Apache Pulsar to 10 PB/day

Scaling Apache Pulsar to 10 Petabytes/Day
Scaling Apache Pulsar to 10 Petabytes/DayScaling Apache Pulsar to 10 Petabytes/Day
Scaling Apache Pulsar to 10 Petabytes/DayScyllaDB
 
Pulsar summit-keynote-final
Pulsar summit-keynote-finalPulsar summit-keynote-final
Pulsar summit-keynote-finalKarthik Ramasamy
 
Toward Scalable and Powerful CloudStack
Toward Scalable and Powerful CloudStackToward Scalable and Powerful CloudStack
Toward Scalable and Powerful CloudStackTakashi Kanai
 
DDN: Protecting Your Data, Protecting Your Hardware
DDN: Protecting Your Data, Protecting Your HardwareDDN: Protecting Your Data, Protecting Your Hardware
DDN: Protecting Your Data, Protecting Your Hardwareinside-BigData.com
 
Sizing Splunk SmartStore - Spend Less and Get More Out of Splunk
Sizing Splunk SmartStore - Spend Less and Get More Out of SplunkSizing Splunk SmartStore - Spend Less and Get More Out of Splunk
Sizing Splunk SmartStore - Spend Less and Get More Out of SplunkPaula Koziol
 
IBM Power Systems - enabling cloud solutions
IBM Power Systems - enabling cloud solutionsIBM Power Systems - enabling cloud solutions
IBM Power Systems - enabling cloud solutionsDavid Spurway
 
MongoDB World 2019: Implementation and Operationalization of MongoDB Sharding...
MongoDB World 2019: Implementation and Operationalization of MongoDB Sharding...MongoDB World 2019: Implementation and Operationalization of MongoDB Sharding...
MongoDB World 2019: Implementation and Operationalization of MongoDB Sharding...MongoDB
 
Open power topics20191023
Open power topics20191023Open power topics20191023
Open power topics20191023Yutaka Kawai
 
HP flash optimized storage - webcast
HP flash optimized storage - webcastHP flash optimized storage - webcast
HP flash optimized storage - webcastCalvin Zito
 
Delivering Apache Hadoop for the Modern Data Architecture
Delivering Apache Hadoop for the Modern Data Architecture Delivering Apache Hadoop for the Modern Data Architecture
Delivering Apache Hadoop for the Modern Data Architecture Hortonworks
 
IBM System Storage DS8000 with SSDs An In-Depth Look at SSD Performance in th...
IBM System Storage DS8000 with SSDs An In-Depth Look at SSD Performance in th...IBM System Storage DS8000 with SSDs An In-Depth Look at SSD Performance in th...
IBM System Storage DS8000 with SSDs An In-Depth Look at SSD Performance in th...IBM India Smarter Computing
 
Accelerate and Scale Big Data Analytics with Disaggregated Compute and Storage
Accelerate and Scale Big Data Analytics with Disaggregated Compute and StorageAccelerate and Scale Big Data Analytics with Disaggregated Compute and Storage
Accelerate and Scale Big Data Analytics with Disaggregated Compute and StorageAlluxio, Inc.
 
#IBMEdge: Flash Storage Session
#IBMEdge: Flash Storage Session#IBMEdge: Flash Storage Session
#IBMEdge: Flash Storage SessionBrocade
 
Aerospike Roadmap Overview - Meetup Dec 2019
Aerospike Roadmap Overview - Meetup Dec 2019Aerospike Roadmap Overview - Meetup Dec 2019
Aerospike Roadmap Overview - Meetup Dec 2019Aerospike
 
MongoDB World 2018: Managing a Mission Critical eCommerce Application on Mong...
MongoDB World 2018: Managing a Mission Critical eCommerce Application on Mong...MongoDB World 2018: Managing a Mission Critical eCommerce Application on Mong...
MongoDB World 2018: Managing a Mission Critical eCommerce Application on Mong...MongoDB
 
Model-driven Telemetry: The Foundation of Big Data Analytics
Model-driven Telemetry: The Foundation of Big Data AnalyticsModel-driven Telemetry: The Foundation of Big Data Analytics
Model-driven Telemetry: The Foundation of Big Data AnalyticsCisco Canada
 
Splunk und Multi-Cloud
Splunk und Multi-CloudSplunk und Multi-Cloud
Splunk und Multi-CloudSplunk
 
Splunk and Multicloud
Splunk and Multicloud Splunk and Multicloud
Splunk and Multicloud Splunk
 

Similar to Scaling Apache Pulsar to 10 PB/day (20)

Scaling Apache Pulsar to 10 Petabytes/Day
Scaling Apache Pulsar to 10 Petabytes/DayScaling Apache Pulsar to 10 Petabytes/Day
Scaling Apache Pulsar to 10 Petabytes/Day
 
Pulsar summit-keynote-final
Pulsar summit-keynote-finalPulsar summit-keynote-final
Pulsar summit-keynote-final
 
Toward Scalable and Powerful CloudStack
Toward Scalable and Powerful CloudStackToward Scalable and Powerful CloudStack
Toward Scalable and Powerful CloudStack
 
DDN: Protecting Your Data, Protecting Your Hardware
DDN: Protecting Your Data, Protecting Your HardwareDDN: Protecting Your Data, Protecting Your Hardware
DDN: Protecting Your Data, Protecting Your Hardware
 
Apache Pulsar @Splunk
Apache Pulsar @SplunkApache Pulsar @Splunk
Apache Pulsar @Splunk
 
Sizing Splunk SmartStore - Spend Less and Get More Out of Splunk
Sizing Splunk SmartStore - Spend Less and Get More Out of SplunkSizing Splunk SmartStore - Spend Less and Get More Out of Splunk
Sizing Splunk SmartStore - Spend Less and Get More Out of Splunk
 
IBM Power Systems - enabling cloud solutions
IBM Power Systems - enabling cloud solutionsIBM Power Systems - enabling cloud solutions
IBM Power Systems - enabling cloud solutions
 
MongoDB World 2019: Implementation and Operationalization of MongoDB Sharding...
MongoDB World 2019: Implementation and Operationalization of MongoDB Sharding...MongoDB World 2019: Implementation and Operationalization of MongoDB Sharding...
MongoDB World 2019: Implementation and Operationalization of MongoDB Sharding...
 
Open power topics20191023
Open power topics20191023Open power topics20191023
Open power topics20191023
 
HP flash optimized storage - webcast
HP flash optimized storage - webcastHP flash optimized storage - webcast
HP flash optimized storage - webcast
 
Delivering Apache Hadoop for the Modern Data Architecture
Delivering Apache Hadoop for the Modern Data Architecture Delivering Apache Hadoop for the Modern Data Architecture
Delivering Apache Hadoop for the Modern Data Architecture
 
IBM System Storage DS8000 with SSDs An In-Depth Look at SSD Performance in th...
IBM System Storage DS8000 with SSDs An In-Depth Look at SSD Performance in th...IBM System Storage DS8000 with SSDs An In-Depth Look at SSD Performance in th...
IBM System Storage DS8000 with SSDs An In-Depth Look at SSD Performance in th...
 
Accelerate and Scale Big Data Analytics with Disaggregated Compute and Storage
Accelerate and Scale Big Data Analytics with Disaggregated Compute and StorageAccelerate and Scale Big Data Analytics with Disaggregated Compute and Storage
Accelerate and Scale Big Data Analytics with Disaggregated Compute and Storage
 
#IBMEdge: Flash Storage Session
#IBMEdge: Flash Storage Session#IBMEdge: Flash Storage Session
#IBMEdge: Flash Storage Session
 
Aerospike Roadmap Overview - Meetup Dec 2019
Aerospike Roadmap Overview - Meetup Dec 2019Aerospike Roadmap Overview - Meetup Dec 2019
Aerospike Roadmap Overview - Meetup Dec 2019
 
IBM PureSystems
IBM PureSystemsIBM PureSystems
IBM PureSystems
 
MongoDB World 2018: Managing a Mission Critical eCommerce Application on Mong...
MongoDB World 2018: Managing a Mission Critical eCommerce Application on Mong...MongoDB World 2018: Managing a Mission Critical eCommerce Application on Mong...
MongoDB World 2018: Managing a Mission Critical eCommerce Application on Mong...
 
Model-driven Telemetry: The Foundation of Big Data Analytics
Model-driven Telemetry: The Foundation of Big Data AnalyticsModel-driven Telemetry: The Foundation of Big Data Analytics
Model-driven Telemetry: The Foundation of Big Data Analytics
 
Splunk und Multi-Cloud
Splunk und Multi-CloudSplunk und Multi-Cloud
Splunk und Multi-Cloud
 
Splunk and Multicloud
Splunk and Multicloud Splunk and Multicloud
Splunk and Multicloud
 

More from Karthik Ramasamy

Apache Pulsar Seattle - Meetup
Apache Pulsar Seattle - MeetupApache Pulsar Seattle - Meetup
Apache Pulsar Seattle - MeetupKarthik Ramasamy
 
Unifying Messaging, Queueing & Light Weight Compute Using Apache Pulsar
Unifying Messaging, Queueing & Light Weight Compute Using Apache PulsarUnifying Messaging, Queueing & Light Weight Compute Using Apache Pulsar
Unifying Messaging, Queueing & Light Weight Compute Using Apache PulsarKarthik Ramasamy
 
Creating Data Fabric for #IOT with Apache Pulsar
Creating Data Fabric for #IOT with Apache PulsarCreating Data Fabric for #IOT with Apache Pulsar
Creating Data Fabric for #IOT with Apache PulsarKarthik Ramasamy
 
Linked In Stream Processing Meetup - Apache Pulsar
Linked In Stream Processing Meetup - Apache PulsarLinked In Stream Processing Meetup - Apache Pulsar
Linked In Stream Processing Meetup - Apache PulsarKarthik Ramasamy
 
Exactly once in Apache Heron
Exactly once in Apache HeronExactly once in Apache Heron
Exactly once in Apache HeronKarthik Ramasamy
 
Streaming Pipelines in Kubernetes Using Apache Pulsar, Heron and BookKeeper
Streaming Pipelines in Kubernetes Using Apache Pulsar, Heron and BookKeeperStreaming Pipelines in Kubernetes Using Apache Pulsar, Heron and BookKeeper
Streaming Pipelines in Kubernetes Using Apache Pulsar, Heron and BookKeeperKarthik Ramasamy
 
Twitter's Real Time Stack - Processing Billions of Events Using Distributed L...
Twitter's Real Time Stack - Processing Billions of Events Using Distributed L...Twitter's Real Time Stack - Processing Billions of Events Using Distributed L...
Twitter's Real Time Stack - Processing Billions of Events Using Distributed L...Karthik Ramasamy
 
Storm@Twitter, SIGMOD 2014 paper
Storm@Twitter, SIGMOD 2014 paperStorm@Twitter, SIGMOD 2014 paper
Storm@Twitter, SIGMOD 2014 paperKarthik Ramasamy
 
Storm@Twitter, SIGMOD 2014
Storm@Twitter, SIGMOD 2014Storm@Twitter, SIGMOD 2014
Storm@Twitter, SIGMOD 2014Karthik Ramasamy
 

More from Karthik Ramasamy (10)

Apache Pulsar Seattle - Meetup
Apache Pulsar Seattle - MeetupApache Pulsar Seattle - Meetup
Apache Pulsar Seattle - Meetup
 
Unifying Messaging, Queueing & Light Weight Compute Using Apache Pulsar
Unifying Messaging, Queueing & Light Weight Compute Using Apache PulsarUnifying Messaging, Queueing & Light Weight Compute Using Apache Pulsar
Unifying Messaging, Queueing & Light Weight Compute Using Apache Pulsar
 
Creating Data Fabric for #IOT with Apache Pulsar
Creating Data Fabric for #IOT with Apache PulsarCreating Data Fabric for #IOT with Apache Pulsar
Creating Data Fabric for #IOT with Apache Pulsar
 
Linked In Stream Processing Meetup - Apache Pulsar
Linked In Stream Processing Meetup - Apache PulsarLinked In Stream Processing Meetup - Apache Pulsar
Linked In Stream Processing Meetup - Apache Pulsar
 
Exactly once in Apache Heron
Exactly once in Apache HeronExactly once in Apache Heron
Exactly once in Apache Heron
 
Streaming Pipelines in Kubernetes Using Apache Pulsar, Heron and BookKeeper
Streaming Pipelines in Kubernetes Using Apache Pulsar, Heron and BookKeeperStreaming Pipelines in Kubernetes Using Apache Pulsar, Heron and BookKeeper
Streaming Pipelines in Kubernetes Using Apache Pulsar, Heron and BookKeeper
 
Modern Data Pipelines
Modern Data PipelinesModern Data Pipelines
Modern Data Pipelines
 
Twitter's Real Time Stack - Processing Billions of Events Using Distributed L...
Twitter's Real Time Stack - Processing Billions of Events Using Distributed L...Twitter's Real Time Stack - Processing Billions of Events Using Distributed L...
Twitter's Real Time Stack - Processing Billions of Events Using Distributed L...
 
Storm@Twitter, SIGMOD 2014 paper
Storm@Twitter, SIGMOD 2014 paperStorm@Twitter, SIGMOD 2014 paper
Storm@Twitter, SIGMOD 2014 paper
 
Storm@Twitter, SIGMOD 2014
Storm@Twitter, SIGMOD 2014Storm@Twitter, SIGMOD 2014
Storm@Twitter, SIGMOD 2014
 

Recently uploaded

SensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving CarsSensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving CarsChristian Birchler
 
Odoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 EnterpriseOdoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 Enterprisepreethippts
 
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
Balasore Best It Company|| Top 10 IT Company || Balasore Software company OdishaBalasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odishasmiwainfosol
 
Unveiling the Future: Sylius 2.0 New Features
Unveiling the Future: Sylius 2.0 New FeaturesUnveiling the Future: Sylius 2.0 New Features
Unveiling the Future: Sylius 2.0 New FeaturesŁukasz Chruściel
 
Salesforce Implementation Services PPT By ABSYZ
Salesforce Implementation Services PPT By ABSYZSalesforce Implementation Services PPT By ABSYZ
Salesforce Implementation Services PPT By ABSYZABSYZ Inc
 
Intelligent Home Wi-Fi Solutions | ThinkPalm
Intelligent Home Wi-Fi Solutions | ThinkPalmIntelligent Home Wi-Fi Solutions | ThinkPalm
Intelligent Home Wi-Fi Solutions | ThinkPalmSujith Sukumaran
 
How To Manage Restaurant Staff -BTRESTRO
How To Manage Restaurant Staff -BTRESTROHow To Manage Restaurant Staff -BTRESTRO
How To Manage Restaurant Staff -BTRESTROmotivationalword821
 
20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...
20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...
20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...Akihiro Suda
 
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024StefanoLambiase
 
Sending Calendar Invites on SES and Calendarsnack.pdf
Sending Calendar Invites on SES and Calendarsnack.pdfSending Calendar Invites on SES and Calendarsnack.pdf
Sending Calendar Invites on SES and Calendarsnack.pdf31events.com
 
CRM Contender Series: HubSpot vs. Salesforce
CRM Contender Series: HubSpot vs. SalesforceCRM Contender Series: HubSpot vs. Salesforce
CRM Contender Series: HubSpot vs. SalesforceBrainSell Technologies
 
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...OnePlan Solutions
 
UI5ers live - Custom Controls wrapping 3rd-party libs.pptx
UI5ers live - Custom Controls wrapping 3rd-party libs.pptxUI5ers live - Custom Controls wrapping 3rd-party libs.pptx
UI5ers live - Custom Controls wrapping 3rd-party libs.pptxAndreas Kunz
 
What is Advanced Excel and what are some best practices for designing and cre...
What is Advanced Excel and what are some best practices for designing and cre...What is Advanced Excel and what are some best practices for designing and cre...
What is Advanced Excel and what are some best practices for designing and cre...Technogeeks
 
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...Angel Borroy López
 
Open Source Summit NA 2024: Open Source Cloud Costs - OpenCost's Impact on En...
Open Source Summit NA 2024: Open Source Cloud Costs - OpenCost's Impact on En...Open Source Summit NA 2024: Open Source Cloud Costs - OpenCost's Impact on En...
Open Source Summit NA 2024: Open Source Cloud Costs - OpenCost's Impact on En...Matt Ray
 
How to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion ApplicationHow to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion ApplicationBradBedford3
 
Simplifying Microservices & Apps - The art of effortless development - Meetup...
Simplifying Microservices & Apps - The art of effortless development - Meetup...Simplifying Microservices & Apps - The art of effortless development - Meetup...
Simplifying Microservices & Apps - The art of effortless development - Meetup...Rob Geurden
 
Machine Learning Software Engineering Patterns and Their Engineering
Machine Learning Software Engineering Patterns and Their EngineeringMachine Learning Software Engineering Patterns and Their Engineering
Machine Learning Software Engineering Patterns and Their EngineeringHironori Washizaki
 
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdfGOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdfAlina Yurenko
 

Recently uploaded (20)

SensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving CarsSensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving Cars
 
Odoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 EnterpriseOdoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 Enterprise
 
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
Balasore Best It Company|| Top 10 IT Company || Balasore Software company OdishaBalasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
 
Unveiling the Future: Sylius 2.0 New Features
Unveiling the Future: Sylius 2.0 New FeaturesUnveiling the Future: Sylius 2.0 New Features
Unveiling the Future: Sylius 2.0 New Features
 
Salesforce Implementation Services PPT By ABSYZ
Salesforce Implementation Services PPT By ABSYZSalesforce Implementation Services PPT By ABSYZ
Salesforce Implementation Services PPT By ABSYZ
 
Intelligent Home Wi-Fi Solutions | ThinkPalm
Intelligent Home Wi-Fi Solutions | ThinkPalmIntelligent Home Wi-Fi Solutions | ThinkPalm
Intelligent Home Wi-Fi Solutions | ThinkPalm
 
How To Manage Restaurant Staff -BTRESTRO
How To Manage Restaurant Staff -BTRESTROHow To Manage Restaurant Staff -BTRESTRO
How To Manage Restaurant Staff -BTRESTRO
 
20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...
20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...
20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...
 
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
 
Sending Calendar Invites on SES and Calendarsnack.pdf
Sending Calendar Invites on SES and Calendarsnack.pdfSending Calendar Invites on SES and Calendarsnack.pdf
Sending Calendar Invites on SES and Calendarsnack.pdf
 
CRM Contender Series: HubSpot vs. Salesforce
CRM Contender Series: HubSpot vs. SalesforceCRM Contender Series: HubSpot vs. Salesforce
CRM Contender Series: HubSpot vs. Salesforce
 
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...
 
UI5ers live - Custom Controls wrapping 3rd-party libs.pptx
UI5ers live - Custom Controls wrapping 3rd-party libs.pptxUI5ers live - Custom Controls wrapping 3rd-party libs.pptx
UI5ers live - Custom Controls wrapping 3rd-party libs.pptx
 
What is Advanced Excel and what are some best practices for designing and cre...
What is Advanced Excel and what are some best practices for designing and cre...What is Advanced Excel and what are some best practices for designing and cre...
What is Advanced Excel and what are some best practices for designing and cre...
 
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
 
Open Source Summit NA 2024: Open Source Cloud Costs - OpenCost's Impact on En...
Open Source Summit NA 2024: Open Source Cloud Costs - OpenCost's Impact on En...Open Source Summit NA 2024: Open Source Cloud Costs - OpenCost's Impact on En...
Open Source Summit NA 2024: Open Source Cloud Costs - OpenCost's Impact on En...
 
How to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion ApplicationHow to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion Application
 
Simplifying Microservices & Apps - The art of effortless development - Meetup...
Simplifying Microservices & Apps - The art of effortless development - Meetup...Simplifying Microservices & Apps - The art of effortless development - Meetup...
Simplifying Microservices & Apps - The art of effortless development - Meetup...
 
Machine Learning Software Engineering Patterns and Their Engineering
Machine Learning Software Engineering Patterns and Their EngineeringMachine Learning Software Engineering Patterns and Their Engineering
Machine Learning Software Engineering Patterns and Their Engineering
 
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdfGOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
 

Scaling Apache Pulsar to 10 PB/day

  • 1. © 2019 SPLUNK INC. Scaling Apache Pulsar to 10 PB/day June 2021 Karthik Ramasamy Splunk
  • 2. © 2020 SPLUNK INC. Karthik Ramasamy Senior Director of Engineering @karthikz streaming @splunk | ex-CEO of @streamlio | co-creator of @heronstreaming | ex @Twitter | Ph.D
  • 3. During the course of this presentation, we may make forward‐looking statements regarding future events or plans of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results may differ materially. The forward-looking statements made in the this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, it may not contain current or accurate information. We do not assume any obligation to update any forward‐ looking statements made herein. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only, and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionalities described or to include any such feature or functionality in a future release. Splunk, Splunk>, Data-to-Everything, D2E and Turn Data Into Doing are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names or trademarks belong to their respective owners. © 2020 Splunk Inc. All rights reserved Forward- Looking Statements © 2020 SPLUNK INC.
  • 4. © 2019 SPLUNK INC. Agenda 1) Introduction to Splunk & DSP 2) Requirements, Use Cases & Deployment 3) Initial Cluster Size Estimation 4) Optimizations 5) Conclusion
  • 5. © 2020 SPLUNK INC. Data 
 Lakes Master Data Management ETL Point Data Management 
 Solutions Data 
 Silos Business Processes The 
 Data-to-Everything Platform IT Security DevOps
  • 6. © 2019 SPLUNK INC. Splunk DSP A real time stream processing solution that collects, processes and delivers data to Splunk and other destinations in milliseconds Splunk Data Stream Processor Detect Data Patterns or Conditions Mask Sensitive Data Aggregate Format Normalize Transform Filter Enhance Turn Raw Data Into 
 High-value Information Protect Sensitive Data Distribute Data To Splunk 
 Or Other Destinations Data 
 Warehouse Public 
 Cloud Message 
 Bus
  • 7. © 2019 SPLUNK INC. DSP - Bird’s Eye View HEC S2S Batch Apache Pulsar Stream Processing Engine External Systems REST Client Forwarders Data Source Splunk Indexer Apache Pulsar is at the core of DSP
  • 8. © 2019 SPLUNK INC. Customer Requirements & Deployment
  • 9. © 2019 SPLUNK INC. Customers ✦ DSP is deployed at several customer s ✦ Some of those customers are marquee customers with large volume of dat a ✦ One such marquee customer is in fi nance and payments
  • 10. © 2019 SPLUNK INC. Use Cases ✦ Microservices and applications emit log s ✦ Logs contain rich informatio n ✦ Process these logs and extract monitoring & tracing informatio n ✦ Filter these logs depending on log volume and if there is high value - justifying retentio n ✦ Compute real time business metrics
  • 11. © 2019 SPLUNK INC. Data Requirements ✦ Environment - Google Cloud Platform ✦ Use of n1-standard-32 VM s ✦ Raw data ingestion of 10 PB/day that translates ~120 GB/se c ✦ Data retention of 3 hour s ✦ Need to handle the entire traf fi c load when a zone fails
  • 12. © 2019 SPLUNK INC. DSP Ingest Cluster DSP Compute Cluster DSP Compute Cluster DSP Compute Cluster DSP Deployment Log Publisher Log Publisher Log Publisher Apache Pulsar Cluster Pipeline 1 Pipeline 2 Pipeline 3 Splunk Enterprise Splunk Observability
  • 13. © 2019 SPLUNK INC. DSP Deployment ✦ Separation of ingestion and computatio n ✦ Pipeline isolation and no noisy neighbor issue s ✦ Troubleshooting single pipeline gets easie r ✦ Might not need over provisioning except for peak load + fudge factor (as compared to deploying a single cluster)
  • 14. © 2019 SPLUNK INC. VM Configuration - n1-standard-32 ✦ 32 vCPU s ✦ 120 GB of memor y ✦ Max number of PDs (EBS equivalent) - 12 8 ✦ Max total PD size - 257 T B ✦ Max egress network bandwidth - 32 Gbps (4 GBps ) ✦ Max 24 L-SSDs for a total of 9 TB
  • 15. © 2020 SPLUNK INC. Storage Options in GCP P-SSD P-HDD L-SSD
  • 16. © 2019 SPLUNK INC. Initial Estimation
  • 17. © 2019 SPLUNK INC. Apache Pulsar Requirements ✦ Replica factor of 3 ✦ Need to handle 120 GBps of raw traf fi c ✦ Need to handle 360 GBps of storage write bandwidt h ✦ With journal required write bandwidth 720 GBp s ✦ Total storage required for retention - 3.9 P B ✦ Total ingress network bandwidth - 480 GBp s ✦ Total egress network bandwidth - 1200 GBps
  • 18. © 2019 SPLUNK INC. Pulsar Cluster Size Estimation ✦ Size of a Pulsar Cluster for a given workload depends on three parameters - ✦ Storage Density - Aggregate storage capacity needed in the cluster and proportional to retention of dat a ✦ Storage Bandwidth - Aggregate write throughput and read throughput needed for data ingestion and consumption. Heavily depends on storage medi a ✦ Network Bandwidth - Aggregate network bandwidth available in the cluster for input traf fi c and output traf fi c.
  • 19. © 2019 SPLUNK INC. Estimating Storage Bandwidth Bookie Bookie Bookie Broker Producer Journal Data Journal Data Journal Data Consumer 1X data 2X data 3X data 2X data 2X data 6X data
  • 20. © 2019 SPLUNK INC. Estimating Network Bandwidth Bookie Bookie Bookie Broker Producer Journal egress ingress egress ingress Data Journal Data Journal Data ingress ingress egress egress egress egress egress Consumer egress egress egress 1X ingress 4X egress 1X ingress 2X egress 4X ingress 10X egress 1X ingress 2X egress 1X ingress 2X egress
  • 21. © 2020 SPLUNK INC. Estimating VMs using P-HDD 0 1000 2000 3000 4000 Storage Bandwidth Storage Density Network Bandwidth 300 444 3686 VMs w/journal Max of 200 MB/sec write throughput per VM Max of 9 TB per instance Max of 4 GBps egress and ingress bandwidth Dominated by Storage Bandwidth
  • 22. © 2020 SPLUNK INC. Estimating VMs using P-SDD 0 500 1000 1500 2000 Storage Bandwidth Storage Density Network Bandwidth 300 444 1843 VMs w/journal Max of 400 MB/sec write throughput per VM Max of 9 TB per instance Max of 4 GBps egress and ingress bandwidth Dominated by Storage Bandwidth
  • 23. © 2019 SPLUNK INC. Estimating Network Bandwidth using L-HDD Bookie Bookie Bookie Broker Producer Journal ingress egress ingress Data Journal Data Journal Data ingress ingress Consumer egress egress egress 1X ingress 4X egress 1X ingress 1X ingress 1X ingress 4X ingress 5X egress
  • 24. © 2020 SPLUNK INC. Estimating VMs using L-SDD 0 225 450 675 900 Storage Bandwidth Storage Density Network Bandwidth 120 444 868 VMs w/journal Max of 850 MB/sec write throughput per VM Max of 9 TB per instance Max of 4 GBps egress and ingress bandwidth Dominated by Storage Bandwidth
  • 25. © 2020 SPLUNK INC. Estimation of VMs - Comparison 0 1000 2000 3000 4000 P-HDD P-SSD L-SSD 868 1843 3686 VMs w/journal
  • 26. © 2019 SPLUNK INC. Optimizations
  • 27. © 2019 SPLUNK INC. Optimization #1 - Eliminating Journal ✦ Different types of durabilit y ✦ Persistent Durability - No data loss in the presence of nodes failures or entire cluster failur e ✦ Replicated Durability - No data loss in the presence of limited nodes failure s ✦ Transient Durability - Data loss in the presence of failure s Since all the data is machine logs, we implemented replicated durability
  • 28. © 2019 SPLUNK INC. Replicated Durability Bookie Bookie Bookie Broker Producer Data Data Data
  • 29. © 2020 SPLUNK INC. Estimating VMs 0 ms 1000 ms 2000 ms 3000 ms 4000 ms P-HDD P-SSD L-SSD 444 922 1843 868 1843 3686 VMs w/journal VMs w/o journal Dominated by Storage Bandwidth Dominated by Storage Bandwidth Dominated by Storage Density
  • 30. © 2019 SPLUNK INC. Optimization #2 - Direct I/O ✦ Overhead of page cache in container environment is pretty hig h ✦ Kernel needs to keep track of the usage quota per container for the page cach e ✦ These translate into maintaining additional data structures and lookups (older kernel had n^2 lookup time for getting pages in & out ) ✦ Bypassed page cache for BookKeeper entry log, using JNI : ✦ We already have in memory caches (write and read-ahead ) ✦ We have better control on what to cache and when to evic t ✦ Avoid double buffering
  • 31. © 2020 SPLUNK INC. Performance of Direct I/O 0 MB/s 400 MB/s 800 MB/s 1200 MB/s 1600 MB/s P-HDD P-SSD L-SSD 1600 600 300 850 400 200 Before Direct I/O After Direct I/O
  • 32. © 2020 SPLUNK INC. Estimating VMs 0 ms 1000 ms 2000 ms 3000 ms 4000 ms P-HDD P-SSD L-SSD 444 614 1228 444 922 1843 868 1843 3686 VMs w/journal VMs w/o journal VMs with direct i/o Dominated by Storage Bandwidth Dominated by Storage Bandwidth Dominated by Storage Density
  • 33. © 2019 SPLUNK INC. Optimization #3 - Use of Compression Bookie Bookie Bookie Broker Producer Data Data Data Compressed data Consumer Compressed data C U
  • 34. © 2019 SPLUNK INC. Employing compression ✦ Compression ratio of 4 x ✦ Need to handle 360 GBps —> 90 GBps of storage write bandwidt h ✦ Total storage required for retention - 3.9 PB —> 975 T B ✦ Total ingress network bandwidth - 480 GBps —> 120 GBps ✦ Total egress network bandwidth - 1200 GBps —> 240 GBps
  • 35. © 2020 SPLUNK INC. Estimating VMs 0 ms 1000 ms 2000 ms 3000 ms 4000 ms P-HDD P-SSD L-SSD 111 154 308 444 614 1228 444 922 1843 868 1843 3686 VMs w/journal VMs w/o journal VMs with direct i/o VMs with compression Dominated by Storage Bandwidth Dominated by Storage Bandwidth Dominated by Storage Density
  • 36. © 2019 SPLUNK INC. Surviving Zone Failure Segment 1 Segment 2 Segment n .
 .
 . Segment 2 Segment 3 Segment n .
 .
 . Segment 3 Segment 1 Segment n .
 .
 . Storage Broker Serving Broker Broker ✦ Zone/Rack Failure s ✦ Bookies provide rack awarenes s ✦ Broker replicate data to different racks/zone s ✦ In the presence of zone/rack failure, data is available in other zone s ✦ One zone failure means two zones should be capable of handling the entire traf fi c ✦ Requires 50% additional VM s Zone A Zone B Zone C
  • 37. © 2020 SPLUNK INC. Estimating VMs 0 ms 1000 ms 2000 ms 3000 ms 4000 ms P-HDD P-SSD L-SSD 222 308 616 111 154 308 444 614 1228 444 922 1843 868 1843 3686 VMs w/journal VMs w/o journal VMs with direct i/o VMs with compression VMs to survive zone failure
  • 38. © 2019 SPLUNK INC. Optimization #4 - C++ Client CPU & Memory Usage ✦ Better round robin across partitions - maximizing the batch size per partitio n ✦ Having bigger batches reduces the cpu usage for client, broker and bookie s ✦ Increases the compression facto r ✦ Reduced client memory usage s ✦ Optimizations to minimize memory allocation overhead ✦ Implemented memory limit in C++ produce r ✦ Simpli fi es the user con fi guration — One single setting instead of multiple queue sizes and complex math
  • 39. © 2019 SPLUNK INC. Finally … Running 200 n1-standard-32 VMs for Pulsar cluster with 24 L-SSDs per VM
  • 40. © 2019 SPLUNK INC. OSS Contributions ✦ Github Pull Request 8283 - C++ Client is allocating buffer bigger than neede d ✦ Github Pull Request 8331 - C++ Client back-pressure is done on batches rather than number of message s ✦ Github Pull Request 8395 - C++ Implement batch aware producer router ✦ Github Pull Request 9679 - C++ Implemented memory limit in C++ producer
  • 41. © 2019 SPLUNK INC. Future Work ✦ Typical operations involve ✦ Upgrade to new version of Pulsa r ✦ Upgrade to new OS versio n ✦ Apply new security patches to O S ✦ New Pulsar version - In-place upgrade s ✦ OS & security patches - Applying one VM at a time is too slow for large cluster
  • 42. Thank You © 2019 SPLUNK INC.