SlideShare a Scribd company logo
1 of 90
Download to read offline
Thoughts on Kafka
Capacity Planning
Jamie Alquiza
Sr. Software Engineer
– Multi-petabyte footprint
– 10s of GB/s in sustained bandwidth
– Globally distributed infrastructure
– Continuous growth
A Lot of Kafka
Motivation
Poor utilization at
scale is $$$🔥
1
Poor utilization at
scale is $$$🔥
1 2
Desire for
predictable
performance ⚡
Kafka Resource
Consumption
CPU
- Message Rate
- Compression
- Compaction (if
used)
Kafka Resource Consumption
Memory Disk Network
CPU
- Message Rate
- Compression
- Compaction (if
used)
Kafka Resource Consumption
Memory
- Efficient, Steady
Heap Usage
- Page Cache
Disk Network
CPU
- Message Rate
- Compression
- Compaction (if
used)
Kafka Resource Consumption
Memory
- Efficient, Steady
Heap Usage
- Page Cache
Disk
- Bandwidth
- Storage
Capacity
Network
CPU
- Message Rate
- Compression
- Compaction (if
used)
Kafka Resource Consumption
Memory
- Efficient, Steady
Heap Usage
- Page Cache
Disk
- Bandwidth
- Storage
Capacity
Network
- Consumers
- Replication
Kafka Makes
Capacity Planning
Easy
Through the lens of the Universal
Scalability Law:
- low contention, crosstalk
- no complex queries
Kafka Makes Capacity Planning Easy
Through the lens of the Universal
Scalability Law:
- low contention, crosstalk
- no complex queries
Exposes mostly bandwidth
problems:
- highly sequential, batched ops
- primary workload is streaming
the reads/writes of bytes
Kafka Makes Capacity Planning Easy
Kafka Makes
Capacity Planning
Hard
The default tools weren’t made for
scaling:
- reassign-partitions focused on
simple partition placement
Kafka Makes Capacity Planning Hard
The default tools weren’t made for
scaling:
- reassign-partitions focused on
simple partition placement
No administrative API:
- no endpoint to inspect or
manipulate resources
Kafka Makes Capacity Planning Hard
A Scaling Model
Created Kafka-Kit (open-source):
- topicmappr for intelligent
partition placement
- registry (WIP): a Kafka
gRPC/HTTP API
A Scaling Model
Created Kafka-Kit (open-source):
- topicmappr for intelligent
partition placement
- registry (WIP): a Kafka
gRPC/HTTP API
Defined a simple workload pattern:
- topics are bound to specific
broker sets (“pools”)
- multiple pools/cluster
- primary drivers: disk capacity &
network bandwidth
A Scaling Model
A Scaling Model
– topicmappr builds optimal
partition -> broker pool
mappings
A Scaling Model
– topicmappr builds optimal
partition -> broker pool
mappings
– topic/pool sets are scaled
individually
A Scaling Model
– topicmappr builds optimal
partition -> broker pool
mappings
– topic/pool sets are scaled
individually
– topicmappr handles repairs,
storage rebalancing, pool
expansion
A large cluster is
composed of
- dozens of pools
- hundreds of brokers
Sizing Pools
Sizing Pools
- Storage utilization is targeted at 60-80%
depending on topic growth rate
Sizing Pools
- Storage utilization is targeted at 60-80%
depending on topic growth rate
- Network capacity depends on several factors
Sizing Pools
- Storage utilization is targeted at 60-80%
depending on topic growth rate
- Network capacity depends on several factors
- consumer demand
+
- MTTR targets (20-40% headroom for replication)
Sizing Pools
Determining broker counts
Sizing Pools
Determining broker counts
storage: n = fullRetention / (storagePerNode * 0.8)
Sizing Pools
Determining broker counts
storage: n = fullRetention / (storagePerNode * 0.8)
network: n = consumerDemand / (bwPerNode * 0.6)
Sizing Pools
Determining broker counts
storage: n = fullRetention / (storagePerNode * 0.8)
network: n = consumerDemand / (bwPerNode * 0.6)
pool size = max(ceil(storage), ceil(network))
Sizing Pools
(we do a pretty good job at actually hitting this)
Instance Types
Instance Types
- If there’s a huge delta between counts required for
network vs storage: probably the wrong type
Instance Types
- If there’s a huge delta between counts required for
network vs storage: probably the wrong type
- Remember: sequential, bandwidth-bound
workloads
Instance Types
- If there’s a huge delta between counts required for
network vs storage: probably the wrong type
- Remember: sequential, bandwidth-bound
workloads
- AWS: d2, i3, h1 class
Instance Types (AWS)
d2: the spinning rust is actually great
Good for:
- Storage/$
- Retention biased workloads
Problems:
- Disk bw far exceeds network
- Long MTTRs
Instance Types (AWS)
h1: a modernized d2?
Good for:
- Storage/$
- Balanced, lower retention / high throughput workloads
Problems:
- ENA network exceeds disk throughput
- Recovery times are disk-bound
- Disk bw / node < d2
Instance Types (AWS)
i3: bandwidth monster
Good for:
- Low MTTRs
- Concurrent i/o outside the page cache
Problems:
- storage/$
Instance Types (AWS)
- r4, c5, etc + gp2 EBS
It actually works well; EBS perf isn’t a problem
Instance Types (AWS)
- r4, c5, etc + gp2 EBS
It actually works well; EBS perf isn’t a problem
Problems:
- low EBS channel bw in relation to instance size
- the burden of running a distributed/replicated store, hinging it
on tech that solves 2009 problems
- may want to consider Kinesis / etc?
Data Placement
Data Placement
topicmappr optimizes for:
- maximum leadership distribution
Data Placement
topicmappr optimizes for:
- maximum leadership distribution
- replica rack.id isolation
Data Placement
topicmappr optimizes for:
- maximum leadership distribution
- replica rack.id isolation
- maximum replica list entropy
Data Placement
Maximum replica list entropy(?)
“For all partitions that a given broker holds,
ensuring that the partition replicas are distributed
among as many other unique brokers as possible”
Data Placement
Maximum replica list entropy
It’s possible to have maximal partition distribution
but a low number of unique broker-to-broker
relationships
Data Placement
Maximum replica list entropy
It’s possible to have maximal partition distribution
but a low number of unique broker-to-broker
relationships
Example: broker A holds 20 partitions, all 20 replica sets contain
only 3 other brokers
Data Placement
Maximum replica list entropy!
- topicmappr expresses this as node degree
distribution
Data Placement
Maximum replica list entropy!
- topicmappr expresses this as node degree
distribution
- broker-to-broker relationships: it’s a graph
Data Placement
Maximum replica list entropy!
- topicmappr expresses this as node degree
distribution
- broker-to-broker relationships: it’s a graph
- replica sets are partial adjacency lists
Data Placement
A graph of replicas
Given the following partition replica sets:
p0: [1, 2, 3]
p1: [ 2, 3, 4]
Data Placement
A graph of replicas
Given the following partition replica sets:
p0: [1, 2, 3]
p1: [ 2, 3, 4]
Broker 3’s adjacency list -> [1, 2, 4]
Data Placement
A graph of replicas
Given the following partition replica sets:
p0: [1, 2, 3]
p1: [ 2, 3, 4]
Broker 3’s adjacency list -> [1, 2, 4] (degree = 3)
Data Placement
Maximizing replica list entropy (is good)
Data Placement
Maximizing replica list entropy (is good)
In broker failure/replacements:
- probabilistically increases replication sources
- faster, lower impact recoveries
Data Placement
topicmappr optimizes for:
- maximum leadership distribution ✅
- replica rack.id isolation ✅
- maximum replica list entropy ✅
Maintaining Pools
Maintaining Pools
Most common tasks:
- ensuring storage balance
- simple broker replacements
Maintaining Pools
Most common tasks:
- ensuring storage balance
- simple broker replacements
Both of these are (also) done with topicmappr
Maintaining Pools
Broker storage balance
Maintaining Pools
Broker storage balance
- finds offload candidates: n distance below
harmonic mean storage free
Maintaining Pools
Broker storage balance
Maintaining Pools
Broker storage balance
Maintaining Pools
Broker storage balance
- finds offload candidates: n distance below
harmonic mean storage free
- plans relocations to least-utilized brokers
Maintaining Pools
Broker storage balance
- finds offload candidates: n distance below
harmonic mean storage free
- plans relocations to least-utilized brokers
- fair-share, first-fit descending bin-packing
Maintaining Pools
Broker storage balance
- finds offload candidates: n distance below
harmonic mean storage free
- plans relocations to least-utilized brokers
- fair-share, first-fit descending bin-packing
- loops until no more relocations can be planned
Maintaining Pools
Broker storage balance (results)
Maintaining Pools
Broker replacements
Maintaining Pools
Broker replacements
- When a single broker fails, how is a replacement
chosen?
Maintaining Pools
Broker replacements
- When a single broker fails, how is a replacement
chosen?
- Goal: retain any previously computed storage
balance (via 1:1 replacements)
Maintaining Pools
Broker replacements
- When a single broker fails, how is a replacement
chosen?
- Goal: retain any previously computed storage
balance (via 1:1 replacements)
- Problem: dead brokers no longer visible in ZK
Maintaining Pools
Broker replacements
- topicmappr can be provided several hot spares
from varying AZs (rack.id)
Maintaining Pools
Broker replacements
- topicmappr can be provided several hot spares
from varying AZs (rack.id)
- infers a suitable replacement (“substitution affinity”
feature)
Maintaining Pools
Broker replacements - inferring replacements
- traverse all ISRs, build a set of all rack.ids:
G = {1a,1b,1c,1d}
Maintaining Pools
Broker replacements - inferring replacements
- traverse all ISRs, build a set of all rack.ids:
G = {1a,1b,1c,1d}
- traverse affected ISRs, build a set of live rack.ids:
L = {1a,1c}
Maintaining Pools
Broker replacements - inferring replacements
- Build a set of suitable rack.ids to choose from:
S = { x ∈ G | x ∉ L }
Maintaining Pools
Broker replacements - inferring replacements
- Build a set of suitable rack.ids to choose from:
S = { x ∈ G | x ∉ L }
- S = {1b,1d}
- automatically chooses a hot spare from 1b or 1d
Maintaining Pools
Broker replacements - inferring replacements
Outcome:
- Keeps brokers bound to specific pools
- Simple repairs that maintain storage balance, high
utilization
Scaling Pools
Scaling Pools
When: >90% storage utilization in 48h
Scaling Pools
How:
- add brokers to pool
- run a rebalance
- autothrottle takes over
autothrottle is a service
that dynamically manages
replication rates
Scaling Pools
Increasing capacity also improves storage balance
What’s Next
What’s Next
- precursor to fully automated capacity management
What’s Next
- precursor to fully automated capacity management
- continued growth, dozens more clusters
What’s Next
- precursor to fully automated capacity management
- continued growth, dozens more clusters
- new infrastructure
What’s Next
- precursor to fully automated capacity management
- continued growth, dozens more clusters
- new infrastructure
- (we’re hiring)
Thank you
Jamie Alquiza
Sr. Software Engineer
twitter.com/jamiealquiza

More Related Content

What's hot

Apache Flink, AWS Kinesis, Analytics
Apache Flink, AWS Kinesis, Analytics Apache Flink, AWS Kinesis, Analytics
Apache Flink, AWS Kinesis, Analytics Araf Karsh Hamid
 
The Rise of ZStandard: Apache Spark/Parquet/ORC/Avro
The Rise of ZStandard: Apache Spark/Parquet/ORC/AvroThe Rise of ZStandard: Apache Spark/Parquet/ORC/Avro
The Rise of ZStandard: Apache Spark/Parquet/ORC/AvroDatabricks
 
Kafka replication apachecon_2013
Kafka replication apachecon_2013Kafka replication apachecon_2013
Kafka replication apachecon_2013Jun Rao
 
From Message to Cluster: A Realworld Introduction to Kafka Capacity Planning
From Message to Cluster: A Realworld Introduction to Kafka Capacity PlanningFrom Message to Cluster: A Realworld Introduction to Kafka Capacity Planning
From Message to Cluster: A Realworld Introduction to Kafka Capacity Planningconfluent
 
Container Performance Analysis
Container Performance AnalysisContainer Performance Analysis
Container Performance AnalysisBrendan Gregg
 
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of Facebook
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of FacebookTech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of Facebook
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of FacebookThe Hive
 
re:Invent 2022 DAT326 Deep dive into Amazon Aurora and its innovations
re:Invent 2022  DAT326 Deep dive into Amazon Aurora and its innovationsre:Invent 2022  DAT326 Deep dive into Amazon Aurora and its innovations
re:Invent 2022 DAT326 Deep dive into Amazon Aurora and its innovationsGrant McAlister
 
Kafka Summit SF 2017 - Kafka Connect Best Practices – Advice from the Field
Kafka Summit SF 2017 - Kafka Connect Best Practices – Advice from the FieldKafka Summit SF 2017 - Kafka Connect Best Practices – Advice from the Field
Kafka Summit SF 2017 - Kafka Connect Best Practices – Advice from the Fieldconfluent
 
How to tune Kafka® for production
How to tune Kafka® for productionHow to tune Kafka® for production
How to tune Kafka® for productionconfluent
 
A Deep Dive into Kafka Controller
A Deep Dive into Kafka ControllerA Deep Dive into Kafka Controller
A Deep Dive into Kafka Controllerconfluent
 
Spark Summit EU 2015: Lessons from 300+ production users
Spark Summit EU 2015: Lessons from 300+ production usersSpark Summit EU 2015: Lessons from 300+ production users
Spark Summit EU 2015: Lessons from 300+ production usersDatabricks
 
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Kafka Tutorial - Introduction to Apache Kafka (Part 1)Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Kafka Tutorial - Introduction to Apache Kafka (Part 1)Jean-Paul Azar
 
Camel Kafka Connectors: Tune Kafka to “Speak” with (Almost) Everything (Andre...
Camel Kafka Connectors: Tune Kafka to “Speak” with (Almost) Everything (Andre...Camel Kafka Connectors: Tune Kafka to “Speak” with (Almost) Everything (Andre...
Camel Kafka Connectors: Tune Kafka to “Speak” with (Almost) Everything (Andre...HostedbyConfluent
 
Getting Started with Confluent Schema Registry
Getting Started with Confluent Schema RegistryGetting Started with Confluent Schema Registry
Getting Started with Confluent Schema Registryconfluent
 
MAA Best Practices for Oracle Database 19c
MAA Best Practices for Oracle Database 19cMAA Best Practices for Oracle Database 19c
MAA Best Practices for Oracle Database 19cMarkus Michalewicz
 
Amazon Aurora Storage Demystified: How It All Works (DAT363) - AWS re:Invent ...
Amazon Aurora Storage Demystified: How It All Works (DAT363) - AWS re:Invent ...Amazon Aurora Storage Demystified: How It All Works (DAT363) - AWS re:Invent ...
Amazon Aurora Storage Demystified: How It All Works (DAT363) - AWS re:Invent ...Amazon Web Services
 
Disaster Recovery Plans for Apache Kafka
Disaster Recovery Plans for Apache KafkaDisaster Recovery Plans for Apache Kafka
Disaster Recovery Plans for Apache Kafkaconfluent
 
RocksDB Performance and Reliability Practices
RocksDB Performance and Reliability PracticesRocksDB Performance and Reliability Practices
RocksDB Performance and Reliability PracticesYoshinori Matsunobu
 

What's hot (20)

Apache Flink, AWS Kinesis, Analytics
Apache Flink, AWS Kinesis, Analytics Apache Flink, AWS Kinesis, Analytics
Apache Flink, AWS Kinesis, Analytics
 
The Rise of ZStandard: Apache Spark/Parquet/ORC/Avro
The Rise of ZStandard: Apache Spark/Parquet/ORC/AvroThe Rise of ZStandard: Apache Spark/Parquet/ORC/Avro
The Rise of ZStandard: Apache Spark/Parquet/ORC/Avro
 
Kafka replication apachecon_2013
Kafka replication apachecon_2013Kafka replication apachecon_2013
Kafka replication apachecon_2013
 
From Message to Cluster: A Realworld Introduction to Kafka Capacity Planning
From Message to Cluster: A Realworld Introduction to Kafka Capacity PlanningFrom Message to Cluster: A Realworld Introduction to Kafka Capacity Planning
From Message to Cluster: A Realworld Introduction to Kafka Capacity Planning
 
Container Performance Analysis
Container Performance AnalysisContainer Performance Analysis
Container Performance Analysis
 
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of Facebook
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of FacebookTech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of Facebook
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of Facebook
 
re:Invent 2022 DAT326 Deep dive into Amazon Aurora and its innovations
re:Invent 2022  DAT326 Deep dive into Amazon Aurora and its innovationsre:Invent 2022  DAT326 Deep dive into Amazon Aurora and its innovations
re:Invent 2022 DAT326 Deep dive into Amazon Aurora and its innovations
 
Kafka Summit SF 2017 - Kafka Connect Best Practices – Advice from the Field
Kafka Summit SF 2017 - Kafka Connect Best Practices – Advice from the FieldKafka Summit SF 2017 - Kafka Connect Best Practices – Advice from the Field
Kafka Summit SF 2017 - Kafka Connect Best Practices – Advice from the Field
 
MyRocks Deep Dive
MyRocks Deep DiveMyRocks Deep Dive
MyRocks Deep Dive
 
How to tune Kafka® for production
How to tune Kafka® for productionHow to tune Kafka® for production
How to tune Kafka® for production
 
A Deep Dive into Kafka Controller
A Deep Dive into Kafka ControllerA Deep Dive into Kafka Controller
A Deep Dive into Kafka Controller
 
Spark Summit EU 2015: Lessons from 300+ production users
Spark Summit EU 2015: Lessons from 300+ production usersSpark Summit EU 2015: Lessons from 300+ production users
Spark Summit EU 2015: Lessons from 300+ production users
 
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Kafka Tutorial - Introduction to Apache Kafka (Part 1)Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
 
Camel Kafka Connectors: Tune Kafka to “Speak” with (Almost) Everything (Andre...
Camel Kafka Connectors: Tune Kafka to “Speak” with (Almost) Everything (Andre...Camel Kafka Connectors: Tune Kafka to “Speak” with (Almost) Everything (Andre...
Camel Kafka Connectors: Tune Kafka to “Speak” with (Almost) Everything (Andre...
 
Getting Started with Confluent Schema Registry
Getting Started with Confluent Schema RegistryGetting Started with Confluent Schema Registry
Getting Started with Confluent Schema Registry
 
Kafka 101
Kafka 101Kafka 101
Kafka 101
 
MAA Best Practices for Oracle Database 19c
MAA Best Practices for Oracle Database 19cMAA Best Practices for Oracle Database 19c
MAA Best Practices for Oracle Database 19c
 
Amazon Aurora Storage Demystified: How It All Works (DAT363) - AWS re:Invent ...
Amazon Aurora Storage Demystified: How It All Works (DAT363) - AWS re:Invent ...Amazon Aurora Storage Demystified: How It All Works (DAT363) - AWS re:Invent ...
Amazon Aurora Storage Demystified: How It All Works (DAT363) - AWS re:Invent ...
 
Disaster Recovery Plans for Apache Kafka
Disaster Recovery Plans for Apache KafkaDisaster Recovery Plans for Apache Kafka
Disaster Recovery Plans for Apache Kafka
 
RocksDB Performance and Reliability Practices
RocksDB Performance and Reliability PracticesRocksDB Performance and Reliability Practices
RocksDB Performance and Reliability Practices
 

Similar to Thoughts on kafka capacity planning

Reference Architecture: Architecting Ceph Storage Solutions
Reference Architecture: Architecting Ceph Storage Solutions Reference Architecture: Architecting Ceph Storage Solutions
Reference Architecture: Architecting Ceph Storage Solutions Ceph Community
 
Revisiting CephFS MDS and mClock QoS Scheduler
Revisiting CephFS MDS and mClock QoS SchedulerRevisiting CephFS MDS and mClock QoS Scheduler
Revisiting CephFS MDS and mClock QoS SchedulerYongseok Oh
 
A Beginner’s Guide to Kafka Performance in Cloud Environments with Steffen Ha...
A Beginner’s Guide to Kafka Performance in Cloud Environments with Steffen Ha...A Beginner’s Guide to Kafka Performance in Cloud Environments with Steffen Ha...
A Beginner’s Guide to Kafka Performance in Cloud Environments with Steffen Ha...HostedbyConfluent
 
Kafka to the Maxka - (Kafka Performance Tuning)
Kafka to the Maxka - (Kafka Performance Tuning)Kafka to the Maxka - (Kafka Performance Tuning)
Kafka to the Maxka - (Kafka Performance Tuning)DataWorks Summit
 
Planning & Best Practice for Microsoft Virtualization
Planning & Best Practice for Microsoft VirtualizationPlanning & Best Practice for Microsoft Virtualization
Planning & Best Practice for Microsoft VirtualizationLai Yoong Seng
 
Optimizing columnar stores
Optimizing columnar storesOptimizing columnar stores
Optimizing columnar storesIstvan Szukacs
 
Optimizing columnar stores
Optimizing columnar storesOptimizing columnar stores
Optimizing columnar storesIstvan Szukacs
 
User-space Network Processing
User-space Network ProcessingUser-space Network Processing
User-space Network ProcessingRyousei Takano
 
Sizing MongoDB on AWS with Wired Tiger-Patrick and Vigyan-Final
Sizing MongoDB on AWS with Wired Tiger-Patrick and Vigyan-FinalSizing MongoDB on AWS with Wired Tiger-Patrick and Vigyan-Final
Sizing MongoDB on AWS with Wired Tiger-Patrick and Vigyan-FinalVigyan Jain
 
(DAT402) Amazon RDS PostgreSQL:Lessons Learned & New Features
(DAT402) Amazon RDS PostgreSQL:Lessons Learned & New Features(DAT402) Amazon RDS PostgreSQL:Lessons Learned & New Features
(DAT402) Amazon RDS PostgreSQL:Lessons Learned & New FeaturesAmazon Web Services
 
Varrow datacenter storage today and tomorrow
Varrow   datacenter storage today and tomorrowVarrow   datacenter storage today and tomorrow
Varrow datacenter storage today and tomorrowpittmantony
 
Presto at Tivo, Boston Hadoop Meetup
Presto at Tivo, Boston Hadoop MeetupPresto at Tivo, Boston Hadoop Meetup
Presto at Tivo, Boston Hadoop MeetupJustin Borgman
 
Devopsconf2015 sebamontini-151117150231-lva1-app6891
Devopsconf2015 sebamontini-151117150231-lva1-app6891Devopsconf2015 sebamontini-151117150231-lva1-app6891
Devopsconf2015 sebamontini-151117150231-lva1-app6891Flavia Marinelli
 
Understanding and building big data Architectures - NoSQL
Understanding and building big data Architectures - NoSQLUnderstanding and building big data Architectures - NoSQL
Understanding and building big data Architectures - NoSQLHyderabad Scalability Meetup
 
How Ceph performs on ARM Microserver Cluster
How Ceph performs on ARM Microserver ClusterHow Ceph performs on ARM Microserver Cluster
How Ceph performs on ARM Microserver ClusterAaron Joue
 
Explore big data at speed of thought with Spark 2.0 and Snappydata
Explore big data at speed of thought with Spark 2.0 and SnappydataExplore big data at speed of thought with Spark 2.0 and Snappydata
Explore big data at speed of thought with Spark 2.0 and SnappydataData Con LA
 
Cassandra's Sweet Spot - an introduction to Apache Cassandra
Cassandra's Sweet Spot - an introduction to Apache CassandraCassandra's Sweet Spot - an introduction to Apache Cassandra
Cassandra's Sweet Spot - an introduction to Apache CassandraDave Gardner
 

Similar to Thoughts on kafka capacity planning (20)

Reference Architecture: Architecting Ceph Storage Solutions
Reference Architecture: Architecting Ceph Storage Solutions Reference Architecture: Architecting Ceph Storage Solutions
Reference Architecture: Architecting Ceph Storage Solutions
 
Revisiting CephFS MDS and mClock QoS Scheduler
Revisiting CephFS MDS and mClock QoS SchedulerRevisiting CephFS MDS and mClock QoS Scheduler
Revisiting CephFS MDS and mClock QoS Scheduler
 
A Beginner’s Guide to Kafka Performance in Cloud Environments with Steffen Ha...
A Beginner’s Guide to Kafka Performance in Cloud Environments with Steffen Ha...A Beginner’s Guide to Kafka Performance in Cloud Environments with Steffen Ha...
A Beginner’s Guide to Kafka Performance in Cloud Environments with Steffen Ha...
 
Kafka to the Maxka - (Kafka Performance Tuning)
Kafka to the Maxka - (Kafka Performance Tuning)Kafka to the Maxka - (Kafka Performance Tuning)
Kafka to the Maxka - (Kafka Performance Tuning)
 
Planning & Best Practice for Microsoft Virtualization
Planning & Best Practice for Microsoft VirtualizationPlanning & Best Practice for Microsoft Virtualization
Planning & Best Practice for Microsoft Virtualization
 
Optimizing columnar stores
Optimizing columnar storesOptimizing columnar stores
Optimizing columnar stores
 
Optimizing columnar stores
Optimizing columnar storesOptimizing columnar stores
Optimizing columnar stores
 
Storage spaces direct webinar
Storage spaces direct webinarStorage spaces direct webinar
Storage spaces direct webinar
 
Masterclass Live: Amazon EMR
Masterclass Live: Amazon EMRMasterclass Live: Amazon EMR
Masterclass Live: Amazon EMR
 
User-space Network Processing
User-space Network ProcessingUser-space Network Processing
User-space Network Processing
 
Sizing MongoDB on AWS with Wired Tiger-Patrick and Vigyan-Final
Sizing MongoDB on AWS with Wired Tiger-Patrick and Vigyan-FinalSizing MongoDB on AWS with Wired Tiger-Patrick and Vigyan-Final
Sizing MongoDB on AWS with Wired Tiger-Patrick and Vigyan-Final
 
(DAT402) Amazon RDS PostgreSQL:Lessons Learned & New Features
(DAT402) Amazon RDS PostgreSQL:Lessons Learned & New Features(DAT402) Amazon RDS PostgreSQL:Lessons Learned & New Features
(DAT402) Amazon RDS PostgreSQL:Lessons Learned & New Features
 
Varrow datacenter storage today and tomorrow
Varrow   datacenter storage today and tomorrowVarrow   datacenter storage today and tomorrow
Varrow datacenter storage today and tomorrow
 
Presto at Tivo, Boston Hadoop Meetup
Presto at Tivo, Boston Hadoop MeetupPresto at Tivo, Boston Hadoop Meetup
Presto at Tivo, Boston Hadoop Meetup
 
Devopsconf2015 sebamontini-151117150231-lva1-app6891
Devopsconf2015 sebamontini-151117150231-lva1-app6891Devopsconf2015 sebamontini-151117150231-lva1-app6891
Devopsconf2015 sebamontini-151117150231-lva1-app6891
 
Devopsconf 2015 sebamontini
Devopsconf 2015 sebamontiniDevopsconf 2015 sebamontini
Devopsconf 2015 sebamontini
 
Understanding and building big data Architectures - NoSQL
Understanding and building big data Architectures - NoSQLUnderstanding and building big data Architectures - NoSQL
Understanding and building big data Architectures - NoSQL
 
How Ceph performs on ARM Microserver Cluster
How Ceph performs on ARM Microserver ClusterHow Ceph performs on ARM Microserver Cluster
How Ceph performs on ARM Microserver Cluster
 
Explore big data at speed of thought with Spark 2.0 and Snappydata
Explore big data at speed of thought with Spark 2.0 and SnappydataExplore big data at speed of thought with Spark 2.0 and Snappydata
Explore big data at speed of thought with Spark 2.0 and Snappydata
 
Cassandra's Sweet Spot - an introduction to Apache Cassandra
Cassandra's Sweet Spot - an introduction to Apache CassandraCassandra's Sweet Spot - an introduction to Apache Cassandra
Cassandra's Sweet Spot - an introduction to Apache Cassandra
 

Recently uploaded

2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shards2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shardsChristopher Curtin
 
Enhancing Supply Chain Visibility with Cargo Cloud Solutions.pdf
Enhancing Supply Chain Visibility with Cargo Cloud Solutions.pdfEnhancing Supply Chain Visibility with Cargo Cloud Solutions.pdf
Enhancing Supply Chain Visibility with Cargo Cloud Solutions.pdfRTS corp
 
Understanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptxUnderstanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptxSasikiranMarri
 
VictoriaMetrics Q1 Meet Up '24 - Community & News Update
VictoriaMetrics Q1 Meet Up '24 - Community & News UpdateVictoriaMetrics Q1 Meet Up '24 - Community & News Update
VictoriaMetrics Q1 Meet Up '24 - Community & News UpdateVictoriaMetrics
 
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...Bert Jan Schrijver
 
Key Steps in Agile Software Delivery Roadmap
Key Steps in Agile Software Delivery RoadmapKey Steps in Agile Software Delivery Roadmap
Key Steps in Agile Software Delivery RoadmapIshara Amarasekera
 
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdfAndrey Devyatkin
 
Osi security architecture in network.pptx
Osi security architecture in network.pptxOsi security architecture in network.pptx
Osi security architecture in network.pptxVinzoCenzo
 
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdfSteve Caron
 
Strategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero resultsStrategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero resultsJean Silva
 
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full RecordingOpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full RecordingShane Coughlan
 
Best Angular 17 Classroom & Online training - Naresh IT
Best Angular 17 Classroom & Online training - Naresh ITBest Angular 17 Classroom & Online training - Naresh IT
Best Angular 17 Classroom & Online training - Naresh ITmanoharjgpsolutions
 
Pros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdf
Pros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdfPros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdf
Pros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdfkalichargn70th171
 
Mastering Project Planning with Microsoft Project 2016.pptx
Mastering Project Planning with Microsoft Project 2016.pptxMastering Project Planning with Microsoft Project 2016.pptx
Mastering Project Planning with Microsoft Project 2016.pptxAS Design & AST.
 
Ronisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited CatalogueRonisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited Catalogueitservices996
 
Introduction to Firebase Workshop Slides
Introduction to Firebase Workshop SlidesIntroduction to Firebase Workshop Slides
Introduction to Firebase Workshop Slidesvaideheekore1
 
GraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4j
GraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4jGraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4j
GraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4jNeo4j
 
Zer0con 2024 final share short version.pdf
Zer0con 2024 final share short version.pdfZer0con 2024 final share short version.pdf
Zer0con 2024 final share short version.pdfmaor17
 
The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...
The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...
The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...kalichargn70th171
 
Keeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository worldKeeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository worldRoberto Pérez Alcolea
 

Recently uploaded (20)

2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shards2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shards
 
Enhancing Supply Chain Visibility with Cargo Cloud Solutions.pdf
Enhancing Supply Chain Visibility with Cargo Cloud Solutions.pdfEnhancing Supply Chain Visibility with Cargo Cloud Solutions.pdf
Enhancing Supply Chain Visibility with Cargo Cloud Solutions.pdf
 
Understanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptxUnderstanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptx
 
VictoriaMetrics Q1 Meet Up '24 - Community & News Update
VictoriaMetrics Q1 Meet Up '24 - Community & News UpdateVictoriaMetrics Q1 Meet Up '24 - Community & News Update
VictoriaMetrics Q1 Meet Up '24 - Community & News Update
 
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...
 
Key Steps in Agile Software Delivery Roadmap
Key Steps in Agile Software Delivery RoadmapKey Steps in Agile Software Delivery Roadmap
Key Steps in Agile Software Delivery Roadmap
 
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
 
Osi security architecture in network.pptx
Osi security architecture in network.pptxOsi security architecture in network.pptx
Osi security architecture in network.pptx
 
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
 
Strategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero resultsStrategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero results
 
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full RecordingOpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
 
Best Angular 17 Classroom & Online training - Naresh IT
Best Angular 17 Classroom & Online training - Naresh ITBest Angular 17 Classroom & Online training - Naresh IT
Best Angular 17 Classroom & Online training - Naresh IT
 
Pros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdf
Pros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdfPros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdf
Pros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdf
 
Mastering Project Planning with Microsoft Project 2016.pptx
Mastering Project Planning with Microsoft Project 2016.pptxMastering Project Planning with Microsoft Project 2016.pptx
Mastering Project Planning with Microsoft Project 2016.pptx
 
Ronisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited CatalogueRonisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited Catalogue
 
Introduction to Firebase Workshop Slides
Introduction to Firebase Workshop SlidesIntroduction to Firebase Workshop Slides
Introduction to Firebase Workshop Slides
 
GraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4j
GraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4jGraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4j
GraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4j
 
Zer0con 2024 final share short version.pdf
Zer0con 2024 final share short version.pdfZer0con 2024 final share short version.pdf
Zer0con 2024 final share short version.pdf
 
The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...
The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...
The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...
 
Keeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository worldKeeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository world
 

Thoughts on kafka capacity planning