Performance improvements in etcd 3.5 release

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Performance Improvements In
Etcd 3.5
Wilson Wang, Jiaxin Shan
Bytedance Infrastructure Computing Team

• Brief Introduction to Etcd
• Problems & Optimizations
• Concurrent Read Tx
• Inefficient Warning Log
• Performance
• New...

Brief Intro to Etcd

• Distributed key-value store provides
strong consistency
• Protocol based on raft consensus
algorithm.
• Best fit for sce...

• Single Raft
• Raft entries are applied in a FIFO by
applierV3
• MVCC(Multi-Version Concurrency
Control)
• KeyIndex: Key ...

• Unary gRPC Calls
• Put()
• Range()
• DeleteRange()
• Txn()
• Compact()
• Stream gRPC Call
• Watch()
Important Etcd APIs
...

• Etcd is widely used in ByteDance to manage different Kubernetes
clusters.
• Our Etcd cluster sizes continue to increase ...

Problems & Optimizations

Concurrent Read Tx

• Introduced in Etcd 3.4
• Concurrent read transaction will not hold the read mutex to block write
transaction. Hence impr...

Etcd 3.3 throughput is much higher than 3.4
Concurrent Read Tx cont.

• Problem: Txn() uses ConcurrentReadTx for read-only ops inside Txn().
Even if the Txn() ops contain write ops, a read-onl...

• Txn() throughput improved to
~2.7x at max
Result

• Problem: Can we further reduce the overhead of txReadBuffer deep
copy?
• Solution: Share the txReadBuffer between multip...

Result
• Txn() throughput improved to
~2.2x at max

Inefficient Warning Log

• Problem: gRPC proto.Size() calls Marshal() which is expensive.
• Solutions:
1. Use gRPC message type's Size() instead of...

New Benchmark Command

• Lack of mixed read/write benchmark in Etcd
• Added in https://github.com/etcd-
io/etcd/pull/13038 , including the comman...

Performance

Testcase: Etcd benchmark
tool mixed txn operations.
With the same workload, Etcd
3.5 both CPU & memory
costs are lower tha...

On-Going & Future Work

• Client gPRC requests dump/replay feature.
• Further improvements in different Etcd components.
• Watch
• Marshal/Unmarsh...

Questions?
Contact me at: wilson.wang@bytedance.com

1. https://etcd.io/
2. https://etcd.io/blog/2021/announcing-etcd-3.5/
3. https://github.com/etcd-io/etcd/pull/10523
4. htt...

THANKS.

Share SlideShare

Facebook
Twitter
LinkedIn

Embed

Size (px)

Show related SlideShares at end

WordPress Shortcode

Link

LinkedIn
Facebook
Twitter
Email

Replay

Top clipped slide

1 of 26 Ad

1 of 26 Ad