Etcd- Mission Critical Key-Value Store

Xiang Li, Head of Distributed Systems at CoreOS SF CoreOS Meetup May 25th, 2016

... ... ...
... ... ...
Unavailable
Uncoordinated Upgrades

Motivation
CoreOS cluster reboot lock
- Decrement a semaphore key atomically
- Reboot and wait...
- After reboot increment the semaphore key

... ... ...
CoreOS updates coordination
0

config
Start / RestartStart / Restart
Store Application Configuration

config
Update
Store Application Configuration

config
Unavailable
Store Application Configuration

Requirements
Strong Consistency
- mutual exclusive at any time for locking purpose
Highly Available
- resilient to single points of failure & network partitions
Watchable
- push configuration updates to application

Requirements
CAP
- We want CP
- We want something like Paxos

Common problem
GFS
Paxos
Big Table
Spanner
CFS
Chubby
Google - “All” infrastructure relies on Paxos

Common problem
Amazon - Replicated log powers ec2
Microsoft - Boxwood powers storage
infrastructure
Hadoop - ZooKeeper is the heart of the ecosystem

ETCD KEY VALUE STORE
Fully Replicated, Highly Available,
Consistent

PUT(foo, bar), GET(foo), DELETE(foo)
Watch(foo)
CAS(foo, bar, bar1)
Key-value Operations

Runtime Reconfiguration
Point-in-time Backup
Extensive Metrics
etcd Operationality

Multi-Version
Put(foo, bar)
Put(foo, bar1)
Put(foo, bar2)
Get(foo) -> bar2

Multi-Version
Put(foo, bar)
Put(foo, bar1)
Put(foo, bar2)
Get(foo, 1) -> bar

Tx.If(
Compare(Value("foo"), ">", "bar"),
Compare(Version("foo"), "=", 2),
...
).Then(
Put("ok","true")...
).Else(
Put("ok","false")...
).Commit()
Mini-Transactions

l = CreateLease(15 * second)
Put(foo, bar, l)
l.KeepAlive()
l.Revoke()
Leases

w = Watch(foo)
for {
r = w.Recv()
print (r.Event) // PUT
print (r.KV) // foo,bar
}
Streaming Watch

ETCD v2
machine coordination -> Thousands

ETCD v3
app/container coordination -> Millions

Performance
Snapshot caused
performance degradation
etcd2 - 600K keys

Performance etcd2 - 600K keys
Snapshot triggered
elections

ZooKeeper Performance
Non-blocking full snapshot
Efficient memory management

Performance
Snapshot triggered
election
ZooKeeper default

Performance
GC
ZooKeeper snapshot disabled

Reliable Performance
- Similar to ZooKeeper with snapshot disabled
- Incremental snapshot
- No Garbage Collection Pauses
- Off-heap storage

Performance etcd3 /ZooKeeper snapshot disabled

Memory
10GB
2.4GB
0.8GB
512MB data - 2M 256B keys

Reliability
99% at small scale is easy
- Failure is infrequent and human manageable
99% at large scale is not enough
- Not manageable by humans
99.99% at large scale
- Reliable systems at bottom layer

HOW DO WE ACHIEVE RELIABILITY
WAL, Snapshots, Testing

Write Ahead Log
Append only
- Simple is good
Rolling CRC protected
- Storage & OSes can be unreliable

Snapshots
Torturing DBs for Fun and Profit (OSDI2014)
- The simpler database is safer
- LMDB was the winner
Boltdb an append only B+Tree
- A simpler LMDB written in Go

Testing Clusters Failure
Inject failures into running clusters
White box runtime checking
- Hash state of the system
- Progress of the system

Testing Cluster Health with Failures
Issue lock operations across cluster
Ensure the correctness of client library

etcd/raft Reliability
Designed for testability and flexibility
Used by large scale db systems and others
- Cockroachdb, TiKV, Dgraph

BETA AVAILABLE TODAY
github.com/coreos/etcd

FUTURE WORK
Proxy, Caching, Watch Coalescing,
Secondary Index

Xiang Li
xiang.li@coreos.com
etcd - mission-critical key-value store
Thank you!

Cloud Foundry and OpenStack are the biggest Open Source projects in their domain. As IaaS and PaaS walk hand in hand the idea of combining both worlds is close. anynines is running their public Cloud Foundry offering on top of OpenStack for more than three years with two years running on a self-hosted OpenStack setup. As head of public Paas operations Julian Weber has gained a lot of knowledge to share about setting up and operating Cloud Foundry installations. This presentation leads the audience through the journey of adopting the Cloud Foundry Open Source version and breeding it to a highly available and production ready Cloud Foundry setup. The listener is guided through the analysis of potential single points of failure in standard CF Open Source setups up to required changes in the Cloud Foundry OS release to reach our goal. As this talk is about Cloud Foundry operations we also need to talk about experiences with BOSH as a general purpose tool for software lifecycle management of big distributed systems and possible improvements to the BOSH tool set and workflows. The talk will enable advanced DevOps to dive deeper into the technical details of setting up production ready Cloud Foundry installations based on Cloud Foundry Open Source.

Delivering a production Cloud Foundry Environment with Bosh | anynines

anynines GmbH

[오픈소스컨설팅] Linux Network Troubleshooting

Open Source Consulting

Running Cloud Foundry for 12 months - An experience report | anynines

anynines GmbH

anynines ran a public PaaS located in a German datacenter based on Cloud Foundry. In more than 12 months of running a Cloud Foundry PaaS man lessons about security, high availability, open stack and many other exciting topics have been learned. See how Bosh can be used and how it shouldn't be used. Learn how to perform Cloud Foundry upgrades and read how to harden Cloud Foundry by adding more fault tolerance with pacemaker.

제4회 한국IBM과 함께하는 난공불락 오픈소스 인프라 세미나-Ranchers

Tommy Lee

[오픈소스컨설팅] 쿠버네티스와 쿠버네티스 on 오픈스택 비교 및 구축 방법

Open Source Consulting

One-click Hadoop Cluster Deployment on OpenPOWER Systems

Pradeep Kumar

Google is making the power of its datacenter, network, and technology innovations available to the world through its Cloud services. This presentation will provide an overview of the Google Cloud Platform and a deeper dive on Google Compute Engine. Google recently made an open source contribution to SaltStack and now you can now use Salt Cloud to manage your Compute Engine resources (IaaS virtual machine services). Come find out more about Google's Cloud Platform and how you can leverage Google scale with SaltStack.

Steve Singer - Managing PostgreSQL with Puppet @ Postgres OpenPostgresOpen

[2018.10.19] Andrew Kong - Tunnel without tunnel (Seminar at OpenStack Korea ...

Monitor PowerKVM using Ganglia, Nagios

Pradeep Kumar

[OpenStack Days Korea 2016] Track1 - All flash CEPH 구성 및 최적화

[OpenInfra Days Korea 2018] Day 2 - E6 - OpenInfra monitoring with Prometheus

[OpenInfra Days Korea 2018] Day 1 - T4-7: "Ceph 스토리지, PaaS로 서비스 운영하기"

Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...

Kubernetes has the concept of resource requests and limits. Pods get scheduled on the nodes based on their requests and optionally limited in how much of the resource they can consume. Understanding and optimizing resource requests/limits is crucial both for reducing resource "slack" and ensuring application performance/low-latency. This talk shows our approach to monitoring and optimizing Kubernetes resources for 80+ clusters to achieve cost-efficiency and reducing impact for latency-critical applications. All shown tools are Open Source and can be applied to most Kubernetes deployments.

Cloud Foundry on OpenStack - An Experience Report | anynines anynines GmbH

(WEB401) Optimizing Your Web Server on AWS | AWS re:Invent 2014

Amazon Web Services

Tuning your EC2 web server will help you to improve application server throughput and cost-efficiency as well as reduce request latency. In this session we will walk through tactics to identify bottlenecks using tools such as CloudWatch in order to drive the appropriate allocation of EC2 and EBS resources. In addition, we will also be reviewing some performance optimizations and best practices for popular web servers such as Nginx and Apache in order to take advantage of the latest EC2 capabilities.

OSv at Cassandra Summit

Don Marti

OSv is a new, high-performance OS for virtual machines in the cloud. Designed to run one application per guest with minimal overhead, OSv eliminates important bottlenecks for NoSQL applications through improvements in memory management, network I/O, and scheduling. And many important bottlenecks for NoSQL applications are tunable on a conventional OS, but do not require tuning in the OSv environment. OSv is fully stateless and can be configured at runtime with cloud-init or through a REST API, with zero configuration files. OSv offers unified tracing from the application layer through the JVM and the OS kernel. Attendees will learn how to boot Cassandra in one second, and create a simple cluster in a minute.

Setup 3 Node Kafka Cluster on AWS - Hands On

hkbhadraa

ZooKeeper - wait free protocol for coordinating processes

Julia Proskurnia

Trove Updates - Kilo Edition

OpenStack Foundation

OpenStack in Enterprise

Nalee Jang

Ensuring Kubernetes Cost Efficiency across (many) Clusters - DevOps Gathering...

Publicis Sapient Engineering

Talk held at DevOps Gathering 2019 in Bochum on 2019-03-13. Abstract: This talk will address one of the most common challenges of organizations adopting Kubernetes on a medium to large scale: how to keep cloud costs under control without babysitting each and every deployment and cluster configuration? How to operate 80+ Kubernetes clusters in a cost-efficient way for 200+ autonomous development teams? This talk provides insights on how Zalando approaches this problem with central cost optimizations (e.g. Spot), cost monitoring/alerting, active measures to reduce resource slack, and automated cluster housekeeping. We will focus on how to ingrain cost efficiency in tooling and developer workflows while balancing rigid cost control with developer convenience and without impacting availability or performance. We will show our use case running Kubernetes on AWS, but all shown tools are open source and can be applied to most other infrastructure environments.

Python Deployment with Fabricandymccurdy

Paris Container Day 2016 : Etcd - overview and future (CoreOS)

Etcd is an open source distributed consistent key-value store developed by CoreOS. It has become a mature cornerstone of a variety of systems in the container ecosystem for networking, service discovery, configuration management and load balancing. This talk will dive into what etcd is and its history, production use cases and how it powers reliable distributed systems including Kubernetes, the container orchestration engine. Par Jonathan Boulle (Head of Containers and Berlin site lead @ CoreOS) Toutes les vidéos des conférences seront disponibles sur Xebia.tv

PhpTek Ten Things to do to make your MySQL servers Happier and Healthier

Dave Stokes

What's hot

Hostvn ceph in production v1.1 dungtq

Viet Stack

[오픈소스컨설팅] EFK Stack 소개와 설치 방법

Open Source Consulting

[2018.10.19] 김용기 부장 - IAC on OpenStack (feat. ansible)

SaltConf14 - Eric johnson, Google - Orchestrating Google Compute Engine with ...

SaltStack

Steve Singer - Managing PostgreSQL with Puppet @ Postgres OpenPostgresOpen

[2018.10.19] Andrew Kong - Tunnel without tunnel (Seminar at OpenStack Korea ...

Monitor PowerKVM using Ganglia, Nagios

Pradeep Kumar

[OpenStack Days Korea 2016] Track1 - All flash CEPH 구성 및 최적화

[OpenInfra Days Korea 2018] Day 2 - E6 - OpenInfra monitoring with Prometheus

[OpenInfra Days Korea 2018] Day 1 - T4-7: "Ceph 스토리지, PaaS로 서비스 운영하기"

Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...

Cloud Foundry on OpenStack - An Experience Report | anynines anynines GmbH

(WEB401) Optimizing Your Web Server on AWS | AWS re:Invent 2014

Amazon Web Services

OSv at Cassandra Summit

Don Marti

Setup 3 Node Kafka Cluster on AWS - Hands On

hkbhadraa

ZooKeeper - wait free protocol for coordinating processes

Julia Proskurnia

Trove Updates - Kilo Edition

OpenStack Foundation

OpenStack in Enterprise

Nalee Jang

Ensuring Kubernetes Cost Efficiency across (many) Clusters - DevOps Gathering...

Publicis Sapient Engineering

Python Deployment with Fabricandymccurdy

What's hot (20)

Hostvn ceph in production v1.1 dungtq

[오픈소스컨설팅] EFK Stack 소개와 설치 방법

[2018.10.19] 김용기 부장 - IAC on OpenStack (feat. ansible)

SaltConf14 - Eric johnson, Google - Orchestrating Google Compute Engine with ...

Steve Singer - Managing PostgreSQL with Puppet @ Postgres Open

[2018.10.19] Andrew Kong - Tunnel without tunnel (Seminar at OpenStack Korea ...

Monitor PowerKVM using Ganglia, Nagios

[OpenStack Days Korea 2016] Track1 - All flash CEPH 구성 및 최적화

[OpenInfra Days Korea 2018] Day 2 - E6 - OpenInfra monitoring with Prometheus

[OpenInfra Days Korea 2018] Day 1 - T4-7: "Ceph 스토리지, PaaS로 서비스 운영하기"

Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...

Cloud Foundry on OpenStack - An Experience Report | anynines

(WEB401) Optimizing Your Web Server on AWS | AWS re:Invent 2014

OSv at Cassandra Summit

Setup 3 Node Kafka Cluster on AWS - Hands On

ZooKeeper - wait free protocol for coordinating processes

Trove Updates - Kilo Edition

OpenStack in Enterprise

Ensuring Kubernetes Cost Efficiency across (many) Clusters - DevOps Gathering...

Python Deployment with Fabric

Similar to Etcd- Mission Critical Key-Value Store

Paris Container Day 2016 : Etcd - overview and future (CoreOS)

PhpTek Ten Things to do to make your MySQL servers Happier and Healthier

Dave Stokes

Oracle Database In-Memory & Multitenant - TROUG Days'16 Istanbul

Mahir M. Quluzade

Fine-grained fault tolerance using device checkpoints

asimkadav

MySQL Monitoring 101

Ronald Bradford

In this presentation I’ll be discussing the following beginner points to understanding and creating monitoring. * Why Monitor? * What’s the minimum to Monitor? * How to monitor? * Monitoring Software Options. * How to use the most basic of monitoring to help * The basics of graphing results * The rule of Everything * The important on Application metrics and timings For a very little investment in time, simple monitoring can be in place, and I can guarantee it will be of benefit to any system. The basis of monitoring are metrics that combined with application measurements can provide trending insights, bottleneck understanding and provide valuable feedback about your growing site.

les12.pdf

VAMSICHOWDARY61

LinuxFest Northwest 2022 - The Evolution of a MySQL Database System

Frederic Descamps

At the beginning of a project, the database is just a single MySQL instance (maybe not even running on its own hardware)... but with the evolution of the business requirements, the database must change to also meet the new targets of data loss and uptime. During this session we will follow the journey of a single MySQL server from the simple instance to a High Available Architecture with multi-site Disaster Recovery. We will discover easy manageable native solutions like MySQL InnoDB ReplicaSet, MySQL InnoDB Cluster and MySQL InnoDB ClusterSet. The session is also illustrated with commands and examples.

Hands-on Lab: How to Unleash Your Storage Performance by Using NVM Express™ B...

Odinot Stanislas

(FR) Voici un excellent document qui explique étape après étape comment installer, monitorer et surtout correctement benchmarker ses SSD PCIe/NVMe (pas si simple que ça). Autre élément clé : comment analyser la charge I/O de véritables applications? Combien d'IOPS, en read, en write, quelle bande passante et surtout quel impact sur la durée de vie des SSD? Bref à mettre en toute les mains, et un merci à mon collègue Andrey Kudryavtsev. (EN) An excellent content which describe step by step how to install, monitor and benchmark PCIe/NVMe SSD (many trick not so simple). Another key learning: how to measure real I/O activities on a real workload? How many R/W IOPS, block size, throughtput, and finally what's the impact on SSD endurance and (real)life? A must read, and a huge thanks to my colleague Andrey Kudryavtsev. Auteurs/Authors: Andrey Kudryavtsev, SSD Solution Architect, Intel Corporation Zhdan Bybin, Application Engineer, Intel Corporation

SiteGround Tech TeamBuilding

Marian Marinov

Oracle 11g R2 RAC setup on rhel 5.0

Santosh Kangane

[db tech showcase Tokyo 2018]　#dbts2018 #B17 『オラクルパフォーマンスチューニング－神話、伝説と解決策』

Insight Technology, Inc.

5 Things You Need to Know About Enterprise Fl

Western Digital

Watch the full webinar here: http://bit.ly/1TUuUCK When considering flash storage, there are many misconceptions and outright myths. Especially when equating consumer-grade flash (USB sticks) to enterprise-grade SSDs. In this webinar SanDisk Chief Architect, Adam Roberts, will discuss 5 myths of flash storage and highlight what you need to look out for when choosing a storage device to accelerate your data center storage. This webinar will cover: 1.Data Protection 2.Power Fail Protection 3.Temperature Throttling/Overheating 4.QoS for Performance 5.SSD Endurance Stay tuned for future webinars which will look at the benefits of flash beyond performance…busting a few more myths on flash.

Accelerating Spark Genome Sequencing in Cloud—A Data Driven Approach, Case St...

Spark Summit

Spark data processing is shifting from on-premises to cloud service to take advantage of its horizontal resource scalability, better data accessibility and easy manageability. However, fully utilizing the computational power, fast storage and networking offered by cloud service can be challenging without deep understanding of workload characterizations and proper software optimization expertise. In this presentation, we will use a Spark based programing framework – Genome Analysis Toolkit version 4 (GATK4, under development), as an example to present a process of configuring and optimizing a proficient Spark cluster on Google Cloud to speed up genome data processing. We will first introduce an in-house developed data profiling framework named PAT, and discuss how to use PAT to quickly establish the best combination of VM configurations and Spark configurations to fully utilize cloud hardware resources and Spark computational parallelism. In addition, we use PAT and other data profiling tools to identify and fix software hotspots in application. We will show a case study in which we identify a thread scalability issue of Java Instanceof operator. The fix in Scala language hugely improves performance of GATK4 and other Spark based workloads.

AHMED JASSAT SOUTH ARICAN ORACLE USER GROUP PRESENTATION

Zahid02

Percona Live 2022 - The Evolution of a MySQL Database System

Frederic Descamps

From a single MySQL instance to multi-site high availability, this is what you will find out in this presentation. You will learn how to make this transition and which solutions best suit changing business requirements (RPO, RTO). Recently, MySQL has extended the possibilities for easy deployment of architecture with integrated tools. Come and discover these open source solutions that are part of MySQL.

Alibaba cloud benchmarking report ecs rds limton xavier

Limton Xavier

ODA Backup Restore Utility & ODA Rescue Live Disk

Ruggero Citton

When applying maintenance to Oracle Database Appliance, it's best practice to back up the ODA system environment (local system boot disk). DBAs have procedures to backup and recover the database but it is also important that you are able to backup and recover the environment that runs the database. This is especially useful if you encounter an issue during patching; you can quickly restore the system disk back to the pre-patch state.

Performance tunninglokesh777

Extreme Linux Performance Monitoring and Tuning

Milind Koyande

How to put 10lbs of functionality into a 5lb package.

Marc Karasek

Similar to Etcd- Mission Critical Key-Value Store (20)

Paris Container Day 2016 : Etcd - overview and future (CoreOS)

PhpTek Ten Things to do to make your MySQL servers Happier and Healthier

Oracle Database In-Memory & Multitenant - TROUG Days'16 Istanbul

Fine-grained fault tolerance using device checkpoints

MySQL Monitoring 101

les12.pdf

LinuxFest Northwest 2022 - The Evolution of a MySQL Database System

Hands-on Lab: How to Unleash Your Storage Performance by Using NVM Express™ B...

SiteGround Tech TeamBuilding

Oracle 11g R2 RAC setup on rhel 5.0

[db tech showcase Tokyo 2018]　#dbts2018 #B17 『オラクルパフォーマンスチューニング－神話、伝説と解決策』

5 Things You Need to Know About Enterprise Fl

Accelerating Spark Genome Sequencing in Cloud—A Data Driven Approach, Case St...

AHMED JASSAT SOUTH ARICAN ORACLE USER GROUP PRESENTATION

Percona Live 2022 - The Evolution of a MySQL Database System

Alibaba cloud benchmarking report ecs rds limton xavier

ODA Backup Restore Utility & ODA Rescue Live Disk

Performance tunning

Extreme Linux Performance Monitoring and Tuning

How to put 10lbs of functionality into a 5lb package.

More from CoreOS

Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes

Tectonic Summit 2016: It's Go Time

Tectonic Summit 2016: Betting on Kubernetes

Tectonic Summit 2016: Multi-Cluster Kubernetes: Planning for Unknowns

Tectonic Summit 2016: Networking for Kubernetes

Sreekanth Pothanis, Cloud Engineering, eBay shares a networking Kubernetes tale from the trenches. Networking is the hardest component in any ones infrastructure, everything depends on it. Specifically when we have web scale infrastructure with tens of thousands of servers. eBay is investing heavily in Kubernetes and networking again is one of the areas we have the most difficulty with. During the course of this talk we will go through various approaches we tried to make container networking conform to Kubernetes networking principles, while ensuring that it adapts to the existing networking models our infrastructure supports. We would also cover how we have automated the process of setting up networking for Kubernetes clusters and how it offers seamless integration with non-Kubernetes workloads. 12/12/16

Tectonic Summit 2016: Multitenant Data Architectures with Kubernetes

Tectonic Summit 2016: Brandon Philips, CTO of CoreOS, Keynote

Tectonic Summit 2016: Ticketmaster's Public Cloud & Kubernetes Strategy

Tectonic Summit 2016: Alex Polvi, CEO of CoreOS, Keynote

Tectonic Summit 2016: The Origins of Kubernetes

Tectonic Summit 2016: Kubernetes 1.5 and Beyond

Tectonic Summit 2016: CoreOS Tectonic on AWS

Tectonic Summit 2016: Preparing for Cloud Native

Kube-AWS

Clair, A Container Image Security Analyzer

Tectonic Summit 2015: Containers Across the Cloud and Data Center

CoreOS in a Nutshell