This talk will cover the design of the FoundationDB operator and lessons learned from operating FoundationDB on Kubernetes. We will discuss some of the missing pieces in Kubernetes to make it easier to operate FoundationDB on top of it and how we solved those challenges in the operator. We will focus on the pieces of the FoundationDB operator that are different to most other operators and why we decided to implement those pieces like they are. We will also discuss how to run an high available FoundationDB cluster on top of Kubernetes and what different choices a user has. We will also cover some challenges that arise when running stateful services at scale on top of Kubernetes and how they can be managed. At the end of this talk we will give an outlook for future design changes and planned features in our operator. This talk was given by Johannes Scheurmann for DoK Day Europe @ KubeCon 2022.

1. DoK Day Europe 2022 @ KubeCon
Johannes M. Scheuermann
Operating FoundationDB on Kubernetes

2. Whoami
• FoundationDB SRE@
• Running FoundationDB at a massive scale
• Working with Kubernetes since the end of 2014
DoK Day Europe 2022 @ KubeCon

3. FoundationDB
• Distributed transactional Key-Value store
• Separate Transaction and Storage systems
• No DNS support (WIP as of 7.1 release)
• Major/minor versions not compatible
• Automatic recovery and data redistribution
• Apache 2 License
• https://www.foundationdb.org
DoK Day Europe 2022 @ KubeCon

4. FDB Kubernetes Operator
• Open Source
• https://github.com/FoundationDB/fdb-kubernetes-operator
• Runs FDB clusters on top of Kubernetes
• Manages bare Pods, PVCs and Services
• Fetches required tooling “injected” by init containers
• FDB libraries and cli tooling
• Different possibilities to run (HA) clusters
• Provides a kubectl plugin for additional steps
• Like monitoring and reparing FDB clusters
DoK Day Europe 2022 @ KubeCon

5. FDB Upgrades
• First we update the sidecar container with the new version
• Sidecar injects new binary
• fdbmonitor configuration file is adjusted to use the new binary
• Restart all processes in the cluster
• They pickup the new config and are running with the new version
• Replace the whole transaction system
• This should be a fairly fast step
• Reduces number of recoveries
• Recreate all storage Pods with the new image
• Per default this is done zone by zone
DoK Day Europe 2022 @ KubeCon

6. FDB HA
• Multiple ways to run FDB on top of Kubernetes
• Running a single FDB cluster across multiple Kubernetes clusters
• Running an HA FDB cluster across multiple Kubernetes clusters
• Running an HA FDB cluster across multiple namespaces in one cluster
• Each strategy requires that all processes in a cluster can communicate (mesh)
• Each Kubernetes cluster has a dedicated operator instance
• Syncing over FDB to do FDB cluster wide actions (kill all processes)
• Requires that all cluster specs have the same config (except for datacenter/
cluster information)
DoK Day Europe 2022 @ KubeCon

7. Make debugging easier
• To help human operators we created a kubectl plugin
• The plugin supports running a variety of actions on a cluster
• Most important command kubectl fdb analyze ...
• This will printout all potential issues with the cluster
• Goal is to make it easy for a human operator to interact/manage FDB
clusters
• In practice this was a very useful pattern
DoK Day Europe 2022 @ KubeCon

8. Gaining confidence
• Unit tests
• Integration tests
• e2e tests (with chaos injection)
• Features disabled by default
• Guarded by flags or by settings
• Safety checks, Safety checks, Safety checks, oh and Safety checks
• Reduce number of possible failure scenarios
• Don't want to lose data/availability
DoK Day Europe 2022 @ KubeCon

9. Future work
• Minimum fault domains for Pod Topology Spread Constraints
• Actually implemented by now: https://github.com/kubernetes/kubernetes/pull/
108362
• Colouring approach
• Will be implemented in fdb-operator
• Idea reduce the number of logical fault domains
• Prevent accidental deletion of clusters and Pods
• Can be achieved with WebHooks
• WebHooks can lead to cluster instability and you might not be able to use one
• Better/easier support for multi-cluster workload
DoK Day Europe 2022 @ KubeCon

10. What’s next?
• FoundationDBProcessGroup CRD
• Allows more advanced things like "exclude a specific instance"
• Better handling of the deletion logic
• Better HA/multi cluster support
• Support for the kubectl plugin
• meta-operator?
• Backup and restore support in the operator
• Currently only basic support for backup and restore
• Make use of the management API
• Don't call external binaries (memory handling)
DoK Day Europe 2022 @ KubeCon

11. DoK Day Europe 2022 @ KubeCon
Johannes M. Scheuermann
Operating FoundationDB on Kubernetes
Thank you!