This document summarizes Haomai Wang's presentation on containers and Ceph. Some key points include: - Ceph can provide block, file, and object storage for containers and virtual machines. Block storage via RBD is commonly used today but file storage via CephFS may be better suited for containers. - CephFS provides POSIX file sharing across containers and clients, with improvements in snapshotting and statistics capabilities. It inherits Ceph's scalability and resilience. - Orchestration tools like Kubernetes can integrate Ceph storage, either using existing volume plugins or new plugins being developed for Ceph block and file storage. This allows containers to easily share storage.

1. Containers and Ceph Haomai 2015.06.04

2. Hi, I’m Haomai Wang
❖ Join Ceph community Since 2013
❖ GSOC 2015 Ceph Mentor
❖ Maintains KeyValueStore and AsyncMessenger
❖ Active in RBD, Performance, ObjectStore things
❖ Newer to Containers!
❖ haomaiwang@gmail.com

3. Agenda
❖ Motivation
❖ Block ? File ?
❖ CephFS Update
❖ Orchestration
❖ Summary

4. Cloud Hodgepodge
❖ Compelling clouds offer options
❖ Compute
❖ VM (KVM, XEN …)
❖ Containers (LXC, Docker, OpenVZ)
❖ Storage
❖ Block
❖ File
❖ Object
❖ Key/Value
❖ NOSQL
❖ SQL
App
Server
App
Server
App
Server
App
Server

5. Containers?
❖ Performance
❖ Shared Kernel
❖ Fast Boot
❖ Lower baseline overhead
❖ Better resource sharing
❖ Storage
❖ Shared Kernel -> Efficient IO
❖ Small Image -> Efficient deployment
❖ Emerging container host OSs
❖ CoreOS
❖ Atomic
❖ Snappy Ubuntu
❖ New app provisioning model
❖ Small, single-service containers
❖ Standalone execution environment

6. Ceph Components

7. Block/File

8. VM + Block(RBD)
❖ Model
❖ Nova → libvirt → KVM →librbd.so
❖ Cinder → rbd.py →librbd.so
❖ Glance → rbd.py → librbd.so
❖ Pros
❖ proven
❖ decent performance good security
❖ Cons
❖ performance could be better
❖ Status
❖ most common deployment model today
(~44% in latest survey)

9. Container + Block(RBD)
❖ The model
❖ libvirt-based lxc containers(Or Docker)
❖ map kernel RBD on host
❖ pass host device to libvirt, container
❖ Pros
❖ fast and efficient
❖ implement existing Nova API Cons
❖ weaker security than VM
❖ Status
❖ lxc is maintained

10. Follow VM to use mature Block(RBD)?

11. Different App Provision Model
❖ Container VS Virtualization
❖ Hardware abstraction
❖ Application Centric
❖ Per VM Isolation, Guest Environment and
lifecycle defined by Application
❖ Application Isolation
❖ Density
❖ New Provision
❖ Micro-Service
❖ Multi-instance, Multi-version, Maximal
flexible, Minimal overhead
❖ Block
❖ Physical block abstraction
❖ Unknown user data layout
❖ Difficult to bind block to container(s)

12. Data Aware

13. RADOS(File alike Interface)
Ceph Storage Layout
Block
Deivc
Block
Deivc
Block
Deivc
Block
Deivc
Block
Deivc
Block
Deivc
Block
Deivc
Block
Deivc
OSD OSD OSD OSD OSD OSD OSD OSD
RBD
OBJECT OBJECT OBJECT OBJECT OBJECT OBJECT OBJECT OBJECT OBJECT OBJECT
OBJECT OBJECT OBJECT OBJECT OBJECT OBJECT OBJECT OBJECT OBJECT OBJECT
BLOCK
CEPHFS(Strict POSIX)
Directory
Directory DirectoryFile
File
File
File
File
File
File
BLOCK BLOCK BLOCK BLOCK BLOCK BLOCK BLOCK BLOCK

14. Containers may like file more than block

15. CephFS Update

16. File Storage
❖ Familiar POSIX semantics(POSIX
is a lingua-franca)
❖ Fully shared volume – many
clients can mount and share data
❖ Elastic storage – amount of data
can grow/shrink without explicit
provisioning
CephFS

17. CephFS Architecture
❖ Inherit resilience and
scalability of RADOS
❖ Multiple metadata daemons
(MDS) handling dynamically
shared metadata
❖ Fuse & kernel clients: POSIX
compatibility
❖ Extra features: Subtree
snapshots, recursive
statistics

18. Detecting failures
❖ MDS
❖ “beacon” pings to RADOS MONs. Logic on MONs
decides when to mark an MDS failed and promote
another daemon to take its place
❖ Clients:
❖ “RenewCaps” pings to each MDS with which it has a
session. MDSs individually decide to drop a client's
session (and release capabilities) if it is too late.

19. The Now
❖ Priority
❖ Complete FSCK & repair
tools
❖ Tenant Security/Auth
❖ Other work:
❖ Multi-MDS hardening
❖ Snapshot hardening

20. Orchestration

21. Existing VM & File
NFS + CEPHFS.KO VIRTFS/9P + CEPHFS.KO

22. Nova-Docker & CephFS
❖ Model
❖ host mounts CephFS directly
❖ mount --bind share into container
namespace
❖ Pros
❖ best performance
❖ full CephFS semantics
❖ Cons
❖ rely on container for security
❖ Status
❖ no prototype

23. Kubernetes & CephFS
❖ Pure Kubernetes
❖ Volume Driver
❖ AWS EBS, Google Block
❖ CephFS
❖ NFS
❖ …
❖ Status
❖ Under review(https://github.com/
GoogleCloudPlatform/kubernetes/pull/
6649)
❖ Drivers expect pre-existing volumes
❖ Expected deploy mode
❖ Pod(Shared File Volume)
❖ Make micro-service ease with shared storage

24. Kubernetes on OpenStack
❖ Provision Nova VMs
❖ KVM or ironic
❖ Atomic or CoreOS
❖ Kubernetes per tenant
❖ Provision storage devices
❖ Cinder for volumes
❖ Manila for shares
❖ Kubernetes binds into pod/container
❖ Status
❖ Prototype Cinder plugin for Kubernetes
(https://github.com/spothanis/
kubernetes/tree/cinder-vol-plugin)

25. Thank You!
Thanks to
Sage Weil(sage@redhat.com)
John Spray(john.spray@redhat.com)