Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Wanting distributed volumes - Experiences with ceph-docker

1,625 views

Published on

Slides of a docker meetup presentation in Cologne (april 28,2016)
The presentation talks about how to run ceph in docker containers and how to use the ceph filesystems for volumes of docker containers in need of persistent storage.

Published in: Software
  • Be the first to comment

Wanting distributed volumes - Experiences with ceph-docker

  1. 1. CEPH-docker by Ewout Prangsma github:ewoutp / twitter:ewoutp Experiences with ceph-docker Wanting distributed volumes
  2. 2. CEPH-docker by Ewout Prangsma github:ewoutp / twitter:ewoutp The problem 1/2 MC MC MC Machine 1 Machine 2 Machine 3
  3. 3. CEPH-docker by Ewout Prangsma github:ewoutp / twitter:ewoutp The problem 2/2 HD MC HD MC HD MC Machine 1 Machine 2 Machine 3 Where to store state?
  4. 4. CEPH-docker by Ewout Prangsma github:ewoutp / twitter:ewoutp Requirements ● No provider specifics – E.g. no AWS EBS ● Use available resources ● Be resilient against failures ● Everything in a container
  5. 5. CEPH-docker by Ewout Prangsma github:ewoutp / twitter:ewoutp Solutions ● CEPH ● Proven in production ● Automatic balancing ● Daunting to setup but wait, ceph-docker ● GlusterFS ● Proven in production ● Manual setup
  6. 6. CEPH-docker by Ewout Prangsma github:ewoutp / twitter:ewoutp CEPH intro ● Distributed object storage ● Block storage ● Filesystem ● Automatic balancing using CRUSH – Object location is calculated, not stored ● Terms – Object Storage Device (OSD) – Cluster map – Monitor
  7. 7. CEPH-docker by Ewout Prangsma github:ewoutp / twitter:ewoutp CEPH-docker ● Source: github.com/ceph/ceph-docker ● Container: ceph/daemon – Contains monitor, osd, mds, rgw – Performs configuration setup/retrieval ● Configuration stored in KV backend – consul – etcd ●
  8. 8. CEPH-docker by Ewout Prangsma github:ewoutp / twitter:ewoutp Running CEPH-docker: monitor task "monitor" { global = true count = 2 image = "ceph/daemon" volumes = "/var/lib/ceph:/var/lib/ceph" args = ["mon"] env { MON_IP = "{{private_ipv4}}" MON_NAME = "{{hostname}}" KV_TYPE = "etcd" KV_IP = "{{private_ipv4}}" CEPH_PUBLIC_NETWORK = "$(echo {{private_ipv4}} | cut -d '.' -f 1,2,3 | awk '{print $1 ".0/24"}')" } docker-args = ["--net=host"] }
  9. 9. CEPH-docker by Ewout Prangsma github:ewoutp / twitter:ewoutp Running CEPH-docker: ODS task "osd" { global = true count = 2 image = "ceph/daemon" volumes = [ "/var/lib/ceph/osd:/var/lib/ceph/osd", "/var/log/ceph:/var/log/ceph" ] args = ["osd"] env { HOSTNAME= "{{hostname}}" OSD_TYPE = "directory" OSD_JOURNAL_SIZE = "20" KV_TYPE = "etcd" KV_IP = "{{private_ipv4}}" } docker-args = ["--net=host", "--privileged=true", "--pid=host"] }
  10. 10. CEPH-docker by Ewout Prangsma github:ewoutp / twitter:ewoutp Running CEPH-docker: MDS task "mds" { count = 2 image = "ceph/daemon" volumes = [ "/var/lib/ceph/:/var/lib/ceph/", "/var/log/ceph:/var/log/ceph" ] args = ["mds"] env { MDS_NAME= "mds-${instance}" CEPHFS_CREATE = "1" KV_TYPE = "etcd" KV_IP = "{{private_ipv4}}" } docker-args = ["--net=host"] }
  11. 11. CEPH-docker by Ewout Prangsma github:ewoutp / twitter:ewoutp Mounting volumes ● Mount CEPH FS once/host – Single point of failure – Bad for upgrades ● Mount CEPH FS / container – May use more resources – Needs recent docker ● Use docker volume plugin – Plugin process cannot be run in container
  12. 12. CEPH-docker by Ewout Prangsma github:ewoutp / twitter:ewoutp Mount CEPH FS / container ● Uses data container pattern docker run -it –-name=mydata --net=host --privileged -v /media/ceph:/data:shared -v /usr/bin/etcdctl:/usr/bin/etcdctl pulcy/ceph-volume:latest docker run -it -–volumes-from=mydata alpine ls -al /data
  13. 13. CEPH-docker by Ewout Prangsma github:ewoutp / twitter:ewoutp The result ● CEPH object storage running on all machines – Using available resources ● Data still available if (some) machines goes down ● Containers get the persistent state they need – on whatever machines they're scheduled on Links ● github.com/ceph/ceph-docker ● github.com/pulcy/ceph-volume ● github.com/pulcy/j2
  14. 14. CEPH-docker by Ewout Prangsma github:ewoutp / twitter:ewoutp Questions ● Is it production ready? – time will tell....

×