ceph – a unified distributed storage system                    sage weil           cloudopen – august 29, 2012
outline●   why you should care●   what is it, what it does●   how it works    ●   architecture●   how you can use it    ● ...
why should you care about another        storage system?
requirements●   diverse storage needs    ●   object storage    ●   block devices (for VMs) with snapshots, cloning    ●   ...
time●   ease of administration●   no manual data migration, load balancing●   painless scaling    ●   expansion and contra...
cost●   linear function of size or performance●   incremental expansion    ●   no fork-lift upgrades●   no vendor lock-in ...
what is ceph?
unified storage system●   objects    ●   native    ●   RESTful●   block    ●   thin provisioning, snapshots, cloning●   fi...
APP      APP                      APP                               APP                  HOST/VM                          ...
open source●   LGPLv2    ●   copyleft    ●   ok to link to proprietary code●   no copyright assignment    ●   no dual lice...
distributed storage system●   data center scale    ●   10s to 10,000s of machines    ●   terabytes to exabytes●   fault to...
ceph object model●   pools    ●   1s to 100s    ●   independent namespaces or object collections    ●   replication level,...
why start with objects?●   more useful than (disk) blocks    ●   names in a single flat namespace    ●   variable size    ...
DISK                   DISK                   DISK                   DISK                   DISK                   DISKHUM...
DISK                   DISK                   DISK                   DISKHUMANHUMAN                   DISK                ...
HUMAN    HUMAN      HUMAN                HUMAN                        HUMAN                         HUMAN HUMAN  HUMAN    ...
COMPUTER        COMPUTER   DISK                   DISK        COMPUTER        COMPUTER   DISK                   DISK      ...
OSD    OSD    OSD    OSD    OSD FS     FS     FS    FS     FS     btrfs                                   xfs             ...
Monitors:    •        Maintain cluster membership and stateM    •        Provide consensus for distributed        decision...
HUMAN        MM           M
data distribution●   all objects are replicated N times●   objects are automatically placed, balanced, migrated    in a dy...
CRUSH•   Pseudo-random placement    algorithm•   Fast calculation, no lookup•   Repeatable, deterministic•   Ensures even ...
10 10 01 01 10 10 01 11 01 10         10 10 01 01 10 10 01 11 01 10                                    hash(object name) %...
10 10 01 01 10 10 01 11 01 10        10 10 01 01 10 10 01 11 01 1010 10   10       10   01             01   01            ...
RADOS●   monitors publish osd map that describes cluster state    ●   ceph-osd node status (up/down, weight, IP)          ...
CLIENTCLIENT         ??
CLIENT         ??
APP      APP                     APP                              APP                   HOST/VM                           ...
APP     APP    LIBRADOS     LIBRADOS                native    M    MMM               M                M
LIBRADOSL    • Provides direct access to      RADOS for applications    • C, C++, Python, PHP, Java    • No HTTP overhead
APP      APP                      APP                               APP                  HOST/VM                          ...
APP  APP                 APP                      APP                                  RESTRADOSGWRADOSGW           RADOSG...
RADOS Gateway:• REST-based interface to  RADOS• Supports buckets,  accounting• Compatible with S3 and  Swift applications
APP      APP                      APP                               APP                  HOST/VM                          ...
COMPUTER                  COMPUTER   DISK                             DISK                  COMPUTER                  COMP...
COMPUTER     COMPUTER   DISK                DISK     COMPUTER     COMPUTER   DISK                DISK     COMPUTER     COM...
VM            VMVIRTUALIZATION CONTAINERVIRTUALIZATION CONTAINER            LIBRBD             LIBRBD         LIBRADOS    ...
CONTAINERCONTAINER         VM                  VM       CONTAINER                           CONTAINER   LIBRBD    LIBRBD  ...
HOST        HOST    KRBD (KERNEL MODULE)     KRBD (KERNEL MODULE)         LIBRADOS          LIBRADOS       M       MMM    ...
RADOS Block Device:• Storage of virtual disks in RADOS• Decouples VMs and containers • Live migration!• Images are striped...
HOW DO YOU    SPIN UPTHOUSANDS OF VMs   INSTANTLY      AND  EFFICIENTLY?
instant copy144      0        0         0   0   = 144
write                          CLIENT                  write                  write                  write144   4   = 148
read             read                    CLIENT             read144   4   = 148
APP      APP                      APP                               APP                  HOST/VM                          ...
CLIENT               CLIENTmetadata           01                    01   data                   10                    10  ...
M    MMM       M        M
Metadata Server• Manages metadata for a  POSIX-compliant shared  filesystem • Directory hierarchy • File metadata (owner, ...
one treethree metadata servers                               ??
DYNAMIC SUBTREE PARTITIONING
recursive accounting●   ceph-mds tracks recursive directory stats    ●   file sizes    ●   file and directory counts    ● ...
snapshots●   volume or subvolume snapshots unusable at petabyte scale    ●   snapshot arbitrary subdirectories●   simple i...
multiple protocols, implementations●   Linux kernel client    ●   mount -t ceph 1.2.3.4:/ /mnt                            ...
APP      APP                      APP                               APP                  HOST/VM                          ...
why we do this●   limited options for scalable open source storage●   proprietary solutions    ●   expensive    ●   dont s...
who we are●   Ceph created at UC Santa Cruz (2007)●   supported by DreamHost (2008-2011)●   Inktank (2012)    ●   Los Ange...
thanksBoF tonight @ 5:15sage weilsage@inktank.com     http://github.com/ceph@liewegas            http://ceph.com/
why we like btrfs●   pervasive checksumming●   snapshots, copy-on-write●   efficient metadata (xattrs)●   inline data for ...
CloudOpen - 08/29/2012
CloudOpen - 08/29/2012
CloudOpen - 08/29/2012
CloudOpen - 08/29/2012
CloudOpen - 08/29/2012
CloudOpen - 08/29/2012
CloudOpen - 08/29/2012
Upcoming SlideShare
Loading in …5
×

CloudOpen - 08/29/2012

548 views

Published on

Sage Weil's slides from CloudOpen - Aug 2012

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
548
On SlideShare
0
From Embeds
0
Number of Embeds
146
Actions
Shares
0
Downloads
28
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

CloudOpen - 08/29/2012

  1. 1. ceph – a unified distributed storage system sage weil cloudopen – august 29, 2012
  2. 2. outline● why you should care● what is it, what it does● how it works ● architecture● how you can use it ● librados ● radosgw ● RBD ● file system● who we are, why we do this
  3. 3. why should you care about another storage system?
  4. 4. requirements● diverse storage needs ● object storage ● block devices (for VMs) with snapshots, cloning ● shared file system with POSIX, coherent caches ● structured data... files, block devices, or objects?● scale ● terabytes, petabytes, exabytes ● heterogeneous hardware ● reliability and fault tolerance
  5. 5. time● ease of administration● no manual data migration, load balancing● painless scaling ● expansion and contraction ● seamless migration
  6. 6. cost● linear function of size or performance● incremental expansion ● no fork-lift upgrades● no vendor lock-in ● choice of hardware ● choice of software● open
  7. 7. what is ceph?
  8. 8. unified storage system● objects ● native ● RESTful● block ● thin provisioning, snapshots, cloning● file ● strong consistency, snapshots
  9. 9. APP APP APP APP HOST/VM HOST/VM CLIENT CLIENT RADOSGW RADOSGW RBD RBD CEPH FS CEPH FSLIBRADOS LIBRADOS A bucket-based A bucket-based A reliable and fully- A reliable and fully- A POSIX-compliant A POSIX-compliant A library allowing A library allowing REST gateway, REST gateway, distributed block distributed block distributed file distributed file apps to directly apps to directly compatible with S3 compatible with S3 device, with aaLinux device, with Linux system, with aa system, with access RADOS, access RADOS, and Swift and Swift kernel client and aa kernel client and Linux kernel client Linux kernel client with support for with support for QEMU/KVM driver QEMU/KVM driver and support for and support for C, C++, Java, C, C++, Java, FUSE FUSE Python, Ruby, Python, Ruby, and PHP and PHPRADOSRADOS A reliable, autonomous, distributed object store comprised of self-healing, self-managing, A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes intelligent storage nodes
  10. 10. open source● LGPLv2 ● copyleft ● ok to link to proprietary code● no copyright assignment ● no dual licensing ● no “enterprise-only” feature set● active community● commercial support
  11. 11. distributed storage system● data center scale ● 10s to 10,000s of machines ● terabytes to exabytes● fault tolerant ● no single point of failure ● commodity hardware● self-managing, self-healing
  12. 12. ceph object model● pools ● 1s to 100s ● independent namespaces or object collections ● replication level, placement policy● objects ● bazillions ● blob of data (bytes to gigabytes) ● attributes (e.g., “version=12”; bytes to kilobytes) ● key/value bundle (bytes to gigabytes)
  13. 13. why start with objects?● more useful than (disk) blocks ● names in a single flat namespace ● variable size ● simple API with rich semantics● more scalable than files ● no hard-to-distribute hierarchy ● update semantics do not span objects ● workload is trivially parallel
  14. 14. DISK DISK DISK DISK DISK DISKHUMANHUMAN COMPUTER COMPUTER DISK DISK DISK DISK DISK DISK DISK DISK
  15. 15. DISK DISK DISK DISKHUMANHUMAN DISK DISKHUMANHUMAN COMPUTER COMPUTER DISK DISK DISK DISKHUMANHUMAN DISK DISK DISK DISK
  16. 16. HUMAN HUMAN HUMAN HUMAN HUMAN HUMAN HUMAN HUMAN DISK DISK HUMAN HUMANHUMAN HUMAN DISK DISK HUMAN HUMAN HUMAN HUMAN DISK DISK DISK DISK HUMAN HUMAN DISK DISK HUMAN HUMANHUMAN HUMAN DISK DISK (COMPUTER)) (COMPUTER HUMAN HUMAN DISK DISK HUMAN HUMAN HUMAN HUMAN DISK DISK HUMAN HUMAN HUMAN HUMAN DISK DISK HUMAN HUMAN DISK DISK HUMAN HUMAN DISK DISK HUMAN HUMAN HUMAN HUMAN DISK DISK HUMAN HUMAN HUMAN HUMAN (actually more like this…)
  17. 17. COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISKHUMANHUMAN COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISKHUMANHUMAN COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISKHUMANHUMAN COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK
  18. 18. OSD OSD OSD OSD OSD FS FS FS FS FS btrfs xfs ext4DISK DISK DISK DISK DISK M M M
  19. 19. Monitors: • Maintain cluster membership and stateM • Provide consensus for distributed decision-making • Small, odd number • These do not serve stored objects to clients Object Storage Daemons (OSDs): • At least three in a cluster • One per disk or RAID group • Serve stored objects to clients • Intelligently peer to perform replication tasks
  20. 20. HUMAN MM M
  21. 21. data distribution● all objects are replicated N times● objects are automatically placed, balanced, migrated in a dynamic cluster● must consider physical infrastructure ● ceph-osds on hosts in racks in rows in data centers● three approaches ● pick a spot; remember where you put it ● pick a spot; write down where you put it ● calculate where to put it, where to find it
  22. 22. CRUSH• Pseudo-random placement algorithm• Fast calculation, no lookup• Repeatable, deterministic• Ensures even distribution• Stable mapping • Limited data migration• Rule-based configuration • specifiable replication • infrastructure topology aware • allows weighting
  23. 23. 10 10 01 01 10 10 01 11 01 10 10 10 01 01 10 10 01 11 01 10 hash(object name) % num pg10 10 10 10 01 01 01 01 10 10 10 10 01 01 11 11 01 01 10 10 CRUSH(pg, cluster state, policy)
  24. 24. 10 10 01 01 10 10 01 11 01 10 10 10 01 01 10 10 01 11 01 1010 10 10 10 01 01 01 01 10 10 10 10 01 01 11 11 01 01 10 10
  25. 25. RADOS● monitors publish osd map that describes cluster state ● ceph-osd node status (up/down, weight, IP) M ● CRUSH function specifying desired data distribution● object storage daemons (OSDs) ● safely replicate and store object ● migrate data as the cluster changes over time ● coordinate based on shared view of reality● decentralized, distributed approach allows ● massive scales (10,000s of servers or more) ● the illusion of a single copy with consistent behavior
  26. 26. CLIENTCLIENT ??
  27. 27. CLIENT ??
  28. 28. APP APP APP APP HOST/VM HOST/VM CLIENT CLIENT RADOSGW RADOSGW RBD RBD CEPH FS CEPH FSLIBRADOS A bucket-based A bucket-based A reliable and fully- A reliable and fully- A POSIX-compliant A POSIX-compliant A library allowing REST gateway, REST gateway, distributed block distributed block distributed file distributed file apps to directly compatible with S3 compatible with S3 device, with aaLinux device, with Linux system, with aa system, with access RADOS, and Swift and Swift kernel client and aa kernel client and Linux kernel client Linux kernel client with support for QEMU/KVM driver QEMU/KVM driver and support for and support for C, C++, Java, FUSE FUSE Python, Ruby, and PHPRADOSRADOS A reliable, autonomous, distributed object store comprised of self-healing, self-managing, A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes intelligent storage nodes
  29. 29. APP APP LIBRADOS LIBRADOS native M MMM M M
  30. 30. LIBRADOSL • Provides direct access to RADOS for applications • C, C++, Python, PHP, Java • No HTTP overhead
  31. 31. APP APP APP APP HOST/VM HOST/VM CLIENT CLIENT RADOSGW RBD RBD CEPH FS CEPH FSLIBRADOS LIBRADOS A bucket-based A reliable and fully- A reliable and fully- A POSIX-compliant A POSIX-compliant A library allowing A library allowing REST gateway, distributed block distributed block distributed file distributed file apps to directly apps to directly compatible with S3 device, with aaLinux device, with Linux system, with aa system, with access RADOS, access RADOS, and Swift kernel client and aa kernel client and Linux kernel client Linux kernel client with support for with support for QEMU/KVM driver QEMU/KVM driver and support for and support for C, C++, Java, C, C++, Java, FUSE FUSE Python, Ruby, Python, Ruby, and PHP and PHPRADOSRADOS A reliable, autonomous, distributed object store comprised of self-healing, self-managing, A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes intelligent storage nodes
  32. 32. APP APP APP APP RESTRADOSGWRADOSGW RADOSGW RADOSGW LIBRADOS LIBRADOS LIBRADOS LIBRADOS native M M M M M M
  33. 33. RADOS Gateway:• REST-based interface to RADOS• Supports buckets, accounting• Compatible with S3 and Swift applications
  34. 34. APP APP APP APP HOST/VM HOST/VM CLIENT CLIENT RADOSGW RADOSGW RBD CEPH FS CEPH FSLIBRADOS LIBRADOS A bucket-based A bucket-based A reliable and fully- A POSIX-compliant A POSIX-compliant A library allowing A library allowing REST gateway, REST gateway, distributed block distributed file distributed file apps to directly apps to directly compatible with S3 compatible with S3 device, with a Linux system, with aa system, with access RADOS, access RADOS, and Swift and Swift kernel client and a Linux kernel client Linux kernel client with support for with support for QEMU/KVM driver and support for and support for C, C++, Java, C, C++, Java, FUSE FUSE Python, Ruby, Python, Ruby, and PHP and PHPRADOSRADOS A reliable, autonomous, distributed object store comprised of self-healing, self-managing, A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes intelligent storage nodes
  35. 35. COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISKCOMPUTERCOMPUTER COMPUTER COMPUTER DISK DISK DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK
  36. 36. COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISKVMVM COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISKVMVM COMPUTER DISK COMPUTER DISK COMPUTER COMPUTER DISK DISKVMVM COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK
  37. 37. VM VMVIRTUALIZATION CONTAINERVIRTUALIZATION CONTAINER LIBRBD LIBRBD LIBRADOS LIBRADOS M M M M M M
  38. 38. CONTAINERCONTAINER VM VM CONTAINER CONTAINER LIBRBD LIBRBD LIBRBD LIBRBD LIBRADOS LIBRADOS LIBRADOS LIBRADOS M M M M M M
  39. 39. HOST HOST KRBD (KERNEL MODULE) KRBD (KERNEL MODULE) LIBRADOS LIBRADOS M MMM M M
  40. 40. RADOS Block Device:• Storage of virtual disks in RADOS• Decouples VMs and containers • Live migration!• Images are striped across the cluster• Snapshots!• Support in • Qemu/KVM • OpenStack, CloudStack • Mainline Linux kernel
  41. 41. HOW DO YOU SPIN UPTHOUSANDS OF VMs INSTANTLY AND EFFICIENTLY?
  42. 42. instant copy144 0 0 0 0 = 144
  43. 43. write CLIENT write write write144 4 = 148
  44. 44. read read CLIENT read144 4 = 148
  45. 45. APP APP APP APP HOST/VM HOST/VM CLIENT CLIENT RADOSGW RADOSGW RBD RBD CEPH FSLIBRADOS LIBRADOS A bucket-based A bucket-based A reliable and fully- A reliable and fully- A POSIX-compliant A library allowing A library allowing REST gateway, REST gateway, distributed block distributed block distributed file apps to directly apps to directly compatible with S3 compatible with S3 device, with aaLinux device, with Linux system, with a access RADOS, access RADOS, and Swift and Swift kernel client and aa kernel client and Linux kernel client with support for with support for QEMU/KVM driver QEMU/KVM driver and support for C, C++, Java, C, C++, Java, FUSE Python, Ruby, Python, Ruby, and PHP and PHPRADOSRADOS A reliable, autonomous, distributed object store comprised of self-healing, self-managing, A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes intelligent storage nodes
  46. 46. CLIENT CLIENTmetadata 01 01 data 10 10 M M M M M M
  47. 47. M MMM M M
  48. 48. Metadata Server• Manages metadata for a POSIX-compliant shared filesystem • Directory hierarchy • File metadata (owner, timestamps, mode, etc.)• Stores metadata in RADOS• Does not serve file data to clients• Only required for shared filesystem
  49. 49. one treethree metadata servers ??
  50. 50. DYNAMIC SUBTREE PARTITIONING
  51. 51. recursive accounting● ceph-mds tracks recursive directory stats ● file sizes ● file and directory counts ● modification time● virtual xattrs present full stats● efficient $ ls ­alSh | head total 0 drwxr­xr­x 1 root            root      9.7T 2011­02­04 15:51 . drwxr­xr­x 1 root            root      9.7T 2010­12­16 15:06 .. drwxr­xr­x 1 pomceph         pg4194980 9.6T 2011­02­24 08:25 pomceph drwxr­xr­x 1 mcg_test1       pg2419992  23G 2011­02­02 08:57 mcg_test1 drwx­­x­­­ 1 luko            adm        19G 2011­01­21 12:17 luko drwx­­x­­­ 1 eest            adm        14G 2011­02­04 16:29 eest drwxr­xr­x 1 mcg_test2       pg2419992 3.0G 2011­02­02 09:34 mcg_test2 drwx­­x­­­ 1 fuzyceph        adm       1.5G 2011­01­18 10:46 fuzyceph drwxr­xr­x 1 dallasceph      pg275     596M 2011­01­14 10:06 dallasceph
  52. 52. snapshots● volume or subvolume snapshots unusable at petabyte scale ● snapshot arbitrary subdirectories● simple interface ● hidden .snap directory ● no special tools $ mkdir foo/.snap/one # create snapshot $ ls foo/.snap one $ ls foo/bar/.snap _one_1099511627776 # parents snap name is mangled $ rm foo/myfile $ ls -F foo bar/ $ ls -F foo/.snap/one myfile bar/ $ rmdir foo/.snap/one # remove snapshot
  53. 53. multiple protocols, implementations● Linux kernel client ● mount -t ceph 1.2.3.4:/ /mnt NFS SMB/CIFS ● export (NFS), Samba (CIFS)● ceph-fuse Ganesha Samba libcephfs libcephfs● libcephfs.so ● your app Hadoop your app libcephfs libcephfs ● Samba (CIFS) ● Ganesha (NFS) ceph-fuse ceph fuse ● Hadoop (map/reduce) kernel
  54. 54. APP APP APP APP HOST/VM HOST/VM CLIENT CLIENT RADOSGW RADOSGW RBD RBD CEPH FS CEPH FSLIBRADOS LIBRADOS A bucket-based A bucket-based A reliable and fully- A reliable and fully- A POSIX-compliant A POSIX-compliant A library allowing A library allowing REST gateway, REST gateway, distributed block distributed block distributed file distributed file apps to directly apps to directly compatible with S3 compatible with S3 device, with aaLinux device, with Linux system, with aa system, with access RADOS, access RADOS, and Swift and Swift kernel client and aa kernel client and Linux kernel client Linux kernel client with support for with support for QEMU/KVM driver QEMU/KVM driver and support for and support for C, C++, Java, C, C++, Java, FUSE FUSE Python, Ruby, Python, Ruby, and PHP and PHP AWESOME AWESOME NEARLY AWESOME AWESOMERADOSRADOS AWESOME A reliable, autonomous, distributed object store comprised of self-healing, self-managing, A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes intelligent storage nodes
  55. 55. why we do this● limited options for scalable open source storage● proprietary solutions ● expensive ● dont scale (well or out) ● marry hardware and software● industry needs to change
  56. 56. who we are● Ceph created at UC Santa Cruz (2007)● supported by DreamHost (2008-2011)● Inktank (2012) ● Los Angeles, Sunnyvale, San Francisco, remote● growing user and developer community ● Linux distros, users, cloud stacks, SIs, OEMs http://ceph.com/
  57. 57. thanksBoF tonight @ 5:15sage weilsage@inktank.com http://github.com/ceph@liewegas http://ceph.com/
  58. 58. why we like btrfs● pervasive checksumming● snapshots, copy-on-write● efficient metadata (xattrs)● inline data for small files● transparent compression● integrated volume management ● software RAID, mirroring, error recovery ● SSD-aware● online fsck● active development community

×