Ceph Community , profile picture
Uploaded byCeph Community
PDF, PPTX1,997 views

Ceph LISA'12 Presentation

Ceph is a unified distributed storage system that provides object storage, block storage, and file storage functionality. It is designed to scale from tens to tens of thousands of servers storing terabytes to exabytes of data. Ceph distributes data across its cluster and replicates it for fault tolerance without a single point of failure using low-cost commodity hardware and an autonomous, self-healing architecture.

Embed presentation

Download as PDF, PPTX
managing a distributed storage system at scale sage weil – inktank LISA – 12.12.12
outline ● the scale-out world ● what is it, what it's for, how it works ● how you can use it ● librados ● radosgw ● RBD, the ceph block device ● distributed file system ● how do you deploy and manage it ● roadmap ● why we do this, who we are
why should you care about another storage system?
the old reality ● direct-attached storage ● raw disks ● RAID controller ● network-attached storage ● a few large arrays ● NFS, CIFS, maybe iSCSI ● SAN ● expensive enterprise array ● scalable capacity and performance, to a point ● poor management scalability
new user demands ● “cloud” ● many disk volumes, with automated provisioning ● larger data sets – RESTful object storage (e.g., S3, Swift) – NoSQL distributed key/value stores (Cassandra, Riak) ● “big data” ● distributed processing ● “unstructured data” ● ...and the more traditional workloads
requirements ● diverse use-cases ● object storage ● block devices (for VMs) with snapshots, cloning ● shared file system with POSIX, coherent caches ● structured data... files, block devices, or objects? ● scale ● terabytes, petabytes, exabytes ● heterogeneous hardware ● reliability and fault tolerance
cost ● near-linear function of size or performance ● incremental expansion ● no fork-lift upgrades ● no vendor lock-in ● choice of hardware ● choice of software ● open
time ● ease of administration ● no manual data migration, load balancing ● painless scaling ● expansion and contraction ● seamless migration ● devops friendly
being a good devops citizen ● seamless integration with infrastructure tools ● Chef, Juju, Puppet, Crowbar, or home-grown ● collectd, statsd, graphite, nagios ● OpenStack, CloudStack ● simple processes for ● expanding or contracting the storage cluster ● responding to failure scenarios: initial healing, or mop-up ● evolution of “Unix philosophy” ● everything is just daemon ● CLI, REST APIs, JSON ● solve one problem, and solve it well
what is ceph?
unified storage system ● objects ● native API ● RESTful ● block ● thin provisioning, snapshots, cloning ● file ● strong consistency, snapshots
APP APP APP APP HOST/VM HOST/VM CLIENT CLIENT RADOSGW RADOSGW RBD RBD CEPH FS CEPH FS LIBRADOS LIBRADOS A bucket-based A bucket-based A reliable and A reliable and A POSIX-compliant A POSIX-compliant A library allowing A library allowing REST gateway, REST gateway, fully-distributed fully-distributed distributed file distributed file apps to directly apps to directly compatible with S3 compatible with S3 block device, with aa block device, with system, with aa system, with access RADOS, access RADOS, and Swift and Swift Linux kernel client Linux kernel client Linux kernel client Linux kernel client with support for with support for and aaQEMU/KVM and QEMU/KVM and support for and support for C, C++, Java, C, C++, Java, driver driver FUSE FUSE Python, Ruby, Python, Ruby, and PHP and PHP RADOS ––the Ceph object store RADOS the Ceph object store A reliable, autonomous, distributed object store comprised of self-healing, self-managing, A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes intelligent storage nodes
distributed storage system ● data center scale ● 10s to 10,000s of machines ● terabytes to exabytes ● fault tolerant ● no single point of failure ● commodity hardware ● self-managing, self-healing
ceph object model ● pools ● 1s to 100s ● independent object namespaces or collections ● replication level, placement policy ● objects ● bazillions ● blob of data (bytes to gigabytes) ● attributes (e.g., “version=12”; bytes to kilobytes) ● key/value bundle (bytes to gigabytes)
why start with objects? ● more useful than (disk) blocks ● names in a simple flat namespace ● variable size ● simple API with rich semantics ● more scalable than files ● no hard-to-distribute hierarchy ● update semantics do not span objects ● workload is trivially parallel
How do you design a storage system that scales?
DISK DISK DISK DISK DISK DISK HUMAN HUMAN COMPUTER COMPUTER DISK DISK DISK DISK DISK DISK DISK DISK
DISK DISK DISK DISK HUMAN HUMAN DISK DISK HUMAN HUMAN COMPUTER COMPUTER DISK DISK DISK DISK HUMAN HUMAN DISK DISK DISK DISK
HUMAN HUMAN HUMAN HUMAN HUMAN HUMAN HUMAN HUMAN DISK DISK HUMAN HUMAN HUMAN HUMAN DISK DISK HUMAN HUMAN HUMAN HUMAN DISK DISK DISK DISK HUMAN HUMAN DISK DISK HUMAN HUMAN HUMAN HUMAN DISK DISK (COMPUTER)) (COMPUTER HUMAN HUMAN DISK DISK HUMAN HUMAN HUMAN HUMAN DISK DISK HUMAN HUMAN HUMAN HUMAN DISK DISK HUMAN HUMAN DISK DISK HUMAN HUMAN DISK DISK HUMAN HUMAN HUMAN HUMAN DISK DISK HUMAN HUMAN HUMAN HUMAN (actually more like this…)
COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK HUMAN HUMAN COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK HUMAN HUMAN COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK HUMAN HUMAN COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK
OSD OSD OSD OSD OSD FS FS FS FS FS btrfs xfs ext4 DISK DISK DISK DISK DISK M M M
Monitors maintain cluster membership and state M ● ● provide consensus for distributed decision-making ● small, odd number ● these do not served stored objects to clients Object Storage Daemons (OSDs) ● 10s to 10000s in a cluster ● one per disk, SSD, or RAID group ● hardware agnostic ● serve stored objects to clients ● intelligently peer to perform replication and recovery tasks
HUMAN M M M
data distribution ● all objects are replicated N times ● objects are automatically placed, balanced, migrated in a dynamic cluster ● must consider physical infrastructure ● ceph-osds on hosts in racks in rows in data centers ● three approaches ● pick a spot; remember where you put it ● pick a spot; write down where you put it ● calculate where to put it, where to find it
CRUSH ● pseudo-random placement algorithm ● fast calculation, no lookup ● repeatable, deterministic ● statistically uniform distribution ● stable mapping ● limited data migration on change ● rule-based configuration ● infrastructure topology aware ● adjustable replication ● allows weighting
hash(object name) % num pg 10 10 10 10 01 01 01 01 10 10 10 10 01 01 11 11 01 01 10 10 CRUSH(pg, cluster state, policy)
10 10 10 10 01 01 01 01 10 10 10 10 01 01 11 11 01 01 10 10
RADOS ● monitors publish osd map that describes cluster state ● ceph-osd node status (up/down, weight, IP) M ● CRUSH function specifying desired data distribution ● object storage daemons (OSDs) ● safely replicate and store object ● migrate data as the cluster changes over time ● coordinate based on shared view of reality ● decentralized, distributed approach allows ● massive scales (10,000s of servers or more) ● the illusion of a single copy with consistent behavior
CLIENT CLIENT ??
CLIENT ??
APP APP APP APP HOST/VM HOST/VM CLIENT CLIENT RADOSGW RADOSGW RBD RBD CEPH FS CEPH FS LIBRADOS A bucket-based A bucket-based A reliable and A reliable and A POSIX-compliant A POSIX-compliant A library allowing REST gateway, REST gateway, fully-distributed fully-distributed distributed file distributed file apps to directly compatible with S3 compatible with S3 block device, with aa block device, with system, with aa system, with access RADOS, and Swift and Swift Linux kernel client Linux kernel client Linux kernel client Linux kernel client with support for and aaQEMU/KVM and QEMU/KVM and support for and support for C, C++, Java, driver driver FUSE FUSE Python, Ruby, and PHP RADOS RADOS A reliable, autonomous, distributed object store comprised of self-healing, self-managing, A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes intelligent storage nodes
APP APP LIBRADOS LIBRADOS native M M M M M M
librados L ● direct access to RADOS from applications ● C, C++, Python, PHP, Java, Erlang ● direct access to storage nodes ● no HTTP overhead
rich librados API ● atomic single-object transactions ● update data, attr together ● atomic compare-and-swap ● efficient key/value storage inside an object ● object-granularity snapshot infrastructure ● embed code in ceph-osd daemon via plugin API ● inter-client communication via object
APP APP APP APP HOST/VM HOST/VM CLIENT CLIENT RADOSGW RBD RBD CEPH FS CEPH FS LIBRADOS LIBRADOS A bucket-based A reliable and A reliable and A POSIX-compliant A POSIX-compliant A library allowing A library allowing REST gateway, fully-distributed fully-distributed distributed file distributed file apps to directly apps to directly compatible with S3 block device, with aa block device, with system, with aa system, with access RADOS, access RADOS, and Swift Linux kernel client Linux kernel client Linux kernel client Linux kernel client with support for with support for and aaQEMU/KVM and QEMU/KVM and support for and support for C, C++, Java, C, C++, Java, driver driver FUSE FUSE Python, Ruby, Python, Ruby, and PHP and PHP RADOS RADOS A reliable, autonomous, distributed object store comprised of self-healing, self-managing, A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes intelligent storage nodes
APP APP APP APP REST RADOSGW RADOSGW RADOSGW RADOSGW LIBRADOS LIBRADOS LIBRADOS LIBRADOS native M M M M M M
RADOS Gateway ● REST-based object storage proxy ● uses RADOS to store objects ● API supports buckets, accounting ● usage accounting for billing purposes ● compatible with S3, Swift APIs
APP APP APP APP HOST/VM HOST/VM CLIENT CLIENT RADOSGW RADOSGW RBD CEPH FS CEPH FS LIBRADOS LIBRADOS A bucket-based A bucket-based A reliable and A POSIX-compliant A POSIX-compliant A library allowing A library allowing REST gateway, REST gateway, fully-distributed distributed file distributed file apps to directly apps to directly compatible with S3 compatible with S3 block device, with a system, with aa system, with access RADOS, access RADOS, and Swift and Swift Linux kernel client Linux kernel client Linux kernel client with support for with support for and a QEMU/KVM and support for and support for C, C++, Java, C, C++, Java, driver FUSE FUSE Python, Ruby, Python, Ruby, and PHP and PHP RADOS RADOS A reliable, autonomous, distributed object store comprised of self-healing, self-managing, A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes intelligent storage nodes
COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER COMPUTER COMPUTER DISK DISK DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK
COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK VM VM COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK VM VM COMPUTER DISK COMPUTER DISK COMPUTER COMPUTER DISK DISK VM VM COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK
VM VM VIRTUALIZATION CONTAINER (KVM) VIRTUALIZATION CONTAINER (KVM) LIBRBD LIBRBD LIBRADOS LIBRADOS M M M M M M
CONTAINER CONTAINER VM VM CONTAINER CONTAINER LIBRBD LIBRBD LIBRBD LIBRBD LIBRADOS LIBRADOS LIBRADOS LIBRADOS M M M M M M
HOST HOST KRBD (KERNEL MODULE) KRBD (KERNEL MODULE) LIBRADOS LIBRADOS M M M M M M
RADOS Block Device ● storage of disk images in RADOS ● decouple VM from host ● images striped across entire cluster (pool) ● snapshots ● copy-on-write clones ● support in ● Qemu/KVM ● mainline Linux kernel (2.6.39+) ● OpenStack, CloudStack
APP APP APP APP HOST/VM HOST/VM CLIENT CLIENT RADOSGW RADOSGW RBD RBD CEPH FS LIBRADOS LIBRADOS A bucket-based A bucket-based A reliable and A reliable and A POSIX-compliant A library allowing A library allowing REST gateway, REST gateway, fully-distributed fully-distributed distributed file apps to directly apps to directly compatible with S3 compatible with S3 block device, with aa block device, with system, with a access RADOS, access RADOS, and Swift and Swift Linux kernel client Linux kernel client Linux kernel client with support for with support for and aaQEMU/KVM and QEMU/KVM and support for C, C++, Java, C, C++, Java, driver driver FUSE Python, Ruby, Python, Ruby, and PHP and PHP RADOS RADOS A reliable, autonomous, distributed object store comprised of self-healing, self-managing, A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes intelligent storage nodes
CLIENT CLIENT metadata 01 01 data 10 10 M M M M M M
M M M M M M
Metadata Server (MDS) ● manages metadata for POSIX shared file system ● directory hierarchy ● file metadata (size, owner, timestamps) ● stores metadata in RADOS ● does not serve file data to clients ● only required for the shared file system
one tree three metadata servers ??
DYNAMIC SUBTREE PARTITIONING
recursive accounting ● ceph-mds tracks recursive directory stats ● file sizes ● file and directory counts ● modification time ● virtual xattrs present full stats ● efficient $ ls ­alSh | head total 0 drwxr­xr­x 1 root            root      9.7T 2011­02­04 15:51 . drwxr­xr­x 1 root            root      9.7T 2010­12­16 15:06 .. drwxr­xr­x 1 pomceph         pg4194980 9.6T 2011­02­24 08:25 pomceph drwxr­xr­x 1 mcg_test1       pg2419992  23G 2011­02­02 08:57 mcg_test1 drwx­­x­­­ 1 luko            adm        19G 2011­01­21 12:17 luko drwx­­x­­­ 1 eest            adm        14G 2011­02­04 16:29 eest drwxr­xr­x 1 mcg_test2       pg2419992 3.0G 2011­02­02 09:34 mcg_test2 drwx­­x­­­ 1 fuzyceph        adm       1.5G 2011­01­18 10:46 fuzyceph drwxr­xr­x 1 dallasceph      pg275     596M 2011­01­14 10:06 dallasceph
snapshots ● volume or subvolume snapshots unusable at petabyte scale ● snapshot arbitrary subdirectories ● simple interface ● hidden '.snap' directory ● no special tools $ mkdir foo/.snap/one # create snapshot $ ls foo/.snap one $ ls foo/bar/.snap _one_1099511627776 # parent's snap name is mangled $ rm foo/myfile $ ls -F foo bar/ $ ls -F foo/.snap/one myfile bar/ $ rmdir foo/.snap/one # remove snapshot
multiple protocols, implementations ● Linux kernel client ● mount -t ceph 1.2.3.4:/ /mnt NFS SMB/CIFS ● export (NFS), Samba (CIFS) ● ceph-fuse Ganesha Samba libcephfs libcephfs ● libcephfs.so ● your app Hadoop your app libcephfs libcephfs ● Samba (CIFS) ● Ganesha (NFS) ceph-fuse ceph fuse ● Hadoop (map/reduce) kernel
How do you deploy and manage all this stuff?
installation ● work closely with distros ● packages in Debian, Fedora, Ubuntu, OpenSUSE ● build releases for all distros, recent releases ● e.g., point to our deb src and apt-get install ● allow mixed-version clusters ● upgrade each component individually, each host or rack one-by-one ● protocol feature bits, safe data type encoding ● facilitate testing, bleeding edge ● automatic build system generates packages for all branches in git ● each to test new code, push hot-fixes
configuration ● minimize local configuration ● logging, local data paths, tuning options ● cluster state is managed centrally by monitors ● specify option via ● config file (global or local) ● command line ● adjusted for running daemon ● flexible configuration management options ● global synced/copied config file ● generated by management tools – Chef, Juju, Crowbar ● be flexible
provisioning ● embrace dynamic nature of the cluster ● disks, hosts, rack may come online at any time ● anything may fail at any time ● simple scriptable sequences 'ceph osd create <uuid>' → allocate osd id 'ceph-osd --mkfs -i <id>' → initialize local data dir 'ceph osd crush …' → add to crush map ● identify minimal amount of central coordination ● monitor cluster membership/quorum ● provide hooks for external tools to do the rest
old school HA ● deploy a pair of servers ● heartbeat ● move a floating IP between then ● clients contact same “server”, which floats ● cumbersome to configure, deploy ● the “client/server” model doesn't scale out
make client, protocol cluster-aware ● clients learn topology of the distributed cluster ● all OSDs, current ips/ports, data distribution map ● clients talk to the server they want ● servers scale out, client traffic distributes too ● servers dynamically register with the cluster ● when a daemon starts, it updates its address in the map ● other daemons and clients learn map updates via gossip ● servers manage heartbeat, replication ● no fragile configuration ● no fail-over pairs
ceph disk management ● label disks ● GPT partition type (fixed uuid) ● udev ● generate event when disk is added, on boot ● 'ceph-disk-activate /dev/sdb' ● mount the disk in the appropriate location (/var/lib/ceph/osd/NNN) ● possibly adjust cluster metadata about disk location (host, rack) ● upstart, sysvinit, … ● start the daemon ● daemon “joins” the cluster, brings itself online ● no manual per-node configuration
distributed system health ● ceph-mon collects, aggregates basic system state ● daemon up/down, current IPs ● disk utilization ● simple hooks query system health ● 'ceph health [detail]' ● trivial plugins for nagios, etc. ● daemons instrumentation ● query state of running daemon ● use external tools to aggregate – collectd, graphite – statsd
many paths ● do it yourself ● use ceph interfaces directly ● Chef ● ceph cookbooks, DreamHost cookbooks ● Crowbar ● Juju ● pre-packaged solutions ● Piston AirFrame, Dell OpenStack-Powered Cloud Solution, Suse Cloud, etc.
Project status and roadmap
APP APP APP APP HOST/VM HOST/VM CLIENT CLIENT RADOSGW RADOSGW RBD RBD CEPH FS CEPH FS LIBRADOS LIBRADOS A bucket-based A bucket-based A reliable and A reliable and A POSIX-compliant A POSIX-compliant A library allowing A library allowing REST gateway, REST gateway, fully-distributed fully-distributed distributed file distributed file apps to directly apps to directly compatible with S3 compatible with S3 block device, with aa block device, with system, with aa system, with access RADOS, access RADOS, and Swift and Swift Linux kernel client Linux kernel client Linux kernel client Linux kernel client with support for with support for and aaQEMU/KVM and QEMU/KVM and support for and support for C, C++, Java, C, C++, Java, driver driver FUSE FUSE Python, Ruby, Python, Ruby, and PHP and PHP AWESOME AWESOME NEARLY AWESOME AWESOME RADOS RADOS AWESOME A reliable, autonomous, distributed object store comprised of self-healing, self-managing, A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes intelligent storage nodes
current status ● argonaut stable release v0.48 ● rados, RBD, radosgw ● bobtail stable release v0.56 ● RBD cloning ● improved performance, scaling, failure behavior ● radosgw API, performance improvements ● release in 1-2 weeks
cuttlefish roadmap ● file system ● pivot in engineering focus ● CIFS (Samba), NFS (Ganesha), Hadoop ● RBD ● Xen integration, iSCSI ● radosgw ● multi-side federation, disaster recovery ● RADOS ● geo-replication, disaster recovery ● ongoing performance improvements
why we do this ● limited options for scalable open source storage ● proprietary solutions ● expensive ● don't scale (well or out) ● marry hardware and software ● users hungry for alternatives ● scalability ● cost ● features ● open, interoperable
licensing <yawn> ● promote adoption ● enable community development ● prevent ceph from becoming proprietary ● allow organic commercialization
LGPLv2 ● “copyleft” ● free distribution ● allow derivative works ● changes you distribute/sell must be shared ● ok to link to proprietary code ● allow proprietary products to incude and build on ceph ● does not allow proprietary derivatives of ceph
fragmented copyright ● we do not require copyright assignment from contributors ● no single person or entity owns all of ceph ● no single entity can make ceph proprietary ● strong community ● many players make ceph a safe technology bet ● project can outlive any single business
why it's important ● ceph is an ingredient ● we need to play nice in a larger ecosystem ● community will be key to success ● truly open source solutions are disruptive ● frictionless integration with projects, platforms, tools ● freedom to innovate on protocols ● leverage community testing, development resources ● open collaboration is efficient way to build technology
who we are ● Ceph created at UC Santa Cruz (2004-2007) ● developed by DreamHost (2008-2011) ● supported by Inktank (2012) ● Los Angeles, Sunnyvale, San Francisco, remote ● growing user and developer community ● Linux distros, users, cloud stacks, SIs, OEMs http://ceph.com/
thanks sage weil sage@inktank.com http://github.com/ceph @liewegas http://ceph.com/
why we like btrfs ● pervasive checksumming ● snapshots, copy-on-write ● efficient metadata (xattrs) ● inline data for small files ● transparent compression ● integrated volume management ● software RAID, mirroring, error recovery ● SSD-aware ● online fsck ● active development community

More Related Content

PDF
Ceph Day Nov 2012 - Sage Weil
byCeph Community
 
PDF
CloudOpen - 08/29/2012
byCeph Community
 
PDF
New features for Ceph with Cinder and Beyond
byCeph Community
 
PPTX
Setting up Storage Features in Windows Server 2012
byLai Yoong Seng
 
PDF
OpenNebulaConf2018 - LINSTOR - Philipp Reisner - LINBIT
byOpenNebula Project
 
PDF
(Free and Net) BSD Xen Roadmap
byThe Linux Foundation
 
PDF
Tandberg Data - Data Protection Solutions Guide
byCustemotion Unternehmensberatung UG (haftungsbeschränkt)
 
PPT
Almoço Natal 2009
byin.oeiras
 
Ceph Day Nov 2012 - Sage Weil
byCeph Community
 
CloudOpen - 08/29/2012
byCeph Community
 
New features for Ceph with Cinder and Beyond
byCeph Community
 
Setting up Storage Features in Windows Server 2012
byLai Yoong Seng
 
OpenNebulaConf2018 - LINSTOR - Philipp Reisner - LINBIT
byOpenNebula Project
 
(Free and Net) BSD Xen Roadmap
byThe Linux Foundation
 
Tandberg Data - Data Protection Solutions Guide
byCustemotion Unternehmensberatung UG (haftungsbeschränkt)
 
Almoço Natal 2009
byin.oeiras
 

Similar to Ceph LISA'12 Presentation

PDF
Storage Developer Conference - 09/19/2012
byCeph Community
 
PDF
Storing VMs with Cinder and Ceph RBD.pdf
byOpenStack Foundation
 
PDF
XenSummit - 08/28/2012
byCeph Community
 
ODP
Block Storage For VMs With Ceph
byThe Linux Foundation
 
PDF
New Features for Ceph with Cinder and Beyond
byOpenStack Foundation
 
PPTX
Inktank:ceph overview
byCeph Community
 
PDF
End of RAID as we know it with Ceph Replication
byCeph Community
 
PDF
Scale 10x 01:22:12
byCeph Community
 
PPTX
Ceph Intro and Architectural Overview by Ross Turk
bybuildacloud
 
PDF
Strata - 03/31/2012
byCeph Community
 
PDF
Ceph Overview for Distributed Computing Denver Meetup
byktdreyer
 
ODP
Ceph: A decade in the making and still going strong
byPatrick McGarry
 
PDF
Ceph Day New York: Ceph: one decade in
byCeph Community
 
PDF
Storage tiering and erasure coding in Ceph (SCaLE13x)
bySage Weil
 
PDF
Openstack with ceph
byIan Colle
 
PDF
Cephfsglusterfs.talk
byUdo Seidel
 
PDF
INFINISTORE(tm) - Scalable Open Source Storage Arhcitecture
byThomas Uhl
 
PDF
Wicked Easy Ceph Block Storage & OpenStack Deployment with Crowbar
byCeph Community
 
PDF
DEVIEW 2013
byPatrick McGarry
 
PDF
Ceph Day New York 2014: Future of CephFS
byCeph Community
 
Storage Developer Conference - 09/19/2012
byCeph Community
 
Storing VMs with Cinder and Ceph RBD.pdf
byOpenStack Foundation
 
XenSummit - 08/28/2012
byCeph Community
 
Block Storage For VMs With Ceph
byThe Linux Foundation
 
New Features for Ceph with Cinder and Beyond
byOpenStack Foundation
 
Inktank:ceph overview
byCeph Community
 
End of RAID as we know it with Ceph Replication
byCeph Community
 
Scale 10x 01:22:12
byCeph Community
 
Ceph Intro and Architectural Overview by Ross Turk
bybuildacloud
 
Strata - 03/31/2012
byCeph Community
 
Ceph Overview for Distributed Computing Denver Meetup
byktdreyer
 
Ceph: A decade in the making and still going strong
byPatrick McGarry
 
Ceph Day New York: Ceph: one decade in
byCeph Community
 
Storage tiering and erasure coding in Ceph (SCaLE13x)
bySage Weil
 
Openstack with ceph
byIan Colle
 
Cephfsglusterfs.talk
byUdo Seidel
 
INFINISTORE(tm) - Scalable Open Source Storage Arhcitecture
byThomas Uhl
 
Wicked Easy Ceph Block Storage & OpenStack Deployment with Crowbar
byCeph Community
 
DEVIEW 2013
byPatrick McGarry
 
Ceph Day New York 2014: Future of CephFS
byCeph Community
 

Recently uploaded

PDF
Why Many Smart Device Platforms Fail to Scale?
byPromeraki Developments
 
PDF
Post-Hackathon-Learnings-Maximizing-Impact-Beyond-the-Event.pdf
byishantyadav1111
 
PDF
GenerationAI Paris 2025 | How Agentic AI is Reinventing Organizations
byapidays
 
PDF
Supercharge Your Copilot-Driven Collaboration with Microsoft 365 Agents SDK
byAntti Koskela
 
PDF
BEP-20 Token on BNB Chain: From Concept to Deployment.pdf
byimoliviabennett
 
PDF
Dev Dives: Build and deploy agentic automations - the unified way
byUiPathCommunity
 
PPTX
Computer architecture, Computer architecture ,Computer architecture
bywaheedqazi123
 
PPTX
Working Session — Build a Document Understanding Automation Using an OOTB ML ...
byDianaGray10
 
PDF
Chapter 4 Network Security in computer security
byGetnet Tigabie Askale -(GM)
 
PDF
UiPath Automation Developer Associate Training Series 2026 - Session 2
byDianaGray10
 
PDF
NTG -Company Profile_Software_Vendor_Company_in_MENA
byMustafa Kuğu
 
PDF
The Enterprise Web3 Landscape - a view from Kaleido based on the past 10 year...
byPeter Broadhurst
 
PPTX
Automation Without Apprentices: How AI Challenges the Open Source Way
byIcinga
 
PDF
Getting Started with Apache Spark: Big Data Made Simple [Free Meetup]
byHaim Michael
 
PDF
The Emergence of Empathic XR: AWE Asia 2026 keynote
byMark Billinghurst
 
PPTX
TechSprint Inauguration at SJB Institute of Technology held on 18 December 2025
bysuhasspgdg
 
PDF
UiPath Automation Developer Associate Training Series 2026 - Session 1
byDianaGray10
 
PDF
Track Phone Location by Number (2026)_ What’s Actually Possible for Free (and...
byEmily Woods
 
PDF
CI CD Observability, Metrics and DORA - Shifting Left and Cleaning Up! - Febr...
byPeter Souter
 
PDF
Mount File Systems using UUID and Label - RHCSA (RH134).pdf
byLinuxCert Guru
 
Why Many Smart Device Platforms Fail to Scale?
byPromeraki Developments
 
Post-Hackathon-Learnings-Maximizing-Impact-Beyond-the-Event.pdf
byishantyadav1111
 
GenerationAI Paris 2025 | How Agentic AI is Reinventing Organizations
byapidays
 
Supercharge Your Copilot-Driven Collaboration with Microsoft 365 Agents SDK
byAntti Koskela
 
BEP-20 Token on BNB Chain: From Concept to Deployment.pdf
byimoliviabennett
 
Dev Dives: Build and deploy agentic automations - the unified way
byUiPathCommunity
 
Computer architecture, Computer architecture ,Computer architecture
bywaheedqazi123
 
Working Session — Build a Document Understanding Automation Using an OOTB ML ...
byDianaGray10
 
Chapter 4 Network Security in computer security
byGetnet Tigabie Askale -(GM)
 
UiPath Automation Developer Associate Training Series 2026 - Session 2
byDianaGray10
 
NTG -Company Profile_Software_Vendor_Company_in_MENA
byMustafa Kuğu
 
The Enterprise Web3 Landscape - a view from Kaleido based on the past 10 year...
byPeter Broadhurst
 
Automation Without Apprentices: How AI Challenges the Open Source Way
byIcinga
 
Getting Started with Apache Spark: Big Data Made Simple [Free Meetup]
byHaim Michael
 
The Emergence of Empathic XR: AWE Asia 2026 keynote
byMark Billinghurst
 
TechSprint Inauguration at SJB Institute of Technology held on 18 December 2025
bysuhasspgdg
 
UiPath Automation Developer Associate Training Series 2026 - Session 1
byDianaGray10
 
Track Phone Location by Number (2026)_ What’s Actually Possible for Free (and...
byEmily Woods
 
CI CD Observability, Metrics and DORA - Shifting Left and Cleaning Up! - Febr...
byPeter Souter
 
Mount File Systems using UUID and Label - RHCSA (RH134).pdf
byLinuxCert Guru
 

Ceph LISA'12 Presentation

  • 1.
    managing a distributed storagesystem at scale sage weil – inktank LISA – 12.12.12
  • 2.
    outline ● the scale-out world ● what is it, what it's for, how it works ● how you can use it ● librados ● radosgw ● RBD, the ceph block device ● distributed file system ● how do you deploy and manage it ● roadmap ● why we do this, who we are
  • 3.
    why should youcare about another storage system?
  • 4.
    the old reality ● direct-attached storage ● raw disks ● RAID controller ● network-attached storage ● a few large arrays ● NFS, CIFS, maybe iSCSI ● SAN ● expensive enterprise array ● scalable capacity and performance, to a point ● poor management scalability
  • 5.
    new user demands ● “cloud” ● many disk volumes, with automated provisioning ● larger data sets – RESTful object storage (e.g., S3, Swift) – NoSQL distributed key/value stores (Cassandra, Riak) ● “big data” ● distributed processing ● “unstructured data” ● ...and the more traditional workloads
  • 6.
    requirements ● diverse use-cases ● object storage ● block devices (for VMs) with snapshots, cloning ● shared file system with POSIX, coherent caches ● structured data... files, block devices, or objects? ● scale ● terabytes, petabytes, exabytes ● heterogeneous hardware ● reliability and fault tolerance
  • 7.
    cost ● near-linear function of size or performance ● incremental expansion ● no fork-lift upgrades ● no vendor lock-in ● choice of hardware ● choice of software ● open
  • 8.
    time ● ease of administration ● no manual data migration, load balancing ● painless scaling ● expansion and contraction ● seamless migration ● devops friendly
  • 9.
    being a gooddevops citizen ● seamless integration with infrastructure tools ● Chef, Juju, Puppet, Crowbar, or home-grown ● collectd, statsd, graphite, nagios ● OpenStack, CloudStack ● simple processes for ● expanding or contracting the storage cluster ● responding to failure scenarios: initial healing, or mop-up ● evolution of “Unix philosophy” ● everything is just daemon ● CLI, REST APIs, JSON ● solve one problem, and solve it well
  • 10.
  • 11.
    unified storage system ● objects ● native API ● RESTful ● block ● thin provisioning, snapshots, cloning ● file ● strong consistency, snapshots
  • 12.
    APP APP APP APP HOST/VM HOST/VM CLIENT CLIENT RADOSGW RADOSGW RBD RBD CEPH FS CEPH FS LIBRADOS LIBRADOS A bucket-based A bucket-based A reliable and A reliable and A POSIX-compliant A POSIX-compliant A library allowing A library allowing REST gateway, REST gateway, fully-distributed fully-distributed distributed file distributed file apps to directly apps to directly compatible with S3 compatible with S3 block device, with aa block device, with system, with aa system, with access RADOS, access RADOS, and Swift and Swift Linux kernel client Linux kernel client Linux kernel client Linux kernel client with support for with support for and aaQEMU/KVM and QEMU/KVM and support for and support for C, C++, Java, C, C++, Java, driver driver FUSE FUSE Python, Ruby, Python, Ruby, and PHP and PHP RADOS ––the Ceph object store RADOS the Ceph object store A reliable, autonomous, distributed object store comprised of self-healing, self-managing, A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes intelligent storage nodes
  • 13.
    distributed storage system ● data center scale ● 10s to 10,000s of machines ● terabytes to exabytes ● fault tolerant ● no single point of failure ● commodity hardware ● self-managing, self-healing
  • 14.
    ceph object model ● pools ● 1s to 100s ● independent object namespaces or collections ● replication level, placement policy ● objects ● bazillions ● blob of data (bytes to gigabytes) ● attributes (e.g., “version=12”; bytes to kilobytes) ● key/value bundle (bytes to gigabytes)
  • 15.
    why start withobjects? ● more useful than (disk) blocks ● names in a simple flat namespace ● variable size ● simple API with rich semantics ● more scalable than files ● no hard-to-distribute hierarchy ● update semantics do not span objects ● workload is trivially parallel
  • 16.
    How do youdesign a storage system that scales?
  • 17.
    DISK DISK DISK DISK DISK DISK HUMAN HUMAN COMPUTER COMPUTER DISK DISK DISK DISK DISK DISK DISK DISK
  • 18.
    DISK DISK DISK DISK HUMAN HUMAN DISK DISK HUMAN HUMAN COMPUTER COMPUTER DISK DISK DISK DISK HUMAN HUMAN DISK DISK DISK DISK
  • 19.
    HUMAN HUMAN HUMAN HUMAN HUMAN HUMAN HUMAN HUMAN DISK DISK HUMAN HUMAN HUMAN HUMAN DISK DISK HUMAN HUMAN HUMAN HUMAN DISK DISK DISK DISK HUMAN HUMAN DISK DISK HUMAN HUMAN HUMAN HUMAN DISK DISK (COMPUTER)) (COMPUTER HUMAN HUMAN DISK DISK HUMAN HUMAN HUMAN HUMAN DISK DISK HUMAN HUMAN HUMAN HUMAN DISK DISK HUMAN HUMAN DISK DISK HUMAN HUMAN DISK DISK HUMAN HUMAN HUMAN HUMAN DISK DISK HUMAN HUMAN HUMAN HUMAN (actually more like this…)
  • 20.
    COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK HUMAN HUMAN COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK HUMAN HUMAN COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK HUMAN HUMAN COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK
  • 21.
    OSD OSD OSD OSD OSD FS FS FS FS FS btrfs xfs ext4 DISK DISK DISK DISK DISK M M M
  • 22.
    Monitors maintain cluster membership and state M ● ● provide consensus for distributed decision-making ● small, odd number ● these do not served stored objects to clients Object Storage Daemons (OSDs) ● 10s to 10000s in a cluster ● one per disk, SSD, or RAID group ● hardware agnostic ● serve stored objects to clients ● intelligently peer to perform replication and recovery tasks
  • 23.
    HUMAN M M M
  • 24.
    data distribution ● all objects are replicated N times ● objects are automatically placed, balanced, migrated in a dynamic cluster ● must consider physical infrastructure ● ceph-osds on hosts in racks in rows in data centers ● three approaches ● pick a spot; remember where you put it ● pick a spot; write down where you put it ● calculate where to put it, where to find it
  • 25.
    CRUSH ● pseudo-random placement algorithm ● fast calculation, no lookup ● repeatable, deterministic ● statistically uniform distribution ● stable mapping ● limited data migration on change ● rule-based configuration ● infrastructure topology aware ● adjustable replication ● allows weighting
  • 26.
    hash(object name) %num pg 10 10 10 10 01 01 01 01 10 10 10 10 01 01 11 11 01 01 10 10 CRUSH(pg, cluster state, policy)
  • 27.
    10 10 10 10 01 01 01 01 10 10 10 10 01 01 11 11 01 01 10 10
  • 28.
    RADOS ● monitors publish osd map that describes cluster state ● ceph-osd node status (up/down, weight, IP) M ● CRUSH function specifying desired data distribution ● object storage daemons (OSDs) ● safely replicate and store object ● migrate data as the cluster changes over time ● coordinate based on shared view of reality ● decentralized, distributed approach allows ● massive scales (10,000s of servers or more) ● the illusion of a single copy with consistent behavior
  • 29.
  • 32.
  • 33.
    APP APP APP APP HOST/VM HOST/VM CLIENT CLIENT RADOSGW RADOSGW RBD RBD CEPH FS CEPH FS LIBRADOS A bucket-based A bucket-based A reliable and A reliable and A POSIX-compliant A POSIX-compliant A library allowing REST gateway, REST gateway, fully-distributed fully-distributed distributed file distributed file apps to directly compatible with S3 compatible with S3 block device, with aa block device, with system, with aa system, with access RADOS, and Swift and Swift Linux kernel client Linux kernel client Linux kernel client Linux kernel client with support for and aaQEMU/KVM and QEMU/KVM and support for and support for C, C++, Java, driver driver FUSE FUSE Python, Ruby, and PHP RADOS RADOS A reliable, autonomous, distributed object store comprised of self-healing, self-managing, A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes intelligent storage nodes
  • 34.
    APP APP LIBRADOS LIBRADOS native M M M M M M
  • 35.
    librados L ● direct access to RADOS from applications ● C, C++, Python, PHP, Java, Erlang ● direct access to storage nodes ● no HTTP overhead
  • 36.
    rich librados API ● atomic single-object transactions ● update data, attr together ● atomic compare-and-swap ● efficient key/value storage inside an object ● object-granularity snapshot infrastructure ● embed code in ceph-osd daemon via plugin API ● inter-client communication via object
  • 37.
    APP APP APP APP HOST/VM HOST/VM CLIENT CLIENT RADOSGW RBD RBD CEPH FS CEPH FS LIBRADOS LIBRADOS A bucket-based A reliable and A reliable and A POSIX-compliant A POSIX-compliant A library allowing A library allowing REST gateway, fully-distributed fully-distributed distributed file distributed file apps to directly apps to directly compatible with S3 block device, with aa block device, with system, with aa system, with access RADOS, access RADOS, and Swift Linux kernel client Linux kernel client Linux kernel client Linux kernel client with support for with support for and aaQEMU/KVM and QEMU/KVM and support for and support for C, C++, Java, C, C++, Java, driver driver FUSE FUSE Python, Ruby, Python, Ruby, and PHP and PHP RADOS RADOS A reliable, autonomous, distributed object store comprised of self-healing, self-managing, A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes intelligent storage nodes
  • 38.
    APP APP APP APP REST RADOSGW RADOSGW RADOSGW RADOSGW LIBRADOS LIBRADOS LIBRADOS LIBRADOS native M M M M M M
  • 39.
    RADOS Gateway ● REST-based object storage proxy ● uses RADOS to store objects ● API supports buckets, accounting ● usage accounting for billing purposes ● compatible with S3, Swift APIs
  • 40.
    APP APP APP APP HOST/VM HOST/VM CLIENT CLIENT RADOSGW RADOSGW RBD CEPH FS CEPH FS LIBRADOS LIBRADOS A bucket-based A bucket-based A reliable and A POSIX-compliant A POSIX-compliant A library allowing A library allowing REST gateway, REST gateway, fully-distributed distributed file distributed file apps to directly apps to directly compatible with S3 compatible with S3 block device, with a system, with aa system, with access RADOS, access RADOS, and Swift and Swift Linux kernel client Linux kernel client Linux kernel client with support for with support for and a QEMU/KVM and support for and support for C, C++, Java, C, C++, Java, driver FUSE FUSE Python, Ruby, Python, Ruby, and PHP and PHP RADOS RADOS A reliable, autonomous, distributed object store comprised of self-healing, self-managing, A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes intelligent storage nodes
  • 41.
    COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER COMPUTER COMPUTER DISK DISK DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK
  • 42.
    COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK VM VM COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK VM VM COMPUTER DISK COMPUTER DISK COMPUTER COMPUTER DISK DISK VM VM COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK COMPUTER COMPUTER DISK DISK
  • 43.
    VM VM VIRTUALIZATION CONTAINER (KVM) VIRTUALIZATION CONTAINER (KVM) LIBRBD LIBRBD LIBRADOS LIBRADOS M M M M M M
  • 44.
    CONTAINER CONTAINER VM VM CONTAINER CONTAINER LIBRBD LIBRBD LIBRBD LIBRBD LIBRADOS LIBRADOS LIBRADOS LIBRADOS M M M M M M
  • 45.
    HOST HOST KRBD (KERNEL MODULE) KRBD (KERNEL MODULE) LIBRADOS LIBRADOS M M M M M M
  • 46.
    RADOS Block Device ● storage of disk images in RADOS ● decouple VM from host ● images striped across entire cluster (pool) ● snapshots ● copy-on-write clones ● support in ● Qemu/KVM ● mainline Linux kernel (2.6.39+) ● OpenStack, CloudStack
  • 47.
    APP APP APP APP HOST/VM HOST/VM CLIENT CLIENT RADOSGW RADOSGW RBD RBD CEPH FS LIBRADOS LIBRADOS A bucket-based A bucket-based A reliable and A reliable and A POSIX-compliant A library allowing A library allowing REST gateway, REST gateway, fully-distributed fully-distributed distributed file apps to directly apps to directly compatible with S3 compatible with S3 block device, with aa block device, with system, with a access RADOS, access RADOS, and Swift and Swift Linux kernel client Linux kernel client Linux kernel client with support for with support for and aaQEMU/KVM and QEMU/KVM and support for C, C++, Java, C, C++, Java, driver driver FUSE Python, Ruby, Python, Ruby, and PHP and PHP RADOS RADOS A reliable, autonomous, distributed object store comprised of self-healing, self-managing, A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes intelligent storage nodes
  • 48.
    CLIENT CLIENT metadata 01 01 data 10 10 M M M M M M
  • 49.
    M M M M M M
  • 50.
    Metadata Server (MDS) ● manages metadata for POSIX shared file system ● directory hierarchy ● file metadata (size, owner, timestamps) ● stores metadata in RADOS ● does not serve file data to clients ● only required for the shared file system
  • 51.
  • 56.
  • 57.
    recursive accounting ● ceph-mds tracks recursive directory stats ● file sizes ● file and directory counts ● modification time ● virtual xattrs present full stats ● efficient $ ls ­alSh | head total 0 drwxr­xr­x 1 root            root      9.7T 2011­02­04 15:51 . drwxr­xr­x 1 root            root      9.7T 2010­12­16 15:06 .. drwxr­xr­x 1 pomceph         pg4194980 9.6T 2011­02­24 08:25 pomceph drwxr­xr­x 1 mcg_test1       pg2419992  23G 2011­02­02 08:57 mcg_test1 drwx­­x­­­ 1 luko            adm        19G 2011­01­21 12:17 luko drwx­­x­­­ 1 eest            adm        14G 2011­02­04 16:29 eest drwxr­xr­x 1 mcg_test2       pg2419992 3.0G 2011­02­02 09:34 mcg_test2 drwx­­x­­­ 1 fuzyceph        adm       1.5G 2011­01­18 10:46 fuzyceph drwxr­xr­x 1 dallasceph      pg275     596M 2011­01­14 10:06 dallasceph
  • 58.
    snapshots ● volume or subvolume snapshots unusable at petabyte scale ● snapshot arbitrary subdirectories ● simple interface ● hidden '.snap' directory ● no special tools $ mkdir foo/.snap/one # create snapshot $ ls foo/.snap one $ ls foo/bar/.snap _one_1099511627776 # parent's snap name is mangled $ rm foo/myfile $ ls -F foo bar/ $ ls -F foo/.snap/one myfile bar/ $ rmdir foo/.snap/one # remove snapshot
  • 59.
    multiple protocols, implementations ● Linux kernel client ● mount -t ceph 1.2.3.4:/ /mnt NFS SMB/CIFS ● export (NFS), Samba (CIFS) ● ceph-fuse Ganesha Samba libcephfs libcephfs ● libcephfs.so ● your app Hadoop your app libcephfs libcephfs ● Samba (CIFS) ● Ganesha (NFS) ceph-fuse ceph fuse ● Hadoop (map/reduce) kernel
  • 60.
    How do youdeploy and manage all this stuff?
  • 61.
    installation ● work closely with distros ● packages in Debian, Fedora, Ubuntu, OpenSUSE ● build releases for all distros, recent releases ● e.g., point to our deb src and apt-get install ● allow mixed-version clusters ● upgrade each component individually, each host or rack one-by-one ● protocol feature bits, safe data type encoding ● facilitate testing, bleeding edge ● automatic build system generates packages for all branches in git ● each to test new code, push hot-fixes
  • 62.
    configuration ● minimize local configuration ● logging, local data paths, tuning options ● cluster state is managed centrally by monitors ● specify option via ● config file (global or local) ● command line ● adjusted for running daemon ● flexible configuration management options ● global synced/copied config file ● generated by management tools – Chef, Juju, Crowbar ● be flexible
  • 63.
    provisioning ● embrace dynamic nature of the cluster ● disks, hosts, rack may come online at any time ● anything may fail at any time ● simple scriptable sequences 'ceph osd create <uuid>' → allocate osd id 'ceph-osd --mkfs -i <id>' → initialize local data dir 'ceph osd crush …' → add to crush map ● identify minimal amount of central coordination ● monitor cluster membership/quorum ● provide hooks for external tools to do the rest
  • 64.
    old school HA ● deploy a pair of servers ● heartbeat ● move a floating IP between then ● clients contact same “server”, which floats ● cumbersome to configure, deploy ● the “client/server” model doesn't scale out
  • 65.
    make client, protocolcluster-aware ● clients learn topology of the distributed cluster ● all OSDs, current ips/ports, data distribution map ● clients talk to the server they want ● servers scale out, client traffic distributes too ● servers dynamically register with the cluster ● when a daemon starts, it updates its address in the map ● other daemons and clients learn map updates via gossip ● servers manage heartbeat, replication ● no fragile configuration ● no fail-over pairs
  • 66.
    ceph disk management ● label disks ● GPT partition type (fixed uuid) ● udev ● generate event when disk is added, on boot ● 'ceph-disk-activate /dev/sdb' ● mount the disk in the appropriate location (/var/lib/ceph/osd/NNN) ● possibly adjust cluster metadata about disk location (host, rack) ● upstart, sysvinit, … ● start the daemon ● daemon “joins” the cluster, brings itself online ● no manual per-node configuration
  • 67.
    distributed system health ● ceph-mon collects, aggregates basic system state ● daemon up/down, current IPs ● disk utilization ● simple hooks query system health ● 'ceph health [detail]' ● trivial plugins for nagios, etc. ● daemons instrumentation ● query state of running daemon ● use external tools to aggregate – collectd, graphite – statsd
  • 68.
    many paths ● do it yourself ● use ceph interfaces directly ● Chef ● ceph cookbooks, DreamHost cookbooks ● Crowbar ● Juju ● pre-packaged solutions ● Piston AirFrame, Dell OpenStack-Powered Cloud Solution, Suse Cloud, etc.
  • 69.
  • 70.
    APP APP APP APP HOST/VM HOST/VM CLIENT CLIENT RADOSGW RADOSGW RBD RBD CEPH FS CEPH FS LIBRADOS LIBRADOS A bucket-based A bucket-based A reliable and A reliable and A POSIX-compliant A POSIX-compliant A library allowing A library allowing REST gateway, REST gateway, fully-distributed fully-distributed distributed file distributed file apps to directly apps to directly compatible with S3 compatible with S3 block device, with aa block device, with system, with aa system, with access RADOS, access RADOS, and Swift and Swift Linux kernel client Linux kernel client Linux kernel client Linux kernel client with support for with support for and aaQEMU/KVM and QEMU/KVM and support for and support for C, C++, Java, C, C++, Java, driver driver FUSE FUSE Python, Ruby, Python, Ruby, and PHP and PHP AWESOME AWESOME NEARLY AWESOME AWESOME RADOS RADOS AWESOME A reliable, autonomous, distributed object store comprised of self-healing, self-managing, A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes intelligent storage nodes
  • 71.
    current status ● argonaut stable release v0.48 ● rados, RBD, radosgw ● bobtail stable release v0.56 ● RBD cloning ● improved performance, scaling, failure behavior ● radosgw API, performance improvements ● release in 1-2 weeks
  • 72.
    cuttlefish roadmap ● file system ● pivot in engineering focus ● CIFS (Samba), NFS (Ganesha), Hadoop ● RBD ● Xen integration, iSCSI ● radosgw ● multi-side federation, disaster recovery ● RADOS ● geo-replication, disaster recovery ● ongoing performance improvements
  • 73.
    why we dothis ● limited options for scalable open source storage ● proprietary solutions ● expensive ● don't scale (well or out) ● marry hardware and software ● users hungry for alternatives ● scalability ● cost ● features ● open, interoperable
  • 74.
    licensing <yawn> ● promote adoption ● enable community development ● prevent ceph from becoming proprietary ● allow organic commercialization
  • 75.
    LGPLv2 ● “copyleft” ● free distribution ● allow derivative works ● changes you distribute/sell must be shared ● ok to link to proprietary code ● allow proprietary products to incude and build on ceph ● does not allow proprietary derivatives of ceph
  • 76.
    fragmented copyright ● we do not require copyright assignment from contributors ● no single person or entity owns all of ceph ● no single entity can make ceph proprietary ● strong community ● many players make ceph a safe technology bet ● project can outlive any single business
  • 77.
    why it's important ● ceph is an ingredient ● we need to play nice in a larger ecosystem ● community will be key to success ● truly open source solutions are disruptive ● frictionless integration with projects, platforms, tools ● freedom to innovate on protocols ● leverage community testing, development resources ● open collaboration is efficient way to build technology
  • 78.
    who we are ● Ceph created at UC Santa Cruz (2004-2007) ● developed by DreamHost (2008-2011) ● supported by Inktank (2012) ● Los Angeles, Sunnyvale, San Francisco, remote ● growing user and developer community ● Linux distros, users, cloud stacks, SIs, OEMs http://ceph.com/
  • 79.
    thanks sage weil sage@inktank.com http://github.com/ceph @liewegas http://ceph.com/
  • 81.
    why we likebtrfs ● pervasive checksumming ● snapshots, copy-on-write ● efficient metadata (xattrs) ● inline data for small files ● transparent compression ● integrated volume management ● software RAID, mirroring, error recovery ● SSD-aware ● online fsck ● active development community