Downloaded 27 times
Uploaded byUdo Seidel
adp.ceph.openstack.talk
The document discusses Ceph, a distributed storage system, highlighting its architecture, the role of object-based storage devices, and integration with OpenStack. Ceph provides a scalable storage solution without single points of failure, utilizing a unique algorithm for data distribution and replication. The presentation emphasizes the benefits of using Ceph with OpenStack for various cloud storage needs.
More Related Content
adp.ceph.openstack.talk
- 1. Petabyte Scale OutRocks CEPH as replacement for Openstack's Swift & Co. Udo Seidel
- 2. 22JAN2014 Agenda â—Ź Background â—Ź Architectureof CEPH â—Ź CEPH Storage â—Ź Openstack â—Ź Dream team: CEPH and Openstack â—Ź Summary
- 3.
- 4. 22JAN2014 CEPH – what? ●So-called parallel distributed cluster file system ● Started as part of PhD studies at UCSC ● Public announcement in 2006 at 7th OSDI ● File system shipped with Linux kernel since 2.6.34 ● Name derived from pet octopus - Cephalopods
- 5. 22JAN2014 Shared file systems– short intro ● Multiple server access the same data ● Different approaches ● Network based, e.g. NFS, CIFS ● Clustered – Shared disk, e.g. CXFS, CFS, GFS(2), OCFS2 – Distributed parallel, e.g. Lustre .. GlusterFS and CEPHFS
- 6.
- 7. 22JAN2014 CEPH and storage ●Distributed file system => distributed storage ● Does not use traditional disks or RAID arrays ● Does use so-called OSD’s – Object based Storage Devices – Intelligent disks
- 8. 22JAN2014 Object Based StorageI â—Ź Objects of quite general nature â—Ź Files â—Ź Partitions â—Ź ID for each storage object â—Ź Separation of meta data operation and storing file data â—Ź HA not covered at all â—Ź Object based Storage Devices
- 9. 22JAN2014 Object Based StorageII ● OSD software implementation ● Usual an additional layer between between computer and storage ● Presents object-based file system to the computer ● Use a “normal” file system to store data on the storage ● Delivered as part of CEPH ● File systems: LUSTRE, EXOFS
- 10. 22JAN2014 CEPH – thefull architecture I ● 4 components ● Object based Storage Devices – Any computer – Form a cluster (redundancy and load balancing) ● Meta Data Servers – Any computer – Form a cluster (redundancy and load balancing) ● Cluster Monitors – Any computer ● Clients ;-)
- 11. 22JAN2014 CEPH – thefull architecture II
- 12.
- 13. 22JAN2014 CEPH and OSD ●User land implementation ● Any computer can act as OSD ● Uses BTRFS as native file system ● Since 2009 ● Before self-developed EBOFS ● Provides functions of OSD-2 standard – Copy-on-write – snapshots ● No redundancy on disk or even computer level
- 14. 22JAN2014 CEPH and OSD– file systems ● BTRFS preferred ● Non-default configuration for mkfs ● XFS and EXT4 possible ● XATTR (size) is key -> EXT4 less recommended ● XFS recommended for Production
- 15. 22JAN2014 OSD failure approach â—ŹAny OSD expected to fail â—Ź New OSD dynamically added/integrated â—Ź Data distributed and replicated â—Ź Redistribution of data after change in OSD landscape
- 16. 22JAN2014 Data distribution ● Filestripped ● File pieces mapped to object IDs ● Assignment of so-called placement group to object ID ● Via hash function ● Placement group (PG): logical container of storage objects ● Calculation of list of OSD’s out of PG ● CRUSH algorithm
- 17. 22JAN2014 CRUSH I â—Ź ControlledReplication Under Scalable Hashing â—Ź Considers several information â—Ź Cluster setup/design â—Ź Actual cluster landscape/map â—Ź Placement rules â—Ź Pseudo random -> quasi statistical distribution â—Ź Cannot cope with hot spots â—Ź Clients, MDS and OSD can calculate object location
- 18.
- 19. 22JAN2014 Data replication ● N-wayreplication ● N OSD’s per placement group ● OSD’s in different failure domains ● First non-failed OSD in PG -> primary ● Read and write to primary only ● Writes forwarded by primary to replica OSD’s ● Final write commit after all writes on replica OSD ● Replication traffic within OSD network
- 20. 22JAN2014 CEPH cluster monitors ●Status information of CEPH components critical ● First contact point for new clients ● Monitor track changes of cluster landscape ● Update cluster map ● Propagate information to OSD’s
- 21. 22JAN2014 CEPH cluster mapI â—Ź Objects: computers and containers â—Ź Container: bucket for computers or containers â—Ź Each object has ID and weight â—Ź Maps physical conditions â—Ź rack location â—Ź fire cells
- 22. 22JAN2014 CEPH cluster mapII ● Reflects data rules ● Number of copies ● Placement of copies ● Updated version sent to OSD’s ● OSD’s distribute cluster map within OSD cluster ● OSD re-calculates via CRUSH PG membership – data responsibilities – Order: primary or replica ● New I/O accepted after information synch
- 23. 22JAN2014 CEPH - RADOS â—ŹReliable Autonomic Distributed Object Storage â—Ź Direct access to OSD cluster via librados â—Ź Support: C, C++, Java, Python, Ruby, PHP â—Ź Drop/skip of POSIX layer (CEPHFS) on top â—Ź Visible to all CEPH cluster members => shared storage
- 24. 22JAN2014 CEPH Block Device â—ŹAka RADOS block device (RBD) â—Ź Upstream since kernel 2.6.37 â—Ź RADOS storage exposed as block device â—Ź /dev/rdb â—Ź qemu/KVM storage driver via librados/librdb â—Ź Alternative to â—Ź shared SAN/iSCSI for HA environments â—Ź Storage HA solutions for qemu/KVM
- 25.
- 26. 22JAN2014 CEPH Object Gateway â—ŹAka RADOS Gateway (RGW) â—Ź RESTful API â—Ź Amazon S3 -> s3 tools work â—Ź SWIFT API's â—Ź Proxy HTTP to RADOS â—Ź Tested with apache and lighthttpd
- 27. 22JAN2014 CEPH Take Aways â—ŹScalable â—Ź Flexible configuration â—Ź No SPOF but build on commodity hardware â—Ź Accessible via different interfaces
- 28.
- 29. 22JAN2014 What? â—Ź Infrastructure asa Service (IaaS) â—Ź 'Open source' version of AWS â—Ź New versions every 6 months â—Ź Current called Havana â—Ź Previous called Grizzly â—Ź Managed by Openstack Foundation
- 30.
- 31. 22JAN2014 Openstack Components ● Keystone– identity ● Glance - image ● Nova - compute ● Cinder - block ● Swift - object ● Neutron - network ● Horizon - dashboard
- 32. 22JAN2014 About Glance â—Ź Theresince almost the beginning â—Ź Image store â—Ź Server â—Ź Disk â—Ź Several formats â—Ź raw â—Ź qcow2 â—Ź ... â—Ź Shared/non-shared
- 33. 22JAN2014 About Cinder â—Ź Kindof new â—Ź Part of Nova before â—Ź Separate since Folsom â—Ź Block storage â—Ź For compute instances â—Ź Managed by Openstack users â—Ź Several storage back-ends possible
- 34. 22JAN2014 About Swift â—Ź Sincethe beginning â—Ź Replace Amazon S3 -> cloud storage â—Ź Scalable â—Ź Redundant â—Ź Object store
- 35. 22JAN2014 Openstack Object Store â—ŹProxy â—Ź Object â—Ź Container â—Ź Account â—Ź Auth
- 36. 22JAN2014 Dream Team: CEPHand Openstack
- 37. 22JAN2014 Why CEPH inthe first place? â—Ź Full blown storage solution â—Ź Support â—Ź Operational model â—Ź Cloud'ish â—Ź Separation of duties â—Ź One solution for different storage needs
- 38. 22JAN2014 Integration â—Ź Full integrationwith RADOS â—Ź Since Folsom â—Ź Two parts â—Ź Authentication â—Ź Technical access â—Ź Both parties must be aware â—Ź Independent for each of the storage components â—Ź Different RADOS pools
- 39. 22JAN2014 Authentication ● CEPH part ●Key rings ● Configuration ● For Cinder and Glance ● Openstack part ● Keystone – Only for Swift – Needs RGW
- 40. 22JAN2014 Access to RADOS/RBDI â—Ź Via API/libraries => no CEPHFS needed â—Ź Easy for Glance/Cinder â—Ź CEPH keyring configuration â—Ź Update of CEPH.conf â—Ź Update of Glance/Cinder API configuration
- 41. 22JAN2014 Access to RADOS/RBDII ● Swift => more work needed ● CEPHFS => again: not needed ● CEPH Object Gateway – Web server – RGW software – Keystone certificates – No support for v2 Swift authentication ● Keystone authentication – Endlist configuration -> RGW
- 42. 22JAN2014 Integration the fullpicture RADOS qemu/kvm CEPH Object Gateway CEPH Block Device SwiftKeystone CinderGlance Nova
- 43. 22JAN2014 Integration pitfalls â—Ź CEPHversions not in sync â—Ź Authentication â—Ź CEPH Object Gateway setup
- 44. 22JAN2014 Why CEPH -reviewed â—Ź Previous arguments still valid :-) â—Ź Modular usage â—Ź High integration â—Ź Built-in HA â—Ź No need for POSIX compatible interface â—Ź Works even with other IaaS implementations
- 45.
- 46. 22JAN2014 Take Aways â—Ź Sophisticatedstorage engine â—Ź Mature OSS distributed storage solution â—Ź Other use cases â—Ź Can be used elsewhere â—Ź Full integration in Openstack
- 47.
- 48.
- 49. 22JAN2014 Petabyte Scale OutRocks CEPH as replacement for Openstack's Swift & Co. Udo Seidel