Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Data storage in clouds
Athens OpenStack User Group #OSATH
12th Meetup, 22th April 2015
Thanassis Parathyras,
aparathyras@s...
Announcements
#OSATH has 200 members !
Greek Mailing List
• openstack-el@lists.openstack.org
• Join at http://lists.openst...
Outline
• Storage types
• OpenStack Storage Services
– Glance
– Cinder
– Swift
• Higher abstractions
– Data Processing aaS...
It’s all about DATA
“often called storage or memory,
is a technology consisting of computer components and
recording media...
File Disk Drive CD-ROM
Object Volume Image
Swift Cinder Glance
Amazon S3 Amazon EBS
AWS Marketplace –
Software Infrastruct...
Glance
OpenStack Image service
• REST API
• CRUD and Search features
• Caching and prefetching
• Supports several formats:...
Cinder
OpenStack Block Storage service
• Volumes
• Snapshots
• Backups
• Modular architecture to support 50+ back-ends
• L...
Swift
OpenStack Object Storage service
• REST API
• Data redundancy (3x or more)
• Drive auditing built-in
• RAID not requ...
Swift in action
Access tier
• Proxy server
Storage tier
• Account server
• Container server
• Object server
• Consistency ...
Swift in detail
Upload
• PUT http://<swift_url>/<acc>/<cont>/fileA
Internals
• Consistent hashing – DHT
• Hash function: m...
Swift in detail
Partition power
• Estimate 100 partitions per
disk at max capacity
• Calculate closest power of 2
rounded ...
Putting it all together
A. Spawn a
new VM from
an Image
B. Attach
volume(s) to VM
C. Store
application objects
D. Retrieve...
Lifting more … data
OpenStack services moving up to stack
• Infrastructure (image, volume, snapshot, backup)
• Platform (d...
OSS storage projects
Lots of them, common goals to support scalable, large, software-
defined storage systems
Ceph
• Based...
Get involved
• Documentation
– http://docs.openstack.org
• Join the community
– http://www.openstack.org/community
• Greek...
Thank you!
Athens OpenStack User Group #OSATH
http://www.meetup.com/Athens-OpenStack-User-Group
Thanassis Parathyras
apara...
Upcoming SlideShare
Loading in …5
×

Data storage in clouds

1,423 views

Published on

Slides from Athens OpenStack User Group meetup.
We visit the concepts of data storage in the cloud computing world and explore the capabilities offered by OpenStack in this regard.

Published in: Technology
  • Be the first to comment

Data storage in clouds

  1. 1. Data storage in clouds Athens OpenStack User Group #OSATH 12th Meetup, 22th April 2015 Thanassis Parathyras, aparathyras@stackmasters.eu, @parathyras
  2. 2. Announcements #OSATH has 200 members ! Greek Mailing List • openstack-el@lists.openstack.org • Join at http://lists.openstack.org OpenStack Summit (https://www.openstack.org) • 18-22 May, Vancouver OpenStack CEE Day (http://openstackceeday.com) • 8 June, Budapest
  3. 3. Outline • Storage types • OpenStack Storage Services – Glance – Cinder – Swift • Higher abstractions – Data Processing aaS (Sahara) – DBaaS (Trove) • Other OSS projects – GlusterFS – Ceph
  4. 4. It’s all about DATA “often called storage or memory, is a technology consisting of computer components and recording media used to retain digital data. It is a core function and fundamental component of computers.” ent. Computer data storage, Wikipedia
  5. 5. File Disk Drive CD-ROM Object Volume Image Swift Cinder Glance Amazon S3 Amazon EBS AWS Marketplace – Software Infrastructure Storage types
  6. 6. Glance OpenStack Image service • REST API • CRUD and Search features • Caching and prefetching • Supports several formats: – raw, qcow2, vmdk, vhd, ami (aki, ari), iso, vdi • Containers: – bare, ovf, ami (aki, ari) • Backend to storage (default: Swift) • Able to aggregate multiple back-ends – Can also increase availability
  7. 7. Cinder OpenStack Block Storage service • Volumes • Snapshots • Backups • Modular architecture to support 50+ back-ends • LVM is the default (iSCSI) • Diverse storage types – iSCSI, Fibre, RBD (Ceph), NFS, GPFS, … • Manages storage resources separately Ephemeral vs Block Storage • Nova manages ephemeral storage coupled with VM state (non-persistent) • Cinder manages block storage decoupled from VM lifecylce (persistent)
  8. 8. Swift OpenStack Object Storage service • REST API • Data redundancy (3x or more) • Drive auditing built-in • RAID not required • Commodity hardware (not low-end) • High availability • Distributed • Eventual consistency CAP theorem • Choose 2 out of 3 (Consistency, Availability, Partition Tolerance) • Swift implements an eventual consistency model
  9. 9. Swift in action Access tier • Proxy server Storage tier • Account server • Container server • Object server • Consistency servers (Auditors, Updaters, Replicators) Zones • Selected per deployment • Determine replica isolation (disk, server, rack, room)
  10. 10. Swift in detail Upload • PUT http://<swift_url>/<acc>/<cont>/fileA Internals • Consistent hashing – DHT • Hash function: md5sum Ring • Static mapping to direct data location • Zones, disks, partitions and replicas • Across every node in the cluster 1 2 3 45 6 7 8 partition
  11. 11. Swift in detail Partition power • Estimate 100 partitions per disk at max capacity • Calculate closest power of 2 rounded up • 2partition_power=partition number Disks and partitions • Swift partition <> filesystem partitions • Presume 4 disks named A, B, C, D A A B BC C D D partition
  12. 12. Putting it all together A. Spawn a new VM from an Image B. Attach volume(s) to VM C. Store application objects D. Retrieve stored objects
  13. 13. Lifting more … data OpenStack services moving up to stack • Infrastructure (image, volume, snapshot, backup) • Platform (database, analytics) Trove – DBaaS • MySQL, Percona, MariaDB, MongoDB, Couchbase, Cassandra, Redis, PostgreSQL, Oracle • Equivelant to Amazon {RDS, DynamoDB} • Backup/Restore, Resize, Replication, User/DB management, etc. Sahara – Big data analytics • Hadoop from Hortonworks or Cloudera, Spark • Amazon Elastic MapReduce (EMR) • Manage and configure cluster, HDFS, MapReduce
  14. 14. OSS storage projects Lots of them, common goals to support scalable, large, software- defined storage systems Ceph • Based on RADOS and CRUSH provides object, block and file-system storage • 10+ years development effort • http://www.meetup.com/Ceph-Athens/ GlusterFS • Simple to use scale-out storage provides unified access to files and objects • Data stored in native format, no metadata completely algorithmic
  15. 15. Get involved • Documentation – http://docs.openstack.org • Join the community – http://www.openstack.org/community • Greek mailing list – http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-el • Contribute – code (development, blueprints, reviews, bugs) – docs writing, translations, infrastructure support
  16. 16. Thank you! Athens OpenStack User Group #OSATH http://www.meetup.com/Athens-OpenStack-User-Group Thanassis Parathyras aparathyras@stackmasters.eu, @parathyras

×