Glusterfs and openstack


Published on

Published in: Technology, Business

Glusterfs and openstack

  1. 1. GlusterFS & OpenStack Vijay Bellur Red Hat
  2. 2. Agenda ➢ What is GlusterFS? ➢ Architecture ➢ Concepts ➢ Algorithms ➢ Access Mechanisms ➢ Implementation ➢ OpenStack and GlusterFS ➢Resources ➢ Q&A
  3. 3. GlusterFS Deployment Global namespace Scale-out storage building blocks Supports thousands of clients Access using GlusterFS native, NFS, SMB and HTTP protocols Runs on commodity hardware
  4. 4. GlusterFS concepts
  5. 5.  Trusted Storage Pool (cluster) is a collection of storage servers.  Trusted Storage Pool is formed by invitation – you “probe” a new member from the cluster and not vice versa.  Logical partition for all data and management operations.  Membership information used for determining quorum.  Members can be dynamically added and removed from the pool. GlusterFS concepts – Trusted Storage Pool
  6. 6. GlusterFS concepts – Trusted Storage Pool Node1 Node2 Probe Probe accepted Node 1 and Node 2 are peers in a trusted storage pool Node2Node1
  7. 7. GlusterFS concepts – Trusted Storage Pool Node1 Node2 Node3Node2Node1 Trusted Storage Pool Node3Node2Node1 Detach
  8. 8.  A brick is the combination of a node and an export directory – for e.g. hostname:/dir  Each brick inherits limits of the underlying filesystem  No limit on the number bricks per node  Ideally, each brick in a cluster should be of the same size /export3 /export3 /export3 Storage Node /export1 Storage Node /export2 /export1 /export2 /export4 /export5 Storage Node /export1 /export2 3 bricks 5 bricks 3 bricks GlusterFS concepts - Bricks
  9. 9. GlusterFS concepts - Volumes ➢ A volume is a logical collection of bricks. ➢ Volume is identified by an administrator provided name. ➢Volume is a mountable entity and the volume name is provided at the time of mounting. ➢ mount -t glusterfs server1:/<volname> /my/mnt/point ➢ Bricks from the same node can be part of different volumes
  10. 10. GlusterFS concepts - Volumes Node2Node1 Node3 /export/brick1 /export/brick2 /export/brick1 /export/brick2 /export/brick1 /export/brick2 Videos music
  11. 11. Volume Types ➢Type of a volume is specified at the time of volume creation ➢ Volume type determines how and where data is placed ➢ Following volume types are supported in glusterfs: a) Distribute b) Stripe c) Replication d) Distributed Replicate e) Striped Replicate f) Distributed Striped Replicate
  12. 12. Distributed Volume ➢Distributes files across various bricks of the volume. ➢Directories are present on all bricks of the volume. ➢Single brick failure will result in loss of data availability. ➢Removes the need for an external meta data server.
  13. 13. How does a distributed volume work? ➢ Distribute works on the basis of a dht algorithm. ➢ A 32-bit hash space is divided into N ranges for N bricks ➢ At the time of directory creation, a range is assigned to each directory. ➢ During a file creation or retrieval, hash is computed on the file name. This hash value is used to locate or place the file. ➢Different directories in the same brick end up with different hash ranges.
  14. 14. How does a distributed volume work?
  15. 15. How does a distributed volume work?
  16. 16. How does a distributed volume work?
  17. 17. Replicated Volume ➢Creates synchronous copies of all directory and file updates. ➢Provides high availability of data when node failures occur. ➢Transaction driven for ensuring consistency. ➢Changelogs maintained for re-conciliation. ➢Any number of replicas can be configured.
  18. 18. How does a replicated volume work?
  19. 19. How does a replicated volume work?
  20. 20. Distributed Replicated Volume ➢ Distribute files across replicated bricks ➢ Number of bricks must be a multiple of the replica count ➢ Ordering of bricks in volume definition matters ➢ Scaling and high reliability ➢ Improved read performance in most environments. ➢ Most preferred model of deployment currently.
  21. 21. Distributed Replicated Volume
  22. 22. Striped Volume ➢Files are striped into chunks and placed in various bricks. ➢Recommended only when very large files greater than the size of the disks are present. ➢Chunks are files with holes – this helps in maintaining offset consistency. ➢A brick failure can result in data loss. Redundancy with replication is highly recommended
  23. 23. Translators in GlusterFS ➢Building blocks for a GlusterFS process. ➢Based on Translators in GNU HURD. ➢Each translator is a functional unit. ➢Translators can be stacked together for achieving desired functionality. ➢ Translators are deployment agnostic – can be loaded in either the client or server stacks.
  24. 24. VFS Server I/O Cache Distribute / Stripe POSIX Ext4 Ext4Ext4 POSIX POSIX Brick 1 ServerServer Read Ahead Brick 2 Brick n-1 Gluster Server Replicate Ext4 POSIX Server Brick n Replicate Customizable GlusterFS Client/Server Stack Client Gluster Server Client GigE, 10GigE – TCPIP / InfiniBand – RDMA Gluster ServerGluster Server Client Client
  25. 25. Dynamic Volume Management Collectively refers to application transparent operations that can be performed in the storage layer. ➢ Addition of Bricks to a volume ➢ Remove brick from a volume ➢ Rebalance data spread within a volume ➢ Replace a brick in a volume ➢ Adaptive Performance / Functionality tuning.
  26. 26. Access Mechanisms: Gluster volumes can be accessed via the following mechanisms: ➢ FUSE ➢ NFS ➢ SMB ➢ libgfapi ➢ ReST ➢ HDFS
  27. 27. ReST based access
  28. 28. UFO Client Proxy Account Container Object HTTP Request (REST) Directory Volume FileClient NFS or GlusterFS Mount Unified File and object view. Entity mapping between file and object building blocks
  29. 29. Interface Possibilities qemu NFS SMB Hadoop FUSE Cinder Swift (UFO) Files Blocks Objects libgfapi Whatever IP RDMA Transports files BD Back ends DB Interface Possibilities
  30. 30. OpenStack and GlusterFS – Current Integration Glance Images Nova Nodes Swift Objects Cinder Data Glance Data Swift Data Swift API Storage Server Storage Server Storage Server… KVM KVM KVM … ● Separate Compute and Storage Pools ● GlusterFS directly provides Swift object service ● Integration with Keystone ● GeoReplication for multi-site support ● Swift data also available via other protocols ● Supports non-OpenStack use in addition to OpenStack use Logical View Physical View
  31. 31. OpenStack and GlusterFS - Future Direction Hadoop Guest Other Guest ... Host Gluster Guest Hadoop Guest Other Guest ... Host Gluster Guest Nova Compute Nodes
  32. 32. Open Stack and GlusterFS - Future Direction ● POC based on proposed OpenStack FaaS (File as a Service) proposal ● Cinder-like virtual NAS service ● Tenant-specific file shares ● Hypervisor mediated for security ● Avoid exposing servers to Quantum tenant network ● Optional multi-site or multi-zone GeoReplication ● FaaS data optionally available to non OpenStack nodes ● Initial focus on Linux guest ● Windows (SMB) and NFS shares also under consideration
  33. 33. Open Stack and GlusterFS - Summary ● Can address storage needs across the stack: ● Block ● Object ● File ● Hadoop/Big Data
  34. 34. Resources Mailing lists: IRC: #gluster and #gluster-dev on freenode Links:
  35. 35. Questions? Thank You!