The document discusses oVirt and GlusterFS for hyperconverged infrastructure. It provides an overview of oVirt architecture including the engine, hosted engine, and storage domains. It also describes GlusterFS architecture with bricks, volumes, and access methods. The document outlines steps for setting up a hyperconverged system with oVirt and GlusterFS including creating replicated volumes, deploying a hosted engine VM, adding nodes, and managing storage.

1. Devconf.cz, February 2016
oVirt and Gluster
Hyperconvergence
Ramesh Nachimuthu
Software Engineer
Red Hat
Devconf.cz, February 2016

2. Devconf.cz, February 2016
Agenda
● Architecture of oVirt
● Hosted Engine
● Gluster in a nutshell
● Hyper converged infrastructure
● How it looks
● Setup
● Management
● Recent enhancements
● Conclusion

3. Devconf.cz, February 2016
Ovirt Architecture
oVirt is a virtualization platform
to manage virtual machines,
storage and networks
● Engine (ovirt-engine)
Manages the oVirt hosts,
and allows system
administrators to create and
deploy new Vms
● Host Agent (VDSM)
oVirt engine communicates
with VSDM to manage the
Vms, storages and
networks
Clusters
Storage Server
StorageEngine
Storage Server
Cluster Hosts

4. Devconf.cz, February 2016
Ovirt Storage Architecture
● Storage Domain
● Centralized storage system
(images, templates, etc.)
● A standalone storage entity
● Stores the images and associated
Metadata
● Only real persistent storage for
VDSM
● Used for synchronization (sanlock)
● Storage Types
● NFS, FCP, iSCSI
● Glusterfs
Clusters
Storage Domain
StorageEngine
Storage Domain
VDSM Hosts

5. Devconf.cz, February 2016
Hosted Engine
● Ovirt Engine running inside a
VM
● Highly Available
● Can be migrated to a
different host
● Special Agent for Monitoring
● Bootstrap deployment
needed Clusters
Storage Domain
StoragePoolEngine
Storage Domain
VDSM Hosts

6. Devconf.cz, February 2016
GlusterFS Architecture
● GlusterFS is a general purpose scale-out distributed file-system
supporting thousands of clients
● Aggregates storage exports over network interconnect to provide
a single unified namespace
● File-system completely in
userspace, runs on
commodity hardware
● Layered on disk file
systems that support
extended attributes

7. Devconf.cz, February 2016
GlusterFS Bricks
● A brick is an export directory located on a specific node
(e.g. host-01:/srv/fs1/brick1)
● Each brick inherits limits of the underlying file-system
● No limit on the number bricks per node (as best-practice each
brick in a cluster should be of the same size)
/srv/fs1/brick1 /srv/fs1/brick2/srv/fs1/brick2
File-System: /srv/fs1 /srv/fs2
Block Device: /dev/dm-1 /dev/dm-2

8. Devconf.cz, February 2016
GlusterFS Volumes
● A volume is a logical collection of bricks
● The mountable entity
● Different types of Volumes
● Distribute, Stripe, Replicate (+ combinations), Quorum
● Type of a volume is specified at the time of volume creation
and determines how and where data is placed
● Accessible via FUSE, NFS, SMB, libgfapi
host01:/srv/fs1/brick1
Volume1
host02:/srv/fs1/brick1 host03:/srv/fs1/brick1
Distribute/Stripe/Replicate

9. Devconf.cz, February 2016
Fuse Access
QEMU
Kernel VFS /dev/fuse
Kernel
Userspace
GlusterFS
Net
File-System
Gluster Brick
Net
Kernel
Userspace
Client Server
FUSE
Access
●
More context switches

10. Devconf.cz, February 2016
QEMU libgfapi Support
● GlusterFS exposes APIs for accessing Gluster volumes
●
Reduces context switches
QEMU
Kernel
Userspace
Net
File-System
Gluster Brick
Net
Kernel
Userspace
Client Server
libgfapi Access
But see: https://bugzilla.redhat.com/show_bug.cgi?id=1247933

11. Devconf.cz, February 2016
Hyperconverged oVirt – GlusterFS
● The Data Center nodes are
used both for virtualization and
serving replicated images from
the GlusterFS Bricks
● Engine runs inside a VM
(Hosted Engine)
● The boxes can be standardized
(hardware and deployment) for
easy addition and replacement
● Support for both scaling up,
adding more disks, and scaling
out, adding more hosts
VMsandStoragewithEngineinsideaVM
GlusterFS Volume
Bricks Bricks Bricks
Experimental?

12. Devconf.cz, February 2016
Improvements needed for HC
● GlusterFS support re-added to hosted-engine setup
● With libgfapi and replica 3 volume
● Ovirt-Engine Appliance
● Preconfigured VM with oVirt Engine
● Cloud-init based customization
● Shared configuration
● All nodes see the same configuration data
● Upgrade path from oVirt 3.5
● Management GUI for the oVirt-engine VM and HE

13. Devconf.cz, February 2016
HC ingredients
● at least 3 virtualization capable hosts (CentOS 7.1+)
● 10 GB of temporary space on the primary host
● two separate partitions for data (20GB+) on all hosts
● DHCP configured to map a MAC address to a fixed IP
● DNS configured with A and PTR names for the IP
● oVirt release package installed on all hosts
http://resources.ovirt.org/pub/yum-repo/ovirt-release36.rpm
● Physical console on the primary host or network access and
screen package installed

14. Devconf.cz, February 2016
HC Setup - Gluster Volume
● 3 Node Gluster cluster
● 2 Volumes with 3 way replication
● One for Hosted Engine VM and other for regular VMs
● Enable Cluster Quorum
● Sharding for better self-heal and geo-rep performace
● Few more tunings
● network.ping-timeout = 10
● cluster.data-self-heal-algorithm = full
● performance.low-prio-threads = 32
● Group = virt
● storage.owner-uid, storage.owner-uid = 36

15. Devconf.cz, February 2016
HC Setup – Hosted Engine
Packages Required
● ovirt-engine-appliance
● ovirt-hosted-engine-setup
● Vdsm-gluster
hosted-engine –-deploy
● 'glusterfs' storage type
● OVA image for hosted-engine VM
● Clound-init to customize the VM
● FQDN and Mac for hosted-engine VM
More details at
http://blogs-ramesh.blogspot.in/2016/01/ovirt-and-gluster-hyperconvergence.html

16. Devconf.cz, February 2016
HC Setup – Final Touch
● Hosted Engine running with single node Cluster
● Enable Gluster service
● Add remaining nodes
● Create and add the main storage domain

17. Devconf.cz, February 2016
Enabling Gluster
● Gluster Service support
is located in the Cluster
properties
● Deploy Hosts with
GlusterFS Server
support
● Enable Bricks and
Volume Management
from oVirt WebAdmin
and REST-API
● Engine is not taking in consideration GlusterFS on
Virtualization Power-Saving policies and Fencing yet

18. Devconf.cz, February 2016
Adding addtional Nodes
● Simple checkbox during in the Add host dialog
● Host deploy script does everything else auto-magically
Removed at the last moment because of unresolved issues.
It can be done through “hosted-engine –deploy” script

19. Devconf.cz, February 2016
Adding Gluster storage
● Gluster volume management using WebAdmin Portal and
REST-API
● Volume Pofiling and Capacity monitoring
● Geo Replication
● Volume Snapshot

20. Devconf.cz, February 2016
Add Storage Domain
● Master Storage Domain using gluster volume
● Imports Hosted Engine Storage Domain and Hosted
Engine VM autmotically.

21. Devconf.cz, February 2016
Dashboard

22. Devconf.cz, February 2016
Hosted engine VM management
● Support for editing the Hosted Engine VM
● Memory and CPU allocation, network configuration
● Work in progress..
● Distributed to all hosted engine nodes using OVF file on
the storage domain
● Hosted engine daemons pick up the configuration when
the management VM is restarted

23. Devconf.cz, February 2016
Hosted engine management
● Reporting configuration
– State transitions
– SMTP details
● Timeout configuration
– Allowed downtime before forced recovery
● Host scoring constants and rules

24. Devconf.cz, February 2016
Recent Enahacements
● Glusterfs Sharding
● Arbiter Volumes
● Replace Brick
● Backup vol file support in libvirt and QEMU-KVM
https://devconfcz2016.sched.org/event/5m1p/improvements-in-gluster-

25. Devconf.cz, February 2016
THANK YOU!
http://www.ovirt.org
users@ovirt.org
devel@ovirt.org
#ovirt irc.oftc.net
http://www.ovirt.org/Features/GlusterFS-Hyperconvergence
http://blogs-ramesh.blogspot.in/2016/01/ovirt-and-gluster-hypercon