Managed block storage integration with Cinderlib allows using Cinder storage drivers outside of Cinder. This provides benefits to oVirt by enabling direct use of storage vendor offloading APIs for fast storage operations with minimal networking. The presentation covers Cinderlib architecture and integration with oVirt, supported operations like creating managed block storage domains and disks, and attaching disks to VMs. It also discusses related changes in VDSM and debugging tools.

1. MANAGED BLOCK STORAGEMANAGED BLOCK STORAGE
CINDERLIB INTEGRATIONCINDERLIB INTEGRATION
PRESENTED BY EYAL SHENITZKYPRESENTED BY EYAL SHENITZKY
SOFTWARE ENGINEER SOFTWARE ENGINEER RED HATRED HAT

2. AGENDAAGENDA
CINDERLIB OVERVIEWCINDERLIB OVERVIEW
MANAGED BLOCK STORAGEMANAGED BLOCK STORAGE
ARCHITECTUREARCHITECTURE
SUPPORTED OPERATIONSSUPPORTED OPERATIONS
VDSM CHANGESVDSM CHANGES
DEBUGGINGDEBUGGING
PREREQUISITESPREREQUISITES
FUTURE WORKFUTURE WORK
LIVE DEMOLIVE DEMO
Q&AQ&A

3. OVERVIEWOVERVIEW
WHAT IS CINDERLIB?WHAT IS CINDERLIB?

4. OVERVIEWOVERVIEW
https://github.com/Akrog/cinderlib
https://gorka.eguileor.com/cinderlib/
CINDER PYTHON LIBRARY THAT ALLOWS USINGCINDER PYTHON LIBRARY THAT ALLOWS USING
STORAGE DRIVERS OUTSIDE OF CINDERSTORAGE DRIVERS OUTSIDE OF CINDER
USE CINDER DRIVERSUSE CINDER DRIVERS
DIRECTLYDIRECTLY

5. OVERVIEWOVERVIEW
BENEFIT TO OVIRTBENEFIT TO OVIRT

6. OVERVIEWOVERVIEW
USE THE STORAGEUSE THE STORAGE
VENDORS OFFLOADING APIVENDORS OFFLOADING API
AND PERFORM FASTAND PERFORM FAST
STORAGE SIDE ACTIONSSTORAGE SIDE ACTIONS
WITH MINIMAL NETWORKWITH MINIMAL NETWORK
USAGE AND USAGE AND MANAGEMENTMANAGEMENT
BENEFIT TO OVIRTBENEFIT TO OVIRT

7. ARCHITECTUREARCHITECTURE
MANAGED BLOCKMANAGED BLOCK
STORAGE BASICSTORAGE BASIC
ARCHITECTUREARCHITECTURE

8. ARCHITECTUREARCHITECTURE

9. ARCHITECTUREARCHITECTURE

10. SUPPORTED OPERATIONSSUPPORTED OPERATIONS
BASIC FLOWSBASIC FLOWS

11. SUPPORTED OPERATIONSSUPPORTED OPERATIONS
CREATE AND MANAGE MBS DOMAINCREATE AND MANAGE MBS DOMAIN
CREATE/DELETE DISK ON MBS DOMAINCREATE/DELETE DISK ON MBS DOMAIN
RUN VMS WITH ATTACHED MBS BASEDRUN VMS WITH ATTACHED MBS BASED
DISKSDISKS

12. ADD MBS DOMAINADD MBS DOMAIN
SUPPORTED OPERATIONSSUPPORTED OPERATIONS
user configures a new storage domain with all needed
parameters for the cinder drivers
CinderlibExecutor
cinderlib-client.py (stats)
Engine stores the storage domain data in the DB
(The driver options saved under new cinder_storage table)

13. ADD MBS DOMAIN - UIADD MBS DOMAIN - UI
SUPPORTED OPERATIONSSUPPORTED OPERATIONS
Drivers options: https://docs.openstack.org/cinder/rocky/configuration/block-storage/volume-
drivers.html

14. ADD ADD MBSMBS DOMAIN - REST DOMAIN - REST
SUPPORTED OPERATIONSSUPPORTED OPERATIONS
<storage_domain>
<name>cinder­kaminario</name>
<type>managed_block_storage</type>
<storage>
<type>managed_block_storage</type>
<driver_options>
<property>
<name>use_multipath_for_image_xfer</name>
<value>true</value>
</property>
<property>
<name>san_ip</name>
<value>10.35.84.1</value>
</property>
...
</driver_options>
<driver_sensitive_options>
<property>
<name>san_password</name>
<value>admin</value>
</property>
...
</driver_sensitive_options>
</storage>
<host>
<name>host_name</name>
</host>
</storage_domain>

15. ADD ADD MBSMBS BASED DISK BASED DISK
SUPPORTED OPERATIONSSUPPORTED OPERATIONS
User configures a 'managed block' disk type
CinderlibExecutor
cinderlib-client.py
(create_volume)
Engine adds new managed block based disk to the DB

16. ADD ADD MBSMBS BASED DISK BASED DISK
SUPPORTED OPERATIONSSUPPORTED OPERATIONS

17. RUN VM - PHASE - 1: CONNECTRUN VM - PHASE - 1: CONNECT
SUPPORTED OPERATIONSSUPPORTED OPERATIONS
Engine fetches the connector info of the host from the DB
(reported in host capabilities)
CinderlibExecutor
cinderlib-client.py
(connect_volume)
Engine now has the connection info to supply to the host

18. RUN VM - PHASE - 2: ATTACHRUN VM - PHASE - 2: ATTACH
SUPPORTED OPERATIONSSUPPORTED OPERATIONS
Engine requests VDSM to attach the volume using the
connection info
VDSM
Engine now has a path of the attached block device
managedvolume-
helper

19. PHASE - 3: RUN VMPHASE - 3: RUN VM
SUPPORTED OPERATIONSSUPPORTED OPERATIONS
Engine creates VM XML
VDSM
Running VM

20. ADDITIONAL FLOWS:ADDITIONAL FLOWS:
Stop VM
Extend MBS disk (when VM is down)
Create VM template with MBS disks
Create VM from template with MBS disks
Clone VM with MBS disks
Hot plug / Hot unplug MBS disk
Live migration
Basic snapshot operations
SUPPORTED OPERATIONSSUPPORTED OPERATIONS

21. ADDITIONAL FLOWS:ADDITIONAL FLOWS:
Stop VM
Extend MBS disk (when VM is down)
Create VM template with MBS disks
Create VM from template with MBS disks
Clone VM with MBS disks
Hot plug / Hot unplug MBS disk
Live migration
Basic snapshot operations
SUPPORTED OPERATIONSSUPPORTED OPERATIONS

22. VDSM CHANGESVDSM CHANGES
VDSM CHANGESVDSM CHANGES

23. VDSM CHANGESVDSM CHANGES
CONNECTOR INFO INCONNECTOR INFO IN
GETCAPABILITIESGETCAPABILITIES
"connector_info": {
"initiator": "iqn.1994-05.com.redhat:b82360563cee",
"ip": null,
"platform": "x86_64",
"host": "vdsm02",
"do_local_attach": false,
"os_type": "linux2",
"multipath": true
}
If os-brick is not installed, the connector information will not be present in GetCapabilites

24. VDSM CHANGESVDSM CHANGES
ATTACH VOLUMEATTACH VOLUME
Vdsm stores the volume information along with the connection information of the attached volumes
in a local DB (sqlite).
The volume ID will be the key of the data in the DB.
The stored data is used to be able to filter them from GetDeviceList result and to detach a volume.
ManagedVolume.attach_volume ( volume ID, connection_info)
input: {
"vol_id": "30a262cf­4489­4748­8b85­2ab52e8dfdc1",
"connection_info": {
"driver_volume_type": "rbd",
"data": {
"secret_type": "ceph",
"name": "volumes/volume­30a262cf­4489­4748­8b85­2ab52e8dfdc1", .....
}
output: {
"path": "/dev/rbd/volumes/volume­30a262cf­4489­4748­8b85­2ab52e8dfdc1",
"vol_id": "30a262cf­4489­4748­8b85­2ab52e8dfdc1",
"attachment": {
"path": "/dev/rbd1",
"type": "block",
"conf": "/tmp/brickrbd_FbSRNv" ...
}

25. VDSM CHANGESVDSM CHANGES
DETACH VOLUMEDETACH VOLUME
The needed data (device info and connection info) for performing the detach is retrieved from the
local DB.
ManagedVolume.detach_volume (volume ID)
input: {
"vol_id"="f036007f-330f-476c-8e65-0beaf4e86dfb"
}
output: {
"status": {
"message": "Done",
"code": 0
}
}

26. VDSM CHANGESVDSM CHANGES
GET VOLUME INFORMATIONGET VOLUME INFORMATION
The data is retrieved from the local DB.
'exist' is set to 'True' if the device is connected and path exists on local machine.
volumes_info support single/multiple volumes ID, or none to get all the volumes.
ManagedVolume.volumes_info ( volume ID)
{
"exists": true,
"connection_info": {
"driver_volume_type": "iscsi",
"data": {
"target_iqn": "iqn.2009­01.com.kaminario:storage.k2.22612", ......
}
},
"attachment": {
"scsi_wwn": "20024f400585403c5", ....
},
"path": "/dev/mapper/20024f400585403c5",
"vol_id": "9eb27e09­990c­4436­a33b­5af548343ed6"
}

27. VDSM CHANGESVDSM CHANGES
MANAGEDVOLUME-HELPERMANAGEDVOLUME-HELPER
os-brick operations are performed via supervdsm, invoking an helper:
/usr/libexec/vdsm/managedvolume-helper
supervdsm.log example:
MainProcess|jsonrpc/6::DEBUG::2019-02-12
12:10:34,724::supervdsm_server::100::SuperVdsm.ServerCallback::(wrapper) call
managedvolume_run_helper with ('attach',) {'cmd_input': {u'driver_volume_type': u'rbd', u'data':
{u'secret_type': u'ceph', u'name': u'volumes/volume-f036007f-330f-476c-8e65-0beaf4e86dfb',
u'discard': True, u'keyring': u'# keyringn[client.admin] nkey =
AQAsV55bOAxeAhAAg5Hy09NP96rp8GMJv4Fazw==an', u'cluster_name': u'ceph', u'secret_uuid':
None, u'auth_enabled': True, u'volume_id': u'f036007f-330f-476c-8e65-0beaf4e86dfb', u'hosts':
[u'10.35.0.252'], u'auth_username': u'admin', u'ports': [u'6789']}}}
MainProcess|jsonrpc/6::DEBUG::2019-02-12
12:10:36,393::supervdsm_server::107::SuperVdsm.ServerCallback::(wrapper) return
managedvolume_run_helper with {u'path': u'/dev/rbd1', u'type': u'block', u'conf':
u'/tmp/brickrbd_4NksPF'}

28. DEBUGGINGDEBUGGING
DEBUGGING MANAGEDDEBUGGING MANAGED
BLOCK RELATEDBLOCK RELATED
OPERATIONOPERATION

29. DEBUGGINGDEBUGGING
CREATE MBS DOMAINCREATE MBS DOMAIN
2019­02­11 11:58:53,499+02 INFO
[org.ovirt.engine.core.bll.storage.domain.GetManagedBlockStorageStatsCommand]
(default task­3) [79c4e6b8­5c09­4f8f­a2cc­d3058a3d84fe] Running command:
GetManagedBlockStorageStatsCommand internal: true.
2019­02­11 11:58:55,966+02 INFO
[org.ovirt.engine.core.common.utils.cinderlib.CinderlibExecutor] (default task­
3) [79c4e6b8­5c09­4f8f­a2cc­d3058a3d84fe] cinderlib output: {"total_volumes":
238, "provisioned_capacity_gb": 85429, "volume_backend_name":
"KaminarioISCSIDriver", "thin_provisioning_support": true, "driver_version":
...
"kaminario:thin_prov_type": "dedup/nodedup", "storage_protocol": "iSCSI"}
2019­02­11 11:58:55,973+02 INFO
[org.ovirt.engine.core.bll.storage.domain.AddManagedBlockStorageDomainCommand]
(default task­3) [79c4e6b8­5c09­4f8f­a2cc­d3058a3d84fe] Running command:
AddManagedBlockStorageDomainCommand internal: false. Entities affected : ID:
aaa00000­0000­0000­0000­123456789aaa Type: SystemAction group
CREATE_STORAGE_DOMAIN with role type ADMIN
ENGINE:ENGINE:

30. DEBUGGINGDEBUGGING
CREATE MBS DOMAIN ERRORCREATE MBS DOMAIN ERROR
2019­02­11 11:55:12,439+02 INFO
[org.ovirt.engine.core.bll.storage.domain.GetManagedBlockStorageStatsCommand]
(default task­3) [5701d492­7361­444a­8d88­e84ce9074658] Running command:
GetManagedBlockStorageStatsCommand internal: true.
2019­02­11 11:55:16,516+02 ERROR
[org.ovirt.engine.core.common.utils.cinderlib.CinderlibExecutor] (default
task­3) [5701d492­7361­444a­8d88­e84ce9074658] cinderlib execution failed:
Traceback (most recent call last):
File "./cinderlib­client.py", line 137, in main
args.command(args)
File "./cinderlib­client.py", line 197, in storage_stats
backend = load_backend(args)
...
ENGINE:ENGINE:

31. DEBUGGINGDEBUGGING
CREATE MBS BASED DISKCREATE MBS BASED DISK
2019­02­11 11:48:22,860+02 INFO
[org.ovirt.engine.core.bll.storage.disk.AddDiskCommand] (default task­3)
[512895ae­3199­4b80­bf86­34ac41f66ed3] Running command: AddDiskCommand
internal: false. Entities affected : ID: 02b10c38­06a6­454f­b490­b2dd182ec0b5
Type: StorageAction group CREATE_DISK with role type USER
2019­02­11 11:48:22,980+02 INFO
[org.ovirt.engine.core.bll.storage.disk.managedblock.AddManagedBlockStorageDis
kCommand] (EE­ManagedThreadFactory­commandCoordinator­Thread­1) [512895ae­
3199­4b80­bf86­34ac41f66ed3] Running command:
AddManagedBlockStorageDiskCommand internal: true.
2019­02­11 11:48:27,195+02 INFO
[org.ovirt.engine.core.common.utils.cinderlib.CinderlibExecutor] (EE­
ManagedThreadFactory­commandCoordinator­Thread­1) [512895ae­3199­4b80­bf86­
34ac41f66ed3] cinderlib output:
ENGINE:ENGINE:

32. DEBUGGINGDEBUGGING
RUN VM - PHASE 1 - CONNECTRUN VM - PHASE 1 - CONNECT
2019-02-11 12:02:49,580+02 INFO [org.ovirt.engine.core.bll.RunVmCommand]
(default task-3) [98f4534e-37f9-416c-b4be-978bb9930cb7] Lock Acquired to
object 'EngineLock:{exclusiveLocks='[4e1d51a7-c3cb-4589-829c-
d0a4e42f2c53=VM]', sharedLocks=''}'
2019-02-11 12:02:49,889+02 INFO
[org.ovirt.engine.core.bll.storage.disk.managedblock.ConnectManagedBlockStorage
DeviceCommand] (EE-ManagedThreadFactory-engine-Thread-204) [5f781d60]
Running command: ConnectManagedBlockStorageDeviceCommand internal: true.
2019-02-11 12:02:53,094+02 INFO
[org.ovirt.engine.core.common.utils.cinderlib.CinderlibExecutor] (EE-
ManagedThreadFactory-engine-Thread-204) [5f781d60] cinderlib output:
{"driver_volume_type": "iscsi", "data": {"target_lun": 141, "target_iqn": "iqn.2009-
01.com.kaminario:storage.k2.22612", "target_portal": "10.35.162.66:3260",
ENGINE:ENGINE:

33. DEBUGGINGDEBUGGING
RUN VM - PHASE 2 - ATTACHRUN VM - PHASE 2 - ATTACH
2019­02­11 12:03:00,833+02 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.AttachManagedBlockStorageVolum
eVDSCommand] (EE­ManagedThreadFactory­engine­Thread­204) [5f781d60]
FINISH, AttachManagedBlockStorageVolumeVDSCommand, return: {attachment=
{path=/dev/dm­5, scsi_wwn=20024f400585403c4, type=block,
multipath_id=20024f400585403c4}, path=/dev/mapper/20024f400585403c4}, log
id: 52545bec
2019­02­11 12:03:00,887+02 INFO
[org.ovirt.engine.core.bll.storage.disk.managedblock.SaveManagedBlockStor
ageDiskDeviceCommand] (EE­ManagedThreadFactory­engine­Thread­204)
[481f9a0a] Running command: SaveManagedBlockStorageDiskDeviceCommand
internal: true.
2019­02­11 12:03:02,730+02 INFO
[org.ovirt.engine.core.common.utils.cinderlib.CinderlibExecutor] (EE­
ManagedThreadFactory­engine­Thread­204) [481f9a0a] cinderlib output:
ENGINE:ENGINE:

34. DEBUGGINGDEBUGGING
RUN VM - PHASE 2 - ATTACHRUN VM - PHASE 2 - ATTACH
attach_volume(vol_id=u'9eb27e09­990c­4436­a33b­5af548343ed6',
connection_info={u'driver_volume_type': u'is
csi', u'data': {u'target_lun': 191, u'target_iqn': u'iqn.2009­
01.com.kaminario:storage.k2.22612', u'target_portal':
u'10.35.162.66:3260', u'target_discovered': True}})
attach_volume return={'status': {'message': 'Done', 'code': 0}, 'result':
{'path': u'/dev/mapper/20024f40
0585403c5', 'vol_id': u'9eb27e09­990c­4436­a33b­5af548343ed6',
'attachment': {u'path': u'/dev/dm­15', u'scsi_wwn': u'20024f400585403c5',
u'type': u'block', u'multipath_id':
u'20024f400585403c5'}}}
VDSM:VDSM:

35. DEBUGGINGDEBUGGING
RUN VM - PHASE 3 - RUNRUN VM - PHASE 3 - RUN
2019-02-11 12:03:06,049+02 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DumpXmlsVDSCommand] (ForkJoinPool-1-worker-2) []
FINISH, DumpXmlsVDSCommand, return: {4e1d51a7-c3cb-4589-829c-d0a4e42f2c53=<domain
type='kvm' id='1'>
...
<devices>
<disk snapshot="no" type="block" device="disk">
<target dev="sda" bus="scsi"/>
<source dev="/dev/rbd/volumes/volume-12f20756-5eef-424b-8b73-e1a9361a2531"/>
<driver name="qemu" cache="none"/>
<alias name="ua-12f20756-5eef-424b-8b73-e1a9361a2531"/>
<address bus="0" controller="0" unit="0" type="drive" target="0"/>
<boot order="1"/>
<serial>12f20756-5eef-424b-8b73-e1a9361a2531</serial>
</disk>
ENGINE:ENGINE:

36. PREREQUISITESPREREQUISITES
MANUAL NEEDED STEPMANUAL NEEDED STEP
FOR MANAGED BLOCKFOR MANAGED BLOCK
STORAGE ENVIRONMENTSTORAGE ENVIRONMENT

37. PREREQUISITESPREREQUISITES
INSTALL CINDERLIBINSTALL CINDERLIB
PACKAGE ON THEPACKAGE ON THE
ENGINE MACHINEENGINE MACHINE

38. PREREQUISITESPREREQUISITES
INSTALL THE OS-BRICKINSTALL THE OS-BRICK
PACKAGE ON THEPACKAGE ON THE
HOSTSHOSTS

39. PREREQUISITESPREREQUISITES
ENABLE THE ENGINEENABLE THE ENGINE
SUPPORT FOR THESUPPORT FOR THE
FEATUREFEATURE

40. PREREQUISITESPREREQUISITES
engine-config -s ManagedBlockDomainSupported=true and
select 4.3 version
ENABLE THE ENGINEENABLE THE ENGINE
SUPPORT FOR THESUPPORT FOR THE
FEATUREFEATURE

41. PREREQUISITESPREREQUISITES
INSTALL DRIVERINSTALL DRIVER
SPECIFIC PACKAGESSPECIFIC PACKAGES

42. PREREQUISITESPREREQUISITES
for example:
krest - for kaminario
ceph-common - for ceph
https://github.com/openstack/cinder/blob/master/driver­requirements.txt
INSTALL DRIVERINSTALL DRIVER
SPECIFIC PACKAGESSPECIFIC PACKAGES

43. PREREQUISITESPREREQUISITES
CREATE CINDERLIBCREATE CINDERLIB
DATABASEDATABASE

44. PREREQUISITESPREREQUISITES
su - postgres -c "psql -d template1 -c "create database
cinder owner engine template template0 encoding 'UTF8'
lc_collate 'en_US.UTF-8' lc_ctype 'en_US.UTF-8';""
host cinder engine ::0/0 md5
host cinder engine 0.0.0.0/0 md5
Make sure pg_hba has an entry for the created CinderLib
database
CREATE CINDERLIBCREATE CINDERLIB
DATABASEDATABASE

45. PREREQUISITESPREREQUISITES
PERMISSIONSPERMISSIONS

46. PREREQUISITESPREREQUISITES
chown ovirt -R /usr/share/ovirt-engine/cinderlib
PERMISSIONSPERMISSIONS

47. PREREQUISITESPREREQUISITES
chown ovirt -R /usr/share/ovirt-engine/cinderlib
PERMISSIONSPERMISSIONS

48. FUTURE WORKFUTURE WORK
FUTURE WORKFUTURE WORK

49. FUTURE WORKFUTURE WORK
SUPPLY MISSING PACKAGESSUPPLY MISSING PACKAGES
cinderlib
os-brick

50. FUTURE WORKFUTURE WORK
Engine MBS related logs
Cinderlib-client logs
IMPROVE LOGGINGIMPROVE LOGGING

51. FUTURE WORKFUTURE WORK
ADD MBS UI CREATIONADD MBS UI CREATION
TEMPLATESTEMPLATES

52. FUTURE WORKFUTURE WORK
NEEDED FLOWSNEEDED FLOWS
Live storage migration
Disaster recovery
Storage migration between MBS to already
existing storage domain types
Monitoring
OVF related features
Upload / Download disk

53. FUTURE WORKFUTURE WORK
NEEDED FLOWSNEEDED FLOWS
Live storage migration
Disaster recovery
Storage migration between MBS to already
existing storage domain types
Monitoring
OVF related features
Upload / Download disk

54. LIVE DEMOLIVE DEMO

55. QUESTIONS?QUESTIONS?

56. LINKSLINKS
Feature page:
Cinderlib :
Driver requirements:
Drivers options:
https://ovirt.org/develop/release-management/features/storage/cinderlib-
integration.html
https://github.com/Akrog/cinderlib
https://github.com/openstack/cinder/blob/master/driver-requirements.txt
https://docs.openstack.org/cinder/rocky/configuration/block-storage/volume-
drivers.html

57. THANK YOU!THANK YOU!
Site: https://www.ovirt.org
Code: https://github.com/oVirt
IRC: #ovirt@oftc #vdsm@freenode
Mailing list: devel@ovirt.org