The document summarizes updates and improvements to OpenStack storage services. Key points include: - Efforts to improve high availability of Cinder APIs and services, including multipath support and active/active deployments. - Enhancements for volume management such as attaching single volumes to multiple hosts, improved volume migration, and retype functionality. - Updates to backup services including incremental backups, support for additional targets like NFS/POSIX, and improved integration with Swift. - Disaster recovery features like consistency group enhancements and planned import/export of snapshots between Cinder installations. - Work on deployment and rolling upgrades, including database cleanup utilities, object-based communication, and standardized driver

1. The Road to Enterprise-Ready
OpenStack Storage as Service

2. Today’s Presenters
Sean Cohen Jon Bernard Flavio Percoco
Principal Product
Manager
Red Hat
Senior Software
Engineer
Red Hat
Senior Software
Engineer
Red Hat
2

3. THANK YOU
High Availability

4. DIVIDERHigh Availability
● High Availability of APIs/Services Goals
○ All services that power the OpenStack APIs should
be always on, and able to always respond even
during failure and massive stress.
○ Provide protection against HW & SW SPOFs (Single
Points of Failure)

5. DIVIDERHigh Availability
● Where are we today?
○ There are cases where the volume is left in
unrecoverable state and it is not possible to delete
the volume without administrator’s intervention
○ If a cinder-volume node dies during volume create
request processing for example, the volume will be in
unresolved state.

6. DIVIDERHigh Availability
● Where are we today?
○ Cinder volume service run in Active/Active state
which is not safe
■ Non-Atomic state transitions in the API may end in
race conditions.
■ Consistent replications of these nodes are currently
not possible.

7. DIVIDERHigh Availability... new in Kilo
● Cinder - iSCSI Multipath
■ nova-compute support multipath for iSCSI volume data
path. However, some arrays only respond to discovery
with a single portal address, even if secondary portals
are available.
■ Cinder now can return multiple iSCSI paths
information so that the connector can attach volumes
even when the primary path is down.
■ Cinder side was completed in Kilo, while the Nova
enablement work is still ahead.

8. Instance Migration
Horizon - Migrate all instances from host
● Allow administrators to migrate all instances from host marked
for maintenance via Horizon in a “Push button” fashion as
available in command line.
● “Migrate instances” button allow administrators to use simpler
way of preparing host for maintenance actions in Horizon
○ Useful in upgrades scenarios.
○ Test/perform manual disaster recovery.

10. DIVIDERThe Road to Active-Active
● Cinder State Enforcer
○ Long standing work to improve Cinder volume’s states
management and reliability, and to improve failure
tolerance.
○ In order to mitigate the concurrent resource access
problems in Cinder, work was done in the last cycles to
refactor the concept of a lock to be a set of allowed and
disallowed state transitions (instead of acquiring local
filesystem locks in the manager processes) by
implementing a new `enforcer` model.

11. DIVIDERThe Road to Active-Active
● Active / Active cinder-volume
○ Effort in Liberty cycle to address issues around:
■ Local file locks in cinder-volume - need to enhance the
lock reporting to Nova based on the volume active
state.
■ DB accesses in drivers - need to be minimized or
limited at all.
■ Nova is inspecting internal state of cinder volumes to
determine if it can take an action, rather than properly
delegating the attach/detach work.

12. DIVIDERThe Road to Active-Active
● Task Flow for managing create volumes tasks
○ There are few corner cases where the volume is left in
unrecoverable state and it is not possible to delete the
volume without administrator’s intervention.
○ The improvements of state management can get us a step
closer to Active-Active safe operations.
○ The road to Active-Active should be spreading the use of
state management (taskflows) to to cover Cinder
operations beyond volume creation tasks

13. DIVIDERVolume Management
● Cinder - Attach a single volume to multiple hosts
○ In order to support Hypervisor/Application clusters level, a
single volume would need to be exported to multiple host.
○ The patch that adds the multiattach flag to volumes was
merged in Cinder during the Kilo release.
○ However we are still missing the Nova and python-
cinderclient patches to provide support for multiple
attachments.

14. DIVIDERVolume Management
● Volume Migration - Retype initiated
○ One of the biggest problems around volume migration is the
confusion around volume migration and retype.
■ Volume Retype will trigger a migration only if a user has
requested it (not by default)

15. DIVIDERVolume Management in Liberty
● Volume Migration with file I/O instead of iSCSI
attachment
○ Currently when migrating a volume between two backends,
the copy_volume_data routine in the source volume's driver
is executed to move the blocks from one volume to another.
This routine assumes that both source and destination
volumes can be attached locally (e.g. iSCSI)
○ Add the ability to migrate volumes of drivers that don’t
support iSCSI such as Ceph RBD.

16. Business Continuity

17. Backup improvements
Incremental backup
● Cinder Backup API was extended to support snapshot based
backups, where the volume can remain online and in-use for
the duration of the operation. (Swift or NFS target)
○ The enhancement also included performing a backup from a
snapshots.
○ New cinder backup CLI was added: --incremental or --incr
○ swift.py creates sha256 file for every backup to calculate deltas
○ During restore, if a differential backup needs to be restored, the
restore process first restores the full backup.

18. Backup improvements
NFS & POSIX Backup
• Cinder Backup has now support to use NFS/POSIX supplied
data repository as backup target with two new drivers in place.
Backup Support for Encrypted Volumes
• The Cinder backup includes now a clone of the volume's
encryption key UUID so that the encryption key is available
when the backup is restored.

19. Backup improvements
Nova - Support for quiescing file-systems during image
snapshot
• Using QEMU guest agent
• With this new feature, users can create a snapshot image with
consistent file systems state while the instances are running
(it requires QEMU Guest Agent to be installed in a KVM
instance).
• Useful for taking a quick backup before installing or upgrading
softwares / Can set to run automatically every night etc.

20. Backup improvements
Swift - Erasure Coding
• Erasure coding is a storage policy designed to reduce
storage costs associated with massive amounts of data (by
providing an option that maintains the same, or better, level of
durability using much less disk space)
• Can be very useful when performing volume backup to a Swift
object storage system, as backups are typically large
compressed objects and are infrequently read once they have
been written to the storage system.

21. DIVIDERBackup improvements… in Liberty
● Cinder - Scaling Backup Service
○ Currently the Backup service must scale up rather than out.
○ The Backup service and cinder volume drivers are coupled
so that all must run together on a single node - By breaking
the coupling between backup service and volume drivers, it
will allow the service to scale out.
● Swift - Fast Posting
○ Where a POST to an object will trigger a container update.
○ Allow for updating objects metadata through POST
semantics and still guarantee data consistency in the
container.

22. Disaster recovery

23. Disaster Recovery
Cinder - Consistency groups enhancements
• Added the ability to add/remove volumes from an existing
consistency group.
• Added the ability to create a consistency group from an
existing consistency group snapshot.
• Support creation of a cg_volume types table (to overcome the
limitation of the current solution which stores all volume type
uuids in one column of the CG table).

24. Disaster Recovery… in Liberty
The OpenStack snapshot mechanism allows you to create new
images from running instances. This is very convenient for
upgrading base images or for taking a published image and
customizing it for local use. But what about external use?
Cinder - Import/Export snapshots
● Allows to import volumes snapshot from one Cinder to
another.
● Allows to import "non" openstack snapshots already on a
backend-device. Where, export snapshots should work the
same way as export volumes.

25. Disaster Recovery… in Liberty
Cinder - Volume Replication V2
● Things we’ve missed in V1:
■ Replication between Cinders
● Currently we have basic replication in a single
Cinder deployment.
■ Consistency data replication
● Align CG design and volume-replication spec, one
CG could support different volume-types, where the
volume-type to decide which volume-replication is
going to be created and added to CG.

26. DIVIDERDeployment & Rolling Upgrades
● Image Introspection
○ A new task has been added to Glance’s v2, which makes
it possible for introspecting image’s metadata and
populate it.
● Image Conversion
○ A new task has been added to Glance’s v2, which makes
it possible for converting images on import.
○ Useful to unify stored image types and use a type that
works better with the hypervisor and the storage backend.
○ Current supported formats are: raw <-> qcow2

27. Deployment & Rolling Upgrades

28. DIVIDERDeployment & Rolling Upgrades
● Cinder DB Purge Utility
○ Very long lived Openstack installations will carry around
database rows for years and years.
○ Operators need to have the ability to purge deleted rows,
possibly on a schedule (cron job) or as needed before an
upgrade, prior to maintenance.
○ The new utility allows you to clean up rows that are
already marked as deleted of a certain specified age.
■ The age is calculated as timedelta in days, which are
given at command line.

29. DIVIDERDeployment & Rolling Upgrades
● Implement force_detach to allow safe cleanup of
stuck volumes
○ For volumes stuck in 'attaching' or 'detaching' , there is no
safe way to cleanup that involves the backend storage.
○ Using python-cinderclient 'reset-state' will only change the
Cinder database, and may leave the volume exported to
the compute host, and may leave an entry in Nova's
database that prevents the volume from being re-used.
○ This also need to be addressed by the Nova side.

30. DIVIDERDeployment & Rolling Upgrades … in Liberty
● Cinder Objects
○ Supporting rolling upgrades by using versioned objects.
○ These objects are isolated from the schema and contain
the required information for communications and
operations.
○ These objects can be sent over RPC.
○ Work started in Kilo.

31. DIVIDERDeployment & Rolling Upgrades … in Liberty
● Cinder Storage Policies - Standard Capabilities
○ Goals:
■ Provide standard capabilities from drivers that the cloud
administrator can specify from volume types.
■ Improve the visibility of what policies are possible with your
storage solution via Cinder client and Horizon.
○ Capabilities need to be exposed to the admin from Cinder, so that
Cinder is not limiting what storage backends can do.
■ These capabilities can include, but are not limited to QoS,
replication factor, bandwidth control, etc.

32. Security

33. DIVIDERSecurity
● Cinder - Private Volume Types
○ With the new Cinder ability for defining private volume types, as
some volume types should only be restricted.
■ Private volumes for special needs where most users should
not be able to select these volumes.
■ Volume types are public by default
■ Private volume types can be created by setting the is_public
boolean field to False at creation time.
■ Access to a private volume type can be controlled by adding
or removing a project from it.

34. DIVIDERSecurity… in Liberty
● Glance - Image Signing and Encryption
○ Right now, there is no way to guarantee that image you asked Glance
for is the image you got in Nova.
○ This feature has been discussed in the past and it looks like it’ll finally
happen.
○ Image signing and encryption using Barbican as a key manager.
○ The goal is to guarantee image’s integrity.
● Horizon - Volume Encryption
■ Support for volume encryption through Horizon is almost there.
Some of the work is done but it was moved out of Kilo at the very
end.

35. DIVIDERSecurity… in Liberty
● Swift - Encryption At Rest
○ Currently objects are typically stored on disk as files in a standard
POSIX filesystem.
○ Provide option for Swift operators to have objects stored in an
encrypted form.
○ When disks reach end-of-life, they are discarded, and if not properly
wiped, may still contain data.
○ Swift will use AES in CTR mode with 256-bit keys, where the entire
object is encrypted as a single byte stream, as well as user
metadata with the same key.
○ Swift will probably want a keymaster that stores things in Barbican
at some point.

36. DIVIDERSecurity… in Liberty
● Swift - Composite Tokens & Service Accounts
○ Composite tokens allow other OpenStack services to store data
in Swift on behalf of a client so that neither the client nor the
service can update the data without both parties consent.
○ Example:
■ User requests that Nova save a snapshot of a VM.
■ Nova passes the request to Glance
■ Glance writes the image to a Swift container as a set of objects.
■ The user cannot modify the snapshot without also having a valid token
from the service.
■ Nor can the service update the data without a valid token from the
user.

37. Q & A