The document discusses deploying Cinder, an OpenStack block storage service, emphasizing the need for a scalable and resilient setup with limited downtime and reduced administrative intervention. It covers high-level architecture, best practices for high availability, and command flow examples for operations like volume creation and extension. The document also highlights recent updates and future directions for Cinder, including improvements in driver stability and ongoing community efforts to address various issues.

1. Deploying Cinder in Production
Avishay Traeger, PhD
Shimshon Zimmerman

2. Copyright 2015
Goals
Deploy Cinder in a way that is scalable and resilient to failures
❖ Limited downtime
❖ Reduce admin intervention

3. Copyright 2015
Available resources
Good documentation exists for
❖ Deploying OpenStack
❖ Many OpenStack HA Guides, for example: http://docs.
openstack.org/high-availability-guide

4. Copyright 2015
What we’ll cover today
❖ What is Cinder?
❖ How is it designed?
❖ What issues in Cinder impact deploying with HA?
❖ How can I best deploy Cinder today?
❖ HA Cinder - live demo
❖ Cinder and Neutron spun off of Nova
➢ Similar architectures
➢ Some lessons from Cinder may be generalized

5. Copyright 2015
Who are we?
❖ Stratoscale R&D
❖ Former Cinder core member
A couple words on what we do at Stratoscale
❖ Hyper-converged infrastructure:
➢ Compute: customized KVM & docker
➢ Storage: high performance scale out block storage
➢ Network: full-featured SDN
❖ Management plane based on OpenStack
❖ Easy install, easy upgrade, no maintenance
Sound interesting? We’re hiring!

6. Copyright 2015
What is Cinder?
❖ Abstraction that enables uniform management of block storage
➢ Exposes northbound API (e.g., create, list, delete volumes
and snapshots)
➢ Storage “drivers” implement southbound API
❖ Enables connections between Nova and storage

7. Copyright 2015
High-level architecture
cinder client
cinder-api
cinder-volume
driver
cinder-scheduler
cinder-backup
driver
storage
REST
SQL DB
Components to make HA:
● Storage
● Database
● RPC messaging
● Cinder services

8. Copyright 2015
HA storage
❖ Almost all storage supported by Cinder is HA
❖ These may not be - by default they use local disks (SPOF)
➢ LVM: Sets up an iSCSI target over LVM on device
➢ NFS: Sets up an NFS server on a file system
❖ Must make sure you have redundant network paths from
compute to the storage

9. Copyright 2015
HA database
❖ An SQL DB is used to store OpenStack metadata
➢ For example, in Cinder, information about volumes,
snapshots, quotas, volume types, etc.
❖ This DB must be replicated
➢ Galera + MySQL/MariaDB/Percona
➢ PostgreSQL replication
➢ Store DB on DRBD backend
❖ Scale-out architectures for SQL exist, but not been tried with
OpenStack as far as we know:
➢ https://github.com/youtube/vitess

10. Copyright 2015
Inter-service messaging
OpenStack projects use oslo messaging for RPCs which wraps
❖ RabbitMQ (AMQP)
➢ By default queues are located on a single node (SPOF)
➢ Configure mirroring: https://www.rabbitmq.com/ha.html
❖ Qpid (AMQP)
➢ Similar to RabbitMQ (configure queue replicas)
❖ ZeroMQ
➢ Allows broker-based reliability like the others
➢ Also allows brokerless peer-to-peer model

11. Copyright 2015
HA Cinder services
Great open source project: www.consul.io
❖ Service discovery: DNS or HTTP interface
❖ Health checking for services and hosts
❖ Key-value store
❖ Scalable, multi-datacenter
Other solutions exist (etcd, zookeeper, pacemaker).

12. Copyright 2015
HA
management
Our Solution
OpenStack services
RabbitMQ
Galera &
MariaDB
consul
service discovery

13. Copyright 2015
Command flow example 1: create
cinder-api:
create DB
record
call cinder-
scheduler
(RPC)
scheduler
chooses
backend
call cinder-
volume
(RPC)
cinder-
volume
works
driver
creates
volume on
storage
DB update
→available

14. Copyright 2015
Command flow example 1: create
cinder-api:
create DB
record
call cinder-
scheduler
(RPC)
scheduler
chooses
backend
call cinder-
volume
(RPC)
cinder-
volume
works
driver
creates
volume on
storage
DB update
→available

15. Copyright 2015
Command flow example 1: create
cinder-api:
create DB
record
call cinder-
scheduler
(RPC)
scheduler
chooses
backend
call cinder-
volume
(RPC)
cinder-
volume
works
driver
creates
volume on
storage
DB update
→available

16. Copyright 2015
Command flow example 2: extend
cinder-api:
check
volume
state and
update
call cinder-
volume
(RPC)
cinder-
volume
works
driver
creates
volume on
storage
DB update
→available

17. Copyright 2015
Command flow example 2: extend
cinder-api:
check
volume
state and
update
call cinder-
volume
(RPC)
cinder-
volume
works
driver
creates
volume on
storage
DB update
→available

18. Copyright 2015
Best practices with Juno
❖ Run cinder-api in active/active mode with a load balancer in front
➢ If you are worried about two processes modifying the same volume
simultaneously, you can work around it with UUID-based routing and
local file locks
❖ Run cinder-scheduler in active/active mode
❖ One cinder-volume per backend in active/passive mode
➢ Cannot run active/active because of local file locks
❖ Make sure you clean up the DB - objects in transient
states and deleted objects that are not purged

19. Copyright 2015
Fresh in Kilo
Kilo was just released
❖ Lots of work on driver stability, including CI
❖ Multi-attaching volumes has been merged into
Cinder, but unfortunately the Nova bits haven’t
gone in - will wait for Liberty
❖ Support for incremental backups, additional
consistency group APIs

20. Copyright 2015
Liberty and beyond
❖ The community is aware of the issues raised here
❖ Some more localized issues like atomic state transitions should
be addressed in Liberty
❖ Recovery and maintaining consistency is a problem with no clear
roadmap at this point

21. Thank You
Avishay Traeger, PhD
Shimshon Zimmerman