The document discusses DRBD, a tool for managing high-availability storage in many-node setups. It introduces the new drbdmanage tool, which handles complexity in multi-node DRBD clusters by managing nodes, resources, volumes and distributing configurations. Drbdmanage also works with OpenStack's Cinder volume service to provide highly-available block storage. Key features of DRBD 9 include multi-node replication, auto-promotion of resources, and a transport abstraction layer enabling high-speed RDMA connectivity.

1. DRBD
drbdmanage
Open Stack
Epilogue
DRBD9: Managing High-Available Storage
in Many-Node Setups
Roland Kammerer,
roland.kammerer@linbit.com,
LINBIT HA-Solutions GmbH
April 22, 2015
Roland Kammerer OSDC 2015 1/ 34

2. DRBD
drbdmanage
Open Stack
Epilogue
Agenda
1 DRBD
2 drbdmanage
3 Open Stack
4 Epilogue
Roland Kammerer OSDC 2015 2/ 34

3. DRBD
drbdmanage
Open Stack
Epilogue
DRBD 8.x
DRBD 9
What is DRBD?
Roland Kammerer OSDC 2015 3/ 34

4. DRBD
drbdmanage
Open Stack
Epilogue
DRBD 8.x
DRBD 9
Key Features of DRBD 8.x
• Automatic resync after node or connectivity failure
direction, amount, no full resync required
• High performance in Linux kernel implementation
160k IOPs measured (on SSDs of course)
• Multiple volumes per resource
Write order fidelity within resource
• Pacemaker integration
• Synchronous and asynchronous replication (LAN and WAN)
• In Linux upstream since 2.6.33 (released 2010)
Roland Kammerer OSDC 2015 4/ 34

5. DRBD
drbdmanage
Open Stack
Epilogue
DRBD 8.x
DRBD 9
Control Plane in DRBD 8.x
• You need to create/provide
block devices for DRBD
• You need to distribute
DRBD config files.
Roland Kammerer OSDC 2015 5/ 34

6. DRBD
drbdmanage
Open Stack
Epilogue
DRBD 8.x
DRBD 9
Promoting a Resource
$ drbdadm primary X
$ mount /dev/drbdY
• When you are done with your work
$ umount /dev/drbdY
$ drbdadm secondary X
• Go to step one on the second node. . .
Roland Kammerer OSDC 2015 6/ 34

7. DRBD
drbdmanage
Open Stack
Epilogue
DRBD 8.x
DRBD 9
Promoting a Resource
$ drbdadm primary X
$ mount /dev/drbdY
• When you are done with your work
$ umount /dev/drbdY
$ drbdadm secondary X
• Go to step one on the second node. . .
• Oh, you did not forget to create the meta-data in the first
place, and drbdadm up the resource, right?
Roland Kammerer OSDC 2015 6/ 34

8. DRBD
drbdmanage
Open Stack
Epilogue
DRBD 8.x
DRBD 9
Promoting a Resource
$ drbdadm primary X
$ mount /dev/drbdY
• When you are done with your work
$ umount /dev/drbdY
$ drbdadm secondary X
• Go to step one on the second node. . .
• Oh, you did not forget to create the meta-data in the first
place, and drbdadm up the resource, right?
• Promote/mount and umount/demote are the reasons you
have to special case DRBD (e.g., in pacemaker setups).
Roland Kammerer OSDC 2015 6/ 34

9. DRBD
drbdmanage
Open Stack
Epilogue
DRBD 8.x
DRBD 9
New Features of DRBD 9
• Multi-Node replication
• Up to 31 connections per resource
• Auto promote
• Transport abstraction layer
• Usual improvements all over the place (kernel/utils split, utils
that support all supported DRBD versions,
dpkg-reconfigure for utils, dkms for the kernel module, . . . )
Roland Kammerer OSDC 2015 7/ 34

10. DRBD
drbdmanage
Open Stack
Epilogue
DRBD 8.x
DRBD 9
Auto Promote
DRBD 8.x:
$ drbdadm primary X
$ mount /dev/drbdY
$ umount /dev/drbdY
$ drbdadm secondary X
DRBD 9:
$ mount /dev/drbdY
$ umount /dev/drbdY
Roland Kammerer OSDC 2015 8/ 34

11. DRBD
drbdmanage
Open Stack
Epilogue
DRBD 8.x
DRBD 9
Transport Abstraction
• Separates DRBD logic from the underlaying transport
• Provides an interface for new transports
• Main drbd.ko
and transports (e.g., drbd_transport_tcp.ko)
cat /proc/drbd
version: 9.0.0 rc2 (api :1/ proto :86 -110)
...
Transports (api :6): tcp (1.0.0)
Roland Kammerer OSDC 2015 9/ 34

12. DRBD
drbdmanage
Open Stack
Epilogue
DRBD 8.x
DRBD 9
Transport Abstraction
• Separates DRBD logic from the underlaying transport
• Provides an interface for new transports
• Main drbd.ko
and transports (e.g., drbd_transport_tcp.ko)
cat /proc/drbd
version: 9.0.0 rc2 (api :1/ proto :86 -110)
...
Transports (api :6): tcp (1.0.0)
So, what is the big deal?
Roland Kammerer OSDC 2015 9/ 34

13. DRBD
drbdmanage
Open Stack
Epilogue
DRBD 8.x
DRBD 9
RDMA
That is the big deal :
cat /proc/drbd
version: 9.0.0 rc2 (api :1/ proto :86 -110)
...
Transports (api :6): tcp (1.0.0) rdma (1.0.0)
Roland Kammerer OSDC 2015 10/ 34

14. DRBD
drbdmanage
Open Stack
Epilogue
DRBD 8.x
DRBD 9
RDMA
That is the big deal :
cat /proc/drbd
version: 9.0.0 rc2 (api :1/ proto :86 -110)
...
Transports (api :6): tcp (1.0.0) rdma (1.0.0)
Currently, we observe about 20 Gbit/s (≈2 GByte/s). More to
come soon, RDMA showed some bottlenecks in the core we were
not aware of (because TCP was slow enough). . .
Roland Kammerer OSDC 2015 10/ 34

15. DRBD
drbdmanage
Open Stack
Epilogue
DRBD 8.x
DRBD 9
Transport Layers to cover
I LINBIT.COM
YOUR WAY TO HIGH AVAILABILITY
TCP SCTP RDMA
SSOCKS SCTP iWARP RoCE
IPoIB
InfiniBandSCI
TCP
IP
Ethernet
Protocol
Transport
Medium
Hardware Dolphin Mellanox etc.Chelsio etc.several suppliers
New features of DRBD9New features of DRBD9
Roland Kammerer OSDC 2015 11/ 34

16. DRBD
drbdmanage
Open Stack
Epilogue
Control Plane
Example
Satellite Nodes
Agenda
1 DRBD
2 drbdmanage
Control Plane
Example
Satellite Nodes
3 Open Stack
4 Epilogue
Roland Kammerer OSDC 2015 12/ 34

17. DRBD
drbdmanage
Open Stack
Epilogue
Control Plane
Example
Satellite Nodes
Why do we need yet another tool?
• Autopromote solves one part of the complexity, but. . .
You still need to distribute your config files
You need to keep them in sync
That is fine in two node clusters, but now we have multiple
nodes.
Roland Kammerer OSDC 2015 13/ 34

18. DRBD
drbdmanage
Open Stack
Epilogue
Control Plane
Example
Satellite Nodes
Why do we need yet another tool?
• Autopromote solves one part of the complexity, but. . .
You still need to distribute your config files
You need to keep them in sync
That is fine in two node clusters, but now we have multiple
nodes.
⇒ We need a tool to. . .
• . . . handle the new complexity of multi-node setups.
• We can use it for additional features.
Roland Kammerer OSDC 2015 13/ 34

19. DRBD
drbdmanage
Open Stack
Epilogue
Control Plane
Example
Satellite Nodes
Requirements and Benefits
• What you need:
Nodes with an LVM VG (drbdpool)
• What you get:
Manges nodes in your cluster
Manges resources and volumes (including replica count)
Manges snapshots
Calls lvmtools
Volumes can be based on thinly provisioned LVM LVs
Distributes config and activates it (by using DRBD 9)
Implemented in Python
Scales to 1000s of nodes (via “Satellite Nodes”, WIP)
Roland Kammerer OSDC 2015 14/ 34

20. DRBD
drbdmanage
Open Stack
Epilogue
Control Plane
Example
Satellite Nodes
Control Plane in DRBD 9
Roland Kammerer OSDC 2015 15/ 34

21. DRBD
drbdmanage
Open Stack
Epilogue
Control Plane
Example
Satellite Nodes
Software Architecture
Roland Kammerer OSDC 2015 16/ 34

22. DRBD
drbdmanage
Open Stack
Epilogue
Control Plane
Example
Satellite Nodes
Our current State
Roland Kammerer OSDC 2015 17/ 34

23. DRBD
drbdmanage
Open Stack
Epilogue
Control Plane
Example
Satellite Nodes
Adding a Node
Roland Kammerer OSDC 2015 18/ 34

24. DRBD
drbdmanage
Open Stack
Epilogue
Control Plane
Example
Satellite Nodes
Rebalancing
Roland Kammerer OSDC 2015 19/ 34

25. DRBD
drbdmanage
Open Stack
Epilogue
Control Plane
Example
Satellite Nodes
Rebalancing finished
Roland Kammerer OSDC 2015 20/ 34

26. DRBD
drbdmanage
Open Stack
Epilogue
Control Plane
Example
Satellite Nodes
Using new Space
Roland Kammerer OSDC 2015 21/ 34

27. DRBD
drbdmanage
Open Stack
Epilogue
Control Plane
Example
Satellite Nodes
Satellite Nodes
• Core Nodes. . .
contain drbdcrtl
volume
contain logic
• Satellite Nodes. . .
no drbdctrl
just execute
commands
OK to shared DRBD res between satellite (and/or core) nodes
Roland Kammerer OSDC 2015 22/ 34

28. DRBD
drbdmanage
Open Stack
Epilogue
Agenda
1 DRBD
2 drbdmanage
3 Open Stack
4 Epilogue
Roland Kammerer OSDC 2015 23/ 34

29. DRBD
drbdmanage
Open Stack
Epilogue
DRBD in OpenStack
DRBD and drbdmange dock on the Cinder part of OpenStack.
Roland Kammerer OSDC 2015 24/ 34

30. DRBD
drbdmanage
Open Stack
Epilogue
Remember the Control Plane in DRBD 9?
Roland Kammerer OSDC 2015 25/ 34

31. DRBD
drbdmanage
Open Stack
Epilogue
drbdmanage Cinder Driver on a Cinder Node
Roland Kammerer OSDC 2015 26/ 34

32. DRBD
drbdmanage
Open Stack
Epilogue
drbdmanage Nova Driver on a Nova Node
If the nova node. . .
• . . . has a local replica of the volume
use it
• . . . does not have a local replica of the volume
it is a DRBD client
it is a diskless primary node that connects to secondaries that
hold the data
Roland Kammerer OSDC 2015 27/ 34

33. DRBD
drbdmanage
Open Stack
Epilogue
Big Vendor View
Roland Kammerer OSDC 2015 28/ 34

34. DRBD
drbdmanage
Open Stack
Epilogue
LINBIT View
Roland Kammerer OSDC 2015 29/ 34

35. DRBD
drbdmanage
Open Stack
Epilogue
Architecture
http://docs.openstack.org/juno/install-guide/install/
yum/content/ch_overview.html
Roland Kammerer OSDC 2015 30/ 34

36. DRBD
drbdmanage
Open Stack
Epilogue
OpenStack Nova vs. Cinder
Low latency storage access possible by aligning Nova and Cinder
allocations.
Roland Kammerer OSDC 2015 31/ 34

37. DRBD
drbdmanage
Open Stack
Epilogue
Get it now!
• http://drbd.linbit.com
• http://oss.linbit.com
• http://git.drbd.org
• To get access to our deb/rpm repos: office@linbit.com
Roland Kammerer OSDC 2015 32/ 34

38. DRBD
drbdmanage
Open Stack
Epilogue
Q&A
Are there any questions?
I will be around, you can also drop me a mail at
roland.kammerer@linbit.com
Roland Kammerer OSDC 2015 33/ 34

39. DRBD
drbdmanage
Open Stack
Epilogue
License
This work is licensed under a
Creative Commons “Attribution-
NonCommercial-ShareAlike 4.0
International ” license.
Roland Kammerer OSDC 2015 34/ 34