This presentation is on the DRBD product. At eNovance, we're using it for several years. In those slides, you will find informations on how we use it, use cases and Ninja tricks.
This document has been realized with a lot of feedbacks and thanks to strong knowledges on that technology that eNovance is able to provide.
3. DRBD: Introduction
How it works
DRBD refers to block devices designed as a building block to form high availability
(HA) clusters. This is done by mirroring a whole block device via an assigned network.
DRBD can be seen as network based raid-1.
Pierre Mavro www.enovance.com
DRBD 3 / 62
4. DRBD: Introduction
Synchronisations
DRBD supports three distinct replication modes, allowing three degrees of replication :
A (Asynchronous replication) : Master node disk write + local TCP send
buffer → operation completed
B (Semi synchronous replication) : Master node disk write + replication
packets reached other node → operation completed
C (Synchronous replication protocol) : Written to both the local and the
remote disk to get operation completed
The Syncronous replication protocol is the most robust protocol tested in
production.
http://www.drbd.org/home/mirroring/
Pierre Mavro www.enovance.com
DRBD 4 / 62
5. DRBD: Introduction
Data accessibility
A consequence of mirroring data on block device level is that you can access your data
(using a file system) only on the active node. This is not a shortcoming of DRBD but
is caused by the nature of most file systems (ext3, XFS, JFS, ext4, ...).
These file systems are designed for one computer accessing one disk, so they
cannot cope with two computers accessing one (virtually) shared disk.
In spite of this limitation, there are still a few ways to access the data on the second
node:
Use DRBD on logical volumes and use LVM’s capabilities to take snapshots on
the standby node, and access the data via the snapshot.
DRBD’s primary-primary mode with a shared disk file system (GFS, OCFS2).
These systems are very sensitive to failures of the replication network.
Mount in read only mode the partition
We are using DRBD in master/slave at eNovance.
Pierre Mavro www.enovance.com
DRBD 5 / 62
7. DRBD: Basic usages and understandings
Configuration
The DRBD configuration should be the same on both nodes. This drbd.conf file is
the global configuration of the DRBD service with fine tuning configuration :
/etc/drbd.conf
global { <-- Global configuration
usage-count no; <-- Do not report statistics usage to LinBit
}
common { <-- All resources inherit the options set in this section
protocol C; <-- C (Synchronous replication protocol)
startup {
wfc-timeout 1 ; <-- Wait for connection timeout (in seconds)
degr-wfc-timeout 1 ; <-- Wait for connection timeout, if this node was a degraded
} cluster (in seconds)
Pierre Mavro www.enovance.com
DRBD 7 / 62
8. DRBD: Basic usages and understandings
Configuration
/etc/drbd.conf
net {
max-buffers 8192; <-- Maximum number of requests to be allocated by DRBD
max-epoch-size 8192; <-- The highest number of data blocks between two write barriers
sndbuf-size 512k; <-- The size of the TCP socket send buffer
unplug-watermark 8192; <-- how often the I/O subsystem’s controller is forced to
process pending I/O requests
cram-hmac-alg sha1; <-- The HMAC algorithm to enable peer authentication at all
shared-secret "xxx"; <-- The shared secret used in peer authentication
# Split brains
after-sb-0pri disconnect; <-- Split brain, resource is not in the Primary role on any host
after-sb-1pri disconnect; <-- Split brain, resource is in the Primary role on one host
after-sb-2pri disconnect; <-- Split brain, resource is in the Primary role on both host
rr-conflict disconnect; <-- Helps to solve the cases when the outcome of the resync
} decision is incompatible with the current role assignment
handlers {
pri-on-incon-degr "echo node is primary, degraded and the local copy of the data is
inconsistent | wall "; <-- If the node is primary, degraded and if the
} local copy of the data is inconsistent
Pierre Mavro www.enovance.com
DRBD 8 / 62
9. DRBD: Basic usages and understandings
Configuration
/etc/drbd.conf
disk {
on-io-error pass_on; <-- The node downgrades the disk status to inconsistent on io errors
no-disk-barrier; <-- Disable protecting data if power failure (done by hardware)
no-disk-flushes; <-- Disable the backing device to support disk flushes
no-disk-drain; <-- Do not let write requests drain before write requests of a new
reordering domain are issued
no-md-flushes; <-- Disables the use of disk flushes and barrier BIOs when accessing
} the meta data device
syncer {
rate 300M; <-- The maximum bandwidth a resource uses for background
re-synchronization
al-extents 3833; <-- Control how big the hot area (= active set) can get
}
}
Pierre Mavro www.enovance.com
DRBD 9 / 62
10. DRBD: Basic usages and understandings
Configuration
Now the resources can be defined. One resource should match a DRBD device. Both
nodes informations should be filled :
/etc/drbd.d/resources.conf
resource drbd1 { <-- DRBD block device name
syncer {after drbd0 } <-- Start drbd1 after drbd0 is up & running
on srv1 { <-- Master node name
device /dev/drbd1; <-- Block device name of the resource being described
disk /dev/sda1; <-- Block device to store and retrieve the data
address x.x.x.x:7789; <-- IP address and port of the local host
meta-disk internal; <-- The last part of the backing device is used to store
the meta-data
}
on srv2 { <-- Slave node name
device /dev/drbd2;
disk /dev/sda1;
address y.y.y.y:7789;
meta-disk internal;
}
}
Pierre Mavro www.enovance.com
DRBD 10 / 62
11. DRBD: Basic usages and understandings
Initialisation
Here is a summary to understand the process to create a drbd replication. You need
to follow that rule to get a working DRBD synchronization :
Init metadata
device
Attach
DRBD device
Connect
DRBD device
Start syn-
chronization
Pierre Mavro www.enovance.com
DRBD 11 / 62
12. DRBD: Basic usages and understandings
Init metadata device
This step must be completed only on initial device creation. It initializes DRBD’s
metadata (replace <drbd_volume> by the drbd device name you want to initialize).
You should complete those steps on both nodes :
Create device metadata on server 1
srv1~$ drbdadm create -md <drbd_volume >
Writing meta data ...
initialising activity log
NOT initializing bitmap
New drbd meta data block sucessfully created.
Create device metadata on server 2
srv2~$ drbdadm create -md <drbd_volume >
...
Pierre Mavro www.enovance.com
DRBD 12 / 62
13. DRBD: Basic usages and understandings
Attach and connect DRBD device
Attach a local backing block device to the DRBD resource’s device :
Attach DRBD device
$ drbdadm attach <drbd_volume >
If the peer device is already configured, the two DRBD devices will connect :
Sets up the network configuration of the resource’s device
$ drbdadm connect <drbd_volume >
Pierre Mavro www.enovance.com
DRBD 13 / 62
14. DRBD: Basic usages and understandings
Start synchronization
To start the first synchronization, you need to ask to the first server to sync all blocks
to the secondary server.
This will erase all blocks on the other server, so all data will be lost :
Start the first synchronization
$ drbdadm -- --overwrite -data -of -peer primary <drbd_volume >
This action may take a while, depending on the network bandwidth and DRBD
volume size.
Pierre Mavro www.enovance.com
DRBD 14 / 62
16. DRBD: Basic usages and understandings
Check replication status
To check the replication status, simply look at DRBD file in /proc :
Check
$ cat /proc/drbd
version: 8.3.7 (api :88/ proto :86 -91)
srcversion: EE47D8BF18AC166BE219757
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/ Inconsistent C r.
ns :912248 nr:0 dw:0 dr :920640 al:0 bm :55 lo:1 pe :388 ua :2048 ap:0
[=== >................] sync ’ed: 21.9% (3283604/4194304) K
finish: 1:08:24 speed: 580 (452) K/sec
Pierre Mavro www.enovance.com
DRBD 16 / 62
17. DRBD: Basic usages and understandings
Check replication status
Another solution consists to launch this command that will provide informations of all
DRBD devices :
Check
$ drbd -overview
0: home Connected Primary/Secondary
UpToDate/UpToDate C r--- /home xfs 200G 158G 43G 79%
Pierre Mavro www.enovance.com
DRBD 17 / 62
18. DRBD: Basic usages and understandings
Check replication status
Several informations need be checked to know the status replication of a DRBD
device :
Replication
status
Connection
state (cs)
Resources
roles (ro)
Disk
states (ds)
Pierre Mavro www.enovance.com
DRBD 18 / 62
19. DRBD: Basic usages and understandings
Check replication status
Connection states (cs) :
StandAlone : No network configuration available. The resource has not yet
been connected, has been administratively disconnected or has dropped its
connection due to failed authentication/split brain.
Unconnected : Temporary state, prior to a connection attempt.
WFConnection : This node is waiting until the peer node becomes visible on
the network.
Connected : A DRBD connection has been established, data mirroring is now
active. This is the normal state.
PausedSync : The local node is the source or target of an ongoing
synchronization, but synchronization is currently paused. This may be due to a
dependency on the completion of another synchronization process, or due to
synchronization having been manually interrupted.
Pierre Mavro www.enovance.com
DRBD 19 / 62
20. DRBD: Basic usages and understandings
Check replication status
Resource roles (ro) :
Primary : The resource is currently in the primary role, and may be read from
and written to. This role only occurs on one of the two nodes, unless
dual-primary mode is enabled.
Secondary : The resource is currently in the secondary role. It normally receives
updates from its peer (unless running in disconnected mode), but may neither be
read from nor written to. This role may occur on one or both nodes.
Unknown : The resource’s role is currently unknown. The local resource role
never has this status. It is only displayed for the peer’s resource role, and only in
disconnected mode.
Pierre Mavro www.enovance.com
DRBD 20 / 62
21. DRBD: Basic usages and understandings
Check replication status
Disk states (ds) :
UpToDate : Consistent, up-to-date state of the data. This is the normal state.
Inconsistent : The data is inconsistent. This status occurs immediately upon
creation of a new resource, on both nodes (before the initial full sync). Also, this
status is found in one node (the synchronization target) during synchronization.
Pierre Mavro www.enovance.com
DRBD 21 / 62
23. DRBD: Basic usages and understandings
Node switching
You may need to perform maintenance tasks. For instance you need to switch a
DRBD volume to the other node.
Simply connect to the secondary node and launch :
Set secondary as primary
$ drbdadm primary <drbd_volume >
You can also switch all volumes at once. Double check if you really can do this
before proceed :
Set secondary as primary
$ drbdadm primary all
Pierre Mavro www.enovance.com
DRBD 23 / 62
24. DRBD: Basic usages and understandings
Node switching
You also can switch a Primary node to Secondary :
Set primary as secondary
$ drbdadm secondary <drbd_volume >
Notes
This will not automatically set the old secondary node to primary state
Pierre Mavro www.enovance.com
DRBD 24 / 62
25. DRBD: Basic usages and understandings
Manual synchronization
To start a manual synchronization, you need to invalidate the DRBD device on the
current host :
Invalidate data on current host
$ drbdadm invalidate <drbd_volume >
You can do the same from one node to the other node :
Invalidate data on remote host
$ drbdadm invalidate_remote <drbd_volume >
You can then check the status in /proc/drbd
Pierre Mavro www.enovance.com
DRBD 25 / 62
27. DRBD: Basic usages and understandings
Remove a DRBD device
Before removing a DRBD device, be sure you’ve dumped all data ! Then disconnect
the device on both nodes :
Disconnect DRBD device
$ drbdadm disconnect <drbd_volume >
And delete it :
Remove DRBD device
$ drbdsetup <volume_number > down
Change volume_number by the number seen with drbd-overview command. To finish,
remove the config files.
Pierre Mavro www.enovance.com
DRBD 27 / 62
29. DRBD: Use cases
Stop, upgrade and commit changes
Sometimes, you may need to perform maintenances tasks on a DRBD device. For
example, you need to upgrade stuffs on a DRBD device and if success, commit
changes to the secondary :
DRBD Primary DRBD Secondary
Set secondary node
in maintenance mode
Resume sync
Upgrade success
Normal state
Pierre Mavro www.enovance.com
DRBD 29 / 62
30. DRBD: Use cases
Stop, upgrade and commit changes
To set the secondary node in maintenance, simply stop the synchronisation on the
secondary node :
Disconnect drbd device
$ drbdadm disconnect <drbd_volume >
Do what you have to do on the DRBD primary device and once you have finished and
everything looks fine, resume the synchronization to get back to the normal state :
Connect drbd device
$ drbdadm connect <drbd_volume >
Pierre Mavro www.enovance.com
DRBD 30 / 62
32. DRBD: Use cases
Stop, upgrade and rollback changes
Sometimes, you may need to perform maintenances tasks on a DRBD device. For
example, you need to upgrade stuffs on a DRBD device and if fail, rollback changes
from the secondary :
DRBD Primary DRBD Secondary
Set secondary node
in maintenance mode
Upgrade failed
Normal state
Rollback sync
Pierre Mavro www.enovance.com
DRBD 32 / 62
33. DRBD: Use cases
Stop, upgrade and rollback changes
To set the secondary node in maintenance, simply stop the synchronisation on the
secondary node :
Disconnect drbd device
$ drbdadm disconnect <drbd_volume >
Then you’ve upgraded and want to rollback. You have to switch your secondary node
to master node :
Promote device as primary
$ drbdadm primary <drbd_volume >
Pierre Mavro www.enovance.com
DRBD 33 / 62
34. DRBD: Use cases
Stop, upgrade and rollback changes
Then you need to invalidate the datas on the old primary :
Invalidate drbd device data
$ drbdadm invalidate <drbd_volume >
To finish, adjust and connect the device to perform the rollback on the new master
node :
Adjust and connect
$ drbdadm adjust <drbd_volume >
$ drbdadm connect <drbd_volume >
Pierre Mavro www.enovance.com
DRBD 34 / 62
36. DRBD: Use cases
Up, connect...but doesn’t want to come UpToDate
If you don’t understand why after several up, connect etc.... you don’t see anything
changing and still got this kind of event :
Check
$ drbd -overview
0: drbd0 StandAlone Primary/Unknown UpToDate/DUnknown r-----
You need to instigate more. The first thing to do is to look at the logs :
/var/log/syslog
[ 6375.849509] block drbd0: Split-Brain detected but unresolved, dropping connection!
[ 6375.850025] block drbd0: helper command: /sbin/drbdadm split-brain minor-0
[ 6375.852859] block drbd0: helper command: /sbin/drbdadm split-brain minor-0 exit code 0 (0x0)
[ 6375.852867] block drbd0: conn( WFReportParams -> Disconnecting )
[ 6375.852873] block drbd0: error receiving ReportState, l: 4!
[ 6375.853274] block drbd0: asender terminated
[ 6375.853277] block drbd0: Terminating drbd0_asender
[ 6375.853506] block drbd0: Connection closed
[ 6375.853514] block drbd0: conn( Disconnecting -> StandAlone )
[ 6375.853523] block drbd0: receiver terminated
[ 6375.853526] block drbd0: Terminating drbd0_receiver
Pierre Mavro www.enovance.com
DRBD 36 / 62
37. DRBD: Use cases
Up, connect...but doesn’t want to come UpToDate
Anyway, it may not be clear enough for you. You can use that command to get a
better understanding on the situation :
Show all drbd informations
$ drbdadm show -gi <drbd_volume >
+--< Current data generation UUID >-
| +--< Bitmap ’s base data generation UUID >-
| | +--< younger history UUID >-
| | | +-< older history >-
V V V V
B63AABEFCFADD87D :2 C30515E7D05B35D :82 C4146DCB8E6BFF :82 C3146DCB8E6BFF :1:1:1:0:0:0:0
^ ^ ^ ^ ^ ^ ^
-< Data consistency flag >--+ | | | | | |
-< Data was/is currently up -to -date >--+ | | | | |
-< Node was/is currently primary >--+ | | | |
-< Node was/is currently connected >--+ | | |
-< Node was in the progress of setting all bits in the bitmap >--+ | |
-< The peer ’s disk was out -dated or inconsistent >--+ |
-< This node was a crashed primary , and has not seen its peer since >--+
flags: Primary , StandAlone , UpToDate
Pierre Mavro www.enovance.com
DRBD 37 / 62
38. DRBD: Use cases
Up, connect...but doesn’t want to come UpToDate
Here the problem is clear : we’re in a split brain situation. So there is no other
solutions than doing a full sync.
But what if no split brain was detected ? There’s a last solution :
Adjust and connect the drbd device
$ drbdadm adjust <drbd_volume >
$ drbdadm connect <drbd_volume >
Now you can check the replication which is resuming.
Pierre Mavro www.enovance.com
DRBD 38 / 62
40. DRBD: Use cases
How to promote a primary when having a dual secondary
If you get the dual secondary state after node’s restart, check which node as the
’UpToDate’ state :
Check
$ drbd -overview
0: drbd0 Connected Secondary/Secondary UpToDate/UpToDate C r-----
Here, both are up to date. Simply select one and promote it as primary :
Set as primary
$ drbdadm primary <drbd_volume >
You should now see one primary and a secondary synced.
Pierre Mavro www.enovance.com
DRBD 40 / 62
42. DRBD: Ninja tricks
Create DRBD replication without syncing data
When you have big devices to sync on a DRBD initialization, it could take several
hours or days in worst cases. There is a solution to only sync bitmap and have your
DRBD device available in a few seconds instead !
Create on both nodes the DRBD device :
Create device metadata
$ drbdadm create -md <drbd_volume >
Writing meta data ...
initialising activity log
NOT initializing bitmap
New drbd meta data block sucessfully created.
Pierre Mavro www.enovance.com
DRBD 42 / 62
43. DRBD: Ninja tricks
Create DRBD replication without syncing data
Then bring it up (still on both nodes) :
Connect the volume
$ drbdadm up <drbd_volume >
Only on the master node, initialize the bitmap and uuid of the DRBD device :
Only sync bitmap
$ drbdadm -- --clear -bitmap new -current -uuid <drbd_volume >
Pierre Mavro www.enovance.com
DRBD 43 / 62
44. DRBD: Ninja tricks
Create DRBD replication without syncing data
Promote this master node as primary node to replicate bitmap :
Set primary volume
$ drbdadm primary <drbd_volume >
Now you can check the DRBD is ready to use :-)
Check
$ drbd -overview
1: drbd1 Connected Primary/Secondary UpToDate/UpToDate C r-----
Pierre Mavro www.enovance.com
DRBD 44 / 62
46. DRBD: Ninja tricks
What to do in Split brain case
If you get this kind of problem in /proc/drbd :
Set secondary as primary
$ cat /proc/drbd
primary/unknown
that mean that you are in a split-brain case. First of all you need to check the version
on both nodes ! You absolutely need to have the same version to avoid errors
(8.0-8.3/8.4/9.0) :
Set secondary as primary
$ drbdadm --version | grep DRBDADM_VERSION =
DRBDADM_VERSION =8.3.13
Pierre Mavro www.enovance.com
DRBD 46 / 62
47. DRBD: Ninja tricks
What to do in Split brain case
You will need to discard all data on the secondary node and reimport all of them from
the master node.
Umount all drbd
Umount all failed DRBD mounted devices before continuing !!!
On the master node, we need to connect all failed DRBD devices first :
Connect DRBD
$ drbdadm connect <drbd_volume >
Pierre Mavro www.enovance.com
DRBD 47 / 62
48. DRBD: Ninja tricks
What to do in Split brain case
And then sync all data to the secondary node :
Sync from master
$ drbdadm -- --discard -my -data connect <drbd_volume >
Check the sync state in /proc/drbd :
Check sync state
$ cat /proc/drbd
version: 8.3.7 (api :88/ proto :86 -91)
srcversion: EE47D8BF18AC166BE219757
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/ Inconsistent C r.
ns :912248 nr:0 dw:0 dr :920640 al:0 bm :55 lo:1 pe :388 ua :2048 ap:0
[=== >................] sync ’ed: 21.9% (3283604/4194304) K
finish: 1:08:24 speed: 580 (452) K/sec
Mount your DRBD device when finished.
Pierre Mavro www.enovance.com
DRBD 48 / 62
49. DRBD: Ninja tricks
What to do in Split brain case
Note : there is a difference between "invalidate" and "discard-my-data" :
drbdadm invalidate : This forces the local device of a pair of connected DRBD
devices into SyncTarget state, which means that all data blocks of the device are
copied over from the peer. This command will fail if the device is not part of a
connected device pair.
drbdadm – –discard-my-data : Use this option to manually recover from a
split-brain situation. In case you do not have any automatic after-split-brain
policies selected, the nodes refuse to connect. By passing this option you make
this node a sync target immediately after successful connect.
Pierre Mavro www.enovance.com
DRBD 49 / 62
51. DRBD: Ninja tricks
Mount secondary node as read only
There are several ways to mount secondary node in a read only state but several ways
to break your data on the secondary node as well. This method is the best to avoid a
full resync.
First, create a snapshot on your logical volume on top of the DRBD device :
Create logical volume
$ lvcreate -s -n <lv >-snapshot -L 2G /dev/<vg >/<lv >
You’re now able to mount the snapshot as a read only device. To make it work, you
absolutely need to specify the filesystem :
Mount filesystem
$ mount -t ext4 -o ro /dev/<vg >/<lv >-snapshot /mountpoint
Backup all your desired data.
Pierre Mavro www.enovance.com
DRBD 51 / 62
52. DRBD: Ninja tricks
Mount secondary node as read only
You can check any time that your data are still under sync with ’drbd-overview’
command. When you’ve finished, you need to umount the mountpoint :
Unmount
$ umount /mountpoint
and remove the logical volume :
Delete LV
$ lvremove /dev/<vg >/<lv >-snapshot
Pierre Mavro www.enovance.com
DRBD 52 / 62
54. DRBD: Ninja tricks
All is good, but still doesn’t want to connect
All your configuration is ok, your logs doesn’t give any errors but the connect method
doesn’t seam to do something.
The last chance is to try the ’adjust’ command :
Adjust
$ drbdadm adjust <drbd_volume >
$ drbdadm connect <drbd_volume >
This should work now
Pierre Mavro www.enovance.com
DRBD 54 / 62
56. DRBD: Ninja tricks
Can’t "create-md" on an old DRBD device
If you get this kind of error when you want to recreate and md device on an old
existing one :
Initiate DRBD device
$ drbdadm create -md <drbd_volume >
Found some data
==> This might destroy existing data! <==
Do you want to proceed?
[need to type ’yes ’ to confirm] yes
drbdadm create -md nameResource : exited with code 40
To resolve the problem, erase all data on the current device :
Flush disk data
$ drbdadm wipe -md <drbd_volume >
Now restart the create procedure.
Pierre Mavro www.enovance.com
DRBD 56 / 62
57. DRBD: Ninja tricks
Can’t "create-md" on an old DRBD device
If it doesn’t work, because you don’t have your configuration any more for example,
you can do it manually :
Flush disk data
$ dd if=/ dev/zero of=/ dev/sdaX bs=1M count =128
Now restart the create procedure.
Pierre Mavro www.enovance.com
DRBD 57 / 62
59. DRBD: Ninja tricks
My DRBD device is still there but not the configuration
You may want to remove the DRBD device but unfortunately you did it in the wrong
way or an automatic script didn’t did it well.
The solution consist to recreate the configuration file to get the device recovered.
Warning
Before creating a configuration file in the drbd.d folder, check that you’ve configured
the correct device and ports to avoid conflicting with already used DRBD devices.
You can check that you’ve recovered your device with drbd-overview.
Then you’ll be able to follow the remove procedure.
Pierre Mavro www.enovance.com
DRBD 59 / 62
61. DRBD: Ninja tricks
Speed up sync transfer rate
If you need to speed up the transfer rate because you need to do a full sync between
to hosts, you can change it on the fly. It does not make sense to set a synchronization
rate higher than the maximum write throughput on your secondary node, so set it up
to a correct rate on the secondary node :
Set sync rate
drbdsetup /dev/<drbd_volume > syncer -r 900M
Then, when you’ve finished and want to revert to the default value :
Set sync rate
drbdadm adjust <drbd_volume >
Pierre Mavro www.enovance.com
DRBD 61 / 62