SlideShare a Scribd company logo
1 of 62
Download to read offline
DRBD
Pierre Mavro
www.enovance.com
September 11, 2013
DRBD: Summary
Summary
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Basic usages and understandings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
3 Use cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4 Ninja tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Pierre Mavro www.enovance.com
DRBD 2 / 62
DRBD: Introduction
How it works
DRBD refers to block devices designed as a building block to form high availability
(HA) clusters. This is done by mirroring a whole block device via an assigned network.
DRBD can be seen as network based raid-1.
Pierre Mavro www.enovance.com
DRBD 3 / 62
DRBD: Introduction
Synchronisations
DRBD supports three distinct replication modes, allowing three degrees of replication :
A (Asynchronous replication) : Master node disk write + local TCP send
buffer → operation completed
B (Semi synchronous replication) : Master node disk write + replication
packets reached other node → operation completed
C (Synchronous replication protocol) : Written to both the local and the
remote disk to get operation completed
The Syncronous replication protocol is the most robust protocol tested in
production.
http://www.drbd.org/home/mirroring/
Pierre Mavro www.enovance.com
DRBD 4 / 62
DRBD: Introduction
Data accessibility
A consequence of mirroring data on block device level is that you can access your data
(using a file system) only on the active node. This is not a shortcoming of DRBD but
is caused by the nature of most file systems (ext3, XFS, JFS, ext4, ...).
These file systems are designed for one computer accessing one disk, so they
cannot cope with two computers accessing one (virtually) shared disk.
In spite of this limitation, there are still a few ways to access the data on the second
node:
Use DRBD on logical volumes and use LVM’s capabilities to take snapshots on
the standby node, and access the data via the snapshot.
DRBD’s primary-primary mode with a shared disk file system (GFS, OCFS2).
These systems are very sensitive to failures of the replication network.
Mount in read only mode the partition
We are using DRBD in master/slave at eNovance.
Pierre Mavro www.enovance.com
DRBD 5 / 62
DRBD: Basic usages and understandings
Plan
2 Basic usages and understandings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Configuration and initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Check replication status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Node switching and manual synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Remove a DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Pierre Mavro www.enovance.com
DRBD 6 / 62
DRBD: Basic usages and understandings
Configuration
The DRBD configuration should be the same on both nodes. This drbd.conf file is
the global configuration of the DRBD service with fine tuning configuration :
/etc/drbd.conf
global { <-- Global configuration
usage-count no; <-- Do not report statistics usage to LinBit
}
common { <-- All resources inherit the options set in this section
protocol C; <-- C (Synchronous replication protocol)
startup {
wfc-timeout 1 ; <-- Wait for connection timeout (in seconds)
degr-wfc-timeout 1 ; <-- Wait for connection timeout, if this node was a degraded
} cluster (in seconds)
Pierre Mavro www.enovance.com
DRBD 7 / 62
DRBD: Basic usages and understandings
Configuration
/etc/drbd.conf
net {
max-buffers 8192; <-- Maximum number of requests to be allocated by DRBD
max-epoch-size 8192; <-- The highest number of data blocks between two write barriers
sndbuf-size 512k; <-- The size of the TCP socket send buffer
unplug-watermark 8192; <-- how often the I/O subsystem’s controller is forced to
process pending I/O requests
cram-hmac-alg sha1; <-- The HMAC algorithm to enable peer authentication at all
shared-secret "xxx"; <-- The shared secret used in peer authentication
# Split brains
after-sb-0pri disconnect; <-- Split brain, resource is not in the Primary role on any host
after-sb-1pri disconnect; <-- Split brain, resource is in the Primary role on one host
after-sb-2pri disconnect; <-- Split brain, resource is in the Primary role on both host
rr-conflict disconnect; <-- Helps to solve the cases when the outcome of the resync
} decision is incompatible with the current role assignment
handlers {
pri-on-incon-degr "echo node is primary, degraded and the local copy of the data is
inconsistent | wall "; <-- If the node is primary, degraded and if the
} local copy of the data is inconsistent
Pierre Mavro www.enovance.com
DRBD 8 / 62
DRBD: Basic usages and understandings
Configuration
/etc/drbd.conf
disk {
on-io-error pass_on; <-- The node downgrades the disk status to inconsistent on io errors
no-disk-barrier; <-- Disable protecting data if power failure (done by hardware)
no-disk-flushes; <-- Disable the backing device to support disk flushes
no-disk-drain; <-- Do not let write requests drain before write requests of a new
reordering domain are issued
no-md-flushes; <-- Disables the use of disk flushes and barrier BIOs when accessing
} the meta data device
syncer {
rate 300M; <-- The maximum bandwidth a resource uses for background
re-synchronization
al-extents 3833; <-- Control how big the hot area (= active set) can get
}
}
Pierre Mavro www.enovance.com
DRBD 9 / 62
DRBD: Basic usages and understandings
Configuration
Now the resources can be defined. One resource should match a DRBD device. Both
nodes informations should be filled :
/etc/drbd.d/resources.conf
resource drbd1 { <-- DRBD block device name
syncer {after drbd0 } <-- Start drbd1 after drbd0 is up & running
on srv1 { <-- Master node name
device /dev/drbd1; <-- Block device name of the resource being described
disk /dev/sda1; <-- Block device to store and retrieve the data
address x.x.x.x:7789; <-- IP address and port of the local host
meta-disk internal; <-- The last part of the backing device is used to store
the meta-data
}
on srv2 { <-- Slave node name
device /dev/drbd2;
disk /dev/sda1;
address y.y.y.y:7789;
meta-disk internal;
}
}
Pierre Mavro www.enovance.com
DRBD 10 / 62
DRBD: Basic usages and understandings
Initialisation
Here is a summary to understand the process to create a drbd replication. You need
to follow that rule to get a working DRBD synchronization :
Init metadata
device
Attach
DRBD device
Connect
DRBD device
Start syn-
chronization
Pierre Mavro www.enovance.com
DRBD 11 / 62
DRBD: Basic usages and understandings
Init metadata device
This step must be completed only on initial device creation. It initializes DRBD’s
metadata (replace <drbd_volume> by the drbd device name you want to initialize).
You should complete those steps on both nodes :
Create device metadata on server 1
srv1~$ drbdadm create -md <drbd_volume >
Writing meta data ...
initialising activity log
NOT initializing bitmap
New drbd meta data block sucessfully created.
Create device metadata on server 2
srv2~$ drbdadm create -md <drbd_volume >
...
Pierre Mavro www.enovance.com
DRBD 12 / 62
DRBD: Basic usages and understandings
Attach and connect DRBD device
Attach a local backing block device to the DRBD resource’s device :
Attach DRBD device
$ drbdadm attach <drbd_volume >
If the peer device is already configured, the two DRBD devices will connect :
Sets up the network configuration of the resource’s device
$ drbdadm connect <drbd_volume >
Pierre Mavro www.enovance.com
DRBD 13 / 62
DRBD: Basic usages and understandings
Start synchronization
To start the first synchronization, you need to ask to the first server to sync all blocks
to the secondary server.
This will erase all blocks on the other server, so all data will be lost :
Start the first synchronization
$ drbdadm -- --overwrite -data -of -peer primary <drbd_volume >
This action may take a while, depending on the network bandwidth and DRBD
volume size.
Pierre Mavro www.enovance.com
DRBD 14 / 62
DRBD: Basic usages and understandings
Plan
2 Basic usages and understandings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Configuration and initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Check replication status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Node switching and manual synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Remove a DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Pierre Mavro www.enovance.com
DRBD 15 / 62
DRBD: Basic usages and understandings
Check replication status
To check the replication status, simply look at DRBD file in /proc :
Check
$ cat /proc/drbd
version: 8.3.7 (api :88/ proto :86 -91)
srcversion: EE47D8BF18AC166BE219757
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/ Inconsistent C r.
ns :912248 nr:0 dw:0 dr :920640 al:0 bm :55 lo:1 pe :388 ua :2048 ap:0
[=== >................] sync ’ed: 21.9% (3283604/4194304) K
finish: 1:08:24 speed: 580 (452) K/sec
Pierre Mavro www.enovance.com
DRBD 16 / 62
DRBD: Basic usages and understandings
Check replication status
Another solution consists to launch this command that will provide informations of all
DRBD devices :
Check
$ drbd -overview
0: home Connected Primary/Secondary
UpToDate/UpToDate C r--- /home xfs 200G 158G 43G 79%
Pierre Mavro www.enovance.com
DRBD 17 / 62
DRBD: Basic usages and understandings
Check replication status
Several informations need be checked to know the status replication of a DRBD
device :
Replication
status
Connection
state (cs)
Resources
roles (ro)
Disk
states (ds)
Pierre Mavro www.enovance.com
DRBD 18 / 62
DRBD: Basic usages and understandings
Check replication status
Connection states (cs) :
StandAlone : No network configuration available. The resource has not yet
been connected, has been administratively disconnected or has dropped its
connection due to failed authentication/split brain.
Unconnected : Temporary state, prior to a connection attempt.
WFConnection : This node is waiting until the peer node becomes visible on
the network.
Connected : A DRBD connection has been established, data mirroring is now
active. This is the normal state.
PausedSync : The local node is the source or target of an ongoing
synchronization, but synchronization is currently paused. This may be due to a
dependency on the completion of another synchronization process, or due to
synchronization having been manually interrupted.
Pierre Mavro www.enovance.com
DRBD 19 / 62
DRBD: Basic usages and understandings
Check replication status
Resource roles (ro) :
Primary : The resource is currently in the primary role, and may be read from
and written to. This role only occurs on one of the two nodes, unless
dual-primary mode is enabled.
Secondary : The resource is currently in the secondary role. It normally receives
updates from its peer (unless running in disconnected mode), but may neither be
read from nor written to. This role may occur on one or both nodes.
Unknown : The resource’s role is currently unknown. The local resource role
never has this status. It is only displayed for the peer’s resource role, and only in
disconnected mode.
Pierre Mavro www.enovance.com
DRBD 20 / 62
DRBD: Basic usages and understandings
Check replication status
Disk states (ds) :
UpToDate : Consistent, up-to-date state of the data. This is the normal state.
Inconsistent : The data is inconsistent. This status occurs immediately upon
creation of a new resource, on both nodes (before the initial full sync). Also, this
status is found in one node (the synchronization target) during synchronization.
Pierre Mavro www.enovance.com
DRBD 21 / 62
DRBD: Basic usages and understandings
Plan
2 Basic usages and understandings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Configuration and initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Check replication status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Node switching and manual synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Remove a DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Pierre Mavro www.enovance.com
DRBD 22 / 62
DRBD: Basic usages and understandings
Node switching
You may need to perform maintenance tasks. For instance you need to switch a
DRBD volume to the other node.
Simply connect to the secondary node and launch :
Set secondary as primary
$ drbdadm primary <drbd_volume >
You can also switch all volumes at once. Double check if you really can do this
before proceed :
Set secondary as primary
$ drbdadm primary all
Pierre Mavro www.enovance.com
DRBD 23 / 62
DRBD: Basic usages and understandings
Node switching
You also can switch a Primary node to Secondary :
Set primary as secondary
$ drbdadm secondary <drbd_volume >
Notes
This will not automatically set the old secondary node to primary state
Pierre Mavro www.enovance.com
DRBD 24 / 62
DRBD: Basic usages and understandings
Manual synchronization
To start a manual synchronization, you need to invalidate the DRBD device on the
current host :
Invalidate data on current host
$ drbdadm invalidate <drbd_volume >
You can do the same from one node to the other node :
Invalidate data on remote host
$ drbdadm invalidate_remote <drbd_volume >
You can then check the status in /proc/drbd
Pierre Mavro www.enovance.com
DRBD 25 / 62
DRBD: Basic usages and understandings
Plan
2 Basic usages and understandings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Configuration and initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Check replication status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Node switching and manual synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Remove a DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Pierre Mavro www.enovance.com
DRBD 26 / 62
DRBD: Basic usages and understandings
Remove a DRBD device
Before removing a DRBD device, be sure you’ve dumped all data ! Then disconnect
the device on both nodes :
Disconnect DRBD device
$ drbdadm disconnect <drbd_volume >
And delete it :
Remove DRBD device
$ drbdsetup <volume_number > down
Change volume_number by the number seen with drbd-overview command. To finish,
remove the config files.
Pierre Mavro www.enovance.com
DRBD 27 / 62
DRBD: Use cases
Plan
3 Use cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Stop, upgrade and commit changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Stop, upgrade and rollback changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Up, connect...but doesn’t want to come UpToDate . . . . . . . . . . . . . . . . . . . . . . . 35
How to promote a primary when having a dual secondary . . . . . . . . . . . . . . . . . . 39
Pierre Mavro www.enovance.com
DRBD 28 / 62
DRBD: Use cases
Stop, upgrade and commit changes
Sometimes, you may need to perform maintenances tasks on a DRBD device. For
example, you need to upgrade stuffs on a DRBD device and if success, commit
changes to the secondary :
DRBD Primary DRBD Secondary
Set secondary node
in maintenance mode
Resume sync
Upgrade success
Normal state
Pierre Mavro www.enovance.com
DRBD 29 / 62
DRBD: Use cases
Stop, upgrade and commit changes
To set the secondary node in maintenance, simply stop the synchronisation on the
secondary node :
Disconnect drbd device
$ drbdadm disconnect <drbd_volume >
Do what you have to do on the DRBD primary device and once you have finished and
everything looks fine, resume the synchronization to get back to the normal state :
Connect drbd device
$ drbdadm connect <drbd_volume >
Pierre Mavro www.enovance.com
DRBD 30 / 62
DRBD: Use cases
Plan
3 Use cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Stop, upgrade and commit changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Stop, upgrade and rollback changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Up, connect...but doesn’t want to come UpToDate . . . . . . . . . . . . . . . . . . . . . . . 35
How to promote a primary when having a dual secondary . . . . . . . . . . . . . . . . . . 39
Pierre Mavro www.enovance.com
DRBD 31 / 62
DRBD: Use cases
Stop, upgrade and rollback changes
Sometimes, you may need to perform maintenances tasks on a DRBD device. For
example, you need to upgrade stuffs on a DRBD device and if fail, rollback changes
from the secondary :
DRBD Primary DRBD Secondary
Set secondary node
in maintenance mode
Upgrade failed
Normal state
Rollback sync
Pierre Mavro www.enovance.com
DRBD 32 / 62
DRBD: Use cases
Stop, upgrade and rollback changes
To set the secondary node in maintenance, simply stop the synchronisation on the
secondary node :
Disconnect drbd device
$ drbdadm disconnect <drbd_volume >
Then you’ve upgraded and want to rollback. You have to switch your secondary node
to master node :
Promote device as primary
$ drbdadm primary <drbd_volume >
Pierre Mavro www.enovance.com
DRBD 33 / 62
DRBD: Use cases
Stop, upgrade and rollback changes
Then you need to invalidate the datas on the old primary :
Invalidate drbd device data
$ drbdadm invalidate <drbd_volume >
To finish, adjust and connect the device to perform the rollback on the new master
node :
Adjust and connect
$ drbdadm adjust <drbd_volume >
$ drbdadm connect <drbd_volume >
Pierre Mavro www.enovance.com
DRBD 34 / 62
DRBD: Use cases
Plan
3 Use cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Stop, upgrade and commit changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Stop, upgrade and rollback changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Up, connect...but doesn’t want to come UpToDate . . . . . . . . . . . . . . . . . . . . . . . 35
How to promote a primary when having a dual secondary . . . . . . . . . . . . . . . . . . 39
Pierre Mavro www.enovance.com
DRBD 35 / 62
DRBD: Use cases
Up, connect...but doesn’t want to come UpToDate
If you don’t understand why after several up, connect etc.... you don’t see anything
changing and still got this kind of event :
Check
$ drbd -overview
0: drbd0 StandAlone Primary/Unknown UpToDate/DUnknown r-----
You need to instigate more. The first thing to do is to look at the logs :
/var/log/syslog
[ 6375.849509] block drbd0: Split-Brain detected but unresolved, dropping connection!
[ 6375.850025] block drbd0: helper command: /sbin/drbdadm split-brain minor-0
[ 6375.852859] block drbd0: helper command: /sbin/drbdadm split-brain minor-0 exit code 0 (0x0)
[ 6375.852867] block drbd0: conn( WFReportParams -> Disconnecting )
[ 6375.852873] block drbd0: error receiving ReportState, l: 4!
[ 6375.853274] block drbd0: asender terminated
[ 6375.853277] block drbd0: Terminating drbd0_asender
[ 6375.853506] block drbd0: Connection closed
[ 6375.853514] block drbd0: conn( Disconnecting -> StandAlone )
[ 6375.853523] block drbd0: receiver terminated
[ 6375.853526] block drbd0: Terminating drbd0_receiver
Pierre Mavro www.enovance.com
DRBD 36 / 62
DRBD: Use cases
Up, connect...but doesn’t want to come UpToDate
Anyway, it may not be clear enough for you. You can use that command to get a
better understanding on the situation :
Show all drbd informations
$ drbdadm show -gi <drbd_volume >
+--< Current data generation UUID >-
| +--< Bitmap ’s base data generation UUID >-
| | +--< younger history UUID >-
| | | +-< older history >-
V V V V
B63AABEFCFADD87D :2 C30515E7D05B35D :82 C4146DCB8E6BFF :82 C3146DCB8E6BFF :1:1:1:0:0:0:0
^ ^ ^ ^ ^ ^ ^
-< Data consistency flag >--+ | | | | | |
-< Data was/is currently up -to -date >--+ | | | | |
-< Node was/is currently primary >--+ | | | |
-< Node was/is currently connected >--+ | | |
-< Node was in the progress of setting all bits in the bitmap >--+ | |
-< The peer ’s disk was out -dated or inconsistent >--+ |
-< This node was a crashed primary , and has not seen its peer since >--+
flags: Primary , StandAlone , UpToDate
Pierre Mavro www.enovance.com
DRBD 37 / 62
DRBD: Use cases
Up, connect...but doesn’t want to come UpToDate
Here the problem is clear : we’re in a split brain situation. So there is no other
solutions than doing a full sync.
But what if no split brain was detected ? There’s a last solution :
Adjust and connect the drbd device
$ drbdadm adjust <drbd_volume >
$ drbdadm connect <drbd_volume >
Now you can check the replication which is resuming.
Pierre Mavro www.enovance.com
DRBD 38 / 62
DRBD: Use cases
Plan
3 Use cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Stop, upgrade and commit changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Stop, upgrade and rollback changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Up, connect...but doesn’t want to come UpToDate . . . . . . . . . . . . . . . . . . . . . . . 35
How to promote a primary when having a dual secondary . . . . . . . . . . . . . . . . . . 39
Pierre Mavro www.enovance.com
DRBD 39 / 62
DRBD: Use cases
How to promote a primary when having a dual secondary
If you get the dual secondary state after node’s restart, check which node as the
’UpToDate’ state :
Check
$ drbd -overview
0: drbd0 Connected Secondary/Secondary UpToDate/UpToDate C r-----
Here, both are up to date. Simply select one and promote it as primary :
Set as primary
$ drbdadm primary <drbd_volume >
You should now see one primary and a secondary synced.
Pierre Mavro www.enovance.com
DRBD 40 / 62
DRBD: Ninja tricks
Plan
4 Ninja tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Create DRBD replication without syncing data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
What to do in Split brain case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Mount secondary node as read only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
All is good, but still doesn’t want to connect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Can’t "create-md" on an old DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
My DRBD device is still there but not the configuration . . . . . . . . . . . . . . . . . . . 58
Speed up sync transfer rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Pierre Mavro www.enovance.com
DRBD 41 / 62
DRBD: Ninja tricks
Create DRBD replication without syncing data
When you have big devices to sync on a DRBD initialization, it could take several
hours or days in worst cases. There is a solution to only sync bitmap and have your
DRBD device available in a few seconds instead !
Create on both nodes the DRBD device :
Create device metadata
$ drbdadm create -md <drbd_volume >
Writing meta data ...
initialising activity log
NOT initializing bitmap
New drbd meta data block sucessfully created.
Pierre Mavro www.enovance.com
DRBD 42 / 62
DRBD: Ninja tricks
Create DRBD replication without syncing data
Then bring it up (still on both nodes) :
Connect the volume
$ drbdadm up <drbd_volume >
Only on the master node, initialize the bitmap and uuid of the DRBD device :
Only sync bitmap
$ drbdadm -- --clear -bitmap new -current -uuid <drbd_volume >
Pierre Mavro www.enovance.com
DRBD 43 / 62
DRBD: Ninja tricks
Create DRBD replication without syncing data
Promote this master node as primary node to replicate bitmap :
Set primary volume
$ drbdadm primary <drbd_volume >
Now you can check the DRBD is ready to use :-)
Check
$ drbd -overview
1: drbd1 Connected Primary/Secondary UpToDate/UpToDate C r-----
Pierre Mavro www.enovance.com
DRBD 44 / 62
DRBD: Ninja tricks
Plan
4 Ninja tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Create DRBD replication without syncing data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
What to do in Split brain case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Mount secondary node as read only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
All is good, but still doesn’t want to connect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Can’t "create-md" on an old DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
My DRBD device is still there but not the configuration . . . . . . . . . . . . . . . . . . . 58
Speed up sync transfer rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Pierre Mavro www.enovance.com
DRBD 45 / 62
DRBD: Ninja tricks
What to do in Split brain case
If you get this kind of problem in /proc/drbd :
Set secondary as primary
$ cat /proc/drbd
primary/unknown
that mean that you are in a split-brain case. First of all you need to check the version
on both nodes ! You absolutely need to have the same version to avoid errors
(8.0-8.3/8.4/9.0) :
Set secondary as primary
$ drbdadm --version | grep DRBDADM_VERSION =
DRBDADM_VERSION =8.3.13
Pierre Mavro www.enovance.com
DRBD 46 / 62
DRBD: Ninja tricks
What to do in Split brain case
You will need to discard all data on the secondary node and reimport all of them from
the master node.
Umount all drbd
Umount all failed DRBD mounted devices before continuing !!!
On the master node, we need to connect all failed DRBD devices first :
Connect DRBD
$ drbdadm connect <drbd_volume >
Pierre Mavro www.enovance.com
DRBD 47 / 62
DRBD: Ninja tricks
What to do in Split brain case
And then sync all data to the secondary node :
Sync from master
$ drbdadm -- --discard -my -data connect <drbd_volume >
Check the sync state in /proc/drbd :
Check sync state
$ cat /proc/drbd
version: 8.3.7 (api :88/ proto :86 -91)
srcversion: EE47D8BF18AC166BE219757
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/ Inconsistent C r.
ns :912248 nr:0 dw:0 dr :920640 al:0 bm :55 lo:1 pe :388 ua :2048 ap:0
[=== >................] sync ’ed: 21.9% (3283604/4194304) K
finish: 1:08:24 speed: 580 (452) K/sec
Mount your DRBD device when finished.
Pierre Mavro www.enovance.com
DRBD 48 / 62
DRBD: Ninja tricks
What to do in Split brain case
Note : there is a difference between "invalidate" and "discard-my-data" :
drbdadm invalidate : This forces the local device of a pair of connected DRBD
devices into SyncTarget state, which means that all data blocks of the device are
copied over from the peer. This command will fail if the device is not part of a
connected device pair.
drbdadm – –discard-my-data : Use this option to manually recover from a
split-brain situation. In case you do not have any automatic after-split-brain
policies selected, the nodes refuse to connect. By passing this option you make
this node a sync target immediately after successful connect.
Pierre Mavro www.enovance.com
DRBD 49 / 62
DRBD: Ninja tricks
Plan
4 Ninja tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Create DRBD replication without syncing data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
What to do in Split brain case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Mount secondary node as read only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
All is good, but still doesn’t want to connect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Can’t "create-md" on an old DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
My DRBD device is still there but not the configuration . . . . . . . . . . . . . . . . . . . 58
Speed up sync transfer rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Pierre Mavro www.enovance.com
DRBD 50 / 62
DRBD: Ninja tricks
Mount secondary node as read only
There are several ways to mount secondary node in a read only state but several ways
to break your data on the secondary node as well. This method is the best to avoid a
full resync.
First, create a snapshot on your logical volume on top of the DRBD device :
Create logical volume
$ lvcreate -s -n <lv >-snapshot -L 2G /dev/<vg >/<lv >
You’re now able to mount the snapshot as a read only device. To make it work, you
absolutely need to specify the filesystem :
Mount filesystem
$ mount -t ext4 -o ro /dev/<vg >/<lv >-snapshot /mountpoint
Backup all your desired data.
Pierre Mavro www.enovance.com
DRBD 51 / 62
DRBD: Ninja tricks
Mount secondary node as read only
You can check any time that your data are still under sync with ’drbd-overview’
command. When you’ve finished, you need to umount the mountpoint :
Unmount
$ umount /mountpoint
and remove the logical volume :
Delete LV
$ lvremove /dev/<vg >/<lv >-snapshot
Pierre Mavro www.enovance.com
DRBD 52 / 62
DRBD: Ninja tricks
Plan
4 Ninja tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Create DRBD replication without syncing data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
What to do in Split brain case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Mount secondary node as read only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
All is good, but still doesn’t want to connect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Can’t "create-md" on an old DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
My DRBD device is still there but not the configuration . . . . . . . . . . . . . . . . . . . 58
Speed up sync transfer rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Pierre Mavro www.enovance.com
DRBD 53 / 62
DRBD: Ninja tricks
All is good, but still doesn’t want to connect
All your configuration is ok, your logs doesn’t give any errors but the connect method
doesn’t seam to do something.
The last chance is to try the ’adjust’ command :
Adjust
$ drbdadm adjust <drbd_volume >
$ drbdadm connect <drbd_volume >
This should work now
Pierre Mavro www.enovance.com
DRBD 54 / 62
DRBD: Ninja tricks
Plan
4 Ninja tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Create DRBD replication without syncing data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
What to do in Split brain case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Mount secondary node as read only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
All is good, but still doesn’t want to connect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Can’t "create-md" on an old DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
My DRBD device is still there but not the configuration . . . . . . . . . . . . . . . . . . . 58
Speed up sync transfer rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Pierre Mavro www.enovance.com
DRBD 55 / 62
DRBD: Ninja tricks
Can’t "create-md" on an old DRBD device
If you get this kind of error when you want to recreate and md device on an old
existing one :
Initiate DRBD device
$ drbdadm create -md <drbd_volume >
Found some data
==> This might destroy existing data! <==
Do you want to proceed?
[need to type ’yes ’ to confirm] yes
drbdadm create -md nameResource : exited with code 40
To resolve the problem, erase all data on the current device :
Flush disk data
$ drbdadm wipe -md <drbd_volume >
Now restart the create procedure.
Pierre Mavro www.enovance.com
DRBD 56 / 62
DRBD: Ninja tricks
Can’t "create-md" on an old DRBD device
If it doesn’t work, because you don’t have your configuration any more for example,
you can do it manually :
Flush disk data
$ dd if=/ dev/zero of=/ dev/sdaX bs=1M count =128
Now restart the create procedure.
Pierre Mavro www.enovance.com
DRBD 57 / 62
DRBD: Ninja tricks
Plan
4 Ninja tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Create DRBD replication without syncing data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
What to do in Split brain case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Mount secondary node as read only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
All is good, but still doesn’t want to connect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Can’t "create-md" on an old DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
My DRBD device is still there but not the configuration . . . . . . . . . . . . . . . . . . . 58
Speed up sync transfer rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Pierre Mavro www.enovance.com
DRBD 58 / 62
DRBD: Ninja tricks
My DRBD device is still there but not the configuration
You may want to remove the DRBD device but unfortunately you did it in the wrong
way or an automatic script didn’t did it well.
The solution consist to recreate the configuration file to get the device recovered.
Warning
Before creating a configuration file in the drbd.d folder, check that you’ve configured
the correct device and ports to avoid conflicting with already used DRBD devices.
You can check that you’ve recovered your device with drbd-overview.
Then you’ll be able to follow the remove procedure.
Pierre Mavro www.enovance.com
DRBD 59 / 62
DRBD: Ninja tricks
Plan
4 Ninja tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Create DRBD replication without syncing data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
What to do in Split brain case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Mount secondary node as read only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
All is good, but still doesn’t want to connect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Can’t "create-md" on an old DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
My DRBD device is still there but not the configuration . . . . . . . . . . . . . . . . . . . 58
Speed up sync transfer rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Pierre Mavro www.enovance.com
DRBD 60 / 62
DRBD: Ninja tricks
Speed up sync transfer rate
If you need to speed up the transfer rate because you need to do a full sync between
to hosts, you can change it on the fly. It does not make sense to set a synchronization
rate higher than the maximum write throughput on your secondary node, so set it up
to a correct rate on the secondary node :
Set sync rate
drbdsetup /dev/<drbd_volume > syncer -r 900M
Then, when you’ve finished and want to revert to the default value :
Set sync rate
drbdadm adjust <drbd_volume >
Pierre Mavro www.enovance.com
DRBD 61 / 62
www.enovance.com

More Related Content

What's hot

Distributed replicated block device
Distributed replicated block deviceDistributed replicated block device
Distributed replicated block deviceChanaka Lasantha
 
High Availability != High-cost
High Availability != High-costHigh Availability != High-cost
High Availability != High-costnormanmaurer
 
High Availability Storage (susecon2016)
High Availability Storage (susecon2016)High Availability Storage (susecon2016)
High Availability Storage (susecon2016)Roger Zhou 周志强
 
Migrating to XtraDB Cluster
Migrating to XtraDB ClusterMigrating to XtraDB Cluster
Migrating to XtraDB Clusterpercona2013
 
Drupal MySQL Cluster
Drupal MySQL ClusterDrupal MySQL Cluster
Drupal MySQL ClusterKris Buytaert
 
Cgroup resource mgmt_v1
Cgroup resource mgmt_v1Cgroup resource mgmt_v1
Cgroup resource mgmt_v1sprdd
 
GlusterFS CTDB Integration
GlusterFS CTDB IntegrationGlusterFS CTDB Integration
GlusterFS CTDB IntegrationEtsuji Nakai
 
Disaster recovery of OpenStack Cinder using DRBD
Disaster recovery of OpenStack Cinder using DRBDDisaster recovery of OpenStack Cinder using DRBD
Disaster recovery of OpenStack Cinder using DRBDViswesuwara Nathan
 
Linux Cluster next generation
Linux Cluster next generationLinux Cluster next generation
Linux Cluster next generationsamsolutionsby
 
Introduction to RCU
Introduction to RCUIntroduction to RCU
Introduction to RCUKernel TLV
 
Corosync and Pacemaker
Corosync and PacemakerCorosync and Pacemaker
Corosync and PacemakerMarian Marinov
 
Romanticos com drbd 2
Romanticos com drbd 2Romanticos com drbd 2
Romanticos com drbd 2eiichi2009
 
Apache Cassandra multi-datacenter essentials
Apache Cassandra multi-datacenter essentialsApache Cassandra multi-datacenter essentials
Apache Cassandra multi-datacenter essentialsJulien Anguenot
 
LXC on Ganeti
LXC on GanetiLXC on Ganeti
LXC on Ganetikawamuray
 
Kernel Recipes 2016 -
Kernel Recipes 2016 - Kernel Recipes 2016 -
Kernel Recipes 2016 - Anne Nicolas
 
HDFS introduction
HDFS introductionHDFS introduction
HDFS introductioninjae yeo
 
CephFS update February 2016
CephFS update February 2016CephFS update February 2016
CephFS update February 2016John Spray
 

What's hot (20)

Distributed replicated block device
Distributed replicated block deviceDistributed replicated block device
Distributed replicated block device
 
High Availability != High-cost
High Availability != High-costHigh Availability != High-cost
High Availability != High-cost
 
High Availability Storage (susecon2016)
High Availability Storage (susecon2016)High Availability Storage (susecon2016)
High Availability Storage (susecon2016)
 
Migrating to XtraDB Cluster
Migrating to XtraDB ClusterMigrating to XtraDB Cluster
Migrating to XtraDB Cluster
 
Drupal MySQL Cluster
Drupal MySQL ClusterDrupal MySQL Cluster
Drupal MySQL Cluster
 
Cgroup resource mgmt_v1
Cgroup resource mgmt_v1Cgroup resource mgmt_v1
Cgroup resource mgmt_v1
 
GlusterFS CTDB Integration
GlusterFS CTDB IntegrationGlusterFS CTDB Integration
GlusterFS CTDB Integration
 
Disaster recovery of OpenStack Cinder using DRBD
Disaster recovery of OpenStack Cinder using DRBDDisaster recovery of OpenStack Cinder using DRBD
Disaster recovery of OpenStack Cinder using DRBD
 
Linux Cluster next generation
Linux Cluster next generationLinux Cluster next generation
Linux Cluster next generation
 
Introduction to RCU
Introduction to RCUIntroduction to RCU
Introduction to RCU
 
Corosync and Pacemaker
Corosync and PacemakerCorosync and Pacemaker
Corosync and Pacemaker
 
Romanticos com drbd 2
Romanticos com drbd 2Romanticos com drbd 2
Romanticos com drbd 2
 
Bluestore
BluestoreBluestore
Bluestore
 
Introduction to HDFS
Introduction to HDFSIntroduction to HDFS
Introduction to HDFS
 
Scale 10x 01:22:12
Scale 10x 01:22:12Scale 10x 01:22:12
Scale 10x 01:22:12
 
Apache Cassandra multi-datacenter essentials
Apache Cassandra multi-datacenter essentialsApache Cassandra multi-datacenter essentials
Apache Cassandra multi-datacenter essentials
 
LXC on Ganeti
LXC on GanetiLXC on Ganeti
LXC on Ganeti
 
Kernel Recipes 2016 -
Kernel Recipes 2016 - Kernel Recipes 2016 -
Kernel Recipes 2016 -
 
HDFS introduction
HDFS introductionHDFS introduction
HDFS introduction
 
CephFS update February 2016
CephFS update February 2016CephFS update February 2016
CephFS update February 2016
 

Viewers also liked

Introduction to DRBD
Introduction to DRBDIntroduction to DRBD
Introduction to DRBDdawnlua
 
DRBD + OpenStack (Openstack Live Prague 2016)
DRBD + OpenStack (Openstack Live Prague 2016)DRBD + OpenStack (Openstack Live Prague 2016)
DRBD + OpenStack (Openstack Live Prague 2016)Jaroslav Jacjuk
 
Ansible meetup-0915
Ansible meetup-0915Ansible meetup-0915
Ansible meetup-0915Pierre Mavro
 
T3UNI12 : SOLR workshop
T3UNI12 : SOLR workshopT3UNI12 : SOLR workshop
T3UNI12 : SOLR workshopPaul Blondiaux
 
Accomplishing redundancy on Lustre based PFS with DRBD
Accomplishing redundancy on Lustre based PFS with DRBDAccomplishing redundancy on Lustre based PFS with DRBD
Accomplishing redundancy on Lustre based PFS with DRBDTyrone Systems
 
What is new in Leap42.2 and SLE12SP2
What is new in Leap42.2 and SLE12SP2What is new in Leap42.2 and SLE12SP2
What is new in Leap42.2 and SLE12SP2Nick Wang
 
Solr Recipes Workshop
Solr Recipes WorkshopSolr Recipes Workshop
Solr Recipes WorkshopErik Hatcher
 
BlackStor - World's fastest & most reliable Cloud Native Software Defined Sto...
BlackStor - World's fastest & most reliable Cloud Native Software Defined Sto...BlackStor - World's fastest & most reliable Cloud Native Software Defined Sto...
BlackStor - World's fastest & most reliable Cloud Native Software Defined Sto...Michal Němec
 
Couchbase live 2016
Couchbase live 2016Couchbase live 2016
Couchbase live 2016Pierre Mavro
 
High Availability With DRBD & Heartbeat
High Availability With DRBD & HeartbeatHigh Availability With DRBD & Heartbeat
High Availability With DRBD & HeartbeatChris Barber
 
أساسيات لغة Php بالعربي
أساسيات لغة Php بالعربيأساسيات لغة Php بالعربي
أساسيات لغة Php بالعربيtahsal99
 
Big Data Analytics Proposal #1
Big Data Analytics Proposal #1Big Data Analytics Proposal #1
Big Data Analytics Proposal #1Ziyad Saleh
 
Big Data Analysis Patterns with Hadoop, Mahout and Solr
Big Data Analysis Patterns with Hadoop, Mahout and SolrBig Data Analysis Patterns with Hadoop, Mahout and Solr
Big Data Analysis Patterns with Hadoop, Mahout and Solrboorad
 
Big Data Airline Project at UAEU
Big Data Airline Project at UAEUBig Data Airline Project at UAEU
Big Data Airline Project at UAEUZiyad Saleh
 
What you need to know about ceph
What you need to know about cephWhat you need to know about ceph
What you need to know about cephEmma Haruka Iwao
 
Storage tiering and erasure coding in Ceph (SCaLE13x)
Storage tiering and erasure coding in Ceph (SCaLE13x)Storage tiering and erasure coding in Ceph (SCaLE13x)
Storage tiering and erasure coding in Ceph (SCaLE13x)Sage Weil
 
Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...
Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...
Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...Odinot Stanislas
 
Ceph Introduction 2017
Ceph Introduction 2017  Ceph Introduction 2017
Ceph Introduction 2017 Karan Singh
 
Drbd9 and drbdmanage_june_2016
Drbd9 and drbdmanage_june_2016Drbd9 and drbdmanage_june_2016
Drbd9 and drbdmanage_june_2016Philipp Reisner
 

Viewers also liked (20)

Introduction to DRBD
Introduction to DRBDIntroduction to DRBD
Introduction to DRBD
 
DRBD + OpenStack (Openstack Live Prague 2016)
DRBD + OpenStack (Openstack Live Prague 2016)DRBD + OpenStack (Openstack Live Prague 2016)
DRBD + OpenStack (Openstack Live Prague 2016)
 
Ansible meetup-0915
Ansible meetup-0915Ansible meetup-0915
Ansible meetup-0915
 
T3UNI12 : SOLR workshop
T3UNI12 : SOLR workshopT3UNI12 : SOLR workshop
T3UNI12 : SOLR workshop
 
Accomplishing redundancy on Lustre based PFS with DRBD
Accomplishing redundancy on Lustre based PFS with DRBDAccomplishing redundancy on Lustre based PFS with DRBD
Accomplishing redundancy on Lustre based PFS with DRBD
 
What is new in Leap42.2 and SLE12SP2
What is new in Leap42.2 and SLE12SP2What is new in Leap42.2 and SLE12SP2
What is new in Leap42.2 and SLE12SP2
 
Solr Recipes Workshop
Solr Recipes WorkshopSolr Recipes Workshop
Solr Recipes Workshop
 
BlackStor - World's fastest & most reliable Cloud Native Software Defined Sto...
BlackStor - World's fastest & most reliable Cloud Native Software Defined Sto...BlackStor - World's fastest & most reliable Cloud Native Software Defined Sto...
BlackStor - World's fastest & most reliable Cloud Native Software Defined Sto...
 
Couchbase live 2016
Couchbase live 2016Couchbase live 2016
Couchbase live 2016
 
High Availability With DRBD & Heartbeat
High Availability With DRBD & HeartbeatHigh Availability With DRBD & Heartbeat
High Availability With DRBD & Heartbeat
 
أساسيات لغة Php بالعربي
أساسيات لغة Php بالعربيأساسيات لغة Php بالعربي
أساسيات لغة Php بالعربي
 
Big Data Analytics Proposal #1
Big Data Analytics Proposal #1Big Data Analytics Proposal #1
Big Data Analytics Proposal #1
 
Big Data Analysis Patterns with Hadoop, Mahout and Solr
Big Data Analysis Patterns with Hadoop, Mahout and SolrBig Data Analysis Patterns with Hadoop, Mahout and Solr
Big Data Analysis Patterns with Hadoop, Mahout and Solr
 
Big Data Airline Project at UAEU
Big Data Airline Project at UAEUBig Data Airline Project at UAEU
Big Data Airline Project at UAEU
 
What you need to know about ceph
What you need to know about cephWhat you need to know about ceph
What you need to know about ceph
 
Big data analysis concepts and references
Big data analysis concepts and referencesBig data analysis concepts and references
Big data analysis concepts and references
 
Storage tiering and erasure coding in Ceph (SCaLE13x)
Storage tiering and erasure coding in Ceph (SCaLE13x)Storage tiering and erasure coding in Ceph (SCaLE13x)
Storage tiering and erasure coding in Ceph (SCaLE13x)
 
Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...
Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...
Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...
 
Ceph Introduction 2017
Ceph Introduction 2017  Ceph Introduction 2017
Ceph Introduction 2017
 
Drbd9 and drbdmanage_june_2016
Drbd9 and drbdmanage_june_2016Drbd9 and drbdmanage_june_2016
Drbd9 and drbdmanage_june_2016
 

Similar to Drbd

OSDC 2015: Roland Kammerer | DRBD9: Managing High-Available Storage in Many-N...
OSDC 2015: Roland Kammerer | DRBD9: Managing High-Available Storage in Many-N...OSDC 2015: Roland Kammerer | DRBD9: Managing High-Available Storage in Many-N...
OSDC 2015: Roland Kammerer | DRBD9: Managing High-Available Storage in Many-N...NETWAYS
 
drbd9_and_drbdmanage_may_2015
drbd9_and_drbdmanage_may_2015drbd9_and_drbdmanage_may_2015
drbd9_and_drbdmanage_may_2015Alexandre Huynh
 
Corpus collapsum: Partition tolerance of Galera in a noisy high load environment
Corpus collapsum: Partition tolerance of Galera in a noisy high load environmentCorpus collapsum: Partition tolerance of Galera in a noisy high load environment
Corpus collapsum: Partition tolerance of Galera in a noisy high load environmentRaghavendra Prabhu
 
Pdn multipath io-pvm-ver1.0.0
Pdn multipath io-pvm-ver1.0.0Pdn multipath io-pvm-ver1.0.0
Pdn multipath io-pvm-ver1.0.0pdnsoftco
 
Upgrade 11gR2 to 12cR1 Clusterware
Upgrade 11gR2 to 12cR1 ClusterwareUpgrade 11gR2 to 12cR1 Clusterware
Upgrade 11gR2 to 12cR1 ClusterwareNikhil Kumar
 
Building android for the Cloud: Android as a Server (AnDevConBoston 2014)
Building android for the Cloud: Android as a Server (AnDevConBoston 2014)Building android for the Cloud: Android as a Server (AnDevConBoston 2014)
Building android for the Cloud: Android as a Server (AnDevConBoston 2014)Ron Munitz
 
Bringing up Android on your favorite X86 Workstation or VM (AnDevCon Boston, ...
Bringing up Android on your favorite X86 Workstation or VM (AnDevCon Boston, ...Bringing up Android on your favorite X86 Workstation or VM (AnDevCon Boston, ...
Bringing up Android on your favorite X86 Workstation or VM (AnDevCon Boston, ...Ron Munitz
 
DevOpSec_DockerNPodMan-20230220.pdf
DevOpSec_DockerNPodMan-20230220.pdfDevOpSec_DockerNPodMan-20230220.pdf
DevOpSec_DockerNPodMan-20230220.pdfkanedafromparis
 
OpenNebulaConf 2016 - The DRBD SDS for OpenNebula by Philipp Reisner, LINBIT
OpenNebulaConf 2016 - The DRBD SDS for OpenNebula by Philipp Reisner, LINBITOpenNebulaConf 2016 - The DRBD SDS for OpenNebula by Philipp Reisner, LINBIT
OpenNebulaConf 2016 - The DRBD SDS for OpenNebula by Philipp Reisner, LINBITOpenNebula Project
 
FOSDEM'17: Disaster Recovery Management with ReaR (relax-and-recover) & DRLM ...
FOSDEM'17: Disaster Recovery Management with ReaR (relax-and-recover) & DRLM ...FOSDEM'17: Disaster Recovery Management with ReaR (relax-and-recover) & DRLM ...
FOSDEM'17: Disaster Recovery Management with ReaR (relax-and-recover) & DRLM ...Didac Oliveira
 
Percona XtraDB 集群文档
Percona XtraDB 集群文档Percona XtraDB 集群文档
Percona XtraDB 集群文档YUCHENG HU
 
Oreilly Webcast 01 19 10
Oreilly Webcast 01 19 10Oreilly Webcast 01 19 10
Oreilly Webcast 01 19 10Sean Hull
 
Hydra: Continuous Integration and Testing for Demanding People: The Details
Hydra: Continuous Integration and Testing for Demanding People: The DetailsHydra: Continuous Integration and Testing for Demanding People: The Details
Hydra: Continuous Integration and Testing for Demanding People: The DetailsSander van der Burg
 
Hands on MapR -- Viadea
Hands on MapR -- ViadeaHands on MapR -- Viadea
Hands on MapR -- Viadeaviadea
 
SUSE Expert Days Paris 2018 - SUSE HA Cluster Multi-Device
SUSE Expert Days Paris 2018 - SUSE HA Cluster Multi-DeviceSUSE Expert Days Paris 2018 - SUSE HA Cluster Multi-Device
SUSE Expert Days Paris 2018 - SUSE HA Cluster Multi-DeviceSUSE
 
Tuning parallelcodeonsolaris005
Tuning parallelcodeonsolaris005Tuning parallelcodeonsolaris005
Tuning parallelcodeonsolaris005dflexer
 
DCEU 18: Tips and Tricks of the Docker Captains
DCEU 18: Tips and Tricks of the Docker CaptainsDCEU 18: Tips and Tricks of the Docker Captains
DCEU 18: Tips and Tricks of the Docker CaptainsDocker, Inc.
 
Building Android for the Cloud: Android as a Server (Mobile World Congress 2014)
Building Android for the Cloud: Android as a Server (Mobile World Congress 2014)Building Android for the Cloud: Android as a Server (Mobile World Congress 2014)
Building Android for the Cloud: Android as a Server (Mobile World Congress 2014)Ron Munitz
 
[DockerCon 2020] Hardening Docker daemon with Rootless Mode
[DockerCon 2020] Hardening Docker daemon with Rootless Mode[DockerCon 2020] Hardening Docker daemon with Rootless Mode
[DockerCon 2020] Hardening Docker daemon with Rootless ModeAkihiro Suda
 
OpenWRT manual
OpenWRT manualOpenWRT manual
OpenWRT manualfosk
 

Similar to Drbd (20)

OSDC 2015: Roland Kammerer | DRBD9: Managing High-Available Storage in Many-N...
OSDC 2015: Roland Kammerer | DRBD9: Managing High-Available Storage in Many-N...OSDC 2015: Roland Kammerer | DRBD9: Managing High-Available Storage in Many-N...
OSDC 2015: Roland Kammerer | DRBD9: Managing High-Available Storage in Many-N...
 
drbd9_and_drbdmanage_may_2015
drbd9_and_drbdmanage_may_2015drbd9_and_drbdmanage_may_2015
drbd9_and_drbdmanage_may_2015
 
Corpus collapsum: Partition tolerance of Galera in a noisy high load environment
Corpus collapsum: Partition tolerance of Galera in a noisy high load environmentCorpus collapsum: Partition tolerance of Galera in a noisy high load environment
Corpus collapsum: Partition tolerance of Galera in a noisy high load environment
 
Pdn multipath io-pvm-ver1.0.0
Pdn multipath io-pvm-ver1.0.0Pdn multipath io-pvm-ver1.0.0
Pdn multipath io-pvm-ver1.0.0
 
Upgrade 11gR2 to 12cR1 Clusterware
Upgrade 11gR2 to 12cR1 ClusterwareUpgrade 11gR2 to 12cR1 Clusterware
Upgrade 11gR2 to 12cR1 Clusterware
 
Building android for the Cloud: Android as a Server (AnDevConBoston 2014)
Building android for the Cloud: Android as a Server (AnDevConBoston 2014)Building android for the Cloud: Android as a Server (AnDevConBoston 2014)
Building android for the Cloud: Android as a Server (AnDevConBoston 2014)
 
Bringing up Android on your favorite X86 Workstation or VM (AnDevCon Boston, ...
Bringing up Android on your favorite X86 Workstation or VM (AnDevCon Boston, ...Bringing up Android on your favorite X86 Workstation or VM (AnDevCon Boston, ...
Bringing up Android on your favorite X86 Workstation or VM (AnDevCon Boston, ...
 
DevOpSec_DockerNPodMan-20230220.pdf
DevOpSec_DockerNPodMan-20230220.pdfDevOpSec_DockerNPodMan-20230220.pdf
DevOpSec_DockerNPodMan-20230220.pdf
 
OpenNebulaConf 2016 - The DRBD SDS for OpenNebula by Philipp Reisner, LINBIT
OpenNebulaConf 2016 - The DRBD SDS for OpenNebula by Philipp Reisner, LINBITOpenNebulaConf 2016 - The DRBD SDS for OpenNebula by Philipp Reisner, LINBIT
OpenNebulaConf 2016 - The DRBD SDS for OpenNebula by Philipp Reisner, LINBIT
 
FOSDEM'17: Disaster Recovery Management with ReaR (relax-and-recover) & DRLM ...
FOSDEM'17: Disaster Recovery Management with ReaR (relax-and-recover) & DRLM ...FOSDEM'17: Disaster Recovery Management with ReaR (relax-and-recover) & DRLM ...
FOSDEM'17: Disaster Recovery Management with ReaR (relax-and-recover) & DRLM ...
 
Percona XtraDB 集群文档
Percona XtraDB 集群文档Percona XtraDB 集群文档
Percona XtraDB 集群文档
 
Oreilly Webcast 01 19 10
Oreilly Webcast 01 19 10Oreilly Webcast 01 19 10
Oreilly Webcast 01 19 10
 
Hydra: Continuous Integration and Testing for Demanding People: The Details
Hydra: Continuous Integration and Testing for Demanding People: The DetailsHydra: Continuous Integration and Testing for Demanding People: The Details
Hydra: Continuous Integration and Testing for Demanding People: The Details
 
Hands on MapR -- Viadea
Hands on MapR -- ViadeaHands on MapR -- Viadea
Hands on MapR -- Viadea
 
SUSE Expert Days Paris 2018 - SUSE HA Cluster Multi-Device
SUSE Expert Days Paris 2018 - SUSE HA Cluster Multi-DeviceSUSE Expert Days Paris 2018 - SUSE HA Cluster Multi-Device
SUSE Expert Days Paris 2018 - SUSE HA Cluster Multi-Device
 
Tuning parallelcodeonsolaris005
Tuning parallelcodeonsolaris005Tuning parallelcodeonsolaris005
Tuning parallelcodeonsolaris005
 
DCEU 18: Tips and Tricks of the Docker Captains
DCEU 18: Tips and Tricks of the Docker CaptainsDCEU 18: Tips and Tricks of the Docker Captains
DCEU 18: Tips and Tricks of the Docker Captains
 
Building Android for the Cloud: Android as a Server (Mobile World Congress 2014)
Building Android for the Cloud: Android as a Server (Mobile World Congress 2014)Building Android for the Cloud: Android as a Server (Mobile World Congress 2014)
Building Android for the Cloud: Android as a Server (Mobile World Congress 2014)
 
[DockerCon 2020] Hardening Docker daemon with Rootless Mode
[DockerCon 2020] Hardening Docker daemon with Rootless Mode[DockerCon 2020] Hardening Docker daemon with Rootless Mode
[DockerCon 2020] Hardening Docker daemon with Rootless Mode
 
OpenWRT manual
OpenWRT manualOpenWRT manual
OpenWRT manual
 

More from Pierre Mavro

Security model for a remote company
Security model for a remote companySecurity model for a remote company
Security model for a remote companyPierre Mavro
 
Compass digital ocean’s customer advisory group 2021_10
Compass digital ocean’s customer advisory group 2021_10Compass digital ocean’s customer advisory group 2021_10
Compass digital ocean’s customer advisory group 2021_10Pierre Mavro
 
Criteo meetup - S.R.E Tech Talk
Criteo meetup - S.R.E Tech TalkCriteo meetup - S.R.E Tech Talk
Criteo meetup - S.R.E Tech TalkPierre Mavro
 
Traefik on Kubernetes at MySocialApp (CNCF Paris Meetup)
Traefik on Kubernetes at MySocialApp (CNCF Paris Meetup)Traefik on Kubernetes at MySocialApp (CNCF Paris Meetup)
Traefik on Kubernetes at MySocialApp (CNCF Paris Meetup)Pierre Mavro
 
Mariadb mysql avancé
Mariadb mysql avancéMariadb mysql avancé
Mariadb mysql avancéPierre Mavro
 

More from Pierre Mavro (6)

Security model for a remote company
Security model for a remote companySecurity model for a remote company
Security model for a remote company
 
Compass digital ocean’s customer advisory group 2021_10
Compass digital ocean’s customer advisory group 2021_10Compass digital ocean’s customer advisory group 2021_10
Compass digital ocean’s customer advisory group 2021_10
 
Criteo meetup - S.R.E Tech Talk
Criteo meetup - S.R.E Tech TalkCriteo meetup - S.R.E Tech Talk
Criteo meetup - S.R.E Tech Talk
 
Traefik on Kubernetes at MySocialApp (CNCF Paris Meetup)
Traefik on Kubernetes at MySocialApp (CNCF Paris Meetup)Traefik on Kubernetes at MySocialApp (CNCF Paris Meetup)
Traefik on Kubernetes at MySocialApp (CNCF Paris Meetup)
 
Mariadb mysql avancé
Mariadb mysql avancéMariadb mysql avancé
Mariadb mysql avancé
 
Puppet slides
Puppet slidesPuppet slides
Puppet slides
 

Recently uploaded

Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Wonjun Hwang
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brandgvaughan
 
Artificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptxArtificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptxhariprasad279825
 
Search Engine Optimization SEO PDF for 2024.pdf
Search Engine Optimization SEO PDF for 2024.pdfSearch Engine Optimization SEO PDF for 2024.pdf
Search Engine Optimization SEO PDF for 2024.pdfRankYa
 
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage CostLeverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage CostZilliz
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr BaganFwdays
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University
 
Unleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubUnleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubKalema Edgar
 
CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):comworks
 
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr LapshynFwdays
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsSergiu Bodiu
 
Vertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering TipsVertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering TipsMiki Katsuragi
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024Stephanie Beckett
 
Vector Databases 101 - An introduction to the world of Vector Databases
Vector Databases 101 - An introduction to the world of Vector DatabasesVector Databases 101 - An introduction to the world of Vector Databases
Vector Databases 101 - An introduction to the world of Vector DatabasesZilliz
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii SoldatenkoFwdays
 
Powerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time ClashPowerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time Clashcharlottematthew16
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
AI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsAI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsMemoori
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024Lorenzo Miniero
 

Recently uploaded (20)

Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brand
 
Artificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptxArtificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptx
 
Search Engine Optimization SEO PDF for 2024.pdf
Search Engine Optimization SEO PDF for 2024.pdfSearch Engine Optimization SEO PDF for 2024.pdf
Search Engine Optimization SEO PDF for 2024.pdf
 
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage CostLeverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!
 
Unleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubUnleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding Club
 
CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):
 
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platforms
 
Vertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering TipsVertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering Tips
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024
 
Vector Databases 101 - An introduction to the world of Vector Databases
Vector Databases 101 - An introduction to the world of Vector DatabasesVector Databases 101 - An introduction to the world of Vector Databases
Vector Databases 101 - An introduction to the world of Vector Databases
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko
 
Powerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time ClashPowerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time Clash
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
AI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsAI as an Interface for Commercial Buildings
AI as an Interface for Commercial Buildings
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio Web
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024
 

Drbd

  • 2. DRBD: Summary Summary 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2 Basic usages and understandings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 3 Use cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 4 Ninja tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Pierre Mavro www.enovance.com DRBD 2 / 62
  • 3. DRBD: Introduction How it works DRBD refers to block devices designed as a building block to form high availability (HA) clusters. This is done by mirroring a whole block device via an assigned network. DRBD can be seen as network based raid-1. Pierre Mavro www.enovance.com DRBD 3 / 62
  • 4. DRBD: Introduction Synchronisations DRBD supports three distinct replication modes, allowing three degrees of replication : A (Asynchronous replication) : Master node disk write + local TCP send buffer → operation completed B (Semi synchronous replication) : Master node disk write + replication packets reached other node → operation completed C (Synchronous replication protocol) : Written to both the local and the remote disk to get operation completed The Syncronous replication protocol is the most robust protocol tested in production. http://www.drbd.org/home/mirroring/ Pierre Mavro www.enovance.com DRBD 4 / 62
  • 5. DRBD: Introduction Data accessibility A consequence of mirroring data on block device level is that you can access your data (using a file system) only on the active node. This is not a shortcoming of DRBD but is caused by the nature of most file systems (ext3, XFS, JFS, ext4, ...). These file systems are designed for one computer accessing one disk, so they cannot cope with two computers accessing one (virtually) shared disk. In spite of this limitation, there are still a few ways to access the data on the second node: Use DRBD on logical volumes and use LVM’s capabilities to take snapshots on the standby node, and access the data via the snapshot. DRBD’s primary-primary mode with a shared disk file system (GFS, OCFS2). These systems are very sensitive to failures of the replication network. Mount in read only mode the partition We are using DRBD in master/slave at eNovance. Pierre Mavro www.enovance.com DRBD 5 / 62
  • 6. DRBD: Basic usages and understandings Plan 2 Basic usages and understandings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Configuration and initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Check replication status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Node switching and manual synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Remove a DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Pierre Mavro www.enovance.com DRBD 6 / 62
  • 7. DRBD: Basic usages and understandings Configuration The DRBD configuration should be the same on both nodes. This drbd.conf file is the global configuration of the DRBD service with fine tuning configuration : /etc/drbd.conf global { <-- Global configuration usage-count no; <-- Do not report statistics usage to LinBit } common { <-- All resources inherit the options set in this section protocol C; <-- C (Synchronous replication protocol) startup { wfc-timeout 1 ; <-- Wait for connection timeout (in seconds) degr-wfc-timeout 1 ; <-- Wait for connection timeout, if this node was a degraded } cluster (in seconds) Pierre Mavro www.enovance.com DRBD 7 / 62
  • 8. DRBD: Basic usages and understandings Configuration /etc/drbd.conf net { max-buffers 8192; <-- Maximum number of requests to be allocated by DRBD max-epoch-size 8192; <-- The highest number of data blocks between two write barriers sndbuf-size 512k; <-- The size of the TCP socket send buffer unplug-watermark 8192; <-- how often the I/O subsystem’s controller is forced to process pending I/O requests cram-hmac-alg sha1; <-- The HMAC algorithm to enable peer authentication at all shared-secret "xxx"; <-- The shared secret used in peer authentication # Split brains after-sb-0pri disconnect; <-- Split brain, resource is not in the Primary role on any host after-sb-1pri disconnect; <-- Split brain, resource is in the Primary role on one host after-sb-2pri disconnect; <-- Split brain, resource is in the Primary role on both host rr-conflict disconnect; <-- Helps to solve the cases when the outcome of the resync } decision is incompatible with the current role assignment handlers { pri-on-incon-degr "echo node is primary, degraded and the local copy of the data is inconsistent | wall "; <-- If the node is primary, degraded and if the } local copy of the data is inconsistent Pierre Mavro www.enovance.com DRBD 8 / 62
  • 9. DRBD: Basic usages and understandings Configuration /etc/drbd.conf disk { on-io-error pass_on; <-- The node downgrades the disk status to inconsistent on io errors no-disk-barrier; <-- Disable protecting data if power failure (done by hardware) no-disk-flushes; <-- Disable the backing device to support disk flushes no-disk-drain; <-- Do not let write requests drain before write requests of a new reordering domain are issued no-md-flushes; <-- Disables the use of disk flushes and barrier BIOs when accessing } the meta data device syncer { rate 300M; <-- The maximum bandwidth a resource uses for background re-synchronization al-extents 3833; <-- Control how big the hot area (= active set) can get } } Pierre Mavro www.enovance.com DRBD 9 / 62
  • 10. DRBD: Basic usages and understandings Configuration Now the resources can be defined. One resource should match a DRBD device. Both nodes informations should be filled : /etc/drbd.d/resources.conf resource drbd1 { <-- DRBD block device name syncer {after drbd0 } <-- Start drbd1 after drbd0 is up & running on srv1 { <-- Master node name device /dev/drbd1; <-- Block device name of the resource being described disk /dev/sda1; <-- Block device to store and retrieve the data address x.x.x.x:7789; <-- IP address and port of the local host meta-disk internal; <-- The last part of the backing device is used to store the meta-data } on srv2 { <-- Slave node name device /dev/drbd2; disk /dev/sda1; address y.y.y.y:7789; meta-disk internal; } } Pierre Mavro www.enovance.com DRBD 10 / 62
  • 11. DRBD: Basic usages and understandings Initialisation Here is a summary to understand the process to create a drbd replication. You need to follow that rule to get a working DRBD synchronization : Init metadata device Attach DRBD device Connect DRBD device Start syn- chronization Pierre Mavro www.enovance.com DRBD 11 / 62
  • 12. DRBD: Basic usages and understandings Init metadata device This step must be completed only on initial device creation. It initializes DRBD’s metadata (replace <drbd_volume> by the drbd device name you want to initialize). You should complete those steps on both nodes : Create device metadata on server 1 srv1~$ drbdadm create -md <drbd_volume > Writing meta data ... initialising activity log NOT initializing bitmap New drbd meta data block sucessfully created. Create device metadata on server 2 srv2~$ drbdadm create -md <drbd_volume > ... Pierre Mavro www.enovance.com DRBD 12 / 62
  • 13. DRBD: Basic usages and understandings Attach and connect DRBD device Attach a local backing block device to the DRBD resource’s device : Attach DRBD device $ drbdadm attach <drbd_volume > If the peer device is already configured, the two DRBD devices will connect : Sets up the network configuration of the resource’s device $ drbdadm connect <drbd_volume > Pierre Mavro www.enovance.com DRBD 13 / 62
  • 14. DRBD: Basic usages and understandings Start synchronization To start the first synchronization, you need to ask to the first server to sync all blocks to the secondary server. This will erase all blocks on the other server, so all data will be lost : Start the first synchronization $ drbdadm -- --overwrite -data -of -peer primary <drbd_volume > This action may take a while, depending on the network bandwidth and DRBD volume size. Pierre Mavro www.enovance.com DRBD 14 / 62
  • 15. DRBD: Basic usages and understandings Plan 2 Basic usages and understandings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Configuration and initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Check replication status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Node switching and manual synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Remove a DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Pierre Mavro www.enovance.com DRBD 15 / 62
  • 16. DRBD: Basic usages and understandings Check replication status To check the replication status, simply look at DRBD file in /proc : Check $ cat /proc/drbd version: 8.3.7 (api :88/ proto :86 -91) srcversion: EE47D8BF18AC166BE219757 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/ Inconsistent C r. ns :912248 nr:0 dw:0 dr :920640 al:0 bm :55 lo:1 pe :388 ua :2048 ap:0 [=== >................] sync ’ed: 21.9% (3283604/4194304) K finish: 1:08:24 speed: 580 (452) K/sec Pierre Mavro www.enovance.com DRBD 16 / 62
  • 17. DRBD: Basic usages and understandings Check replication status Another solution consists to launch this command that will provide informations of all DRBD devices : Check $ drbd -overview 0: home Connected Primary/Secondary UpToDate/UpToDate C r--- /home xfs 200G 158G 43G 79% Pierre Mavro www.enovance.com DRBD 17 / 62
  • 18. DRBD: Basic usages and understandings Check replication status Several informations need be checked to know the status replication of a DRBD device : Replication status Connection state (cs) Resources roles (ro) Disk states (ds) Pierre Mavro www.enovance.com DRBD 18 / 62
  • 19. DRBD: Basic usages and understandings Check replication status Connection states (cs) : StandAlone : No network configuration available. The resource has not yet been connected, has been administratively disconnected or has dropped its connection due to failed authentication/split brain. Unconnected : Temporary state, prior to a connection attempt. WFConnection : This node is waiting until the peer node becomes visible on the network. Connected : A DRBD connection has been established, data mirroring is now active. This is the normal state. PausedSync : The local node is the source or target of an ongoing synchronization, but synchronization is currently paused. This may be due to a dependency on the completion of another synchronization process, or due to synchronization having been manually interrupted. Pierre Mavro www.enovance.com DRBD 19 / 62
  • 20. DRBD: Basic usages and understandings Check replication status Resource roles (ro) : Primary : The resource is currently in the primary role, and may be read from and written to. This role only occurs on one of the two nodes, unless dual-primary mode is enabled. Secondary : The resource is currently in the secondary role. It normally receives updates from its peer (unless running in disconnected mode), but may neither be read from nor written to. This role may occur on one or both nodes. Unknown : The resource’s role is currently unknown. The local resource role never has this status. It is only displayed for the peer’s resource role, and only in disconnected mode. Pierre Mavro www.enovance.com DRBD 20 / 62
  • 21. DRBD: Basic usages and understandings Check replication status Disk states (ds) : UpToDate : Consistent, up-to-date state of the data. This is the normal state. Inconsistent : The data is inconsistent. This status occurs immediately upon creation of a new resource, on both nodes (before the initial full sync). Also, this status is found in one node (the synchronization target) during synchronization. Pierre Mavro www.enovance.com DRBD 21 / 62
  • 22. DRBD: Basic usages and understandings Plan 2 Basic usages and understandings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Configuration and initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Check replication status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Node switching and manual synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Remove a DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Pierre Mavro www.enovance.com DRBD 22 / 62
  • 23. DRBD: Basic usages and understandings Node switching You may need to perform maintenance tasks. For instance you need to switch a DRBD volume to the other node. Simply connect to the secondary node and launch : Set secondary as primary $ drbdadm primary <drbd_volume > You can also switch all volumes at once. Double check if you really can do this before proceed : Set secondary as primary $ drbdadm primary all Pierre Mavro www.enovance.com DRBD 23 / 62
  • 24. DRBD: Basic usages and understandings Node switching You also can switch a Primary node to Secondary : Set primary as secondary $ drbdadm secondary <drbd_volume > Notes This will not automatically set the old secondary node to primary state Pierre Mavro www.enovance.com DRBD 24 / 62
  • 25. DRBD: Basic usages and understandings Manual synchronization To start a manual synchronization, you need to invalidate the DRBD device on the current host : Invalidate data on current host $ drbdadm invalidate <drbd_volume > You can do the same from one node to the other node : Invalidate data on remote host $ drbdadm invalidate_remote <drbd_volume > You can then check the status in /proc/drbd Pierre Mavro www.enovance.com DRBD 25 / 62
  • 26. DRBD: Basic usages and understandings Plan 2 Basic usages and understandings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Configuration and initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Check replication status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Node switching and manual synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Remove a DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Pierre Mavro www.enovance.com DRBD 26 / 62
  • 27. DRBD: Basic usages and understandings Remove a DRBD device Before removing a DRBD device, be sure you’ve dumped all data ! Then disconnect the device on both nodes : Disconnect DRBD device $ drbdadm disconnect <drbd_volume > And delete it : Remove DRBD device $ drbdsetup <volume_number > down Change volume_number by the number seen with drbd-overview command. To finish, remove the config files. Pierre Mavro www.enovance.com DRBD 27 / 62
  • 28. DRBD: Use cases Plan 3 Use cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Stop, upgrade and commit changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Stop, upgrade and rollback changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Up, connect...but doesn’t want to come UpToDate . . . . . . . . . . . . . . . . . . . . . . . 35 How to promote a primary when having a dual secondary . . . . . . . . . . . . . . . . . . 39 Pierre Mavro www.enovance.com DRBD 28 / 62
  • 29. DRBD: Use cases Stop, upgrade and commit changes Sometimes, you may need to perform maintenances tasks on a DRBD device. For example, you need to upgrade stuffs on a DRBD device and if success, commit changes to the secondary : DRBD Primary DRBD Secondary Set secondary node in maintenance mode Resume sync Upgrade success Normal state Pierre Mavro www.enovance.com DRBD 29 / 62
  • 30. DRBD: Use cases Stop, upgrade and commit changes To set the secondary node in maintenance, simply stop the synchronisation on the secondary node : Disconnect drbd device $ drbdadm disconnect <drbd_volume > Do what you have to do on the DRBD primary device and once you have finished and everything looks fine, resume the synchronization to get back to the normal state : Connect drbd device $ drbdadm connect <drbd_volume > Pierre Mavro www.enovance.com DRBD 30 / 62
  • 31. DRBD: Use cases Plan 3 Use cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Stop, upgrade and commit changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Stop, upgrade and rollback changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Up, connect...but doesn’t want to come UpToDate . . . . . . . . . . . . . . . . . . . . . . . 35 How to promote a primary when having a dual secondary . . . . . . . . . . . . . . . . . . 39 Pierre Mavro www.enovance.com DRBD 31 / 62
  • 32. DRBD: Use cases Stop, upgrade and rollback changes Sometimes, you may need to perform maintenances tasks on a DRBD device. For example, you need to upgrade stuffs on a DRBD device and if fail, rollback changes from the secondary : DRBD Primary DRBD Secondary Set secondary node in maintenance mode Upgrade failed Normal state Rollback sync Pierre Mavro www.enovance.com DRBD 32 / 62
  • 33. DRBD: Use cases Stop, upgrade and rollback changes To set the secondary node in maintenance, simply stop the synchronisation on the secondary node : Disconnect drbd device $ drbdadm disconnect <drbd_volume > Then you’ve upgraded and want to rollback. You have to switch your secondary node to master node : Promote device as primary $ drbdadm primary <drbd_volume > Pierre Mavro www.enovance.com DRBD 33 / 62
  • 34. DRBD: Use cases Stop, upgrade and rollback changes Then you need to invalidate the datas on the old primary : Invalidate drbd device data $ drbdadm invalidate <drbd_volume > To finish, adjust and connect the device to perform the rollback on the new master node : Adjust and connect $ drbdadm adjust <drbd_volume > $ drbdadm connect <drbd_volume > Pierre Mavro www.enovance.com DRBD 34 / 62
  • 35. DRBD: Use cases Plan 3 Use cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Stop, upgrade and commit changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Stop, upgrade and rollback changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Up, connect...but doesn’t want to come UpToDate . . . . . . . . . . . . . . . . . . . . . . . 35 How to promote a primary when having a dual secondary . . . . . . . . . . . . . . . . . . 39 Pierre Mavro www.enovance.com DRBD 35 / 62
  • 36. DRBD: Use cases Up, connect...but doesn’t want to come UpToDate If you don’t understand why after several up, connect etc.... you don’t see anything changing and still got this kind of event : Check $ drbd -overview 0: drbd0 StandAlone Primary/Unknown UpToDate/DUnknown r----- You need to instigate more. The first thing to do is to look at the logs : /var/log/syslog [ 6375.849509] block drbd0: Split-Brain detected but unresolved, dropping connection! [ 6375.850025] block drbd0: helper command: /sbin/drbdadm split-brain minor-0 [ 6375.852859] block drbd0: helper command: /sbin/drbdadm split-brain minor-0 exit code 0 (0x0) [ 6375.852867] block drbd0: conn( WFReportParams -> Disconnecting ) [ 6375.852873] block drbd0: error receiving ReportState, l: 4! [ 6375.853274] block drbd0: asender terminated [ 6375.853277] block drbd0: Terminating drbd0_asender [ 6375.853506] block drbd0: Connection closed [ 6375.853514] block drbd0: conn( Disconnecting -> StandAlone ) [ 6375.853523] block drbd0: receiver terminated [ 6375.853526] block drbd0: Terminating drbd0_receiver Pierre Mavro www.enovance.com DRBD 36 / 62
  • 37. DRBD: Use cases Up, connect...but doesn’t want to come UpToDate Anyway, it may not be clear enough for you. You can use that command to get a better understanding on the situation : Show all drbd informations $ drbdadm show -gi <drbd_volume > +--< Current data generation UUID >- | +--< Bitmap ’s base data generation UUID >- | | +--< younger history UUID >- | | | +-< older history >- V V V V B63AABEFCFADD87D :2 C30515E7D05B35D :82 C4146DCB8E6BFF :82 C3146DCB8E6BFF :1:1:1:0:0:0:0 ^ ^ ^ ^ ^ ^ ^ -< Data consistency flag >--+ | | | | | | -< Data was/is currently up -to -date >--+ | | | | | -< Node was/is currently primary >--+ | | | | -< Node was/is currently connected >--+ | | | -< Node was in the progress of setting all bits in the bitmap >--+ | | -< The peer ’s disk was out -dated or inconsistent >--+ | -< This node was a crashed primary , and has not seen its peer since >--+ flags: Primary , StandAlone , UpToDate Pierre Mavro www.enovance.com DRBD 37 / 62
  • 38. DRBD: Use cases Up, connect...but doesn’t want to come UpToDate Here the problem is clear : we’re in a split brain situation. So there is no other solutions than doing a full sync. But what if no split brain was detected ? There’s a last solution : Adjust and connect the drbd device $ drbdadm adjust <drbd_volume > $ drbdadm connect <drbd_volume > Now you can check the replication which is resuming. Pierre Mavro www.enovance.com DRBD 38 / 62
  • 39. DRBD: Use cases Plan 3 Use cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Stop, upgrade and commit changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Stop, upgrade and rollback changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Up, connect...but doesn’t want to come UpToDate . . . . . . . . . . . . . . . . . . . . . . . 35 How to promote a primary when having a dual secondary . . . . . . . . . . . . . . . . . . 39 Pierre Mavro www.enovance.com DRBD 39 / 62
  • 40. DRBD: Use cases How to promote a primary when having a dual secondary If you get the dual secondary state after node’s restart, check which node as the ’UpToDate’ state : Check $ drbd -overview 0: drbd0 Connected Secondary/Secondary UpToDate/UpToDate C r----- Here, both are up to date. Simply select one and promote it as primary : Set as primary $ drbdadm primary <drbd_volume > You should now see one primary and a secondary synced. Pierre Mavro www.enovance.com DRBD 40 / 62
  • 41. DRBD: Ninja tricks Plan 4 Ninja tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Create DRBD replication without syncing data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 What to do in Split brain case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Mount secondary node as read only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 All is good, but still doesn’t want to connect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Can’t "create-md" on an old DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 My DRBD device is still there but not the configuration . . . . . . . . . . . . . . . . . . . 58 Speed up sync transfer rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 Pierre Mavro www.enovance.com DRBD 41 / 62
  • 42. DRBD: Ninja tricks Create DRBD replication without syncing data When you have big devices to sync on a DRBD initialization, it could take several hours or days in worst cases. There is a solution to only sync bitmap and have your DRBD device available in a few seconds instead ! Create on both nodes the DRBD device : Create device metadata $ drbdadm create -md <drbd_volume > Writing meta data ... initialising activity log NOT initializing bitmap New drbd meta data block sucessfully created. Pierre Mavro www.enovance.com DRBD 42 / 62
  • 43. DRBD: Ninja tricks Create DRBD replication without syncing data Then bring it up (still on both nodes) : Connect the volume $ drbdadm up <drbd_volume > Only on the master node, initialize the bitmap and uuid of the DRBD device : Only sync bitmap $ drbdadm -- --clear -bitmap new -current -uuid <drbd_volume > Pierre Mavro www.enovance.com DRBD 43 / 62
  • 44. DRBD: Ninja tricks Create DRBD replication without syncing data Promote this master node as primary node to replicate bitmap : Set primary volume $ drbdadm primary <drbd_volume > Now you can check the DRBD is ready to use :-) Check $ drbd -overview 1: drbd1 Connected Primary/Secondary UpToDate/UpToDate C r----- Pierre Mavro www.enovance.com DRBD 44 / 62
  • 45. DRBD: Ninja tricks Plan 4 Ninja tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Create DRBD replication without syncing data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 What to do in Split brain case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Mount secondary node as read only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 All is good, but still doesn’t want to connect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Can’t "create-md" on an old DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 My DRBD device is still there but not the configuration . . . . . . . . . . . . . . . . . . . 58 Speed up sync transfer rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 Pierre Mavro www.enovance.com DRBD 45 / 62
  • 46. DRBD: Ninja tricks What to do in Split brain case If you get this kind of problem in /proc/drbd : Set secondary as primary $ cat /proc/drbd primary/unknown that mean that you are in a split-brain case. First of all you need to check the version on both nodes ! You absolutely need to have the same version to avoid errors (8.0-8.3/8.4/9.0) : Set secondary as primary $ drbdadm --version | grep DRBDADM_VERSION = DRBDADM_VERSION =8.3.13 Pierre Mavro www.enovance.com DRBD 46 / 62
  • 47. DRBD: Ninja tricks What to do in Split brain case You will need to discard all data on the secondary node and reimport all of them from the master node. Umount all drbd Umount all failed DRBD mounted devices before continuing !!! On the master node, we need to connect all failed DRBD devices first : Connect DRBD $ drbdadm connect <drbd_volume > Pierre Mavro www.enovance.com DRBD 47 / 62
  • 48. DRBD: Ninja tricks What to do in Split brain case And then sync all data to the secondary node : Sync from master $ drbdadm -- --discard -my -data connect <drbd_volume > Check the sync state in /proc/drbd : Check sync state $ cat /proc/drbd version: 8.3.7 (api :88/ proto :86 -91) srcversion: EE47D8BF18AC166BE219757 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/ Inconsistent C r. ns :912248 nr:0 dw:0 dr :920640 al:0 bm :55 lo:1 pe :388 ua :2048 ap:0 [=== >................] sync ’ed: 21.9% (3283604/4194304) K finish: 1:08:24 speed: 580 (452) K/sec Mount your DRBD device when finished. Pierre Mavro www.enovance.com DRBD 48 / 62
  • 49. DRBD: Ninja tricks What to do in Split brain case Note : there is a difference between "invalidate" and "discard-my-data" : drbdadm invalidate : This forces the local device of a pair of connected DRBD devices into SyncTarget state, which means that all data blocks of the device are copied over from the peer. This command will fail if the device is not part of a connected device pair. drbdadm – –discard-my-data : Use this option to manually recover from a split-brain situation. In case you do not have any automatic after-split-brain policies selected, the nodes refuse to connect. By passing this option you make this node a sync target immediately after successful connect. Pierre Mavro www.enovance.com DRBD 49 / 62
  • 50. DRBD: Ninja tricks Plan 4 Ninja tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Create DRBD replication without syncing data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 What to do in Split brain case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Mount secondary node as read only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 All is good, but still doesn’t want to connect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Can’t "create-md" on an old DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 My DRBD device is still there but not the configuration . . . . . . . . . . . . . . . . . . . 58 Speed up sync transfer rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 Pierre Mavro www.enovance.com DRBD 50 / 62
  • 51. DRBD: Ninja tricks Mount secondary node as read only There are several ways to mount secondary node in a read only state but several ways to break your data on the secondary node as well. This method is the best to avoid a full resync. First, create a snapshot on your logical volume on top of the DRBD device : Create logical volume $ lvcreate -s -n <lv >-snapshot -L 2G /dev/<vg >/<lv > You’re now able to mount the snapshot as a read only device. To make it work, you absolutely need to specify the filesystem : Mount filesystem $ mount -t ext4 -o ro /dev/<vg >/<lv >-snapshot /mountpoint Backup all your desired data. Pierre Mavro www.enovance.com DRBD 51 / 62
  • 52. DRBD: Ninja tricks Mount secondary node as read only You can check any time that your data are still under sync with ’drbd-overview’ command. When you’ve finished, you need to umount the mountpoint : Unmount $ umount /mountpoint and remove the logical volume : Delete LV $ lvremove /dev/<vg >/<lv >-snapshot Pierre Mavro www.enovance.com DRBD 52 / 62
  • 53. DRBD: Ninja tricks Plan 4 Ninja tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Create DRBD replication without syncing data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 What to do in Split brain case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Mount secondary node as read only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 All is good, but still doesn’t want to connect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Can’t "create-md" on an old DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 My DRBD device is still there but not the configuration . . . . . . . . . . . . . . . . . . . 58 Speed up sync transfer rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 Pierre Mavro www.enovance.com DRBD 53 / 62
  • 54. DRBD: Ninja tricks All is good, but still doesn’t want to connect All your configuration is ok, your logs doesn’t give any errors but the connect method doesn’t seam to do something. The last chance is to try the ’adjust’ command : Adjust $ drbdadm adjust <drbd_volume > $ drbdadm connect <drbd_volume > This should work now Pierre Mavro www.enovance.com DRBD 54 / 62
  • 55. DRBD: Ninja tricks Plan 4 Ninja tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Create DRBD replication without syncing data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 What to do in Split brain case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Mount secondary node as read only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 All is good, but still doesn’t want to connect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Can’t "create-md" on an old DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 My DRBD device is still there but not the configuration . . . . . . . . . . . . . . . . . . . 58 Speed up sync transfer rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 Pierre Mavro www.enovance.com DRBD 55 / 62
  • 56. DRBD: Ninja tricks Can’t "create-md" on an old DRBD device If you get this kind of error when you want to recreate and md device on an old existing one : Initiate DRBD device $ drbdadm create -md <drbd_volume > Found some data ==> This might destroy existing data! <== Do you want to proceed? [need to type ’yes ’ to confirm] yes drbdadm create -md nameResource : exited with code 40 To resolve the problem, erase all data on the current device : Flush disk data $ drbdadm wipe -md <drbd_volume > Now restart the create procedure. Pierre Mavro www.enovance.com DRBD 56 / 62
  • 57. DRBD: Ninja tricks Can’t "create-md" on an old DRBD device If it doesn’t work, because you don’t have your configuration any more for example, you can do it manually : Flush disk data $ dd if=/ dev/zero of=/ dev/sdaX bs=1M count =128 Now restart the create procedure. Pierre Mavro www.enovance.com DRBD 57 / 62
  • 58. DRBD: Ninja tricks Plan 4 Ninja tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Create DRBD replication without syncing data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 What to do in Split brain case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Mount secondary node as read only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 All is good, but still doesn’t want to connect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Can’t "create-md" on an old DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 My DRBD device is still there but not the configuration . . . . . . . . . . . . . . . . . . . 58 Speed up sync transfer rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 Pierre Mavro www.enovance.com DRBD 58 / 62
  • 59. DRBD: Ninja tricks My DRBD device is still there but not the configuration You may want to remove the DRBD device but unfortunately you did it in the wrong way or an automatic script didn’t did it well. The solution consist to recreate the configuration file to get the device recovered. Warning Before creating a configuration file in the drbd.d folder, check that you’ve configured the correct device and ports to avoid conflicting with already used DRBD devices. You can check that you’ve recovered your device with drbd-overview. Then you’ll be able to follow the remove procedure. Pierre Mavro www.enovance.com DRBD 59 / 62
  • 60. DRBD: Ninja tricks Plan 4 Ninja tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Create DRBD replication without syncing data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 What to do in Split brain case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Mount secondary node as read only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 All is good, but still doesn’t want to connect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Can’t "create-md" on an old DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 My DRBD device is still there but not the configuration . . . . . . . . . . . . . . . . . . . 58 Speed up sync transfer rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 Pierre Mavro www.enovance.com DRBD 60 / 62
  • 61. DRBD: Ninja tricks Speed up sync transfer rate If you need to speed up the transfer rate because you need to do a full sync between to hosts, you can change it on the fly. It does not make sense to set a synchronization rate higher than the maximum write throughput on your secondary node, so set it up to a correct rate on the secondary node : Set sync rate drbdsetup /dev/<drbd_volume > syncer -r 900M Then, when you’ve finished and want to revert to the default value : Set sync rate drbdadm adjust <drbd_volume > Pierre Mavro www.enovance.com DRBD 61 / 62