MLAG provides invisible Layer 2 redundancy across switches by making them appear as a single logical switch. It establishes dual-connected ports across switches and synchronizes MAC address tables and BPDUs to eliminate duplicate packets and prevent spanning tree loops. MLAG configuration involves bonding dual-connected ports with a common CLAG ID and running the CLAGD protocol over a peer link to synchronize state.

1. ®
®
MLAG: Invisible Layer 2 Redundancy
Scott Emery
Cumulus Networks
May 20, 2015

2. ®
Agenda
u What is MLAG?
u How does MLAG work?
u How to set up an MLAG
u Tools for MLAG analysis and debugging

3. ®
MLAG Introduction
You need to set up a rack of servers for a new
application
u Add some extra servers for redundancy
u Uplink to redundant core switches
u Redundant Internet connections
u Backup power with batteries and generators
u Over-provisioned cooling
You receive a midnight call that everything is
down

4. ®
MLAG Introduction
MLAG – A LAG across more than one node
u Multi-homing for redundancy
u Active-active to utilize all links which otherwise
may get blocked by Spanning Tree
u No modification of LAG partner

5. ®
MLAG Terminology
S1 S2
H1 H2 H3 H4 H5
Secondary Role
ISL – Inter-Switch Link
Dually Connected
Primary Role
Singly Connected

6. ®
MLAG Partner View
S1 S2
H1 H2 H3 H4 H5
Switch

7. ®
The Fundamental Job of MLAG
S1 S2
S1 S2
Make this:
Look like this:
Switch

8. ®
MLAG and LACP
u Both ends must run LACP
u Normally, when connected
to two different systems,
only one link is used
• Common system ID is used on
each switch
u Identification of which ports
on each system are dual-
connected pairs
S1 S2
H1 H2 H3 H4 H5

9. ®
Eliminating Duplicate Packets
u BUM1 packets are flooded and result in:
§ Duplicate packets at dual-connected hosts
§ A dual-connected host receives packets which
it transmitted
1 BUM packets are: Broadcast, Unknown unicast, and Multicast

10. ®
Eliminating Duplicate Packets
S1 S2
H1 H2 H3 H4 H5
H2 sends a BUM packet which goes up the link to S1

11. ®
Eliminating Duplicate Packets
S1 sends the packet out all interfaces in the bridge, except the
interface on which the packet arrived
S1 S2
H1 H2 H3 H4 H5

12. ®
Eliminating Duplicate Packets
S2 sends the packet out all interfaces in the bridge, except the
interface on which the packet arrived
S1 S2
H1 H2 H3 H4 H5

13. ®
Eliminating Duplicate Packets
u Dual-connected hosts receive duplicate
copies of the packet
u Dual-connected hosts which send BUM
packets receive the packet they sent
u To fix this: Packets received on the ISL are
not forwarded to dual-connected ports

14. ®
Eliminating Duplicate Packets
S2 only sends packet out singly-connected interfaces
S1 S2
H1 H2 H3 H4 H5

15. ®
MAC Address Learning
u To act as a single logical switch, both switches
must synchronize their MAC address tables
§ Addresses learned on dual-connected ports are
added to the corresponding port on the other
switch
§ Addresses learned on singly-connected ports are
added to the ISL on the other switch
§ Address learning is disabled on the ISL

16. ®
MAC Address Learning
H2 sends a BUM packet, S1 learns the port to H2
S1 S2
H1 H2 H3 H4 H5
H2

17. ®
MAC Address Learning
S1 sends the packet out all interfaces in the bridge, except the
interface on which the packet arrived
S1 S2
H1 H2 H3 H4 H5
H2

18. ®
MAC Address Learning
S2 would ordinarily learn H2 on the ISL and forward the
packet out all singly-connected ports
S1 S2
H1 H2 H3 H4 H5
H2
H2

19. ®
MAC Address Learning
But, learning is disabled on the ISL. Instead, S1 sends a MAC
sync message to S2 which adds H2 to the dual connected port
S1 S2
H1 H2 H3 H4 H5
H2 H2

20. ®
MAC Address Learning
For singly-connected hosts, the MAC sync message causes the
address to be added to the ISL
S1 S2
H1 H2 H3 H4 H5
H1
H1

21. ®
MAC Address Learning
Final MAC address tables may look like this. Red: Address
originally learned on switch. Blue: Address added by MAC sync
S1 S2
H1 H2 H3 H4 H5
H2 H5H1 H4H3
H5
H2
H1
H4H3

22. ®
Switch-Switch MLAG
u Just like a host can be connected to two
switches, a pair of MLAG'd switches can be
connected to another pair of MLAG'd
switches
§ Used to create larger redundant L2 networks
§ Each pair of MLAG'd switches views the other
switches as a single logical switch

23. ®
Switch-Switch MLAG
S3 S4
S1 S2

24. ®
Switch-Switch MLAG
S3 S4Switch
S1 S2Switch

25. ®
Spanning Tree
u One switch is set as the primary,
the other is secondary
u Both switches use the same
bridge ID, dual connected ports
have the same port ID
u Only primary sends BPDUs on
dual-connected ports
u BPDUs received on dual
connected ports are sent to the
peer unmodified
u BPDUs received on the root port
are sent to the peer unmodified
u Source MACs of BPDUs received
on peer link are checked
u Peer link never blocks
S1 S2
M1
R1

26. ®
Split Brain
u If one switch sees that
the ISL is down it
cannot distinguish
between the link going
down (split brain) and
the peer switch going
down (solo)
u A backup link is used to
make this distinction
S1 S2
H1 H2 H3 H4 H5
S1 S2
H1 H2 H3 H4 H5
??? Which One ???

27. ®
Split Brain
u When the ISL goes
down, the backup
link can determine
if the peer switch is
still alive
S1 S2
H1 H2 H3 H4 H5

28. ®
Configuring MLAG
In /etc/network/interfaces put all dual-connected ports
in an 802.3ad bond and assign them a clag-id
auto bond1
iface bond1 inet static
bond-slaves swp48
bond-mode 802.3ad
bond-miimon 100
bond-use-carrier 1
bond-lacp-rate 1
bond-min-links 1
bond-xmit_hash_policy layer3+4
clag-id 1
auto bond11
iface bond11 inet static
bond-slaves swp4
bond-mode 802.3ad
bond-miimon 100
bond-use-carrier 1
bond-lacp-rate 1
bond-min-links 1
bond-xmit_hash_policy layer3+4
clag-id 1
Switch S1 Switch S2

29. ®
Configuring MLAG
In /etc/network/interfaces assign clagd
parameters on a VLAN sub-interface of the ISL link
auto peer6.4000
iface peer6.4000 inet static
address 169.254.0.1
netmask 255.255.255.0
clagd-peer-ip 169.254.0.2
clagd-sys-mac 44:38:39:ff:bb:01
clagd-backup-ip 192.168.1.101
auto peer16.4000
iface peer16.4000 inet static
address 169.254.0.2
netmask 255.255.255.0
clagd-peer-ip 169.254.0.1
clagd-sys-mac 44:38:39:ff:bb:01
clagd-backup-ip 192.168.1.100
Switch S1 Switch S2

30. ®
MLAG Tools
clagctl can be used to get the current state of
the MLAG
# clagctl
The peer is alive
Peer Priority, ID, and Role: 32768 00:02:00:00:00:17 primary
Our Priority, ID, and Role: 32768 70:72:cf:e9:f0:76 secondary
Peer Interface and IP: peer6.4000 169.254.0.2
Backup IP: 192.168.1.101 (active)
System MAC: 44:38:39:ff:bb:01
Dual Attached Ports
Our Interface Peer Interface CLAG Id
---------------- ---------------- -------
bond4 bond14 4
bond5 bond15 5
bond1 bond11 1
bond2 bond12 2
bond3 bond13 3
$ clagctl
The peer is alive
Our Priority, ID, and Role: 32768 00:02:00:00:00:17 primary
Peer Priority, ID, and Role: 32768 70:72:cf:e9:f0:76 secondary
Peer Interface and IP: peer16.4000 169.254.0.1
Backup IP: 192.168.1.100 (active)
System MAC: 44:38:39:ff:bb:01
Dual Attached Ports
Our Interface Peer Interface CLAG Id
---------------- ---------------- -------
bond14 bond4 4
bond15 bond5 5
bond12 bond2 2
bond13 bond3 3
bond11 bond1 1
Switch S1 Switch S2

31. ®
MLAG Tools
/var/log/syslog contains MLAG status changes
# grep clagd /var/log/syslog
May 19 16:25:31 act-5712-08 clagd[7253]: Beginning execution of clagd version 1.1.0
May 19 16:25:31 act-5712-08 clagd[7253]: Invoked with: /usr/sbin/clagd --daemon 169.254.0.2
peer6.4000 44:38:39:ff:bb:01
May 19 16:25:31 act-5712-08 clagd[7258]: Role is now secondary
May 19 16:25:32 act-5712-08 clagd[7258]: Initial config loaded
May 19 16:25:33 act-5712-08 clagd[7258]: The peer switch is active.
May 19 16:25:33 act-5712-08 clagd[7258]: Initial data sync from peer done.
May 19 16:25:33 act-5712-08 clagd[7258]: Initial handshake done.
May 19 16:25:33 act-5712-08 clagd[7258]: Initial data sync to peer done.
May 19 16:25:37 act-5712-08 clagd[7258]: bond2 is now dual connected.
May 19 16:25:37 act-5712-08 clagd[7258]: bond3 is now dual connected.
May 19 16:25:37 act-5712-08 clagd[7258]: bond1 is now dual connected.
May 19 16:25:37 act-5712-08 clagd[7258]: bond5 is now dual connected.
May 19 16:25:37 act-5712-08 clagd[7258]: bond4 is now dual connected.

32. ®
© 2014 Cumulus Networks. Cumulus Networks, the Cumulus Networks Logo, and Cumulus Linux are trademarks or registered trademarks of Cumulus Networks, Inc.
or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. The registered trademark Linux® is used pursuant to a
sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis.
§ Thank You!
®
cumulusnetworks.com 32