The document discusses Cisco's Application Centric Infrastructure (ACI) Multi-Pod, which allows a single ACI fabric to span multiple physical locations using virtualization. It connects multiple pods, each with their own spine and leaf architecture, using an Inter-Pod Network. The control plane is maintained by a single APIC cluster running MP-BGP EVPN to advertise routes between pods and provide a unified network view, while data traffic flows directly between pods over the Inter-Pod Network.

1. ACI Multi-Pod
2018/07/20
( tayokois@cisco.com )
SDN

2. © 2 0 1 8 C i s c o a n d / o r i t s a f f i l i a t e s . A l l r i g h t s r e s e r v e d . C i s c o P u b l i c
• ACI
• ACI Multi Pod
•
•

3. © 2 0 1 8 C i s c o a n d / o r i t s a f f i l i a t e s . A l l r i g h t s r e s e r v e d . C i s c o P u b l i c
ACI
Multi-Pod

4. © 2 0 1 8 C i s c o a n d / o r i t s a f f i l i a t e s . A l l r i g h t s r e s e r v e d . C i s c o P u b l i c

5. © 2 0 1 8 C i s c o a n d / o r i t s a f f i l i a t e s . A l l r i g h t s r e s e r v e d . C i s c o P u b l i c
ACI C osfu a P a
S
du rui P a
S
3 e u I
supL
B p i A
0
9 rui

6. © 2 0 1 8 C i s c o a n d / o r i t s a f f i l i a t e s . A l l r i g h t s r e s e r v e d . C i s c o P u b l i c
ACI /
(MAC + )
COOP
MP-BGP

7. © 2 0 1 8 C i s c o a n d / o r i t s a f f i l i a t e s . A l l r i g h t s r e s e r v e d . C i s c o P u b l i c
ACI A )
L2 DVXLANL2 D L2 D
(

8. © 2 0 1 8 C i s c o a n d / o r i t s a f f i l i a t e s . A l l r i g h t s r e s e r v e d . C i s c o P u b l i c
APIC / APIC /
Pod ‘A’ Pod ‘n’
MP-BGP - EVPN
Multi-Pod (2.0 )
…
IPN
APIC Cluster
Floor1 Floor2
ACI Fabric
Stretched Fabric
APIC Cluster
IP
Site ‘A’ Site ‘n’
MP-BGP - EVPN
Multi-Site (3.0 )
…
ACI Fabric 2ACI Fabric 1
Multi-Fabric (with L2 and L3 DCI)
L2/L3 DCI
Inter-Site
App
8
ACI
Multi Site
ACI

9. © 2 0 1 8 C i s c o a n d / o r i t s a f f i l i a t e s . A l l r i g h t s r e s e r v e d . C i s c o P u b l i c
• Pod: Spine Leaf
• Multi-Pod: "1APIC " Pod
ACI Multi-Pod
MP-BGP - EVPN
APIC Cluster
Availability Zone ‘A’
Inter Pod Network
VXLAN
Pod 1 Pod 2
Control Plane
Data Plane

10. © 2 0 1 8 C i s c o a n d / o r i t s a f f i l i a t e s . A l l r i g h t s r e s e r v e d . C i s c o P u b l i c
ACI Multi-Pod
1
L3
L3
( 50m sec RTT**)
10G*/40G/100G
10G*/40G/100G
10G*/40G/100G
10G*/40G/100G
POD 1 POD n
…
DC
10G*/40G/100G 10G*/40G/100G
APIC Cluster
2 DC
POD 1 POD 2Dark fiber/DWDM
( 50 m sec RTT**)
10G*/40G/100G 10G*/40G/100G
1G/10G/40G/100G
APIC Cluster
* 10G EX/FX 9364C Spine
POD1 POD2
POD3
3 ( ) DC
Dark fiber/DWDM
( 50 m sec RTT**)
1 0 G * / 4 0 G / 1 0 0 G 1 0 G * / 4 0 G / 1 0 0 G
1 0 G * / 4 0 G / 1 0 0 G
1G/10G/40G/100G
** 50 msec 2.3(1)

11. © 2 0 1 8 C i s c o a n d / o r i t s a f f i l i a t e s . A l l r i g h t s r e s e r v e d . C i s c o P u b l i c
Multi-Pod

12. © 2 0 1 8 C i s c o a n d / o r i t s a f f i l i a t e s . A l l r i g h t s r e s e r v e d . C i s c o P u b l i c
Multi-Pod
Inter Pod Network
Pod 1 Pod 2
IS-IS, COOP
MP-BGP(L3out)
OSPF, MP-BGP
IS-IS, COOP
MP-BGP(L3out)

13. © 2 0 1 8 C i s c o a n d / o r i t s a f f i l i a t e s . A l l r i g h t s r e s e r v e d . C i s c o P u b l i c
Multi-Pod Spine
IPN
Routed Sub-Interface
Pod Spine
MP-BGP Loopback Interface
( Spine )
MP-BGP Next Hop
Loopback Interface
(Pod Spine )

14. © 2 0 1 8 C i s c o a n d / o r i t s a f f i l i a t e s . A l l r i g h t s r e s e r v e d . C i s c o P u b l i c
Spine
• OSPF
• Spine
10G*, 40G, 100G
APIC
• DHCP Relay
• RTT 50msec (2.3 )
• QoS
• Multicast BiDir PIM
• Jumbo MTU (9150Byte)
Inter Pod Network
* 10G EX/FX 9364C Spine
QSA

15. © 2 0 1 8 C i s c o a n d / o r i t s a f f i l i a t e s . A l l r i g h t s r e s e r v e d . C i s c o P u b l i c
• DHCP Relay
• APIC IP
• RTT 50msec (2.3 )
• QoS
• APIC APIC Read Only
APIC
DHCP Discover
SYNC SYNC

16. © 2 0 1 8 C i s c o a n d / o r i t s a f f i l i a t e s . A l l r i g h t s r e s e r v e d . C i s c o P u b l i c
• Multicast BiDir PIM
• ACI Fabric
BUM (Broadcast, Unknown Unicast, Multicast)
Multicast
• Jumbo MTU (9150Byte)
• ACI Jumbo
• ACI Fragmentation

17. © 2 0 1 8 C i s c o a n d / o r i t s a f f i l i a t e s . A l l r i g h t s r e s e r v e d . C i s c o P u b l i c

18. © 2 0 1 8 C i s c o a n d / o r i t s a f f i l i a t e s . A l l r i g h t s r e s e r v e d . C i s c o P u b l i c
• Inter Pod
• Pod
IPN
Inter Pod Network
Pod 1 Pod 2

19. © 2 0 1 8 C i s c o a n d / o r i t s a f f i l i a t e s . A l l r i g h t s r e s e r v e d . C i s c o P u b l i c

20. © 2 0 1 8 C i s c o a n d / o r i t s a f f i l i a t e s . A l l r i g h t s r e s e r v e d . C i s c o P u b l i c
Pod 1
(10.0.0.0/16)
Pod 2
(10.1.0.0/16)
APIC1
10.0.0.1
APIC2
10.0.0.2
APIC3
10.0.0.3
172.31.0.1/31
172.31.0.3/31 172.31.0.11/31
172.31.255.1/32
172.31.255.2/32 172.31.255.11/32
172.31.254.1/32
Routing Sub-Interface
MP-BGP Loopback
MP-BGP next hop Loopback
172.31.254.2/32
172.31.255.255/32

21. © 2 0 1 8 C i s c o a n d / o r i t s a f f i l i a t e s . A l l r i g h t s r e s e r v e d . C i s c o P u b l i c
IPN
10.0.0.0/16
10.0.0.1/32
10.0.0.2/32
10.0.0.3/32
10.0.0.33/32
10.0.0.34/32
10.0.0.35/32
10.0.120.65/32
10.0.120.69/32
172.31.254.1/32
172.31.254.2/32
172.31.255.1/32
172.31.255.2/32
172.31.255.11/32
Pod1
IPN
APIC
Spine
Loopback
Spine Loopback
10.1.0.0/16
10.1.0.33/32
10.1.0.34/32
10.1.0.35/32
10.1.8.64/32
Pod2
Spine
Loopback
Spine Loopback
Pod Loopback
(MP-BGP NextHop)
Spine Loopback
(BGP )

22. © 2 0 1 8 C i s c o a n d / o r i t s a f f i l i a t e s . A l l r i g h t s r e s e r v e d . C i s c o P u b l i c
• APIC Pod SSH
• APIC Pod ping
Multi-Pod
apic1# attach Pod2-Leaf1 apic1# ssh Pod2-Leaf1 –b {{APIC IP}}
apic1# ping -I bond0.3967 Pod2-Leaf1

23. © 2 0 1 8 C i s c o a n d / o r i t s a f f i l i a t e s . A l l r i g h t s r e s e r v e d . C i s c o P u b l i c
APIC Pod
Multi-Pod
admin@apic1:~> ip route
default via 172.16.255.254 dev oobmgmt metric 16
10.0.0.0/16 via 10.0.0.30 dev bond0.3967 src 10.0.0.1
10.0.0.30 dev bond0.3967 scope link src 10.0.0.1
10.0.8.65 via 10.0.0.30 dev bond0.3967
10.0.8.66 via 10.0.0.30 dev bond0.3967
169.254.1.0/24 dev teplo-1 proto kernel scope link src 169.254.1.1
169.254.254.0/24 dev lxcbr0 proto kernel scope link src 169.254.254.254
172.16.0.0/16 dev oobmgmt proto kernel scope link src 172.16.10.1
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1

24. © 2 0 1 8 C i s c o a n d / o r i t s a f f i l i a t e s . A l l r i g h t s r e s e r v e d . C i s c o P u b l i c
• Multi-Pod 1 APIC
•

25. © 2 0 1 8 C i s c o a n d / o r i t s a f f i l i a t e s . A l l r i g h t s r e s e r v e d . C i s c o P u b l i c
ü ACI Multi-Pod White Paper
•http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-
infrastructure/white-paper-c11-737855.html?cachemode=refresh
ü ACI Multi-Pod Configuration White Paper
•https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-
infrastructure/white-paper-c11-739714.html
ü ACI Multi-Pod Cisco Live Las Vegas 2017
•https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-
infrastructure/white-paper-c11-739714.html
ü ACI Multi-Pod and Network Services Integration White Paper
•Coming Soon!
2