ACI MultiPod 구성 방법에 대한 소개 자료입니다.

1. Date : 2018.03.12
DC TSA / (whchoi@cisco.com)

2. 구성도

3. ACI MultiPoD Topology
[N5K-A-e1/5] 56.56.56.1/30,2/30 [N6K-A-e1/5]
[N5K-B-e1/6] 56.56.56.13/30,14/30 [N6K-A-e1/6]
[LEAF102-e1/49]-
[SPINE201-e1/12]
[LEAF103-e1/49]-
[SPINE201-e1/13]
[LEAF103-e1/50]-
[SPINE202-e1/13]
[LEAF101-e1/5]-
[FI6200-A-e1/5]
[LEAF101-e1/6]-
[FI6200-B-e1/6]
[LEAF102-e1/5]-
[FI6200-A-e1/6]
[LEAF101-e1/6]-
[FI6200-B-e1/5]
[N5K-A-e1/25]-
[LEAF101-e1/25]
[N5K-B-e1/26]-
[LEAF102-e1/26]
[N5K-A-e1/26]-
[LEAF102-e1/25]
[N6K-A-e1/25]-
[LEAF301-e1/25]
[N6K-B-e1/26]-
[LEAF302-e1/25]
[N6K-A-e1/26]-
[LEAF302-e1/25]
[LEAF301-e1/49]-
[SPINE301-e1/1]
[LEAF302-e1/50]-
[SPINE302-e1/2]
[LEAF302-e1/49]-
[SPINE302-e1/2]
[LEAF301-e1/50]-
[SPINE302-e1/1]
[LEAF303-e1/49]-
[SPINE301-e1/3]
[LEAF303-e1/50]-
[SPINE302-e1/3]
[LEAF304-e1/49]-
[SPINE301-e1/4]
[LEAF304-e1/50]-
[SPINE302-e1/4]

4. IPN Switch 구성

5. IPN Switch 구성
Nexus 9K 1st, Nexus 3K Broadcom T2 ,PIM BiDir , IPN Switch .

6. IPN Switch 구성 – Step 1 : VRF 생성 (Option)
vrf context IPN vrf context IPN vrf context IPN vrf context IPN

7. IPN Switch 구성 – Step 2 : OSPF Process 구성
router ospf 1
vrf IPN
router-id 1.1.1.1
router ospf 1
vrf IPN
router-id 1.1.1.2
router ospf 1
vrf IPN
router-id 1.1.1.3
router ospf 1
vrf IPN
router-id 1.1.1.4

8. IPN Switch 구성 – Step 3 : Multicast 구성
interface loopback100
description "IPN-RP"
vrf member IPN
ip address 100.100.100.1/30
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
vrf context IPN
ip pim rp-address 100.100.100.2
group-list 225.0.0.0/8 bidir
ip pim rp-address 100.100.100.2
group-list 239.0.0.0/8 bidir
ip pim ssm range 232.0.0.0/8
interface loopback100
description "IPN-RP"
vrf member IPN
ip address 100.100.100.1/30
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
vrf context IPN
ip pim rp-address 100.100.100.2
group-list 225.0.0.0/8 bidir
ip pim rp-address 100.100.100.2
group-list 239.0.0.0/8 bidir
ip pim ssm range 232.0.0.0/8
interface loopback100
description "IPN-RP"
vrf member IPN
ip address 100.100.100.1/29
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
vrf context IPN
ip pim rp-address 100.100.100.2
group-list 225.0.0.0/8 bidir
ip pim rp-address 100.100.100.2
group-list 239.0.0.0/8 bidir
ip pim ssm range 232.0.0.0/8
interface loopback100
description "IPN-RP"
vrf member IPN
ip address 100.100.100.1/29
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
vrf context IPN
ip pim rp-address 100.100.100.2
group-list 225.0.0.0/8 bidir
ip pim rp-address 100.100.100.2
group-list 239.0.0.0/8 bidir
ip pim ssm range 232.0.0.0/8
PIM Bidirectional RP phantom RP .
https://supportforums.cisco.com/t5/network-infrastructure-documents/rp-redundancy-with-pim-bidir-phantom-rp/ta-p/3117191

9. IPN Switch 구성 – Step 4 : IPN Interface 구성
interface Ethernet2/5
no switchport
mtu 9150
interface Ethernet2/5.4
description "TO-POD1-SPINE201-5"
mtu 9150
encapsulation dot1Q 4
vrf member IPN
ip address 201.1.1.2/30
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
interface Ethernet2/6
no switchport
mtu 9150
interface Ethernet2/6.4
description "TO-POD1-SPINE202-5"
mtu 9150
encapsulation dot1Q 4
vrf member IPN
ip address 201.1.1.10/30
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
interface Ethernet2/5
no switchport
mtu 9150
interface Ethernet2/5.4
description "TO-SPINE-201-6"
mtu 9150
encapsulation dot1Q 4
vrf member IPN
ip address 201.1.1.6/30
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
interface Ethernet2/6
no switchport
mtu 9150
interface Ethernet2/6.4
description "TO-POD1-SPINE-202-6"
mtu 9150
encapsulation dot1Q 4
vrf member IPN
ip address 201.1.1.14/30
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
interface Ethernet2/3
no switchport
mtu 9150
interface Ethernet2/3.4
description "TO-POD2-SPINE-401-3"
mtu 9150
encapsulation dot1Q 4
vrf member IPN
ip address 202.1.1.2/30
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
interface Ethernet2/4
no switchport
mtu 9150
interface Ethernet2/4.4
description "TO-POD2-SPINE-402-3"
mtu 9150
encapsulation dot1Q 4
vrf member IPN
ip address 202.1.1.10/30
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
interface Ethernet2/3
no switchport
mtu 9150
interface Ethernet2/3.4
description "TO-POD2-SPINE-401-4"
mtu 9150
encapsulation dot1Q 4
vrf member IPN
ip address 202.1.1.6/30
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
interface Ethernet2/4
no switchport
mtu 9150
interface Ethernet2/4.4
description "TO-POD2-SPINE-404-4"
mtu 9150
encapsulation dot1Q 4
vrf member IPN
ip address 202.1.1.14/30
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
MultiPoD VLAN VLAN 4 , Reserve .

10. IPN Switch 구성 – Step 5 : DHCP Relay 구성
feature dhcp
Ip dhcp relay
interface Ethernet2/5.4
description "TO-POD1-SPINE201-5"
ip dhcp relay address 10.0.0.1
ip dhcp relay address 10.0.0.2
ip dhcp relay address 10.0.0.3
ip dhcp relay address 10.0.0.4
ip dhcp relay address 10.0.0.5
interface Ethernet2/6.4
description "TO-POD1-SPINE202-5"
ip dhcp relay address 10.0.0.1
ip dhcp relay address 10.0.0.2
ip dhcp relay address 10.0.0.3
ip dhcp relay address 10.0.0.4
ip dhcp relay address 10.0.0.5
feature dhcp
Ip dhcp relay
interface Ethernet2/5.4
description "TO-SPINE-201-6"
ip dhcp relay address 10.0.0.1
ip dhcp relay address 10.0.0.2
ip dhcp relay address 10.0.0.3
ip dhcp relay address 10.0.0.4
ip dhcp relay address 10.0.0.5
interface Ethernet2/6.4
description "TO-POD1-SPINE-202-6"
ip dhcp relay address 10.0.0.1
ip dhcp relay address 10.0.0.2
ip dhcp relay address 10.0.0.3
ip dhcp relay address 10.0.0.4
ip dhcp relay address 10.0.0.5
feature dhcp
Ip dhcp relay
interface Ethernet2/3.4
description "TO-POD2-SPINE-401-3"
ip dhcp relay address 10.0.0.1
ip dhcp relay address 10.0.0.2
ip dhcp relay address 10.0.0.3
ip dhcp relay address 10.0.0.4
ip dhcp relay address 10.0.0.5
interface Ethernet2/4.4
description "TO-POD2-SPINE-402-3"
ip dhcp relay address 10.0.0.1
ip dhcp relay address 10.0.0.2
ip dhcp relay address 10.0.0.3
ip dhcp relay address 10.0.0.4
ip dhcp relay address 10.0.0.5
feature dhcp
Ip dhcp relay
interface Ethernet2/3.4
description "TO-POD2-SPINE-401-4"
ip dhcp relay address 10.0.0.1
ip dhcp relay address 10.0.0.2
ip dhcp relay address 10.0.0.3
ip dhcp relay address 10.0.0.4
ip dhcp relay address 10.0.0.5
interface Ethernet2/4.4
description "TO-POD2-SPINE-404-4"
ip dhcp relay address 10.0.0.1
ip dhcp relay address 10.0.0.2
ip dhcp relay address 10.0.0.3
ip dhcp relay address 10.0.0.4
ip dhcp relay address 10.0.0.5

11. APIC Setup

12. APIC Setup

13. APIC 구성
APIC1 APIC2 APIC3 APIC4 APIC5
Fabric Name DC1 DC1 DC1 DC1 DC1
Fabric ID 1 1 1 1 1
Number Controllers 5 5 5 5 5
POD ID 1 1 1 2 2
Controller ID 1 2 3 4 5
Controller Name apic1 apic2 apic3 apic4 apic5
TEP address Pool 10.0.0.0/16 10.0.0.0/16 10.0.0.0/16 10.0.0.0/16 10.0.0.0/16
VLAN ID for Infra 4093 4093 4093 4093 4093
Admin Credentials admin / password
IPV6 Enable Out of Band N N N N N
Management IP 10.72.86.21/24 10.72.86.22/24 10.72.86.23/24 10.72.86.24/24 10.72.86.25/24
Default Gateway 10.72.86.1 10.72.86.1 10.72.86.1 10.72.86.1 10.72.86.1

14. APIC Setup
Cluster configuration ...
Enter the fabric name [ACI Fabric1 #1]: DC1
Enter the fabric ID (1-128) [1]:
Enter the number of controllers in the fabric (1-9) [3]: 5
Enter the POD ID (1-9): [1]
Enter the controller ID (1-3) [1]:
Enter the controller name [apic1]: apic1
Enter address pool for TEP addresses [10.0.0.0/16]: 10.0.0.0/16
Note: The infra VLAN ID should not be used elsewhere in your environment
and should not overlap with any other reserved VLANs on other platforms.
Enter the VLAN ID for infra network (2-4094): 4093
Enter address pool for BD multicast addresses (GIPO) [225.0.0.0/15]:
Out-of-band management configuration ...
Enter the IP address for out-of-band management: 10.72.86.21/24
Enter the IP address of the default gateway [None]: 10.72.86.1
Enter the interface speed/duplex mode [auto]:
Administrator user configuration...
Enable strong passwords? [Y]
Enter the password for admin:
[ APIC Setup ] [ Cluster Size – . PoD1 APIC Cluster Size ]

15. POD Fabric Setup

16. POD Fabric Setup

17. POD Fabric Setup – Step1 : PoD Setup

18. POD Fabric Setup – Step2 : MultiPoD Setup

19. POD Fabric Setup – Step3 : Remote SPINE Switch 추가
Remote SPINE ,
Discovery .

20. POD Fabric Setup – Step4 : EVPN 연결 구성 설정

21. POD Fabric Setup – Step4 : EVPN 연결 구성 설정

22. POD Fabric Setup – Step5 : Remote POD 연결 구성 설정.

23. POD Fabric Setup – 설정 확인

24. Access Policy Setup

25. Access Policy Setup

26. Access Policy Setup – Step1 : IPN VLAN 설정
MultiPoD VLAN VLAN 4 , Reserve .

27. Access Policy Setup – Step2 : IPN AEP 설정

28. Access Policy Setup – Step3 : IPN용 L3 Outside 설정

29. Access Policy Setup – Step4 : Switch Policy – Spine Profiles 설정

30. Access Policy Setup
– Step5 : Interface Policy – Policy Group – Spine Policy Group

31. Access Policy Setup
– Step6 : Interface Policy – Policy Group – Spine Policy Group

32. Access Policy Setup – Step7 : External Routed Domain 변경

33. Fabric Inventory 설정

34. Fabric Inventory 설정

35. Fabric Inventory 설정 – Remote POD SPINE/LEAF 등록

36. APIC Cluster 서비스 확인

37. ACI Topology 정상 여부 확인

38. Trouble Shooting

39. IPN OSPF Neighbor 점검.
N5600-A# show ip ospf neighbors vrf IPN
OSPF Process ID 1 VRF IPN
Total number of neighbors: 4
Neighbor ID Pri State Up Time Address Interface
1.1.1.3 1 FULL/ - 6d20h 56.56.56.2 Eth1/5
1.1.1.4 1 FULL/ - 6d15h 56.56.56.6 Eth1/6
201.1.1.21 1 FULL/ - 6d18h 201.1.1.1 Eth2/5.4
201.1.1.22 1 FULL/ - 6d18h 201.1.1.9 Eth2/6.4
N5600-B# show ip ospf neighbors vrf IPN
OSPF Process ID 1 VRF IPN
Total number of neighbors: 4
Neighbor ID Pri State Up Time Address Interface
1.1.1.3 1 FULL/ - 6d15h 56.56.56.10 Eth1/5
1.1.1.4 1 FULL/ - 6d15h 56.56.56.14 Eth1/6
201.1.1.21 1 FULL/ - 6d15h 201.1.1.5 Eth2/5.4
201.1.1.22 1 FULL/ - 6d15h 201.1.1.13 Eth2/6.4
N6001-A# show ip ospf neighbors vrf IPN
OSPF Process ID 1 VRF IPN
Total number of neighbors: 4
Neighbor ID Pri State Up Time Address Interface
1.1.1.1 1 FULL/ - 6d20h 56.56.56.1 Eth1/5
1.1.1.2 1 FULL/ - 6d15h 56.56.56.9 Eth1/6
202.1.1.41 1 FULL/ - 6d15h 202.1.1.1 Eth2/3.4
202.1.1.42 1 FULL/ - 6d15h 202.1.1.9 Eth2/4.4
N6001-B# show ip ospf neighbors vrf IPN
OSPF Process ID 1 VRF IPN
Total number of neighbors: 4
Neighbor ID Pri State Up Time Address Interface
1.1.1.1 1 FULL/ - 6d15h 56.56.56.5 Eth1/5
1.1.1.2 1 FULL/ - 6d15h 56.56.56.13 Eth1/6
202.1.1.41 1 FULL/ - 6d15h 202.1.1.5 Eth2/3.4
202.1.1.42 1 FULL/ - 6d15h 202.1.1.13 Eth2/4.4

40. IPN OSPF Neighbor 점검.
DC1-SPINE-201# show ip interface vrf overlay-1 | more
IP Interface Status for VRF "overlay-1"
eth1/5.5, Interface status: protocol-up/link-up/admin-up, iod: 45,
IP address: 201.1.1.1, IP subnet: 201.1.1.0/30
IP broadcast address: 255.255.255.255
IP primary address route-preference: 1, tag: 0
eth1/6.6, Interface status: protocol-up/link-up/admin-up, iod: 46,
IP address: 201.1.1.5, IP subnet: 201.1.1.4/30
IP broadcast address: 255.255.255.255
DC1-SPINE-202# show ip interface vrf overlay-1 | more
IP Interface Status for VRF "overlay-1"
eth1/5.5, Interface status: protocol-up/link-up/admin-up, iod: 45,
IP address: 201.1.1.9, IP subnet: 201.1.1.8/30
IP broadcast address: 255.255.255.255
IP primary address route-preference: 1, tag: 0
eth1/6.6, Interface status: protocol-up/link-up/admin-up, iod: 46,
IP address: 201.1.1.13, IP subnet: 201.1.1.12/30
IP broadcast address: 255.255.255.255
IP primary address route-preference: 1, tag: 0
DC2-SPINE-401# show ip interface vrf overlay-1 | more
IP Interface Status for VRF "overlay-1"
IP unnumbered interface (lo0)
IP unnumbered interface (lo0)
eth1/3.39, Interface status: protocol-up/link-up/admin-up, iod: 44,
IP address: 202.1.1.1, IP subnet: 202.1.1.0/30
IP broadcast address: 255.255.255.255
IP primary address route-preference: 1, tag: 0
eth1/4.40, Interface status: protocol-up/link-up/admin-up, iod: 43,
IP address: 202.1.1.5, IP subnet: 202.1.1.4/30
IP broadcast address: 255.255.255.255
DC2-SPINE-402# show ip interface vrf overlay-1 | more
IP Interface Status for VRF "overlay-1"
IP unnumbered interface (lo0)
IP unnumbered interface (lo0)
eth1/3.39, Interface status: protocol-up/link-up/admin-up, iod: 44,
IP address: 202.1.1.9, IP subnet: 202.1.1.8/30
IP broadcast address: 255.255.255.255
IP primary address route-preference: 1, tag: 0
eth1/4.40, Interface status: protocol-up/link-up/admin-up, iod: 43,
IP address: 202.1.1.13, IP subnet: 202.1.1.12/30
IP broadcast address: 255.255.255.255
IP primary address route-preference: 1, tag: 0

41. MP-BGP EVP 점검
DC1-SPINE-201# show bgp l2vpn evpn summary vrf overlay-1
BGP summary information for VRF overlay-1, address family L2VPN EVPN
BGP router identifier 201.1.1.21, local AS number 100
BGP table version is 501, L2VPN EVPN config peers 2, capable peers 2
61 network entries and 65 paths using 10492 bytes of memory
BGP attribute entries [3/432], BGP AS path entries [0/0]
BGP community entries [0/0], BGP clusterlist entries [0/0]
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
202.1.1.41 4 100 9724 107279 501 0 0 6d15h 4
202.1.1.42 4 100 9727 107310 501 0 0 6d15h 4
DC1-SPINE-202# show bgp l2vpn evpn summary vrf overlay-1
BGP summary information for VRF overlay-1, address family L2VPN EVPN
BGP router identifier 201.1.1.22, local AS number 100
BGP table version is 497, L2VPN EVPN config peers 2, capable peers 2
61 network entries and 65 paths using 10492 bytes of memory
BGP attribute entries [3/432], BGP AS path entries [0/0]
BGP community entries [0/0], BGP clusterlist entries [0/0]
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
202.1.1.41 4 100 9723 107264 497 0 0 6d15h 4
202.1.1.42 4 100 9726 107297 497 0 0 6d15h 4
DC2-SPINE-401# show bgp l2vpn evpn summary vrf overlay-1
BGP summary information for VRF overlay-1, address family L2VPN EVPN
BGP router identifier 202.1.1.41, local AS number 100
BGP table version is 1336, L2VPN EVPN config peers 2, capable peers 2
110 network entries and 163 paths using 18920 bytes of memory
BGP attribute entries [9/1296], BGP AS path entries [0/0]
BGP community entries [0/0], BGP clusterlist entries [2/8]
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
201.1.1.21 4 100 105793 9574 1336 0 0 6d15h 53
201.1.1.22 4 100 105786 9574 1336 0 0 6d15h 53
DC2-SPINE-402# show bgp l2vpn evpn summary vrf overlay-1
BGP summary information for VRF overlay-1, address family L2VPN EVPN
BGP router identifier 202.1.1.42, local AS number 100
BGP table version is 1502, L2VPN EVPN config peers 2, capable peers 2
110 network entries and 163 paths using 18920 bytes of memory
BGP attribute entries [9/1296], BGP AS path entries [0/0]
BGP community entries [0/0], BGP clusterlist entries [2/8]
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
201.1.1.21 4 100 105767 9572 1502 0 0 6d15h 53
201.1.1.22 4 100 105760 9572 1502 0 0 6d15h 53

42. 감사합니다.