Brkaci 1002

Introduction to ACI for Network
Admins
Steve Sharman
BRKACI-1002

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Abstract
ACI for the Network Administrator takes the attendee through building an
ACI network through the eyes of the network administrator.
The session will focus on logical and concrete models, how to use bridge
domains and VLANs, how to configure external connectivity from the
fabric, and how to integrate third party devices.
BRKACI-1002 3

Session Objectives
• Understand ACI through the eyes of the network administrator
• Understand ACI building blocks
• Understand external and services integration
• Consuming ACI with Automation
• Getting started with ACI
BRKACI-1002 4

Before We Start, Let’s Get to Know
Each Other …

• How do we sell ACI?
• Understanding ACI Building Blocks
• VMware Integration
• External Connectivity
• Service Graph Integration
• Consuming ACI with Automation
• Getting Started with ACI
Agenda

Let me talk to you
about Cisco ACI…
ACI is all about
applications and I
don’t know
applications…
Are all applications
based on three
tiers…?
BRKACI-1002

In Reality ACI is all About
Networking and How You Deploy
Applications Onto the Network!

At a Very Basic Level ACI is Really Just a
Clos Network of Nexus 9k Switches with a
Management Platform
Charles Clos – 1952
https://en.wikipedia.org/wiki/Clos_network

The Network Management Platform
(APIC) Provides You With a Single
Place From Which to Manage the
Network

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKACI-1002
Is ACI an Overlay or Underlay Network?
12

ACI is a Software Defined Network Which
Uses VXLAN to Transport Packets Between
Switches Across an Automated IP Fabric with
End to End Header Visibility

IETF Draft
BRKACI-1002 14

ACI Can Transport Any IP (and non IP)
Traffic Including “Overlay” Networks
Based on VXLAN*, NVGRE* etc.
* ACI has visibility of the outer header

To Help Understand ACI, Let’s Look at a
Real Customer Example

OSPF Area 30OSPF Area 20
OSPF Area 10
(stub)
BRKACI-1002
CPoC – Large Financial Organisation
APIC
APIC
APIC
Spirent Test
Centre
Spirent Test
Centre
Spirent Test
Centre
ESX-01ESX-02
c3850
n7706 n9504n7706-01 n7706-02
n5672-01 n5672-02
L2
L3
OSPF
Area 0
e1/3
e1/1 e1/2 e1/1 e1/2
e1/7 e1/8
e1/15 e1/15 e1/15e1/5 e1/6 e1/11 e1/12
17

“ACI Has to be Operationally Simple. Our Ops Team are
Used to Using the CLI, if They’re Not Comfortable with
Troubleshooting ACI it Won’t be Accepted!”
18

Step 1 – Building the Network and
Provisioning Interfaces

Physically Building the ACI Network
BRKACI-1002
APIC
APIC
APIC
Management options:
• GUI (basic/advanced)
• CLI
• XML/JSON
• Scripting
• Open API
• Automation
Benefits:
• Distributed, Centralised Management
• Full traffic visibility*
• Self documenting
• Integrated virtual and physical network
• Integrated L4-7 device management
• Policy defined network
20

Network Provisioning
Manual
setup
Quick Start
wizard
21

Switch Policies
Leaf Profiles
Leafs_101_and_102
Which switches
should be
configured?
Interface Policies
Policies
CDP_enabled
LACP_Active
Interface Policies
Leaf Policy Groups
vPC_to_UCS_FI_A
SVI_to_outside
AAEP
(Allowed VLANs)
vCenter-01-DVS-01
UCS-phys-svrs
Outside-Fabric
Pools
VLAN/VXLAN
vCenter-01-DVS-01
UCS-phys-svrs
Outside-Fabric
Virtual Machine
Domains
(vSwitches)
vCenter-01-DVS-01
Phy/Out Domains
(VLAN mgmt)
UCS-phys-svrs
Outside-Fabric
What interface
settings do I want
to configure?
What type of
interface do I want
to configure?
Group my VLANs
together to allow
them on an interface
What “function” do
I want to allocate
VLANs for?
Which DVS do I
want to configure?
Where do I want
to use my VLANs?
BRKACI-1002
Policy Defined Network
Concrete Model
Logical Model
APIC
APIC
APIC
Interface Selector
1/21
Leaf Profile
vPC_to_UCS_FI_A
Security Domain
(optional)
Interface Policies
Leaf Profiles
Which interfaces
should be configured?
22

A Consistent Naming Convention is
Critical for Simple Troubleshooting

Example Rack Layout
24

Example Naming Approach
• VLAN Pool
• Domains (L2, L3, Phys)
• AAEP (allowed VLANs)
• Interface Polices (settings)
• Leaf Policy Groups (aggregated settings)
• Leaf Profiles (settings mapped to interfaces)
• Switch Profiles (interfaces mapped to
switches)
• Customer_A_01
• Customer_A_L3_01
• Customer_A_01
• 10G, CDP_enabled
• 10G_access_c3850-01
• 101_to_c3850-01
• A1_101
• Tenant_Name
• Tenant_Name
• Tenant_Name
• Enabled/Disabled
• PortSpeed_PortType_Usage
• Rack_ID/Switch_ID_to_ConnectedDevice
• Rack_ID or Rack_ID_SwitchID
25

Example Rack DetailsLegend TenantName TenantName Tenant Comment TenantName Settings PortSpeed_PortType_Usage Rack_PortSpeed_PortType_Tenant_ConnectedDevice InterfaceNumber RackID (vPC)
Tenant RackID_Switch (single connection)
Tenant
Tenant_vDS_Number
Connected Device Type Tenant (Consumer) VLAN Pool Domain Domain Type AAEP Interface Policies Leaf Policy Groups Leaf Profiles Access Port Selector Switch ID(s)
(allowed VLANs) (Interface Settings) (Interface Type) (Interface number) (Switch Profiles)
1Gbps
cdp_enabled
10Gbps
cdp_enabled
1Gbps
cdp_enabled
10Gbps
cdp_enabled
10Gbps
cdp_enabled
LCAP_active
10Gbps
cdp_enabled
LCAP_active
10Gbps
cdp_enabled
LCAP_active
10Gbps
cdp_enabled
LCAP_active
10Gbps
cdp_enabled
10Gbps
cdp_enabled
B3_10Gbps_acc_Linux 1/21-30 B3
10Gbps_acc_Linux
A2_10Gbps_acc_Linux 1/21-30 A2
B2_10Gbps_acc_Linux 1/21-30 B2
A3_10Gbps_acc_Linux 1/21-30 A3
Linux Host Tenant_01 Tenant_01 Tenant_01 Physical Tenant_01
ESX Host Tenant_01 Tenant_01 Tenant_01 Physical Tenant_01 10Gbps_acc_ESX
A3_10Gbps_acc_ESX 1/1-20 A3
B3_10Gbps_acc_ESX 1/1-20 B3
A2_10Gbps_acc_ESX 1/1-20 A2
B2_10Gbps_acc_ESX 1/1-20 B2
F5 IO Tenant_01 Tenant_01 Tenant_01 Outside Routed Tenant_01 10Gbps_acc_F5_io
Outside Routed
F5 Management Tenant_01 Tenant_01 Tenant_01 Physical
1Gbps_acc_ASA_mgmtASA Firewall Management Tenant_01 Tenant_01 Tenant_01 Physical Tenant_01
1/2
A1_101
B1_121
A1_101
B1_121
B1_1Gbps_acc_F5_mgmt
B1_10Gbps_acc_ASA_io 1/3 B1
A1
10Gbps_acc_ASA_ioTenant_01ASA IO Tenant_01 Tenant_01 Tenant_01
A1_10Gbps_acc_ASA_io
B1_10Gbps_acc_F5_io 1/4 B1
A1_10Gbps_acc_F5_io 1/4 A1
A1_1Gbps_acc_F5_mgmt 1/2
Tenant_01 1Gbps_acc_F5_mgmt
B1_10Gbps_vPC_to_N5k_02 1/10
Tenant_01 10Gbps_vPC_N7k_02 B1_10Gbps_vPC_to_N7k_02 1/11 B1
Tenant_01 10Gbps_vPC_N7k_01 A1_10Gbps_vPC_to_N7k_01 1/11 A1
Outside Bridged Tenant_01
Nexus 7k Tenant_01 Tenant_01 Tenant_01 Outside Routed
Tenant_01 10Gbps_vPC_N5k_02
1/3
B1_1Gbps_acc_ASA_mgmt 1/1
A1_1Gbps_acc_ASA_mgmt 1/1
B1
Nexus 7k Tenant_01 Tenant_01 Tenant_01 Outside Routed
10Gbps_vPC_N5k_01 A1_10Gbps_vPC_to_N5k_01 1/10 A1
Nexus 5k Tenant_01 Tenant_01 Tenant_01 Outside Bridged
Nexus 5k Tenant_01 Tenant_01 Tenant_01
26

How Does it Look When we Apply
the Naming Convention?

OSPF Area 10
(stub)
BRKACI-1002
APIC
APIC
APIC
c3850
n7706 n9504n7706-01 n7706-02
n5672-01 n5672-02
L2
L3
OSPF
Area 0
e1/3
e1/1 e1/2 e1/1 e1/2
e1/7 e1/8
e1/15 e1/15 e1/15e1/5 e1/6 e1/11 e1/12
Spirent Test
Centre
Spirent Test
Centre
Spirent Test
Centre
ESX-01ESX-02
28

Interface Policies
CDP_enabled
VLAN Pool
Customer_A_01
External Routed
Domain
Customer_A_L3_01
AAEP
Customer_A_01
Leaf Policy Group
10G_acc_c3850
Interface Policies
10G
Leaf Profile
li07_to_
ld04-c3850-01
Leaf Profile
Leafs_101_and_102
Interface Selector
1/3
BRKACI-1002
10G_acc_c3850
Concrete Model
Logical Model
Rack/Switch to
connected device
Interface setting
group
29

Interface Policies
CDP_enabled
BRKACI-1002
10G_acc_n7706
VLAN Pool
Customer_A_01
External Routed
Domain
Customer_A_L3_01
AAEP
Customer_A_01
Leaf Policy Group
10G_acc_n7706
Interface Policies
10G
Leaf Profile
li07_to_
lg05-n7706-01
Leaf Profile
Leafs_101_and_102
Interface Selector
1/7
Concrete Model
Logical Model
Rack/Switch to
connected device
Interface setting
group
30

Interface Policies
CDP_enabled
BRKACI-1002
10G_acc_n9504
VLAN Pool
Customer_A_01
External Routed
Domain
Customer_A_L3_01
AAEP
Customer_A_01
Leaf Policy Group
10G_acc_n9504
Interface Policies
10G
Leaf Profile
li07_to_
lg11-n9504-01
Leaf Profile
Leafs_101_and_102
Interface Selector
1/8
Concrete Model
Logical Model
Rack/Switch to
connected device
Interface setting
group
31

Interface Policies
CDP_enabled
BRKACI-1002
10G_acc_Spirent_Test_Center
Leaf Profile
Leaf_101
VLAN Pool
Customer_A_01
Physical Domain
Customer_A_Phys_01
AAEP
Customer_A_01
Leaf Policy Group
10G_acc_Spirent_
Test_Center
Interface Policies
10G
Leaf Profile
li08_104_to_
Spirent_Test_Center
Leaf Profile
li08_103_to_
Spirent_Test_Center
Leaf Profile
li07_101_to_
Spirent_Test_Center
Leaf Profile
Leaf_103
Leaf Profile
Leaf_104
Interface Selector
1/15
Interface Selector
1/15
Interface Selector
1/15
Concrete Model
Logical Model
Rack/Switch to
connected device
Interface setting
group
32

Interface Policies
LLDP_enabled
BRKACI-1002
10G_vPC_esx_li07-c220m4-01
VLAN Pool
Customer_A_01
Physical Domain
Customer_A_Phys_01
AAEP
Customer_A_01
Leaf Policy Group
10G_vPC_esx_
li07-c220m4-01
Interface Policies
10G
Leaf Profile
li08_to_
li07-c220m4-01
Leaf Profile
Leafs_103_and_104
Interface Selector
1/11
Interface Policies
LACP_active
Concrete Model
Logical Model
Rack/Switch to
connected device
Unique Interface
setting group
33

Interface Policies
LLDP_enabled
BRKACI-1002
10G_vPC_esx_li07-c220m4-02
VLAN Pool
Customer_A_01
Physical Domain
Customer_A_Phys_01
AAEP
Customer_A_01
Leaf Policy Group
10G_vPC_esx_
li07-c220m4-02
Interface Policies
10G
Leaf Profile
li07_to_
li07-c220m4-02
Leaf Profile
Leafs_101_and_102
Interface Selector
1/12
Interface Policies
LACP_active
Concrete Model
Logical Model
Rack/Switch to
connected device
Unique Interface
setting group
34

Couldn’t we Reduce the Number of
Leaf Policy Groups?

Yes – Provided That They Are
“Access” Policy Groups With The
Same Interface Policies

Interface Policies
CDP_enabled
BRKACI-1002
10G_acc_ c3850 | n7706 | n9504
Leaf Profile
Leafs_101_and_102
VLAN Pool
Customer_A_01
External Routed
Domain
Customer_A_L3_01
AAEP
Customer_A_01
Interface Policies
10G
Leaf Profile
li07_to_
lg11-n9504-01
Leaf Profile
li07_to_
lg05-n7706-01
Leaf Profile
li07_to_
ld04-c3850-01
Leaf Profile
Leafs_101_and_102
Leaf Profile
Leafs_101_and_102
Interface Selector
1/3
Interface Selector
1/7
Interface Selector
1/8
Leaf Policy Group
10G_acc_c3850
Leaf Policy Group
10G_acc_n7706
Leaf Policy Group
10G_acc_n9504
All Leaf Policy Groups use the
same Interface Policies
(Settings and allowed VLANs)
Concrete Model
Logical Model
37

Interface Policies
CDP_enabled
BRKACI-1002
10G_acc_to_external_L3_switch
Leaf Profile
Leafs_101_and_102
VLAN Pool
Customer_A_01
External Routed
Domain
Customer_A_L3_01
AAEP
Customer_A_01
Interface Policies
10G
Leaf Profile
li07_to_
lg11-n9504-01
Leaf Profile
li07_to_
lg05-n7706-01
Leaf Profile
li07_to_
ld04-c3850-01
Leaf Profile
Leafs_101_and_102
Leaf Profile
Leafs_101_and_102
Interface Selector
1/3
Interface Selector
1/7
Interface Selector
1/8
Leaf Policy Group
10G_acc_to_external_
L3_switch
Consolidated Leaf Policy Group
for Interfaces which use the
same Interface Policies
Concrete Model
Logical Model
38

Couldn’t We Reduce The Number of
Leaf Profiles?

Yes – Provided That They Use The
Same Interfaces On The Physical
Switch(es)

Interface Policies
CDP_enabled
BRKACI-1002
Leaf Profile
Leafs_101_and_102
VLAN Pool
Customer_A_01
External Routed
Domain
Customer_A_L3_01
AAEP
Customer_A_01
Interface Policies
10G
Leaf Profile
li07_to_
lg11-n9504-01
Leaf Profile
li07_to_
lg05-n7706-01
Leaf Profile
li07_to_
ld04-c3850-01
Leaf Profile
Leafs_101_and_102
Leaf Profile
Leafs_101_and_102
Interface Selector
1/3
Interface Selector
1/7
Interface Selector
1/8
Leaf Policy Group
L3_switch
Multiple Leaf Profiles / Interface
Selectors consume the same
Leaf Policy Group
Concrete Model
Logical Model
41

Interface Policies
CDP_enabled
BRKACI-1002
VLAN Pool
Customer_A_01
External Routed
Domain
Customer_A_L3_01
AAEP
Customer_A_01
Leaf Policy Group
L3_switch
Interface Policies
10G
Leaf Profile
li07_to_external
L3_switch
Leaf Profile
Leafs_101_and_102
Interface Selector
1/3, 1/7, 1/8
Consolidated Leaf Profiles /
Interface Selectors consume
the same Leaf Policy Group
Concrete Model
Logical Model
42

Automating “Access Policies” Abstracts the Naming
Rules Away From APIC Thus Ensuring Configuration
Conformance
43

In Large Organisations Having an Automated Approach
to Interface Configuration Could Allow the “rack/stack”
Team to Configure the Switches From a Simple IT
Services Catalogue
44

Notes to Remember:
• Interface Policies can be reused across any interface type
• Leaf Policy Groups for “Access” ports can be used by different Leaf Profiles
• Leaf Policy Groups for PC/vPC cannot be used by different Leaf Profiles
• Leaf Profiles can be used by different Switch Profiles
45

Step 2 – VRF, SVI, Bridge Domain
Configuration

OSPF Area 10
(stub)
BRKACI-1002
APIC
APIC
APIC
Spirent Test
Centre
Spirent Test
Centre
Spirent Test
Centre
ESX-01ESX-02
c3850
n7706 n9504n7706-01 n7706-02
n5672-01 n5672-02
L2
L3
OSPF
Area 0
e1/3
e1/1 e1/2 e1/1 e1/2
e1/7 e1/8
e1/15 e1/15 e1/15e1/5 e1/6 e1/11 e1/12
47

Network Consumption
Quick Start
wizard
Tenants
48

ACI Tenants are Network Wide Administrative
Containers
BRKACI-1002
Tenant: Common
Tenant: Production Tenant: Pre-Production
Objects created in “Common” can be
consumed by other Tenants
BD: 01 BD: 02 BD: 03
VRF: A VRF: B VRF: C
AD DHCPDNS
APIC
APIC
APIC
Tenant: ESX-Hosts
BD: 01 BD: 02 BD: 03
VRF: A
49

Looking Under the Covers at Tenants
BRKACI-1002
apic1# show tenant
Tenant Tag Description
--------------- --------------- ----------------------------------------
avanker
common
fgandola
hyper-v
infra
mgmt
nickmart
nvermand
nvermand-vRA-01
openstack
robvand
rwhitear
ssharman
vmware
apic1#
50

ACI VRFs (aka Private Networks, aka Contexts) Provide
the Routing Function Within a Given Tenant
BRKACI-1002
VRF: VRF-01 (Anycast gateway)
Tenant: Common
APIC
APIC
APIC
51

Multiple VRFs Allow Overlapping IP Address Space and
Integration with External Devices
BRKACI-1002
VRF: VRF-01 (Anycast gateway) VRF: VRF-02 (Anycast gateway)
Tenant: Common
APIC
APIC
APIC
52

Looking Under the Covers at VRFs
BRKACI-1002
apic1# show vrf
Tenant Vrf
---------- ----------
common default
common inside_enforced
common inside_unenforced
common outside_ospf
common outside_static
common outside_vlans
fgandola VRF-01
mgmt inb
mgmt oob
nickmart nickmart
nvermand VRF-01
nvermand VRF-02
nvermand VRF-AVS
53

ACI Bridge Domains are Pervasive Layer 2 Boundaries with
Defined Forwarding Characteristics
BRKACI-1002
Bridge Domain: BD-01
APIC
APIC
APIC
Tenant: Common
BD: 03
Hardware Proxy: No
ARP Flooding: Yes
Unknown Unicast Flooding: Yes
IP Routing: No
BD: 01
Hardware Proxy: No
ARP Flooding: Yes
IP Routing: No
BD: 02
Hardware Proxy: No
ARP Flooding: Yes
IP Routing: No
The Bridge Domain to VRF
association is always required,
even if the VRF is not routing
54

Display Details of a Single Bridge Domain
BRKACI-1002
apic1# show bridge-domain outside_infra-ssharman
Tenant : common
Interface : outside_infra-ssharman
MAC Address : 00:22:BD:F8:19:FF
MTU : inherit
Description :
Multi-Destination Action : bd-flood
Unknown Multicast Action : flood
Unknown MAC Unicast Action : flood
Tenant : ssharman
Interface : Internal_Fabric_02
MTU : inherit
Description :
Unknown Multicast Action : opt-flood
Unknown MAC Unicast Action : proxy
55

A Bridge Domain Uses a Locally Significant VLAN ID on
Each Leaf which Dynamically Maps to a VXLAN ID
BRKACI-1002
Bridge Domain: outside_infra-ssharman
APIC
APIC
APIC
Tenant: Common
Leaf 101
Tenant: Common
BD: outside_infra-ssharman
Leaf 102
Tenant: Common
BD: outside_infra-ssharman
The Bridge Domain to VRF
association is always required,
even if the VRF is not routing
Layer 2 Bridge Domain
carried over VXLAN
56

A Bridge Domain Uses a Locally Significant VLAN ID
Underneath
BRKACI-1002
apic1# fabric 101 show vlan
----------------------------------------------------------------
Node 101 (Leaf-1)
----------------------------------------------------------------
VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
9 infra:default active Eth1/1, Eth1/21, Eth1/22, Po3, Po4
11 common:outside_infra-robvand active Eth1/11, Eth1/21, Eth1/22, Po3,
14 fgandola:www-zone1 active Eth1/33, Po2
15 ssharman:192.168.66.0 active Eth1/21, Eth1/22, Po3, Po4
26 common:outside_infra-ssharman active Eth1/11, Eth1/21, Eth1/22, Po3, Po4, Po8
apic1# fabric 102 show vlan
----------------------------------------------------------------
Node 102 (Leaf-2)
----------------------------------------------------------------
---- -------------------------------- --------- -------------------------------
9 infra:default active Eth1/1, Eth1/21, Eth1/22, Po1, Po2
11 ssharman:L2-to-outside:Group-05 active Eth1/21, Eth1/22, Po1, Po2
14 fgandola:app-zone2 active Eth1/33, Po8
15 -- active Eth1/69, Po7
35 common:outside_infra-ssharman active Eth1/11, Eth1/21, Eth1/22, Po1, Po2, Po4
Leaf 101
Leaf 102
57

VXLANs Require VTEPs
BRKACI-1002
VRF: 01 (Anycast gateway)
BD: 01
Hardware Proxy: Yes
ARP Flooding: No
Unknown Unicast Flooding: No
IP Routing: Yes
APIC
APIC
APIC
Known unicast traffic forwarded directly
between Leaf VTEP’s
Unknown unicast traffic is forwarded to
anycast spine proxy VTEP’s
Logical vPC switch is represented by
anycast Leaf vPC VTEP’s
Multicast and any allowed broadcast
traffic is forwarded to a Group VTEP that
exists on any leaf with membership for
that specific group
VTEP’s may exist in physical or virtual
switches
VTEP VTEP VTEP VTEP
VTEP VTEP VTEP VTEP VTEP VTEP
Tenant: Common
VTEPs are dynamically
created as required
58

A Bridge Domain Uses a VXLAN to Transport Data
Between Leaf Switches
BRKACI-1002
apic1# fabric 101 show vlan id 26 extended
----------------------------------------------------------------
Node 101 (Leaf-1)
----------------------------------------------------------------
---- -------------------------------- --------- -------------------------------
26 common:outside_infra-ssharman active Eth1/11, Eth1/21, Eth1/22, Po3,
Po4, Po8
VLAN Type Vlan-mode Encap
---- ----- ---------- -------------------------------
26 enet CE vxlan-15433637
apic1# fabric 102 show vlan id 35 extended
----------------------------------------------------------------
Node 102 (Leaf-2)
----------------------------------------------------------------
---- -------------------------------- --------- -------------------------------
35 common:outside_infra-ssharman active Eth1/11, Eth1/21, Eth1/22, Po1,
Po2, Po4
VLAN Type Vlan-mode Encap
---- ----- ---------- -------------------------------
35 enet CE vxlan-15433637
Leaf 101
Leaf 102
59

ACI SVIs are Configured on a Given Bridge Domain and
Instantiated on the Associated VRF
BRKACI-1002
APIC
APIC
APIC
Tenant: Common
BD: 01
Hardware Proxy: Yes
ARP Flooding: No
IP Routing: 192.168.10.1/24
60

ACI Bridge Domains can be Configured with Multiple
Subnets/Default Gateways (Secondary)
BRKACI-1002
APIC
APIC
APIC
Tenant: Common
BD: 01
Hardware Proxy: Yes
ARP Flooding: No
IP Routing: 192.168.10.1/24
: 192.168.20.1/24
61

apic1# show bridge-domain outside_infra-ssharman
Tenant : common
MTU : inherit
Description :
Unknown Multicast Action : flood
Unknown MAC Unicast Action : flood
Tenant : ssharman
Interface : Internal_Fabric_02
MTU : inherit
Description :
Unknown Multicast Action : opt-flood
Unknown MAC Unicast Action : proxy
Display Details of a Single Bridge Domain
BRKACI-1002
apic1# show ip interface bridge-domain outside_infra-ssharman
----- IPv4 Bridge-Domain Information: -----
Tenant : common
VRF Member : outside_vlans
IP Addresses : 192.168.29.254/24
192.168.30.254/24
Bridge Domain + SVIBridge Domain + SVI
VRF name
62

ACI Nomenclature
• A Tenant is just an Administrative boundary
• A VRF is a VRF as you know it today
• A Bridge Domain is a L2 segment where flooding rules apply – think VLAN but
without a VLAN ID
• A Bridge Domain is the scope of one or more subnets – think SVI and IP
Secondary
BRKACI-1002 63

Step 3 – Consume the Configured
Interfaces

Leaf Profiles
(Target Switches)
Leafs_101_and_102
AAEP
(Allowed VLANs)
UCS-phys-svrs
Interface Policies
CDP_enabled
LACP_Active
VLAN/VXLAN
(Pools)
UCS-phys-svrs
VLAN mgmt
(Phy/Out Domain)
UCS-phys-svrs
BRKACI-1002
Network Interfaces Must be Configured First!
Concrete Model
Logical Model
APIC
APIC
APIC
Interface Selector
1/21
Security Domain
(optional)
ANP: My_App
EPG: Web
Domain: UCS-phys-svrs
Path: vPC_to_UCS_FI_A
VLAN_10
Path: vPC_to_UCS_FI_B
VLAN_10
Interface Selector
1/22
Leaf Policy Group
vPC_to_UCS_FI_A
Leaf Policy Group
vPC_to_UCS_FI_B
Leaf Profile
vPC_to_UCS_FI_A
Leaf Profile
vPC_to_UCS_FI_B
Leaf Profiles
65

Application Network Profiles – a
Collection of Endpoint Groups

Endpoint Groups – a Collection of
Interfaces and VLANs

EPG Tag: DB (VLAN 12)
Security Zone
EPG Tag: App (VLAN 11)
Security Zone
EPG Tag: Web (VLAN 10)
Security Zone
BRKACI-1002
Option 1: Single EPG on a Single BD with a Single Subnet
– “Standard Networking”
ANP:
My_App
APIC
APIC
APIC
Tenant: My_Tenant
Communication allowed within EPG Communication allowed within EPGCommunication allowed within EPG
BD:192.168.30.x
Hardware Proxy: Yes
ARP Flooding: No
IP Routing: Yes
BD: 192.168.10.X
Hardware Proxy: Yes
ARP Flooding: No
IP Routing: Yes
192.168.20.11/24 192.168.20.12/24 192.168.30.11/24 192.168.30.12/24192.168.10.11/24 192.168.10.12/24
BD: 192.168.20.x
Hardware Proxy: Yes
ARP Flooding: No
IP Routing: Yes Endpoints in EPG identified by
Switch/Interface and VLAN ID
68

Display the Mac Addresses Contained in the EPG
BRKACI-1002
apic1# fabric 101 show mac address-table vlan 37
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 37 0000.0c07.ac08 dynamic - F F po2
* 37 001a.a2d5.c080 dynamic - F F po2
* 37 02a0.981c.b2be dynamic - F F po2
* 37 0026.0bf1.f002 dynamic - F F po2
* 37 0014.384e.26e1 dynamic - F F po2
* 37 0016.355b.ddda dynamic - F F po2
* 37 0060.1646.97da dynamic - F F po2
* 37 0010.18cf.c318 dynamic - F F po2
* 37 0018.74e2.1540 dynamic - F F po2
* 37 0004.02f6.1f13 dynamic - F F po2
* 37 0025.b506.006d dynamic - F F po2
* 37 001b.21be.fa68 dynamic - F F po2
* 37 0025.b501.04af dynamic - F F po2
* 37 0025.b501.049f dynamic - F F po2
* 37 0025.b501.04bf dynamic - F F po2
* 37 0025.b506.007c dynamic - F F po2
* 37 0025.b501.04df dynamic - F F po2
* 37 0025.b506.0027 dynamic - F F po2
* 37 0025.b506.0068 dynamic - F F po2
69

Displaying the Endpoints on the Network
BRKACI-1002
apic1# show endpoints
Tenant Application AEPg End Point MAC IP Address Node Interface Encap
---------- ----------------- ---------------------------------------- ---------- ------------------------------ ----------
vmware ESXi- Host-mgmt 00:25:B5:06:00:1F 192.168.29.43 101 102 vpc 1Gbps_vPC_to_ucs-02-b vlan-8
ssharman
vmware ESXi- Host-mgmt 00:25:B5:06:00:3E 192.168.29.44 101 102 vpc 1Gbps_vPC_to_ucs-02-b vlan-8
ssharman
vmware ESXi- Host-mgmt 00:25:B5:06:00:47 192.168.29.46 101 102 vpc 1Gbps_vPC_to_ucs-02-b vlan-8
ssharman
vmware ESXi- Host-mgmt 00:50:56:86:81:1D 192.168.29.102 101 102 vpc 1Gbps_vPC_to_ucs-02-b vlan-8
ssharman
vmware ESXi- Host-mgmt 00:50:56:86:F7:6A 192.168.29.106 101 102 vpc 1Gbps_vPC_to_ucs-02-b vlan-8
ssharman
70

Displaying the Endpoints on a Leaf
BRKACI-1002
apic1# fabric 101 show endpoint
Legend:
O - peer-attached H - vtep a - locally-aged S - static
V - vpc-attached p - peer-aged L - local M - span
s - static-arp B - bounce
+-----------------------------------+---------------+-----------------+--------------+-------------+
VLAN/ Encap MAC Address MAC Info/ Interface
Domain VLAN IP Address IP Info
+-----------------------------------+---------------+-----------------+--------------+-------------+
common:outside_ospf 101.1.1.1 L
44/common:outside_ospf vxlan-15302582 0000.0c07.ac30 L eth1/96
44/common:outside_ospf vxlan-15302582 0018.74e2.1540 L eth1/96
44/common:outside_ospf vxlan-15302582 001a.a2d5.c080 L eth1/96
13 vlan-2022 0025.b506.0062 LV po3
common:outside_vlans vlan-2022 192.168.22.14 LV
13 vlan-2022 0025.b506.0002 LV po3
32 vlan-22 0000.0c07.ac16 LV po2
32 vlan-22 001a.a2d5.c080 LV po2
32/common:outside_vlans vlan-22 0018.74e2.1540 LV po2
32 vlan-22 0050.5699.9099 LV po2
32 vlan-22 0050.5699.7e05 LV po2
71

Security Zone
Security Zone
Security Zone
BRKACI-1002
Option 2: Multiple EPGs on a Single BD with a Single
Subnet – µSegmentation in IP space
Bridge Domain: 192.168.10.X_24
Gateway: 192.168.10.1
ANP:
My_App
Bridge Domain
Hardware Proxy: Yes
ARP Flooding: No
IP Routing: Yes
APIC
APIC
APIC
Tenant: My_Tenant
192.168.10.11/24 192.168.10.12/24 192.168.10.13/24 192.168.10.14/24 192.168.10.15/24 192.168.10.16/24
Endpoints in EPG identified by
Layer 2 Segment
72

Just Because You Can Doesn't
Always Mean You Should

Security Zone
Security Zone
Security Zone
BRKACI-1002
Option 3a: Multiple EPGs on a Single BD with Multiple
Subnets – IP Secondary
Bridge Domain: multiple_subnets
Gateway: 192.168.10.1
192.168.20.1
192.168.30.1
ANP:
My_App
Bridge Domain
Hardware Proxy: Yes
ARP Flooding: No
IP Routing: Yes
APIC
APIC
APIC
Tenant: My_Tenant
192.168.10.11/24 192.168.10.12/24 192.168.20.11/24 192.168.20.12/24 192.168.30.11/24 192.168.30.12/24
74

Security Zone
Security Zone
Security Zone
BRKACI-1002
Option 3b: Multiple EPGs on a Single BD with Multiple
Subnets – IP Secondary
Bridge Domain: multiple_subnets
Gateway: 192.168.10.1
192.168.20.1
ANP:
My_App
Bridge Domain
Hardware Proxy: Yes
ARP Flooding: No
IP Routing: Yes
APIC
APIC
APIC
Tenant: My_Tenant
192.168.10.11/24 192.168.20.11/24 192.168.10.12/24 192.168.20.12/24 192.168.10.15/24 192.168.10.16/24
75

What About Segmenting Inside an
EPG?

Security Zone
BRKACI-1002
Options 1, 2, and 3 – µSegmentation within an EPG/Port
Group (no East/West traffic flows)
Gateway: 192.168.10.1
ANP:
My_App
Bridge Domain
Hardware Proxy: Yes
ARP Flooding: No
IP Routing: Yes
APIC
APIC
APIC
Tenant: My_Tenant
192.168.10.11/24 192.168.10.12/24 192.168.10.13/24 192.168.10.14/24 192.168.10.15/24 192.168.10.16/24
Communication not allowed within EPG
Layer 2 Segment
77

EPG Tag: All_Web_Servers (VLAN 10)
Security Zone
BRKACI-1002
Options 1, 2, and 3 – µSegmentation within an EPG/Port
Group Based on Machine Attribute
Gateway: 192.168.10.1
ANP:
My_App
Bridge Domain
Hardware Proxy: Yes
ARP Flooding: No
IP Routing: Yes
APIC
APIC
APIC
Tenant: My_Tenant
192.168.10.11/24 192.168.10.12/24 192.168.10.13/24 192.168.10.14/24 192.168.10.15/24 192.168.10.16/24
Layer 2 Segment
Name Contains:
Web_1
Name Contains:
Web_2
Name Contains:
Web_3
Communication allowed within uSeg EPG
78

Advanced Query: How to Find If/Where any VLAN has
Been Used
BRKACI-1002
apic1# moquery -c fvIfConn | grep dn | grep common | grep vlan
dn: uni/epp/br-[uni/tn-common/l2out-outside_infra-robvand/instP-EPG_outside_infra-robvand]/node-102/stpathatt-
[1Gbps_vPC_to_n5548]/conndef/conn-[vlan-47]-[0.0.0.0]
dn: uni/epp/br-[uni/tn-common/l2out-outside_infra-robvand/instP-EPG_outside_infra-robvand]/node-101/stpathatt-
dn: uni/epp/br-[uni/tn-common/l2out-outside_infra-anvanker/instP-EPG_outside_infra-anvanker]/node-102/stpathatt-
dn: uni/epp/br-[uni/tn-common/l2out-outside_infra-anvanker/instP-EPG_outside_infra-anvanker]/node-101/stpathatt-
Interface
Connection
Distinguished
Name
Tenant
Name
VLAN
Managed
Object
Class
79

Where are IP/Mac Addresses Stored?
BRKACI-1002
BD: 01
Hardware Proxy: Yes
ARP Flooding: No
IP Routing: Yes
APIC
APIC
APIC
Proxy Proxy Proxy Proxy
FIB FIB FIB FIB FIB FIB
Tenant: Common
Leaf Local Station Table
contains addresses of ‘all’
hosts attached directly to the
Leaf
10.1.3.11 Port 9
Leaf Global Station Table
contains a local cache of the
fabric endpoints
10.1.3.35 Leaf 3
Proxy A*
10.1.3.35 Leaf 3
10.1.3.11 Leaf 1
Leaf 4
Leaf 6
fe80::8e5e
fe80::5b1a
Spine Proxy Station Table contains
addresses of ‘all’ hosts attached to the
fabric
81

High Level Packet Walk
BRKACI-1002
ANP:
ESXi-Hosts
BD: ESXi
Hardware Proxy: Yes
ARP Flooding: No
IP Routing: Yes
EPG: Host-Mgmt
Security Zone
Leaf-101/1/10
vlan-8
Leaf-102/1/10
vlan-8
APIC
APIC
APIC
Tenant: ESXi-Hosts
Endpoints identified by
Interface and VLAN ID
PayloadIP
Packet Sourced from
physical server1
PayloadIPVXLANL1 VTEP
Leaf swaps ingress encapsulation with VXLAN
(EPG) ID and performs any required policy functions
2
Leaf-103/1/10
vlan-8
Leaf-104/1/10
vlan-8
Leaf-105/1/10
vlan-8
Leaf-106/1/10
vlan-8
3a
If the ingress Leaf has learned the
destination IP to egress VTEP binding
it will set required destination VTEP
address and forward
If the ingress Leaf has NOT learned the
destination IP to egress VTEP binding
it will set required destination VTEP to
the Spine Proxy VTEP
3b
PayloadIPVXLANS1 VTEP
PayloadIP
Packet Delivered to
physical server5
Communication allowed within EPG
Leaf removes ingress VXLAN (EPG) ID and
performs any required policy functions
4
There is no requirement to use
the same VLAN on every Leaf
82

Host-mgmt EPG –
Access Encap VLAN 8
Alternate command:
show vlan extended
Remember for troubleshooting use
the Internal VLAN ID not the
Access Encap VLAN ID
apic1# fabric 101 show system internal epm vlan all
+----------+---------+-----------------+----------+------+----------+---------
VLAN ID Type Access Encap Fabric H/W id BD VLAN Endpoint
(Type Value) Encap Count
+----------+---------+-----------------+----------+------+----------+---------
9 Infra BD 802.1Q 3967 16777209 11 9 3
10 Ext. BD 802.1Q 2050 15269816 12 10 0
11 Ext. BD 802.1Q 49 15531935 111 11 2
12 Tenant BD NONE 0 15662984 14 12 0
13 FD vlan 802.1Q 2022 8814 15 12 2
14 Ext. BD 802.1Q 2020 14909414 16 14 0
15 Tenant BD NONE 0 15171524 17 15 0
16 FD vlan 802.1Q 33 8324 19 15 1
17 FD vlan 802.1Q 2131 9023 20 15 0
18 Tenant BD NONE 0 15138760 18 18 0
19 FD vlan 802.1Q 2125 9017 21 18 0
20 FD vlan 802.1Q 47 8338 22 18 4
34 Tenant BD NONE 0 15302581 29 34 0
35 FD vlan 802.1Q 14 8305 40 34 4
36 Tenant BD NONE 0 15400873 30 36 0
37 FD vlan 802.1Q 8 8299 41 36 19
38 Ext. BD 802.1Q 115 15269817 31 38 1
Lets Look at Which VLANs/VXLANs Have Been Used by
Bridge Domains and EPGs on a Given Leaf
BRKACI-1002
BD_CTRL_VLAN: The infrastructure vlan which was configured during the
APIC setup script.
BD_EXT_VLAN: Bridge Domain to represent external VLAN
BD_VLAN: An internal Bridge Domain construct which is represented by
the grouping of multiple FD_VLANs/VXLANs – i.e many FD_VLANs can
map to one BD_VLAN
FD_VLAN: A VLAN backed EPG identified by the “Access encap” VLAN
ID mapped to the Bridge Domain – a FD_VLAN can only map to a single
BD_VLAN
FD_VXLAN: Used to communicate with hosts behind hypervisors using
VXLAN
Access encap: The Access_enc is significant outside the ACI network as
it is the VLAN that is programmed on a front panel port mapping inbound
frames to an EPG (FD_VLAN)
Fabric Encap: The VXLAN ID for a given EPG/BD
HW_VlanId: The VLAN used to encapsulate incoming traffic from
Access_enc to send to the ALE
VlanId: The VlanId is significant for troubleshooting, most (if not all) show
commands use the VlanId not the Access_enc VLAN ID
83

External VLANs – L2 Connection to
Legacy Networks

Option 1: Same VLANs Outside/Inside
(No Contract Required)
ANP:
Outside_VLANs
Bridge Domain
Hardware Proxy: No
ARP Flooding: Yes
IP Routing: Yes
vPC_to_UCS_a
vlan-10
vPC_to_UCS_b
vlan-10
EPG: Host-Mgmt
192.168.10.11 192.168.10.10
vPC_to_n5ks
vlan-10
vlan-10
APIC
APIC
APIC
Tenant: My_Tenant
Bridge Domain: outside_vlan_10
Gateway: 192.168.10.1
85

Option 2: Different VLANs Outside/Inside
(Contract Required)
ANP:
Outside_VLANs
EPG
Bridge Domain
Hardware Proxy: No
ARP Flooding: Yes
IP Routing: Yes
L2out
vPC_to_n5ks
vlan-10
vlan-10
APIC
APIC
APIC
Tenant: My_Tenant
Bridge Domain: outside_vlan_10
Gateway: 192.168.10.1
vPC_to_UCS_a
vlan-100
vPC_to_UCS_b
vlan-100
EPG: Host-Mgmt
192.168.10.10 192.168.10.11
Communication allowed to External EPG
86

External Routed Connections
Bridge Domain
Hardware Proxy: Yes
ARP Flooding: No
IP Routing: Yes
L3out: Area0
101/1/96: 192.168.30.1/30
102/1/96: 192.168.30.5/30
Outside
Security Import Subnet*
i.e which external subnets can
be accessed through this EPG
APIC
APIC
APIC
EPG
0.0.0.0/0
OSPF
Configuration
Security Zone
Security Zone
ANP:
My_App
192.168.10.11/22 192.168.10.12/22 192.168.10.21/22 192.168.10.22/24
Communication allowed to 10.1.1.0/24
Bridge Domain: 192.168.10.x_22
Gateway: 192.168.10.1
Permit access to all
remote subnets:
0.0.0.0/0
Tenant: My_Tenant
Communication allowed to all External Subnets
EPG
10.1.1.0/24
Permit access to
remote subnet:
10.1.1.0/24
88

OSPF Area 10
(stub)
BRKACI-1002
APIC
APIC
APIC
Spirent Test
Centre
Spirent Test
Centre
Spirent Test
Centre
ESX-01ESX-02
c3850
n7706 n9504n7706-01 n7706-02
n5672-01 n5672-02
L2
L3
OSPF
Area 0
e1/3
e1/1 e1/2 e1/1 e1/2
e1/7 e1/8
e1/15 e1/15 e1/15e1/5 e1/6 e1/11 e1/12
89

Transit Routing – Multiple L3 Out per VRF
BRKACI-1002
Outside Outside
MP BGP
L3out
Area 10
EPG EPG
L3out
Area 20
Contract = Allow Communication
Use a 0.0.0.0/0 subnet with
the ‘aggregate export’ option
checked to export all routes
VRF: Production
70.1.1.0/24
80.1.1.0/24
60.1.1.0/24
Tenant: Common
BD: Inside
Hardware Proxy: Yes
ARP Flooding: No
IP Routing: 192.168.10.1/24
90

Let’s consider the consumers of a cloud provider. The
consumers don’t concern themselves with server
connectivity…
91

They simply concern themselves with the IP
addresses/gateway for their applications, and the
security rules which allow access to those applications
92

Automating “Tenant” configuration allows teams other
than the network team to consume network services
93

ACI Nomenclature
• An EPG is just a logical grouping of devices – think interfaces and VLANs
• An EPG is a Port Group in VMware
• An EPG can contain different VLANs, e.g. when mixing dynamic Virtual Port Groups and Physical
machines – think hardware VTEP
• Devices in an EPG are allowed to communicate (by default)
• Isolated EPGs block communication within the EPG – think PVLAN
• Micro Segmentation (µSeg) EPGs are used to dynamically move devices from a “base” EPG into a
more specific EPG
• An Application Network Profile is a group of one or more EPGs – remember an EPG can only be a
member of one ANP
• Communication between EPGs and/or from devices off the ACI fabric require Contracts (ACLs)
BRKACI-1002 94

Step 4 – Allow Communication Between
EPGs with Contracts

Filter: Any-TrafficFilter: 80, 443 etc EPG:
Clients
Contract: Any-to-Any
Contracts are “directional” Access Lists between Provider and Consumer EPGs. They comprise of one or more Filters
(ACEs) to identify traffic, e.g:
• Contract: Any-to-Any | Filter: Any-Traffic
• Contract: Web | Filter: 80, 443, 8000
• Contract: DNS | Filter: 53
Contracts (ACLs)
BRKACI-1002
Provider Consumer
Contract: Clients-to-Web
Filter: none
Flags:
• Apply in both directions (single contract
which allows return traffic)
• Reverse filter ports (dynamically permits
return flow based on src/dst ports)
Flags:
• IP Protocol
• Ports
• Stateful
• Etc.
EPG: Web
External
Subnet
ANP:
My-Web-App
L3out:
Clients
96

Contracts Permit Communication Between EPGs
Tenant: My_Tenant
VRF: 01
ANP: DB
EPG: DB_1
192.168.10.11/24 192.168.10.12/24
EPG: Web_1
192.168.10.11/24 192.168.10.12/24
EPG: App_1
192.168.20.11/24 192.168.20.12/24
ANP: MyApp_2
BD: 192.168.10.X
BD: 192.168.20.x
EPG: App_1 BD:192.168.30.xEPG: Web_1
192.168.10.11/24 192.168.10.12/24 192.168.10.11/24 192.168.10.12/24
ANP: MyApp_1
97

Contracts are “scoped” at:
• Global
• Tenant
• Context (aka Private Network, aka VRF)
Web_to_App
• Application Profile
App_to_DB
Contracts Scope
BRKACI-1002
ANP: 01
EPG: Web
EPG: App
EPG: DB
ANP: 02
EPG: Web
EPG: App
EPG: DB
VRF: 01
Tenant: Web_Hosting
BD: 01
Hardware Proxy: Yes
IP Routing: Yes
98

What Happens If I Don’t Know The
Required Filter Ports?

• Ask the Application Owner – it’s their application, they will (ok should) know
• Ask the Security Admin for the firewall rules
• Use an “any-any” Filter between EPGs ß Most customers start here
• Use Wireshark
• Configure “Unenforced” mode on the VRF
Filter Discovery
BRKACI-1002 100

Once the ACI Fabric is Up and Running
How Does it Integrate with VMware’s
Virtual Switches?

Firstly, why should you care about integrating
with VMware’s Virtual Switches?
102

A perceived barrier to timely delivery of new services
(from Virtualisation Teams) is that it takes too long to
provision Network Services i.e. VLANs, Subnets, and
L4-7 Devices
103

The reality was that until the release of Cisco ACI there
was no turnkey SDN solution for both Physical
Machines, Virtual Machines, and L4-7 Devices
104

1. Manually configure the vSwitch/vDS as you do today
2. Dynamically configure the vDS (VMware) by pushing Port Groups
(VLAN) from APIC to vCentre
3. Dynamically configure the vDS (Cisco AVS) by pushing Port Groups
(VLAN/VXLAN) from APIC to vCentre
4. Build NSX overlay networks (VXLAN) between different hosts –
requires additional (costly) NSX licenses from VMware
There are Four Integration Options with VMware
BRKACI-1002 105

Traditional Networking
SVI | VLAN | Port Group Relationship
BRKACI-1002
Layer 2 VLAN: VLAN10
VRF: VRF-01 (HSRP gateway)
Interface VLAN10
IP Address 192.168.10.1/24
vDS-01
VM VM VM VM VM VM VM VM VM VM VM VM VM VM VM VM VM VM
Port Group: Web
(VLAN 10)
Host-01 Host-02 Host-03 Host-04
106

Single EPG on a Single BD with a Single Subnet –
“Standard Networking”
BRKACI-1002
ANP: My-App-01
vCentre
Service Request:
Create Application
Create vDS Port Groups
Tenant: Tenant-01
APIC
APIC
BD: Apps
IP Routing: 192.168.10.1/24
Outside
EPG: Web (Dynamic VLAN 2001)
vDS-01
APIC
Port Group: VMware|My-App-01|Web
(Dynamic VLAN 2001)
107

Multiple EPGs on a Single BD with a Single Subnet –
µSegmentation in IP space
BRKACI-1002
ANP: My-App-01
No Contract = No Communication
vCentre
Service Request:
Create Application
Create vDS Port Groups
Tenant: Tenant-01
APIC
APIC
BD: Apps
IP Routing: 192.168.10.1/24
Outside
EPG: App (Dynamic VLAN 2002)EPG: Web (Dynamic VLAN 2001) EPG: DB (Dynamic VLAN 2003)
vDS-01
Contract = Allow Communication Contract = Allow Communication
APIC
Port Group: VMware|My-App-01|Web
(Dynamic VLAN 2001)
Port Group: VMware|My-App-01|App
(Dynamic VLAN 2002)
Port Group: VMware|My-App-01|DB
(Dynamic VLAN 2003)
PS PS
(Eth1/50, 51 VLAN 3600)
108

NSX Overlay
BRKACI-1002
ANP: Overlay_Network
vCentre
Tenant: Tenant-01
APIC
APIC
Outside
EPG: NSX_Transport (VLAN 1000)
APIC
vDS-01
(not managed by APIC)
VLAN 1000 VTEP 10.0.0.4VTEP 10.0.0.3VTEP 10.0.0.2VTEP 10.0.0.1
VM VM VM VM VM VM VM
Dedicated Hosts for
“Edge” Functionality
NSX Logical Switch:
Layer 2 segment carried over
VXLAN, carried over a
dedicated VLAN
DLR DLR B/U
NSX ESG Routers Peer
with the Physical Network
NSX Manager
APIC Configures fabric with an NSX
Transport EPG (VLAN) across all hosts
ESG ESG B/U
NSX DLR informs
controllers of learnt routes
VRF: VRF-01
EPG
VM VM VM VM VM
BD: NSX
IP Routing: Yes
Controllers push
routes to Hosts
L3out
Interface: VLAN 2000
IP: 192.168.30.1
IP: 192.168.30.2
NSX Controller
Cluster
109

Virtual Switching Comparison
BRKACI-1002
Feature / Requirement Standard
vSwitch
VMware NSX APIC Managed
vDS (VMware)
APIC Managed
vDS (Cisco)
Manual port group / EPG Configuration
N/A
Automated port group / EPG configuration pushed from APIC
N/A
VLAN backed port groups
VXLAN backed port groups
Integrated Physical and Virtual Machine security (inc FW, SLB)
Micro-segmentation – VM/VM/Physical separation within the same IP address space
Micro-segmentation – VM to VM separation within a port group (attribute based)
No requirement for dedicated ESX hosts to provide L2/L3 Controllers/Gateways between
Virtual and Physical environments
Traffic visibility between Virtual and Physical Environments
Simple Troubleshooting
110

Cisco AVS is a Partner Supported VIB
• Let’s look at vSphere 6.0 Official Documentation about kernel
Virtual Installation Bundles (VIB) - http://vmw.re/1Ta1Zz0
111

• Cisco AVS Statement of Support
Customers Call Cisco for AVS Support
BRKACI-1002
OpFlex
VM VM VM
VMware ESXi Server
VM VM VM
VMware ESXi Server
OpFlex
VMware vCentreCisco APIC
VMM Domain
AVS AVS
http://www.cisco.com/c/dam/en/us/products/collateral/switches/application-
virtual-switch/avs-support-statement-an.pdf
112

Adding L4-7 Devices to the Network –
Service Graphs and Service Chains

Service Graph Contracts Connect two EPGs
and Optionally Provide Configuration
Parameters to the FW and SLB Which Sit
Between the EPGs
Note: Normal L2/L3 rules still apply, you still have to direct the traffic to the FW/SLB

In “Managed” Mode the APIC Pushes the
Required VLANs and Configuration to the
FW/SLB

In “Unmanaged” Mode the APIC Only Pushes the
Required VLANs to the EPG

Service Chains are Two L4-7 Devices Linked
in a Series

It is Possible to use L4-7 Devices Without
Service Graphs, in this Mode the Fabric Only
Provides L2 Connectivity

Transparent Firewall – Server’s Default Gateway is the
Bridge Domain on the ACI Fabric
BRKACI-1002
EPG: Servers_Inside
L3out
EPG: Servers_Outside
Standard_Contract
ANP: My-App-01 Service_Graph_Contract
BD: Outside
Hardware Proxy: No
ARP Flooding: Yes
IP Routing: Yes
BD: Inside
Hardware Proxy: No
ARP Flooding: Yes
IP Routing: No
Connector type must
be specified as L2
Connector type must
be specified as L2
Tenant: Common
192.168.10.x/24192.168.10.x/24
Servers_Outside can
communicate externally via
the contract to the L3out
Servers_Outside can communicate
with Servers_Inside via the Service
Graph Contract
VRF not used
Server default
gateway
VRF: 01 VRF: 02
119

Transparent Firewall – Server’s Default Gateway is the
Bridge Domain on the ACI Fabric
BRKACI-1002
EPG: Servers_InsideANP: My-App-01
L3out
BD: Outside
Hardware Proxy: No
ARP Flooding: Yes
IP Routing: Yes
BD: Inside
Hardware Proxy: No
ARP Flooding: Yes
IP Routing: No
Service_Graph_Contract
Tenant: Common
192.168.10.x/24
Server default
gateway
Connector type must
be specified as L3
Connector type must
be specified as L2
Servers_Inside can communicate to
the “outside world” via the Service
Graph Contract to the L3out
192.168.10.x/24
VRF not used
VRF: 01 VRF: 02
120

Routed Firewall – Server’s Default Gateway is the
Firewall Attached to the ACI Fabric
BRKACI-1002
EPG: Servers_InsideANP: My-App-01
BD: Inside
Hardware Proxy: Yes
ARP Flooding: Yes
IP Routing: No
L3out
L3out
Tenant: Common
Connector type must
be specified as L3
Connector type must
be specified as L2
Servers_Inside can communicate to
Server default
gateway
Static route to firewall
“inside” subnet via
L3out ot Firewall
VRF has Static route to
firewall “inside” subnet
via L3out to Firewall
192.168.10.x/24
10.1.1.0/30
VRF not used
VRF: 01 VRF: 02
121

Routed Firewall – Server’s Default Gateway is the Bridge
Domain on the ACI Fabric
BRKACI-1002
EPG: Servers_Inside
Server default
gateway
ANP: My-App-01
BD: Inside
Hardware Proxy: Yes
ARP Flooding: No
IP Routing: Yes
L3out
L3out VRF: 01
L3out
VRF: 02
Connector type must
be specified as L3
Connector type must
be specified as L3
Tenant: Common Servers_Inside can communicate to
10.1.1.0/30 10.1.2.0/30
192.168.10.x/24
Static route to firewall
“inside” subnet via
L3out ot Firewall
VRFs peer with Firewall
via L3out
122

Install a L4-7 device once (e.g the ASA firewall) and deploy it multiple
times in different logical topologies
The benefits of the service graph are:
• Reusable configuration templates
• Automatic management of VLAN assignments
• Health score collection from the L4-7 device
• Statistics collection from the L4-7 device
• Automatic ACLs and Pools configuration with endpoint discovery
Service Graph Benefits
BRKACI-1002 123

ADC Device Package Status (as of 09/02/2016)
BRKACI-1002
Device
Package
Status
Virtual
and
physical
Mode Function
Profile
HA Multi-context on physical appliance Dynamic
Routing
Dynamic
EPG
IPv6 Feature Operational
model
Citrix
NetScaler
FCS Yes Go-To
(one-arm and
two-arm)
Yes No
(manual
OOB)
Yes
Create Virtual instance on SDX
manually
Yes Yes
member of
pool for VIP
Yes ADC Everything via
APIC
F5
BIG-IP LTM
FCS Yes Go-To
(one-arm and
two-arm)
Yes Yes Yes
Create route-domain on physical LTM
automatically or create vCMP
manually (no HA)
No Yes
member of
pool for VIP
No ADC Everything via
APIC
or BIG-IQ
F5
Big-IQ cloud
Q1CY16 Yes - - - - - - - - -
A10
Thunder
FCS Yes Go-To
(one-arm and
two-arm)
No No
(manual
OOB)
No No No No ADC Everything via
APIC
Radware
Alteon
FCS Physical Go-To No No No No No No ADC Everything via
APIC
Avi Networks FCS Virtual
only
Go-To Yes Yes - No No No ADC Avi controller is
required.
124

FW Device Package Status (as of 09/02/2016)
BRKACI-1002
Device
Package
Status
Virtual
and
physic
al
Mode Functio
n
Profile
HA Multi-context on physical appliance Dynamic
Routing
Dynamic EPG IPv6 Feature Operational
model
Cisco
ASA
FCS Yes Go-To
Go-Through
Yes Yes Yes
Create context on ASA5500X manually
allocate-interface to each context is done
by APIC
Yes Yes
object-group for
ACE
Yes FW,
ACL,NAT
Everything
via APIC
Palo Alto CA Yes Go-To Yes No No No
1HCY16
planning
No No FW Panorama is
required
Cisco
FirePOWER
FCS Oct
2015, in
controlled
introduction
Yes Go-Through Yes No No - - - IPS Everything
via APIC
Checkpoint Q2CY16 Yes Go-To
Go-Through
Yes Yes
(manual
OOB)
Yes No No Yes FW Everything
via APIC
Fortinet Q2CY16 Yes Go-To
Go-Through
Yes Yes Yes No No Yes FW Everything
via APIC
125

Three Tier Application
BRKACI-1002
Bridge Domain:
Clients
192.168.14.x
Bridge Domain:
Web_
192.168.30.x
Bridge Domain:
Service_chain_clients_
to_web
EPG:
WebServers
Tenant: VMware_AVS
Tenant: Common
EPG:
Clients
IP: 192.168.14.254
Zone: external
IP: 192.168.100.254
Zone: internal
Ext SIP: 192.168.100.2
Int SIP: 192.168.30.254
V
M
IP: 192.168.14.11
GW: 192.168.14.254
V
M
IP: 192.168.30.13
GW: 192.168.30.1
V
M
IP: 192.168.30.14
GW: 192.168.30.1
V
M
IP: 192.168.30.15
GW: 192.168.30.1
PA-FW
Service_chain_clients_to_web
PA-VM-01
Bridge Domain:
Service_chain_web_to_a
pp
IP: 192.168.30.1
Zone: external
IP: 192.168.150.254
Zone: internal
Bridge Domain:
Application_
192.168.40.x
EPG:
AppServersService_chain_web_to_app
Ext SIP: 192.168.150.2
Int SIP: 192.168.40.254
V
M
IP: 192.168.40.11
GW: 192.168.40.1
V
M
IP: 192.168.40.12
GW: 192.168.40.1
vIP: 192.168.150.150
vIP: 192.168.100.100 I06-vCMP-01 I06-vCMP-02 PA-VM-02
Bridge Domain:
Service_chain_app_to_d
b
IP: 192.168.40.1
Zone: external
IP: 192.168.200.254
Zone: internal
Bridge Domain:
Database_
192.168.50.x
Ext SIP: 192.168.200.2
Int SIP: 192.168.50.254
vIP: 192.168.200.200
I06-vCMP-03
EPG:
DBServers
V
M
IP: 192.168.50.11
GW: 192.168.50.1
V
M
IP: 192.168.50.12
GW: 192.168.50.1
Service_chain_app_to_db
https://cisco.box.com/s/fn47le5r5um091fynbds43r32kwdcrxf
126

Now That We Have a Better
Understanding of ACI, Lets Consider
How Customers Can Consume ACI
With Automation

Customer Use Cases
Credit Services
• Multi-Tier application
Deployments
• Tenants
• VRFs
• Bridge Domains
• Endpoint Groups
• Contracts
• Load Balancing (Citrix)
• VM creation
Media
• Tenants
• VRFs
• Bridge Domains
• Endpoint Groups
• Contracts
• Switch Interfaces
Banking
• VRFs
• Bridge Domains
• Endpoint Groups
• Contracts
• Switch Interfaces
• VM creation
• OS Installation
128

What Should You Look to do First?
A. Automate the building of networking infrastructure
B. Automate the consumption of networking resources
• Blueprints for Tenants, L2 (EPG/VLAN/VXLAN), L3, L4-7 services
• IP Address Management (IPAM)
• Summary routes into the fabric
• Virtual machine creation
• Containers
• Application Provisioning
• Self service offering
C. Automate both infrastructure and consumption
D. Automate application deployment
BRKACI-1002 129

Take a Step Back, Most Customers
Actually Require a Number of Pre-
Defined Functional “Blueprints”

Sample Network Blueprints
Clients
ACI
Gateway
(not used)
External Router
to WAN
Gateway
192.168.10.1
L2 Fabric (external g/w)
Clients
ACI
Gateway
External Router
to WAN
L3 Fabric
Clients
ACI
Gateway
External Router
to WAN
L3 Fabric with external firewall
131

Sample Network Blueprints
Clients
ACI
Internal Gateway
External Router
to WAN
L3 Fabric with firewall on fabric
ACI
External Gateway
Clients
ACI
Internal Gateway
External Router
to WAN
L3 Fabric with SLB on fabric
ACI
External Gateway
SLB
Clients
ACI
Gateway
External Router
to WAN
L3 Fabric with firewall and SLB
SLB
132

If We Now Understand The “Why”…

We Next Need To Understand The
“How”…

How Many of You....
• Are already scripting and automating common tasks?
• In my experience, most of us are not
• Are really good at copy and paste?
• That’s me that is!!
BRKACI-1002 135

Congratulations!
136

Being Serious For A Moment
• We talk to a lot of partner and customer engineers all over the world
• It is clear that some knowledge of programming concepts is quite valuable these
days
• The top question is always “Do I need to learn programming to keep doing my
job?”
• I’ve got some good news for you...
• In a nutshell, the answer is No....
• But only if you learn to consume the easy-to-use tools and processes out there
BRKACI-1002 137

What is ACI?
It is all about the API and Object Model
BRKACI-1002
APIC
APIC
APIC
139

ACI and REST API
• REST is fundamental to APIC interaction
• All other tools are built around it
• Understand REST, understand ACI automation
• The second time you need to do something, think about
automating it instead!!
BRKACI-1002 140

Using REST
• HTTP(S) to the URL or Address of an object
• Select an Action to perform (GET, POST etc)
• Send the Payload (in XML or JSON format)
BRKACI-1002 141

Common (Free) Tools For The Network Engineer
Use these to automate things in ACI
• Postman Plugin for Google Chrome
• API Inspector
• APIC GUI
• COBRA SDK
• Python IDE (Pycharm, Atom, others)
• Git / Github
• ARYA
• ACI Toolkit
• Many Others
BRKACI-1002 142

Different Engineers, Different Tools
APIC CLI
APIC GUI
REST API
SDK
BRKACI-1002
Powerful/Complex
Simple/Rigid
143

API Inspector – a REST API Sniffer
• Record your GUI interaction as JSON
• Modify and replay with tools like Postman
BRKACI-1002 144

Postman Plugin for Google Chrome
BRKACI-1002 145

Python SDK (aka “Cobra”) + ARYA
• Full featured access to entire APIC
REST API
• Native ACI language – configure in
GUI and turn into Cobra SDK
• Contributors include: Business Unit
Engineers, Technical Services
Engineers, Advanced Services
Engineers
• Complete user use cases all
possible
• http://github.com/datacenter/cobra
• http://github.com/datacenter/arya
BRKACI-1002
XML/JSON
arya.py
Python
code
{"fvTenant":{"attributes":{"dn":"uni/tn-
Cisco","name":"Cisco","rn":"tn-
Cisco","status":"created"},"children":[{"fvBD":{"attribut
es":{"dn":"uni/tn-Cisco/BD-
CiscoBd","mac":"00:22:BD:F8:19:FF","name":"CiscoBd","rn":
"BD-
CiscoBd","status":"created"},"children":[{"fvRsCtx":{"att
ributes":{"tnFvCtxName":"CiscoNetwork","status":"created,
modified"},"children":[]}},{"fvSubnet":{"attributes":{"dn
":"uni/tn-Cisco/BD-CiscoBd/subnet-
[10.0.0.1/8]","ip":"10.0.0.1/8","rn":"subnet-
[10.0.0.1/8]","status":"created"},"children":[]}}]}},{"fv
Ctx":{"attributes":{"dn":"uni/tn-Cisco/ctx-
CiscoNetwork","name":"CiscoNetwork","rn":"ctx-
CiscoNetwork","status":"created"},"children":[]}}]}}
fvTenant = cobra.model.fv.Tenant(topMo, name='Cisco')
fvCtx = cobra.model.fv.Ctx(fvTenant, name='CiscoNetwork')
fvBD = cobra.model.fv.BD(fvTenant,
mac='00:22:BD:F8:19:FF', name='CiscoBd')
fvRsCtx = cobra.model.fv.RsCtx(fvBD,
tnFvCtxName=fvCtx.name)
fvSubnet = cobra.model.fv.Subnet(fvBD, ip='10.0.0.1/8')
146

Practical example of tool usage
BRKACI-1002 147

Cisco on Github
• https://github.com/datacenter
• https://github.com/datacenter/ACI
• https://github.com/datacenter/aci-examples
• https://github.com/datacenter/sparci
• https://github.com/datacenter/acitoolkit
148

How Should I Get Started with ACI?

Choose Your Management Method(s)
BRKACI-1002 152

Connect the Old to the New
BRKACI-1002
APIC
APIC
APIC
Layer 2 vPC to existing
network
Layer 3 (OSPF etc) to
existing network
Connect new workloads
to the ACI fabric and
route out
Separate “border leafs”
shown for clarity
vDS-02vDS-01vDS-01
Separate “border leafs”
shown for clarity
153

Leaf Profiles
(Target Switches)
Leafs_101_and_102
AAEP
(Allowed VLANs)
UCS-phys-svrs
Interface Policies
CDP_enabled
LACP_Active
VLAN/VXLAN
(Pools)
UCS-phys-svrs
VLAN mgmt
(Phy/Out Domain)
UCS-phys-svrs
BRKACI-1002
Understand the Interface Policies
Concrete Model
Logical Model
APIC
APIC
APIC
Interface Selector
1/21
Security Domain
(optional)
ANP: My_App
EPG: Web
Domain: UCS-phys-svrs
Path: vPC_to_UCS_FI_A
VLAN_10
Path: vPC_to_UCS_FI_B
VLAN_10
Interface Selector
1/22
Leaf Policy Group
vPC_to_UCS_FI_A
Leaf Policy Group
vPC_to_UCS_FI_B
Leaf Profile
vPC_to_UCS_FI_A
Leaf Profile
vPC_to_UCS_FI_B
Leaf Profiles
155

Understand the Managed Object Hierarchy
BRKACI-1002
EP EP
EPGEPG
EP EP
Bridge Domain
(Flood)
EP EP
EPGEPG
EP EP EP EP
EPGEPG
EP EP
Bridge Domain
(Hardware Proxy)
Tenant “Private”
Private Network
(VRF)
Private Network
(VRF)
Tenant “Common”
Bridge Domain
(Hardware Proxy)
Application Network Profile
OutsideOutside
156

Requirements Hardware Proxy no ARP flooding IP Routing Subnet Check
Routed traffic, no silent hosts Yes Yes Yes Yes
Routed traffic, silent hosts Yes ARP flooding (optional
since Subnet is present)
(*)
Yes Yes
non-IP switched traffic, silent hosts No N/A No No
non-IP switched traffic, no silent hosts Yes N/A No No
IP L2 switched traffic, silent hosts Yes ARP flooding (optional if
Subnet is present) (*)
Yes (for advanced
functions and aging)
Yes (for aging and ARP
gleaning)
IP L2 switched traffic, no silent hosts Yes no ARP flooding (if hosts
send DHCP requests or
gratuitous ARP)
Yes (for advanced
functions and aging)
Yes (for aging and ARP
gleaning)
Bridge Domain Options
BRKACI-1002
(*) if the Subnet is configured ACI can do ARP gleaning so ARP flooding is not strictly needed
157

1. You must have at least one Tenant or use the
Common Tenant
2. VRFs are constrained within Tenants
3. VRFs provide external L3 connectivity (with a
contract)
4. You must have at least one Bridge Domain
5. Bridge Domains determine the L2 forwarding
characteristics
6. Bridge Domains provide internal L3
connectivity (default gateways)
7. Bridge Domains to outside VLANs must be
mapped 1:1
8. Endpoint Groups map to a single Bridge Domain
9. Endpoint Groups are security zones where
communication is allowed
10. Communication between Endpoint Groups is
allowed through contracts (ACLs)
11. Endpoint Groups must be bound to a virtual,
physical, or outside domain
12. Endpoint Groups allow you to mix and match
VLANs/VXLANs/interfaces (access, port channel,
virtual port channel)
13. Endpoints can only be a member of a single
Endpoint Group
14. AAEP’s allow VLANs on interfaces or VMM
domains
ACI Networking Rules!
BRKACI-1002 158

Cisco Spark
Ask Questions, Get Answers, Continue the Experience
Use Cisco Spark to communicate with the Speaker and fellow
participants after the session
Download the Cisco Spark app from iTunes or Google Play
1. Go to the Cisco Live Melbourne 2017 Mobile app
2. Find this session
3. Click the Spark button under Speakers in the session description
4. Enter the room, room name = BRKACI-1002
5. Join the conversation!
The Spark Room will be open for 2 weeks after Cisco Live
160BRKACI-1002

Other Sessions of Interest
• BRKACI-2603 – ACI Operation and Troubleshooting
• BRKACI-2016 – ACI L4-7 Integration
• BRKACI-3502 – ACI Multisite Deployment
• BRKACI-2004 – How to Setup an ACI Fabric from Scratch
• LABDC-1011 – ACI with VMware Integration
BRKACI-1002 161

Complete Your Online Session Evaluation
162BRKACI-1002
Learn online with Cisco Live!
Visit us online after the conference
for full access to session videos and
presentations.
www.CiscoLiveAPAC.com
Give us your feedback and receive a
Cisco Live 2017 Cap by completing the
overall event evaluation and 5 session
evaluations.
All evaluations can be completed via
the Cisco Live Mobile App.
Caps can be collected Friday 10 March
at Registration.

• fabric <#> show system internal epm vlan all ßalways use this command first
• fabric <#> show interface vlan <#>
• fabric <#> show vlan brief
• fabric <#> show vlan extended
• fabric <#> show interface trunk
• fabric <#> show interface ethernet <#/#>
• fabric <#> show port-channel summary
• fabric <#> show cdp neighbors
• fabric <#> show lldp neighbors
Layer 2 Commands
BRKACI-1002 166

• show endpoints vpc context <#> <#> interface vpc <#>
Layer 2 Commands
BRKACI-1002 167

• fabric <#> show ip interface brief
• fabric <#> show ip interface brief vrf <tenant>:<vrf>
• fabric <#> show ip route vrf <tenant>:<vrf>
• fabric <#> show ip route vrf <tenant>:<vrf> <route>
• fabric <#> show ip route ospf vrf <tenant>:<vrf>
• fabric <#> show ip ospf neighbors vrf <tenant>:<vrf>
• fabric <#> show ip ospf neighbors detail vrf <tenant>:<vrf>
• fabric <#> show bgp ipv4 unicast vrf <tenant>:<vrf>
L3 Commands
BRKACI-1002 168

• fabric <#> show ip igmp interface brief vrf <tenant>:<vrf>
• fabric <#> show ip igmp group vrf <tenant>:<vrf>
• fabric <#> show ip mroute vrf <tenant>:<vrf>
• fabric <#> show ip pim vrf <tenant>:<vrf>
• fabric 101 show ip pim neighbor vrf Production:VRF-01
Multicast Commands
BRKACI-1002 169

• show running-config leaf <#> interface ethernet <#/#>
• show running-config template policy-group <#>
• show running-config template port-channel <#>
• show running-config leaf-interface-profile <#>
• show running-config leaf-profile <#>
• show running-config leaf <#> vrf context tenant <#> vrf <#>
• show running-config leaf <#> router ospf
Show Run Commands
BRKACI-1002 170

• show running-config tenant <#> vrf context <#>
• show running-config tenant <#> interface bridge-domain <#>
• show running-config tenant <#> external-l3
• show running-config tenant <#> application <#>
• show running-config tenant <#> application <#> epg <#>
Show Run Tenant Commands
BRKACI-1002 171

• show tenant <#> detail
• show tenant <#> vrf <#> detail
• show tenant <#> bridge-domain <#> detail
• show tenant <#> epg <#> detail
• show tenant <#> contract <#>
• show tenant <#> access-list <#>
Show Tenant Commands
BRKACI-1002 172

li08-apic-svr-01# sh run leaf 101 interface e 1/15
leaf 101
interface ethernet 1/15
# Policy-group configured from leaf-profile ['Leaf_101'], leaf-interface-profile
li07_101_to_Spirent_Test_Center
# policy-group 10G_acc_Spirent_Test_Center
switchport trunk allowed vlan 10 tenant Production application ANP-01 epg vlan-10__10.161.10.x_24
exit
exitl
i08-apic-svr-01#
How To Find What EPG Is On An Interface
BRKACI-1002 173

How To Find All Interfaces For An EPG
li08-apic-svr-01# show epg vlan-18__10.181.18.x_24 detail
[snip]
Static Paths:
Encap: (P):Primary VLAN, (S):Secondary VLAN
Node Interface Encap
---------- ------------------------------ -------------------------
101 eth1/30 unknown(P),vlan-18(S)
101 102 vpc 10G_vPC_esx_li07-c220m4-02 unknown(P),vlan-18(S)
103 104 vpc 10G_vPC_esx_li07-c220m4-01 unknown(P),vlan-18(S)
[snip]
Untagged EPG
BRKACI-1002 174

• moquery -c fvLocale | grep dn | grep <epg name> - finds which node an epg is applied
• moquery -c fvIfConn | grep dn | grep vlan-<#> - finds where a VLAN has been applied
Advanced Commands
BRKACI-1002 175

li08-apic-svr-01# configure
tenant <#>
application <#>
epg <#>
bridge-domain member <#>
contract consumer <#>
contract provider <#>
exit
exit
exit
Configure: Tenant, Application, EPG
BRKACI-1002 176

Brkaci 1002

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Brkaci 1002

Similar to Brkaci 1002 (20)

Recently uploaded

Recently uploaded (20)

Brkaci 1002