2. Virtual Networking concept
www.silantia.com2
Each ESXi host has mainly two types of network adaptor
VmKernel port – Handles traffic for ESXi services like
vMotion, iSCSI, NFS and N1Kv Control.
Virtual machine port – Handles traffic for virtual machines.
Vmware ESXi has two types of virtual switches.
Standard vSwitch: managed as individual entity and cannot be centrally
configured
Distributed virtual switch (DVS) : Managed as single entity across
multiple host and configured centrally.
Each switch can have its own set of uplinks which connects
ESXi host to external world.
All virtual machines and vmkernal ports connects to vswitch
and/or DVS.
3. Virtual Networking concept
www.silantia.com3
vSwitch/DVS are dot1q capable switchs which can switch traffic
between virtual machine or Vmk ports.
vSwitch/DVS DOES not run spanning tree.
vSwitch/DVS can have multiple VLANs and none of them could
do L3 routing.
vSwitch/DVS looks up each frame’s destination MAC when it
arrives. It builds mac address table for local VMs.
If destination mac address is not present in MAC address table
then traffic is sent to one of the uplink.
Traffic between two VMs gets pinned to uplink and switched via
physical switch and goes to destination ESXi host where VM
resides and vSwitch/DVS on that host switches that frame to
desired VM.
4. Introduction to Nexus 1000v
www.silantia.com4
First CLI based virtual distributed switch.
It’s a Cisco Nexus switch.
It has two components
VSM ,Virtual Supervisor Module : VM which controls VEM
VEM Virtual ethernet module : Software agent / module running inside
ESXi host.
All components in a single vSphere cluster operate logically as
a single virtualised access switch
6. Introduction to Nexus 1000v
www.silantia.com6
Protocols such as CDP, LACP, SNMP, IGMP operate as a single
switch, These control protocols are run on the VSM and carried
over the ―packet VLAN‖ to the VEMs.
Because the VSM and VEM are not physically connected, the
VSM (supervisor) must program the VEM (linecards) over a
network. It does this by using Control and Packet network.
There are two ways to extend communication between VSM
and VEM:
Over Layer 2 network using Control and Packet VLANs called L2 mode
Over Layer 3 network using Layer 3 Control Capability called L3 mode
VEM has special VMK port which it uses to communicate with
VSM.
L3 mode is more preferred because you can then put VEM and
VSM in separate subnet.
7. More on L3 control mode
www.silantia.com7
Requires an IP address be assigned to the VEM
L3 uses UDP port 4785 for both source and destination
Uses Mgmt or Control interface of the VSM
VSM mgmt 0 is default interface for L3, it can also use ―control
0‖ interface
Ties to control adapter of the VSM VM (Adapter 1)
Control0 interface uses default VRF and mgmt0 uses
management VRF.
Primary and secondary VSM still need to be L2 adjacent
8. L3 VEM Deployment Requirements
www.silantia.com8
VSM setup for L3 control
Create VMK Interface on ESXi host
Test ping from control0 or mgmt0 to VMK of ESXi host
Create uplink port-profile
Create veth port-profile with ―capability l3control‖
Migrate host and VMK interface
VEM gets installed via VUM, manual, or installer app
Host will not show up until VMK interface is moved to port-profile
9. L3 Control mode configuration
www.silantia.com9
svs-domain
domain id 50
svs mode L3 interface control0
interface control0
ip address 172.20.186.11/24
port-profile type vethernet N1K-Control
capability l3control
vmware port-group
switchport mode access
switchport access vlan 260
no shutdown
system vlan 260
description N1KV Control
state enabled
port-profile type ethernet NON-LACP-UPLINKS
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 1,100,260-261
channel-group auto mode on mac-pinning
no shutdown
system vlan 1,100,260-261
state enabled
10. Introduction to Nexus 1000v
www.silantia.com10
The ports that connects to VMs are called veth ports and ports
that are connected to Physical network switches are called
uplink ports.
Ports are never configured directly, instead it is configured with
port-profile (a set of ports with similar properties such as VLAN
ID, QoS, ACL, etc.)
These port profiles appear in vcenter as a port group. Server
admin can then select a port group defined by network admin
for a network adaptor while creating a VM.
Editing an enabled profile will cause config changes to
propagate to all interfaces using that profile (unlike a static one-
time smartport macro in catalyst switch)
Two types
Type Ethernet used for physical NIC uplinks
Type Vethernet used for VM network connectivity
11. Example port-profiles.
www.silantia.com11
port-profile type ethernet UPLINK
vmware port-group
switchport mode trunk
switchport trunk native vlan 1
switchport trunk allowed vlan 1-3967,4048-4093
service-policy type queuing output UPLINK_QUEUING
channel-group auto mode on mac-pinning
no shutdown
system vlan 1,100,260-261
state enabled
port-profile type vethernet Management
vmware port-group
switchport mode access
switchport access vlan 1
no shutdown
system vlan 1
description ESXi management
state enabled
12. Nexus 1000v System Vlans
www.silantia.com12
A system vlan should be used whenever need to pass traffic
through a VEM even when it can’t connect to VSM.
A system vlan is any vlan needed for functionality in a corner
case, that is, when the VSM is offline and the ESXi host with the
VEM is rebooted.
A system vlan is used to config and bring up physical or veth
ports BEFORE the VSM has established communication with
VEM.
Examples are VLANs that are used for control, packet, ESXi
Management and iSCSI.
You define these vlans as system vlans under corresponding
port-profile and uplink port-profile.
13. vCenter to VSM communication
www.silantia.com13
VSM connects to vCenter using SSL connection using mgmt0 IP
address as source.
VSM configures vCenter using its API
VSM creates N1KV Port-Groups in vCenter
VSM also stores opaque data in vCenter
VSM pulls information from vCenter (DC, DVS, VM, …)
VSM continuously able to talk to vCenter alltime there is config
changes is done.
VSM registers to vCenter using extension key plug-in
Contains a public SSL Certificate
Extension key of the VSM
14. vCenter to VSM communication
www.silantia.com14
N1KV-VSM# sh svs connections
connection vcenter:
ip address: 192.168.2.99
remote port: 80
protocol: vmware-vim https
certificate: default
datacenter name: CCIEDC
admin: n1kUser(user)
max-ports: 8192
DVS uuid: cd 26 09 50 e5 55 9e d3-1b 17 d4 9c 1c ae 4f 6e
config status: Enabled
operational status: Connected
sync status: Complete
version: VMware vCenter Server 5.1.0 build-947673
vc-uuid: EE1F2CA8-2DEE-43E4-826D-472D18E6BF89
15. Nexus 1000v Port-channels
www.silantia.com15
You can bundle uplinks into a port-channel. Uplinks from
different ESXi hosts cannot be bundled into a port-channel.
Veth cannot be bundled into a port-channel.
If Physical switch supports LACP then it can be used in Nexus
1000v to dynamically form port-channel.
If upstream switch does not support LACP then you will need to
use a concept called MAC pinning (more in later slide).
LACP offload feature: This feature allows LACP negotiation off
loaded to VEM code i.e. VEM can do LACP negotiation with
upstream switch without even connecting to VSM.
19. Nexus 1000v Port-channels
www.silantia.com19
Scenario II upstream switches DOES NOT support LACP.
port-profile type ethernet NON-LACP-UPLINKS
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 1-3967,4048-4093
switchport trunk native vlan 1
mtu 1500
service-policy type queuing output UPLINK_QUEUING
switchport access vlan 1
channel-group auto mode on mac-pinning
no shutdown
system vlan 1,100,260-261
state enabled
20. Nexus 1000v Port-channels
www.silantia.com20
Scenario II upstream switches DOES NOT support
LACP.
N1KV-VSM# sh port-channel summary
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
S - Switched R - Routed
U - Up (port-channel)
--------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------
1 Po1(SU) Eth NONE Eth3/1(P) Eth3/2(P)
22. Nexus 1000v and UCS
www.silantia.com22
End host mode
Enable CDP on Network control policy.
Enable Host control full in QoS policy.
No Fabric failover.
Use one, two or four vNIC template for each fabric.
Use On Nexus 1000v channel-group auto mode on
pinning on uplink port-profile.
L3 control mode.
Use ESXi mgmt, Control and iSCSI VLANs as
System VLANs.
23. Nexus 1000v and QoS
www.silantia.com23
Nexus 1000v leverage the MQC qos-group
capabilities to identify and define traffic in policy
configuration.
Define matching criteria via a class-map
Associate action with each defined class via a policy-map
Apply policy to entire system or an interface via a service-
policy
Nexus1000v: 64 classes (8 pre-defined) .
Since Nexus 1000v is access layer switch so
Classification, Marking and Queuing is done.
24. Nexus 1000v and QoS
www.silantia.com24
Classification Criteria:
CoS
IP precedence
DSCP
Layer 2 to Layer 4 parameters
ACL
QoS group
Discard class
Protocol
Marking allowed:
CoS
IP precedence
DSCP
QoS group
Discard class
Policing Conditions:
Conforms to rate limits
Exceeds rate limit maximum
Violates rate limit Policing Types:
Single rate (CIR)
Dual rate (CIR and PIR)
Color aware
25. Nexus 1000v and QoS
www.silantia.com25
Ingress QoS policies per interface:
One type QoS
One type queuing
Egress QoS policies per interface:
One type QoS
One type queuing
26. Nexus 1000v and QoS Configuration
www.silantia.com26
N1KV-VSM# sh class-map
Type queuing class-maps
========================
class-map type queuing match-any iSCSI
match protocol vmw_iscsi
class-map type queuing match-any vMotion
match protocol vmw_vmotion
class-map type queuing match-all Management
match protocol vmw_mgmt
match protocol n1k_mgmt
class-map type queuing match-all N1KV-Control-Packet
match protocol n1k_packet
match protocol n1k_control
27. Nexus 1000v and QoS Configuration
www.silantia.com27
Type qos policy-maps
====================
policy-map type qos SET_COS2
class class-default
set cos 2
policy-map type qos SET_COS4
class class-default
set cos 4
Type queuing policy-maps
========================
policy-map type queuing UPLINK_QUEUING
class type queuing Management
bandwidth percent 5
class type queuing N1KV-Control-Packet
bandwidth percent 5
class type queuing iSCSI
bandwidth percent 30
class type queuing vMotion
bandwidth percent 30
28. Nexus 1000v and QoS Configuration
www.silantia.com28
Type qos policy-maps
====================
policy-map type qos SET_COS2
class class-default
set cos 2
policy-map type qos SET_COS4
class class-default
set cos 4
Type queuing policy-maps
========================
policy-map type queuing UPLINK_QUEUING
class type queuing Management
bandwidth percent 5
class type queuing N1KV-Control-Packet
bandwidth percent 5
class type queuing iSCSI
bandwidth percent 30
class type queuing vMotion
bandwidth percent 30
30. Nexus 1000v Network Monitoring
www.silantia.com30
Each Veth port has interface statistics just like physical ports.
These statistics are preserved when attached VM moved
from one host to another host.
Veth ports are brought up on demand it should never be
configured individually.
Uplink physical ports also has interface statistics.
Ethanalyzer can be used to capture control0 and mgmt0
interfaces.
ethanalyzer local sniff-interface control dump-
pkt write bootflash:control0.pcap
Netflow version 5 and version 9 are supported.
31. Nexus 1000v Netflow
www.silantia.com31
Feature netflow
flow exporter NetflowExporter
destination 192.168.2.11 use-vrf management
source mgmt0
version 9
flow record Inbound
description Netflow record for inbound traffic
match ipv4 destination address
match ip protocol
match ip tos
match transport destination-port
collect transport tcp flags
collect counter bytes
collect counter packets
flow monitor NetflowMonitor
record Inbound
exporter NetflowExporter
port-profile type vethernet NON-NLB
ip flow monitor NetFlowMonitor input
32. Nexus 1000v ERSPAN
www.silantia.com32
Encapsulated Remote Switched Port Analyzer
Nexus 1000v does not support destination erspan
A veth port can be captured and sent accross IP network to
destination in to a GRE encapsulated packet where it can bet
decapsulated and analysed.
Capability L3-control has to be specified for port-profile of ERSPAN
source. This port profile is applied to vmk port of hypervisor and it
is used as source of a GRE tunnel.
N1KV-VSM(config)# monitor session 1 type erspan-
source
N1KV-VSM(config-erspan-src)# source ?
interface Configure interfaces
port-profile Port profile name
vlan Vlan type
N1KV-VSM(config-erspan-src)# destination ip
192.168.2.169
33. Nexus 1000v IGMP snooping
www.silantia.com33
Nexus 1000v is full IGMP snooping bridge but it cannot
perform IGMP querier job.
ip igmp snooping ! Enables at global level
vlan 200
ip igmp snooping ! Enables per vlan IGMP snooping
ip igmp snooping explicit-tracking
ip igmp snooping mrouter interface ethernet 2/1
! vEths are not supported as router ports
ip igmp snooping static-group 230.0.0.1 interface vethernet 21
show ip igmp snooping vlan 200
34. Nexus 1000v Layer 2 Security feature
www.silantia.com34
Works exactly similar to Layer 2 security features in physical
switches.
Security features requires Nexus 1000v Advanced License.
Layer 2 security is important in Virtual Desktop Infrastructure
type of environment where each virtual machine is a user
desktop.
Unmanaged VMs can bring down the whole Layer 2 network if
it is not protected at VEM level.
35. Nexus 1000v Port-security
www.silantia.com35
MAC-to-Port Mapping
Don't allow any MAC addresses other than those mapped to
pass traffic
Static = Static MAC-to-Port Mapping
Dynamic = Learn the MAC, map to the port, then don't allow
anyone ELSE
Can also age this dynamic mapping out
Sticky = Same as dynamic, but store mapping in NVRAM
Violations
Shutdown = Shuts port down. Simple, done.
Restrict = Drops traffic from any Other MAC addys
Protect = Basically drops traffic from any other MACs like
Restrict, but first it learns the MAC of the 1st violator and logs
him (still drops his traffic too)
and doesn't learn any other violators' MACs
36. Nexus 1000v Port-security
www.silantia.com36
You can configure port security only on Layer 2 interfaces
Details about port security and different types of interfaces or
ports are as follows:
Access ports
You can configure port security on interfaces that you have
configured as Layer 2 access ports
On an access port, port security applies only to the access VLAN
Trunk ports
You can configure port security on interfaces that you have
configured as Layer 2 trunk ports
The device allows VLAN maximums only for VLANs associated with
the trunk port
SPAN ports
You can configure port security on SPAN source ports but not on
SPAN destination ports
37. Nexus 1000v DHCP snooping
www.silantia.com37
DHCP snooping functions like a firewall between
untrusted hosts and trusted DHCP servers by doing the
following:
Validates DHCP messages received from untrusted sources
and filters out invalid response messages from DHCP
servers.
Builds and maintains the DHCP snooping binding database,
which contains information about untrusted hosts with leased
IP addresses.
Uses the DHCP snooping binding database to validate
subsequent requests from untrusted hosts.
Dynamic ARP Inspection (DAI) and IP Source Guard
also use information stored in the DHCP snooping
binding database.
When you enable DHCP snooping, by default, all
vEthernet (vEth) ports are untrusted and all Ethernet
38. Nexus 1000v DHCP snooping
www.silantia.com38
DHCP operations are categorized into four basic phases:IP Discovery
IP Lease Offer
IP Request
IP Lease Acknowledgement
Only DHCP messages that come from a server that is connected to a
trusted port are accepted.
Any DHCP message on UDP port 68 that is data from the server to
the client that is received on an untrusted port is dropped. Nexus
1000v VEM builds and maintains the DHCP snooping binding
database, which contains information about clients with leased IP
addresses.
Uses the DHCP snooping binding database to validate subsequent
requests from clients.
39. Nexus 1000v DHCP snooping
www.silantia.com39
Configuration
Enable the DHCP feature.
feature dhcp
Enable DHCP snooping globally.
ip dhcp snooping
Enable DHCP snooping on at least one VLAN.By default,
DHCP snooping is disabled on all VLANs. Ensure that the
ip dhcp snooping vlan vlan-list
DHCP server is connected to the device using a trusted
interface.
N1KV-VSM(config)# port-profile profilename
N1KV-VSM(config-if)# ip dhcp snooping trust
Configuring the Rate Limit for DHCP Packets
N1KV-VSM(config-if)#[no] ip dhcp snooping limit rate
rate
40. Nexus 1000v DHCP snooping
www.silantia.com40
Configuration
Error disable detection and recovery
errdisable detect cause dhcp-rate-limit
Enables DHCP error-disabled detection.
errdisable recovery cause dhcp-rate-limit
Enables DHCP error-disabled detection.
errdisable recovery interval time interval
Sets the DHCP error-disabled recovery interval, where time interval is
the number of seconds from 30 to 65535.
41. Nexus 1000v DHCP snooping
www.silantia.com41
Verification
N1KV-VSM# show ip dhcp snooping
Switch DHCP snooping is enabled
DHCP snooping is configured on the following VLANs:
100,200,250-252
DHCP snooping is operational on the following VLANs:
100,200,250-252
Insertion of Option 82 is disabled
Verification of MAC address is enabled
DHCP snooping trust is configured on the following
interfaces:
Interface Trusted Pkt Limit
------------ ------- ---------
Vethernet1 No Unlimited
Vethernet2 No Unlimited
Vethernet3 Yes 15
Vethernet4 No Unlimited
Vethernet5 No Unlimited
42. Nexus 1000v DHCP snooping
www.silantia.com42
Verification
N1KV-VSM# show ip dhcp snooping statistics
Packets processed 0
Packets forwarded 0
Total packets dropped 0
Packets dropped from untrusted ports 0
Packets dropped due to MAC address check failure 0
Packets dropped due to Option 82 insertion failure 0
Packets dropped due to o/p intf unknown 0
Packets dropped which were unknown 0
Packets dropped due to service dhcp not enabled 0
Packets dropped due to no binding entry 0
Packets dropped due to interface error/no interface 0
Packets dropped due to max hops exceeded 0
43. Nexus 1000v Dynamic ARP inspection
www.silantia.com43
DAI ensures that only valid ARP requests and responses are
relayed by intercepting all ARP requests and responses on
untrusted ports and verifying that each of these intercepted
packets has a valid IP-to-MAC address binding before
updating the local ARP cache or before forwarding the packet
to the appropriate destination.
DAI depends on the entries in the DHCP snooping binding
database to verify IP-to-MAC address bindings in incoming
ARP requests and ARP responses.
DAI is supported on vEthernet interfaces and private VLAN
ports
44. Nexus 1000v Dynamic ARP inspection
www.silantia.com44
Configuration:
N1KV-VSM(config)# ip arp inspection vlan #
N1KV-VSM(config)# port-profile profilename
N1KV-VSM(config-port-profile)# ip arp
inspection trust
Verification
switch# show ip arp inspection interfaces vethernet 3
Interface Trust State Pkt Limit Burst Interval
------------- ----------- --------- ----------------
Vethernet9 Untrusted 30 5
45. Nexus 1000v Dynamic ARP inspection
www.silantia.com45
Rate limiting
ip arp inspection limit {rate pps [burst
interval l bint] | none}
Configures the specified ARP inspection limit on the
interface or the port profile as follows.
rate—Specifies that allowable values are between 1 and 2048
packets per second (pps).
The untrusted interface default is15 packets per second.
The trusted interface default is15 packets per second.
burst interval—Specifies that allowable values are between 1
and 15 seconds (the default is 5 seconds).
none—Specifies an unlimited number of packets per second.
46. Nexus 1000v Dynamic ARP inspection
www.silantia.com46
Can enable additional validation checks
ip arp inspection validate ?
src-mac: Checks the source MAC address in the Ethernet header against the
sender MAC address in the ARP body for ARP requests and responses
dst-mac: Checks the destination MAC address in the Ethernet header against
the target MAC address in the ARP body for ARP responses
ip: Checks the ARP body for invalid and unexpected IP addresses. Addresses
include 0.0.0.0, 255.255.255.255, and all IP multicast addresses
arp access-list UNK-SW
permit ip host 10.0.0.1 mac host 0000.0000.0001
ip arp inspection filter UNK-SW vlan 10
Error disable
Port may go into error disable when ARP inspection is violated.
N1KV-VSM(config)# errdisable detect cause arp-inspection
N1KV-VSM(config)# errdisable recovery cause arp-inspection
You can shut no shut port or configure error disable recovery to recover automatically.
47. Nexus 1000v IP Source Guard
www.silantia.com47
IP SG is a per-interface traffic filter that permits IP traffic only
when the IP address and MAC address of each packet
matches one of two sources of IP and MAC address bindings
Entries in DHCP snooping binding table.
Static IP source entries that you configure.
You can enable IP Source Guard on Layer 2 interfaces that
are not trusted by DHCP snooping.
When you initially enable IP Source Guard, all inbound IP
traffic on the interface is blocked except for the following:
DHCP packets, which DHCP snooping inspects and then forwards or
drops, depending upon the results of inspecting the packet.
IP traffic from a source whose static IP entries are configured in the
Cisco Nexus 1000V.
48. Nexus 1000v IP Source Guard
www.silantia.com48
Configuration: (can be done under port-profile)
N1KV-VSM(config)# interface vethernet 31
N1KV-VSM(config-if)# ip verify source dhcp-
snooping-vlan
Verification:
switch (config-if)# show ip verify source interface vethernet 3
Filter Mode(for static bindings): IP-MAC IP source guard is
enabled on this interface.
Interface Filter-mode IP-address Mac-address Vlan
---------- ----------- ---------- ----------- ----
Vethernet3 active 1.182.56.137 00:50:56:82:56:3e 1053
Adding Static entry in for IP SG.
N1KV-VSM(config)# ip source binding 10.5.22.17
001f.28bd.0013 vlan 100 interface vethernet 3
50. Nexus 1000v ACL
www.silantia.com50
Two types of ACLs are supported in Nexus 1000v
IP ACL- Applied only to IP traffic
MAC ACL – Applied only to Non IP traffic
Order of ACL application
Ingress port ACL
Egress port ACL
MAC ACL supports following additional filtering options.
Layer 3 protocol
VLAN ID
Class of Service (CoS)
51. Nexus 1000v IP ACL
www.silantia.com51
IP ACL supports following additional filtering options.
Layer 4 protocol
TCP and UDP ports
ICMP types and codes
IGMP types
Precedence level
Differentiated Services Code Point (DSCP) value
TCP packets with the ACK, FIN, PSH, RST, SYN, or URG bit set
All ACL are configured via CLI on VSM and when ACLs are
applied to port-profile or veth/ethernet port it will be processed
at VEM level.
52. Nexus 1000v IP ACL
www.silantia.com52
Configuration example:
ip access-list DENY_OSPF
10 deny ip any 224.0.0.5/32
20 deny ip any 224.0.0.6/32
30 permit ip any any
ip access-list DENY_TELNET
10 deny tcp any 150.10.2.1/32 eq telnet
20 permit ip any any
port-profile type veth SERVERFARM1
ip access-group DENY_TELNET in
53. Nexus 1000v Private VLANs
www.silantia.com53
Private VLANs partition a regular VLAN domain into
subdomains and can have multiple VLAN pairs.
All VLAN pairs in a private VLAN share the same Primary
VLAN. The secondary VLAN ID differentiates one
subdomain from another.
All members in the private VLAN share a common address
space, which is allocated to the primary VLAN.
Private VLANs can span multiple switches. A trunk port
carries the primary VLAN and secondary VLANs to a
neighboring switch. (uplink ports in case of Nexus 1000v)
54. Nexus 1000v Private Vlans
www.silantia.com54
Enable private vlan and configure primary
and secondary vlans
feature private-valn
vlan 153
private-vlan primary
private-vlan association 154-155
vlan 154
private-vlan community
vlan 155
private-vlan isolated
55. Nexus 1000v Private Vlans
www.silantia.com55
! Private vlan configured on port-profile
port-profile type vethernet pv154
vmware port-group
switchport mode private-vlan host
switchport private-vlan host-association 153 154
no shutdown
state enabled
! You can configure private vlan on Veth port itself.
port-profile type vethernet pv155
vmware port-group
switchport mode private-vlan host
switchport private-vlan host-association 153 155
no shutdown
state enabled
!
56. Nexus 1000v Private Vlans
www.silantia.com56
Create uplink port-profile carrying private vlans.
port-profile type ethernet pcpvtrunk
vmware port-group
switchport mode private-vlan trunk promiscuous
switchport private-vlan mapping trunk 153 154-155
switchport private-vlan trunk allowed vlan 153-155
channel-group auto mode on mac-pinning
no shutdown
state enabled