The document outlines the design and best practices for using VMware's vSphere Distributed Switch (VDS), highlighting its new capabilities and enhancements in version 5.5. It covers various aspects such as deployment strategies, infrastructure design goals, security measures, performance issues, and the importance of using features like Network I/O Control (NIOC) and Load Based Teaming (LBT). Additionally, it provides recommendations for ensuring reliable, secure, and scalable virtual network architecture.

1. vSphere Distributed Switch –
Design and Best Practices
Vyenkatesh (Venky) Deshpande, VMware
Marcos Hernandez, VMware
NET5521
#NET5521

2. 2
Session Objective
 New capabilities in VDS
 VDS can meet your design requirements
 Provide Common best practices while designing with VDS

3. 3
Recommended Sessions & Labs
 VSVC4966 – vSphere Distributed Switch – Technical Deep Dive
 VSVC5103 - vSphere Networking and vCloud Networking Suite
Best Practices and Troubleshooting
 You can check out VSS to VDS Migration workflow and new VDS
features in the lab HOL-SDC-1302
 NET5266 - Bringing Network Virtualization to VMware environments
with NSX
 NET5654 - Troubleshooting VXLAN and Network Services in a
Virtualized Environment

4. 4
Agenda
 Overview of VDS and New Features in 5.5
 Common Customer Deployments
 Design and Best Practices
 NSX and VDS

5. 5
VDS Overview and 5.5 Features

6. 6
vSphere Distributed Switch (VDS)
vSphere Distributed Switch
 Manage a Datacenter wide switch vs. Individual switches per host
 Advanced feature support
 Higher Scale
 Foundation for your Network Virtualization Journey

7. 7
vSphere Distributed Switch (VDS) Architecture
vSphere vSphere
vSphere Distributed Switch
Host 1 Host 2
Legend :
dvPG-A
dvPG-B
Data Plane Data Plane
Data Plane : Handles the packet switching function
VMware vCenter Server
Management Plane
vSphere Distributed Switch
Management Plane : Allows to configure various parameters of the distributed switch
vmnic0 vmnic1 vmnic0 vmnic1
dvUplink PG
dvUplink
dvuplink1 dvuplink2

8. 8
VDS Enhancements in vSphere 5.5
Visibility & Troubleshooting
Performance and Scale
 Host Level Packet Capture
Tool (tcpdump). Available
for Standard Switch as well
 Enhanced LACP
 Enhanced SR-IOV
 40 Gig NIC support
Packet Classification
 Traffic Filtering (ACLs)
 DSCP Marking (QoS)
vSphere Distributed Switch

9. 9
LACP Enhancements
vSphere
vSphere Distributed Switch
Host
Physical switches
LACP
Communication
 Link Aggregation Control
Protocol
 Standards based – 802.3ad
 Automatic negotiation of link aggregation
parameters
 Advantages
 Aggregates link BW and provides
redundancy
 Detects link failures and cabling mistakes
and automatically reconfigures
 Enhancements
 Support for 64 LAGs per VDS and per
Host
 Support for 22 different hashing
algorithms

10. 10
Common Customer Deployments

11. 11
VDS in the Enterprise
VMware vCentServervCenter Server
 Multiple VDS per VC (128)
 VDS can span multiple Clusters
 Hundreds of Hosts per VDS
 Central Management for DC and
ROBO environments
 Role Based management control
VDS VDS
ROBO 1 ROBO 2
VDSVDS VDS
Cluster 1 Cluster 2 Cluster 3 Cluster 4
DataCenter

12. 12
Design Best Practices

13. 13
Infrastructure Design Goals
 Reliable
 Secure
 Performance
 Scalable
 Operational

14. 14
Infrastructure Types Influence Your Design Decisions
 Available Infrastructure
• Type of Servers
• Type of Physical Switches
 Servers
• Rack mount or Blade
• Number of Ports and Speed. For example, Multiple 1 Gig or 2 – 10 Gig
 Physical Switches
• Managed and un-managed
• Protocol and features support
 Example Deployment – 2 – 10 Gig Server configuration

15. 15
Reliable - Connectivity

16. 16
Physical Connection Options
vSphere
VDS
vSphere
VDS
vSphere
VDS
vSphere
VDS
Port Group – Teaming
Port ID, MAC Hash,
Explicit Failover, LBT
One Physical Switch Two Physical Switches One Physical Switch
with Ether Channel
Two Physical Switches
in MLAG configuration
Port Group – Teaming
IP Hash
Port Group – Teaming
LACP
Port Group – Teaming
Port ID, MAC Hash,
Explicit Failover, LBT
MLAG/vPC

17. 17
Connectivity Best Practices
 Avoid Single point of Failure
• Connect two or more physical NICs to a VDS
• Preferably connect those physical NICs to separate physical switches
 Configure Port groups with appropriate teaming setting based on
the physical switch connectivity and configuration. For example
• Use IP hash when Ether channel is configured on Physical Switch
 Configure Port Fast and BPDU guard on Access Switch Ports
• No STP running on virtual switches
• No loop created by virtual switch
 Trunk all Port group VLANs on Access Switch ports

18. 18
Spanning Tree Protocol Boundary
vSphere vSphere
vSphere Distributed Switch
Switch Port
Configuration:
Port Fast
BPDU Guard
VLAN 10,20
Switch Port
Configuration:
Port Fast
BPDU Guard
VLAN 10.20
Physical Network
Virtual Network
Spanning Tree Protocol Boundary
No Spanning Tree
Support
No BPDU
generated

19. 19
Teaming Best Practices
 Link Aggregation mechanisms do not double the BW
• Hashing algorithm performs better in some scenarios. For example
• Web servers accessed by different users have enough variation in IP Src and Dest
fields and can utilize links effectively
• However, few workloads accessing a NAS array doesn’t have any variation in
the packet header fields. Traffic might end up on only one physical NIC
 Why Load Based Teaming is better ?
 Takes into account link utilization
 Checks Utilization of Links every 30 seconds
 No special configuration required on the physical switches

20. 20
Load Based Teaming
1 2 3 4
10 11
VM1 VM2
vMotion
1 2 3 4
10 11
VM2
Network Traffic Bandwidth
vMotion traffic 7 Gig
VM1 traffic 5 Gig
VM2 traffic 2 Gig
10 Gig 2 Gig 7 Gig 7 Gig
VDS VDS
VM1
vMotion
Rebalance

21. 21
Security/Isolation

22. 22
Traffic Types Running on a vSphere Host
vSphere
PG-A PG-B
Host
VDS
PG-C PG-EPG-D
Mgmt
Traffic
vmk3
vMotion
Traffic
vmk4
FT
Traffic
vmk2
NFS
Traffic
vmk1
10 Gig 10 Gig

23. 23
Security Best Practices
 Provide Traffic Isolation using VLANs
• Each Port group can be associated with different VLAN
 Keep default Security settings on the Port group
• Promiscuous Mode – Reject
• MAC address Changes – Reject
• Forged Transmit – Reject
 While utilizing PVLAN feature make sure Physical Switches are
also configured with Primary, Secondary VLAN configuration
 Enable BPDU filter property at Host level to prevent DoS attack
situation due to compromised virtual machines
 Make use Access Control List Feature (5.5)

24. 24
Performance

25. 25
Why Should You Care About Performance?
 As more workloads are getting virtualized, 10 Gig pipes
are getting filled
 Some workloads have specific BW and latency requirements
• Business Critical applications
• VOIP applications
• VDI application
 Noisy Neighbors problem has to be addressed
• vMotion is very BW intensive and can impact other traffic types
• General Purpose VM traffic can impact other critical applications such
as VOIP application

26. 26
Administrator
MgmtvMotion
Teaming Policy
vSphere Distributed Switch
vSphere Distributed
Port groups
Network I/O Control
VM
Traffic
Scheduler
Shaper
Scheduler
Shaper
FT NFS
Traffic Shares Limit
(Mbps)
802.1p
VM Traffic 30 - 4
vMotion 20 - 3
Mgmt 5 - 7
FT 10 - 6
NFS 20 - 5
Port 1
Port 2
10 Gig 10 Gig
Infrastructure Traffics
4000
Limits
Host
Shares %
BW
Link BW
10 Gig
30 30/50 3/5*10 = 6
20 20/50 2/5*10 = 4
Total 50

27. 27
Administrator
MgmtvMotion
Teaming Policy
vSphere Distributed Switch
vSphere Distributed Port groups
Business Critical Applications and User Defined Traffic Types
VM
Traffic
Scheduler
Shaper
Scheduler
FT NFS
Traffic Shares Limit
(Mbps)
802.1p
App1 10 - 7
App2 10 - 6
VM Traffic 10 - 4
vMotion 20 - 3
Mgmt 5 - 7
FT 10 - 6
NFS 20 - 5
Port 1
Port 2
10 Gig 10 Gig
App 2
Traffic
App 1
Traffic
Shaper
Host

28. 28
End to End QoS
 How to make sure that the Application traffic flowing through
Physical Network Infrastructure is also Prioritized ?
 Two types of Tagging or Marking supported
• COS – Layer 2 Tag
• DSCP Marking – Layer 3 Tag
0x8100 COS VLAND
16 bits 3 bits 12 bits1 bit
802.1Q Header
DSCP ECN
6 bits 2 bits
Version H Length TOS/DS P Length …..
IP Header

29. 29
Tagging at Different Level
vSphere
vSphere Switch
Physical
Network
DSCP
COS
vSphere
vSphere Switch
Physical
Network
DSCP
COS
vSphere
vSphere Switch
Physical
Network
DSCP
COS
Guest Tagging Virtual Switch Tagging Physical Switch Tagging
VDS can pass VM QoS
markings downstream
NIOC can’t assign
separate queue based
on the tag
Admins lose control
VDS implements 802.1p and/or
DSCP marking
Preferred option
Single Edge QoS enforcement
point
QoS marking or remarking
done in the physical switch
and/or router
Burdensome QoS management
on each edge device (e.g. ToR)

30. 30
Congestion Scenario in the Physical Network
vSphere
vSphere Switch
vSphere
vSphere Switch
Higher Tagged Traffic
Un Tagged Traffic
Lower Tagged Traffic
Congested Switch
Physical Network

31. 31
MgmtvMotion
Per Port Traffic Shaping
VM
Traffic
10 Gig 10 Gig
Ingress Egress
Time
BW
Average BW
Peak BW
Burst Size
 Ingress and Egress
Parameters
 Average Bandwidth
 Kbps
 Peak Bandwidth
 Kbps
 Burst Size
 Kbytes
Token
Bucket

32. 32
Other Performance Related Decisions
 Need more BW for Storage
• If iSCSI, utilize Multi-Pathing.
• MTU configuration – Jumbo frame
• LBT can’t work for iSCSI traffic because of port binding requirements
 Need more BW for vMotion
• Use Multi-NIC vMotion.
• LBT doesn’t split the vMotion traffic to multiple Physical NICs.
 Latency Sensitive application – Care about Micro seconds
• Utilize SR-IOV
• Doesn’t support vMotion, HA and DRS features

33. 33
Scalable

34. 34
Scale
 Scaling Compute Infrastructure
 Adding Hosts to Clusters
 Adding new Clusters
 Impact on VDS Design
 VDS can span across 500 hosts
VDS
Cluster 1 Cluster 2 Cluster 3 Cluster 4
DataCenter
VDS
Cluster 1 Cluster 2 Cluster 3 Cluster 4
DataCenter
 Scaling number of users or
applications
 More Virtual Machines connected to
isolated networks (VLANs)
 Impact on VDS Design
 Separate port groups for each application
– 10,000 port groups support
 Number of virtual ports - 60,000
 Dynamic Port management (Static Ports)

35. 35
Operational

36. 36
How to Operate Your Virtual Network?
 Major concerns
• Lost visibility into traffic from VM to VM on the same Host
• How do I troubleshoot configuration issues?
• How do I troubleshoot connectivity issues?
 Make use of VDS features
• Netflow and Port Mirroring
• Network Health Check detects mis-configuration across virtual
and physical switches
• Host level Packet Capture allows you to monitor traffic at vnic,
vmknic and vmnic level

37. 37
NSX and VDS

38. 38
VMware NSX Functional System Overview
vSphere vSphere vSphere vSphere
vSwitch vSwitch vSwitch vSwitch
Hosts
Data Plane
Operations
UI
Logs/Stats
CMP
Consumption
Tenant UI
API
Control Plane Run-time state
Management Plane
API
API, config, etc.
HA, scale-out
NSX Manager
NSX Controller
vCenter Server

39. 39
VXLAN Protocol Overview
 Ethernet in IP overlay network
 Entire L2 frame encapsulated in
UDP
 50+ bytes of overhead
 Decouples Physical network
from the Logical
 24 bits VXLAN ID identifies 16 M
Logical networks
 VMs do NOT see VXLAN ID
 Physical Network devices don’t see
VMs MAC and IP address
 VTEP (VXLAN Tunnel End
Point)
 VMkernel interface which serves as
the endpoint for encapsulation/de-
encapsulation of VXLAN traffic
 VXLAN can cross Layer 3
network boundaries
 Technology submitted to IETF
for standardization
• With Cisco, Citrix, Red Hat,
Broadcom, Arista and Others

40. 40
VXLAN Configuration on VDS
vSphere Host
VM1
VXLAN Transport Network
vSphere Host
VM2
vSphere Host
VXLAN 5001
VTEP1 10.20.10.10 VTEP2 10.20.10.11 VTEP3 10.20.11.10
vSphere Host
VTEP4 10.20.11.11
VM3 VM4
VXLAN Transport Subnet A 10.20.10.0/24 VXLAN Transport Subnet B 10.20.11.0/24
vSphere Distributed Switch

41. 41
For More Details on VXLAN attend
NET5654 - Troubleshooting VXLAN and Network
Services in a Virtualized Environment

42. 42
Key Takeaways
 VDS is flexible and scalable to meet your design requirements.
 VDS simplifies the deployment and operational aspects
of virtual network
 Make use of NIOC and LBT feature to improve utilization
of your I/O resources
 VDS is a key component of NSX Platform

43. 43
Q&A
Paper: http://www.vmware.com/resources/techresources/10250
http://blogs.vmware.com/vsphere/networking
@VMWNetworking

44. 44
Other VMware Activities Related to This Session
 HOL:
HOL-SDC-1302
vSphere Distributed Switch from A to Z
 Group Discussions:
NET1000-GD
vSphere Distributed Switch with Vyenkatesh Deshpande

45. THANK YOU

47. vSphere Distributed Switch –
Design and Best Practices
Vyenkatesh Deshpande, VMware
Marcos Hernandez, VMware
NET5521
#NET5521

48. 48
Backup: Example Design

49. 49
VDS in Rack Server Deployment: Two 10 Gig Ports
Access
Layer
Aggregation
Layer
. . . . . . . . . . . . . . . . .
Cluster 1 Cluster 2
ESXi ESXi ESXi ESXi
vSphere Distributed Switch
Legend :
PG-A
PG-BL2 Switch
Router

50. 50
Option1: Static Design – Port Group to NIC Mapping
Traffic Type
Port
Group
Teaming
Option
Active
Uplink
Standby
Uplink
Unused
Uplink
Virtual Machine PG-A LBT
dvuplink1/
dvuplink2
None None
NFS PG-B
Explicit
Failover
dvuplink1 dvuplink2 None
FT PG-C
Explicit
Failover dvuplink2 dvuplink1 None
Management PG-D
Explicit
Failover dvuplink2 dvuplink1 None
vMotion PG-E
Explicit
Failover dvuplink2 dvuplink1 None

51. 51
Option2: Dynamic Design –
Use NIOC and Configure Shares and Limits
 Need Bandwidth information for different traffic types
• NetFlow
 Bandwidth Assumption
• Management – Less than 1 Gig
• vMotion – 2 Gig
• NFS – 2 Gig
• FT – 1 Gig
• Virtual Machine – 2 Gig
 Shares calculation
• Equal shares to vMotion, NFS and Virtual Machine
• Lower shares to Management and FT

52. 52
Option2: Dynamic Design –
Use NIOC and Configure Shares and Limits
Traffic
Type
Port
Group
Teaming
Option
Active
Uplink
Standby
Uplink
NIOC
Shares
NIOC
Limits
Virtual
Machine
PG-A LBT dvuplink1,2 None 20 -
NFS PG-B LBT dvuplink1,2 None 20 -
FT PG-C
LBT
dvuplink1,2 None 10 -
Mgmt. PG-D
LBT
dvuplink1,2 None 5 -
vMotion PG-E LBT dvuplink1,2 None 20 -

53. 53
Dynamic Design Option with NIOC and LBT – Pros and Cons
 Pros
• Better utilized I/O resources through traffic management
• Logical separation of traffic through VLAN
• Traffic SLA maintained through NIOC shares
• Resiliency through Active-Active Paths
 Cons
• Dynamic traffic movement across physical infrastructure need all paths
to be available and handle any traffic characteristics.
• VLAN expertise