1. NSX for Small Data Centers - Breaking Boundaries
Shahzad Ali, VMware, Inc
NET8935
#NET8935
2. Security
Inherently secure infrastructure
Automation
IT at the speed of business
Application continuity
Data center anywhere
NSX customer use cases
Micro-segmentation
DMZ anywhere
Secure end user
IT automating IT
Multi-tenant infrastructure
Developer cloud
Disaster recovery
Cross cloud
Multi data center pooling
Shahzad Ali NSX For Small DC
3. • This presentation may contain product features that are currently under development.
• This overview of new technology represents no commitment from VMware to deliver these
features in any generally available product.
• Features are subject to change, and must not be included in contracts, purchase orders, or
sales agreements of any kind.
• Technical feasibility and market demand will affect final delivery.
• Pricing and packaging for any new technologies or features discussed or presented have not
been determined.
Disclaimer
3
4. Abstract / Motivation
• NSX Reference Architecture Recommends:
Dedicated Mgmt., Edge and Compute clusters
– http://tinyurl.com/nsxdg3-0
• Limiting Factors
– Budget
– Staffing
– Small scale deployment
– Small number of hosts for dedicated clusters
SMALL DC DOES NOT MEAN SMALL CUSTOMER
Break
Boundaries
&
Design and
Deploy NSX in
Small DC with a
Single Cluster
Shahzad Ali NSX For Small DC
5. Agenda
5
1 Introduction
2 Deployment Models
3 Design and Deployment Considerations
4 Growth – Business Needs
5 Closing / QA
Disclaimer: Not all possible Small DC designs are discussed
Only few common options are shown
Shahzad Ali NSX For Small DC
6. Understanding of vSphere and NSXv Components
NSX Advance Technical Session
6
NSX-MGR
Logical Switch
vCenter (VC)
Management Plane
vCenter: VDS, DRS, HA, vMotion etc.
NSX-MGR: API Entry Point
Control Plane
Separation of control and data plane
Manages Logical networks
Control Plane protocol (VXLAN, Routing)
Data Plane
Distributed Functions
Scale-out Model
Data Plane
NSX Edge Service Gateway (ESG)
Functions
VM Form Factor
NSX
EDGE
NAT
Firewall Load Balancer
(LB)
Router
NSX-Controller ClusterDLR Control VM
Distributed Logical Router
(DLR)
Distributed Firewall
(DFW)
Reference
Shahzad Ali NSX For Small DC
VDS
7. Large DC Cluster Design
• Typical number of hosts > 100
– NSX Design guide - http://tinyurl.com/nsxdg3-0
• North-South (N-S) BW requirement > 10G
7
Large
DC
Medium
DC
Small
DC
Cluster
Type
Number of
Hosts
Features
Mgmt. 3
• VC, NSX and other mgmt. VMs
• Less I/O requirements
Edge 4 (ECMP)
• ESG, DLR Control VMs
• On/Off-ramp, P/V, ECMP
• Higher I/O requirement
Compute As needed
• Application/Workload
• vMotion boundary
• Variable CPU, Memory & I/O
requirement
Management
WAN
Internet
L3
L2
Compute
Host 1
Host 3
Host 2
Host 6
Host 5
Host 4
Host 1
Host 3
Host 2
Host 6
Host 5
Host 4
L3
L2
DC Fabric
Edge
NSX
EDGE
NSX
EDGE
NSX
EDGE
NSX
EDGE
Shahzad Ali NSX For Small DC
8. Medium DC Cluster Design
• Typical number of hosts: 10-100
• North-South (N-S) BW requirement < 10G
8
Medium
DC
Large
DC
Small
DC
Cluster
Type
Number of
Hosts
Features
Collapse
Mgmt.
Edge
3
• VC, NSX and other mgmt. VMs
• ESG, DLR Control VMs
• Mix of less I/O and High I/O
requirement VMs
Compute As needed
• Application/Workload
• vMotion boundary
• Variable CPU, Memory & I/O
requirement
Management
&
Edge Clusters
Collapsed Edge and Management
but separate Compute
WAN
Internet
L3
L2
Compute
Cluster
Host 1
Host 3
Host 2
Host z
Host y
Host x
NSX
EDGE
NSX
EDGE
Shahzad Ali NSX For Small DC
9. Small DC Cluster Design
• Typical number of hosts: 3 - 10
• North-South (N-S) BW requirement < 10G
9
Medium
DC
Large
DC
Small
DC
Single Cluster hosting Mgmt., Edge & Compute resources
Resource reservation is the key to meet SLA in Small DC
Cluster
Type
Number of
Hosts
Components
Collapse
Mgmt.
Edge
Compute
3
• VC, NSX and other mgmt. VMs
• ESG, DLR Control VMs
• Mix of less I/O and High I/O
requirement VMs
• Application/Workload
• Variable CPU, Memory & I/O
requirement
Shahzad Ali NSX For Small DC
11. VXLAN Backed
Port Groups
(LS)
NSX Deployment Models in Small DC
12
VDS
DFW
VLAN Backed
Port Groups
Physical
NSX Edge
Routing
LB
FW
DFW
Physical
DLR
Transit LS
Uplink Port Group
Uplink Port Group
Security Focused Deployment Model
• Distributed Firewall
• Non disruptive
• VXLAN is not a requirement
• Agentless Anti-Virus (AV)
Full Stack Deployment Model
• Security Focused Deployment +
• Logical Switching (VXLAN)
• Distributed Routing (DLR)
• ESG Services (NAT, LAB, VPN etc.)
LB
Bridge
Shahzad Ali NSX For Small DC
12. Centralized Edge Deployment Model
• Could be used as
– Intermediate Step: Security Focused Full Stack deployment
– Where not much East/West traffic required
– Multi-function gateway
• Highlights
– No DLR, VXLAN and Controllers needed
– VLAN backed-port groups directly attached to ESG VM
– No physical routing/MTU changes needed
– Availability improved by Edge HA and vSphere
13
VDS
NSX ESG
Routing
Firewall
LB
NAT
VPN GW
DFW
VLAN Backed
Port Groups
Physical
WAN/Internet
L3
L2
Host 1
Host 2
Host 3
NSX
EDGE
NSX
EDGE
Single Collapsed Cluster
Shahzad Ali NSX For Small DC
13. Security Focused Model: Design Considerations
• Use-Cases
• Micro-Segmentation (DFW)
• Agentless Anti-Virus (AV)
• Highlights
• No physical routing/MTU change needed
• Use existing VLAN backed-port groups
• Security Services requires Service VMs
• DFW enabled on all hosts
14
Management and Compute collapsed in a single cluster
Single
Cluster
Components
Management
Plane
NSX Manager, VC, LogInsight, vROps
and other management VMs
Compute
Compute VMs
Service VMs
Data Plane
ESXi Kernel Component
Distributed Firewall (DFW)
vSphere Distributed Switch (VDS)
Shahzad Ali NSX For Small DC
14. WAN
Internet
Security Focused Model: Deployment Considerations
• Small footprint
– Min: 2 hosts required
– Easy expansion for additional workload
– Deploy more hosts to sustain a single host failure
– Recommendation: At least 3 hosts in production
15
Use-Case: Micro-Segmentation (DFW)
Single Cluster with
NSX
L3
L2
Host 1
Host 3
Host 2
Function vCPU MEM (GB) Storage (GB) VMs
Tiny vCenter Appliance
with Embedded PSC
2 8 116 1
NSX Manager 4 16 60 1
Total 6 24 176 2
NSX Footprint
Shahzad Ali NSX For Small DC
15. WAN
Internet
Security Focused Model: Deployment Considerations
• Agentless-AV requires additional Service VMs
– NSX GI-SVM (Guest Introspection Service VM)
– Partner Service VM (SVM)
– Cluster based SVM deployment
– Don’t move SVM (manual, vMotion or Storage vMotion)
• Small footprint
– Min: 2 hosts required
– Recommendation: At least 3 hosts in production
16
Use-Case: DFW with Agentless Anti-Virus (AV)
Single Cluster with
NSX
L3
L2
Host 1
Host 3
Host 2
NET8022 – Implementing Agentless AV and IPS/IDS with NSX
NSX GI
SVM
Partner
SVM
NSX GI
SVM
Partner
SVM
NSX GI
SVM
Partner
SVM
Shahzad Ali NSX For Small DC
16. Full Stack Model: Design Considerations
• Use-Cases
– Full abstraction from underlying hardware
– Networking and Security closest to the workload
– Disaster avoidance and recovery (DR)
• Highlights
– VXLAN based L2 over L3 overlay
– Optimized routing (DLR) and logical switching (LS)
– Separation of control and data plane
– DFW and VXLAN enabled on all hosts
– Connectivity to physical network may require
additional changes
• MTU of >=1600 for VTEP segment
17
Management, Edge and Compute collapsed in a single cluster
Cluster
Function
Components
Management
Plane
NSX Manager, Controllers, VC, DB
Server and other management VMs
Compute
Compute VMs
Service VMs
Data Plane
East-West
ESXi Kernel Component
(VXLAN, DLR, DFW, VDS)
Data Plane
North-South
Active/Standby DLR Control VM
ESG VM (HA or ECMP Mode)
Shahzad Ali NSX For Small DC
17. Full Stack Model: Deployment Considerations
• At least 3 hosts needed
– Design to sustain at least a single host failure
– Management and Edge functions can co-exist with Compute
– No DLR Control VM needed with static routing
– Recommendation: 4 ESXi hosts in Production
18
Single Cluster
WAN
Internet
L3
L2
Host 1
Host 3
Host 2
Host 4
Function vCPU MEM (GB) Storage (GB) VMs
Tiny vCenter Appliance with
Embedded PSC
2 8 116 1
NSX Manager 4 16 60 1
Controllers 4 x 3 4 x 3 20 x 3 3
Edge VM (Large)* 2 x 2 0.5 x 2 ~1 x 2 2*
Total 22 37 ~ 238 7
* ESG with High Availability with static routing
NSX
EDGE
NSX
EDGE
Shahzad Ali NSX For Small DC
19. vCenter (VC)
• Tiny vCenter (VC) Appliance with Embedded PSC
– If reduced resource utilization are key factors for the environment
• Majority Small DC Customers:
– Deploy Small VC appliance
– Future growth
vSphere / VC is the foundation
Options Hosts VM Potential
NSX Deployment Type
vCPU MEM (GB) Disk (GB)
Embedded PSC
Tiny 10 100 Small DC 2 8 116
Small 100 1000 Small DC 4 16 136
Medium 400 4000 Medium DC 8 24 275
Large 1000 10,000 Large DC 16 32 325
http://tinyurl.com/DeployVC6
http://tinyurl.com/PerformanceVC6
Reference
Shahzad Ali NSX For Small DC
20. License Considerations
– NSX supported for all vSphere licenses
– VDS included with NSX (vSphere 5.5 U3 or 6.0+)
21
NSX
vSphere Enterprise is EoA: https://kb.vmware.com/kb/2143987
Compare License Options: http://www.vmware.com/products/vsphere.html#compare
Essential+ • Up to 3 hosts, vSphere HA
Standard • 1000 hosts per vCenter, vSphere HA
Enterprise or Enterprise+ • vSphere Standard + DRS Related Features
vSphere
Features Standard Advance Enterprise
Distributed Routing and Switching (DLR/VXLAN) ✓ ✓ ✓
NSX ESG (except load balancer) ✓ ✓ ✓
SW L2 bridging ✓ ✓ ✓
Distributed Firewall (DFW – Micro-Segmentation) ✓ ✓
NSX Edge load balancing ✓ ✓
Cross vCenter NSX ✓
Reference
Shahzad Ali NSX For Small DC
21. Design Considerations: vCenter
• VC with embedded PSC is recommended for small DC
– 1 single sign-on domain with single site
– No growth plans in near future
• External PSC is recommended for medium-large
environments with multiple vCenters
– Consider this option if planning to grow
• VC should be first to boot
22
• Add management VMs in the NSX “VM Exclusion List”
• Or create fine grained rules in DFW
• NSX components are automatically part of exclusion list
vCenter
Server
Platform
Services
Controller
(PSC)
Virtual Machine
Shahzad Ali NSX For Small DC
22. Design Considerations: NSX Manager
• vCPU and Mem modification allowed
– Recommended to stick with the defaults
• Second in VM boot order
• Management plane only
– Never in the data path
23
16 GB
reserved by
default
Schedule
Backup
Shahzad Ali NSX For Small DC
23. Design Considerations: NSX Controllers
• Must deploy 3
– Each on separate hosts
– Use “SHOULD” anti-affinity rules
– Use 4 hosts for additional redundancy
– Controller VM (vCPU/MEM) modification not
possible (4 vCPU, 4GB Mem)
• Only needed for VXLAN and DLR
• 3rd in VM boot order
• Never in the data-path
24
Default
2GB reserved
4GB total
Shahzad Ali NSX For Small DC
24. Design Considerations: DLR Control VM
• Needed for dynamic routing
• Deploy in HA mode (Active/Standby)
• vCPU/MEM modification disabled
• Anti-affinity rule is created automatically
25
No vCPU or
Mem reserved
by default
Shahzad Ali NSX For Small DC
26. Deployment Consideration: ESG (1/2)
• ESG VM Form factor
– Large: Good for majority design/features
– X-Large: For L7 NSX Load Balancer (LB)
– Reserves vCPU and Mem at creation
– Form factor can be upgraded any time later
• ESG VMs have reservation enabled by default
– Locked down VM
• ESG Deployed in HA
– Anti-affinity rules automatically created (DRS)
– Avoid: Active ESG and Active DLR Control VM on
same host
– Example config:
• Host1: Active ESG + Standby DLR Control VM
• Host2: Standby ESG + Active DLR Control VM
27
Automatic
Rule
VM Size vCPU
Memory
(GB)
HD
(GB)
Suitable For
Large 2 1 1 Small DC
X-Large 6 8 2.5 L7 LB
Shahzad Ali NSX For Small DC
27. Deployment Consideration: ESG (2/2)
• ESG Deployed in ECMP
– Avoid: ESG VM and Active DLR Control VM on same host
– Example config:
• Host1: ESG-1 + ESG-2
• Host2: ESG-3 + ESG-4
• Host3: Active DLR Control VM
• Host4: Standby DLR Control VM
• Manually create anti-affinity rules
28
Host 1
Host 3
Host 2
Host 4
NSX
EDGE
NSX
EDGE
NSX
EDGE
NSX
EDGE
Active DLR
Control VM
Standby DLR
Control VM
Shahzad Ali NSX For Small DC
28. VDS (vSphere Distributed Switch) Considerations
• VDS requires vSphere Enterprise+
– Free with NSX (vSphere 5.5 U3 or 6.0+)
• Use single VDS – keep it simple
• Recommended VTEP vmknic teaming policy is
Route Based on Originating Port (Source-ID)
– Provides VXLAN multipath with multiple VTEPs
per host
– VM-to-VTEP pinning based on the VM source
virtual port ID
– For single VTEP without VXLAN multipath - use
“Fail Over”
Shahzad Ali NSX For Small DC
31. DFW Service Insertion Full Stack
32
Enhancing DC Security Beyond DFW
Note: Other topologies are possible – the pictures shown are representative only
Partner
SVM
GI
SVM
VDS
Distributed
Firewall
Partner
SVM
GI
SVM
VLAN Backed Port Groups
NSX
EDGE
VXLAN Backed Port Groups
NSX
EDGE
VXLAN Transit
Logical Switch
Uplink Port Group
Uplink Port Group
Shahzad Ali NSX For Small DC
32. ESG HA DFW L2 Bridging
33Note: Other topologies are possible – the pictures shown are representative only
DLR
NSX Edge Features
Routing
Firewall
LB
Distributed
Firewall
Shahzad Ali NSX For Small DC
33. Single Site Multi-Site (Cross-VC NSX)
34
Site-A Site-B
DLR Universal DLR
Shahzad Ali NSX For Small DC
35. Shahzad Ali NSX For Small DC
NSX Already Deployed In Small DCs
Modular and Flexible
Any Size
Any Vertical
Any Use-Case BeyondLargeMediumSmall
36. Learn More
Connect & Engage
communities.vmware.com
NSX Product Page & Technical Resources
vmware.com/products/nsx
Network Virtualization Blog
blogs.vmware.com/networkvirtualization
VMware NSX on YouTube
youtube.com/user/vmwarensx
Where to get started
At VMworld
70+ Unique NSX Sessions
Spotlights, breakouts, quick talks & group discussions
Visit the VMware Booth
View use case demos and chat with NSX experts
Visit NSX Technical Partner Booths
Integration demos – EPSec & NetX, Hardware VTEP,
Ops & Visibility
Test Drive NSX with free Hands-on Labs
Expert-led or Self-paced. labs.hol.vmware.com
VMware Services for NSX
NSX Proactive Support Service
Optimize performance based on data monitoring
and analytics to help resolve problems, mitigate
risk and improve operational efficiency.
vmware.com/products/nsx/services.html
Training and Certification
Several paths to professional certifications. Learn
more at the Education & Certification Lounge.
vmware.com/go/nsxtraining
Reference
37. NSX partner ecosystem
Physical Infrastructure
Security
Application Delivery
Operations and Visibility
DYNAMIC INSERTION OF
PARTNER SERVICES
Reference
