The document introduces logical routing concepts in NSX-T, including the logical router, distributed router (DR), services router (SR), edge node, and multi-tier routing topology. It provides terminology, examples of packet flows, and discusses concepts like encapsulation/decapsulation and the role of the edge node. The edge node architecture including virtual machine and multi-N-VDS deployment models are also summarized.

1. Confidential │ ©2019 VMware, Inc.
NSX-T Data Center
2.4 Logical Routing
Amit Aneja
Senior Technical Product Manager, NSBU
May 2019

2. Terminology: Introducing the Logical Router(LR)
2
Confidential │ ©2019 VMware, Inc.
Logical Router
• Provides E-W routing
between different Logical
Switches.
• Peers with the physical
infrastructure for N-S
Routing.
• Can provides network
services like Network
Address Translation(NAT),
Load-Balancing, Perimeter
Firewall, VPN etc.
Logical Switch
1
O R Segment
1
Phys ical
Router
Downlink/
Connected to
Segment
Uplink/
External
10.1.1.0/
24
Logical Switch
2
O R Segment
2
10.2.2.2.0/2
4
10.2.2.1/
24
Log ical
Router
10.1.1.1/
24

3. • Runs locally in the transport nodes
participating in the NSX fabric.
• Typically runs as kernel module in the
hypervisor.
• Provides distributed E-W routing
• Traffic between different subnets on
same hypervisor doesn’t leave the
hypervisor
• Responsible for providing on/off ramp
gateway services including N/S routing.
• Provides centralized services like
NAT, BGP, LB, Edge Firewall, Connectivity to
the physical
• The SR is instantiated as a service on an
appliance called the Edge Node.
Distributed Router (DR) Services Router (SR)
Terminology: Logical Router Components
Distributed Router (DR) and Services Router (SR)
D R S
R
3
Confidential │ ©2019 VMware, Inc.

4. Logical Router: Distributed Component(DR)
Distributed Routing
Tier-0
Log ical
Router
ES Xi- ES Xi-
2
K V M
1
D R D
R
D R
10.1.1.0/
24
10.2.2.0/24
10.1.1.10 10.2.2.10/2 10.2.2.20/2 10.1.1.20/
/24 4 4 24
4
Confidential │ ©2019 VMware, Inc.

5. Logical Router: Services Component(SR)
Services Router (SR)
Load
Balancing
Pto V
Gateway
Router Perimeter
Firewall
VPN
Whenever a service which cannot be distributed is enabled on a Logical Router, an SR or
Services Component is instantiated.
SR is instantiated for the following services:
SR is instantiated on an appliance called the Edge Node.
NAT DHCP
5
Confidential │ ©2019 VMware, Inc.

6. • Edge-nodes are appliances with pools of capacity for hosting any services
which are not distributed.
• Form Factor Choice - Virtual Machine or Bare Metal
Both OVA and ISO both flavors available
• Sizing Choice - 3 sizes available (Small, Medium, Large)
• Built for resiliency - A/A and A/S models available
• Leverages DPDK technology for fast packet processing.
NSX-T Terminology: Edge Node
Introducing the “Edge Node”
BM
*Edge Node in NSX-T = ! Edge service Gateway (ESG) in N S X for
vSphere
6
Confidential │ ©2019 VMware, Inc.

7. Compute Hypervisors
(vSphere / K V M )
Confidential │ ©2019 VMware, Inc. 7
Logical Routing Topology
Spin
e
WAN
Infrastructure Clusters:
Edge Nodes, Management
Nodes
Lea
f
Edge Node
hosting S R
D R on every
hypervisor
(in kernel)

8. Confidential │ ©2019 VMware, Inc. 8
Topology View : Putting it all together
DR/SR Interaction
Phys ical
Router
10.1.1.10/24
ES X i
Transport
Network
10.1.1.10
W eb
1
External
Interface
169.254.0.1 169.254.0.2
169.254.0.1
DR SR
W eb
1
W eb
S egmen
t
Tier-0
G atewa
y
10.1.1.1
Web Segment
Transit Segment
DR
10.1.1.1
Create
External
interface
N S X Management
plane auto-plumbs
this link (internal VNI)
and routing between
D R and S R
P hys ical
Router
Edge Node
NSX user configuration What happened behind the scenes?

9. Logical Routing
Packet Walk (South-North traffic)
Edge Node
ES X i
Transport
Network
External Segment
WEB Segment
Transit Segment
SR
DR
Physical
Router
DR
10.1.1.10,
MAC1
W eb1
192.168.100.0/24
192.168.240.1
192.168.240.3
• TEP
10.10.10.1
0
DR Routing table
Network
0.0.0.0/0
10.1.1.0/24
Gateway
169.254.0.2
0.0.0.0
DR ARP Table
Network Mac
169.254.0.2
02:50:56:56:53:00
Logical Switch MAC Table
Outer IP
Inner MAC
02:50:56:56:53:00
30.30.30.30
169.254.0.2
Payload
Src= 10.1.1.10
D s t =
192.168.100.1
• TEP
30.30.30.30
SR Routing table
Next hop
Type Network
t0s 0.0.0.0/0
t0c 10.1.1.0/24
192.168.240.1
0.0.0.0
GENEVE
Src=10.10.10.
10
Dst=30.30.30.
30
9
Confidential │ ©2019 VMware, Inc.

10. Logical Routing
Packet Walk (North–South traffic)
Edge Node
ES X i
Transport
Network
External Segment
WEB Segment
Transit Segment
SR
DR
Physical
Router
DR
10.1.1.10,
MAC1
Web1
192.168.100.0/24
192.168.240.1
• TEP
10.10.10.1
0
• TEP
30.30.30.30
SR Routing table
Type Network Next hop
t0s 0.0.0.0/0 192.168.240.1
t0c 10.1.1.0/24 0.0.0.0
IP
ARP Table
: 10.1.1.10
MAC : 00:50:56:b7:2c:79
MAC Table of Web-Segment
MAC : 00:50:56:b7:2c:79
LOCAL : 30.30.30.30
REMOTE: 10.10.10.10
ENCAP : GENEVE
Payload
Src=
192.168.100.1
D s t = 10.1.1.10
GENEVE
Src=30.30.30.
30
Dst=10.10.10.
10
10
Confidential │ ©2019 VMware, Inc.

11. Multi-Tier Routing
11
Confidential │ ©2019 VMware, Inc.

12. •
•
Tenant Isolation
Separate control for Infra
and Tenant admin
Eliminates dependency on
physical infrastructure
when a new tenant is
provisioned
•
• Role- Connects to physical infra
• Manual Management
Tier-0 Logical Router
Benefit
Logical Routing- Multi Tier Topology
Tier-0
Logical Router
Phys ical
Router
Tier-1
Log ical
Router
12
Confidential │ ©2019 VMware, Inc.
Tier-1 Logical Router
•
•
Role- Per tenant first hop router
Cloud Management Platform
(CMP) driven Management
Tier0-Tier1 Transit
link
(100.64.0.0/31)
Uplink
Tier-1
Logical Router
D ownlink
Tenant-1 Tenant-2

13. Logical Routing- Multi Tier Topology
Interface and Route types
Interface Types
Uplink/External
▪ Used to connect to physical infrastructure
Tier0-Tier1 Transit link
▪ Used to interconnect Tier0 and Tier1 Logical Routers.
DownLink
▪ Used to connect overlay Logical Switches
Service Port
▪ Interface connecting Vlan backed logical switch to provide
connectivity to vlan backed physical or virtual workloads.
Tier-0
Log ical
Router
P hys ica
l
Router
Tier-1
Logical Router
13
Confidential │ ©2019 VMware, Inc.
20.20.20.0/2
4
30.30.30.0/24
100.64.224.0/31
100.64.224.1/31

14. Logical Routing- Multi Tier Topology
Route Advertisement and Route Redistribution- Auto Plumbing
Phys ica
l
14
Confidential │ ©2019 VMware, Inc.
Router
20.20.20.0/2
4
30.30.30.0/24
100.64.224.0/31
100.64.224.1/31
Tier-1
Logical Router
20.20.20.0/24 & 30.30.30.0/24
are seen a s Tier-1 Connected
routes * with next hop IP a s
100.64.224.1/31
Tier-0 redistributes Tier-1
Connected routes * (20.20.20.0/24
& 30.30.30.0/24)
Default route with next hop
IP a s 100.64.224.0/31
20.20.20.0/24 &
30.30.30.0/24
are seen a s eBGP routes
with next hop IP a s
192.168.240.3/24
24
Tier-0
Log ical
Router
192.168.240.1/
24
192.168.240.3/
* NSX-static –NSX-T 2.3 or
earlier

15. Logical Routing- Multi Tier Topology
Multi Tier Distributed Routing
ESXi-1
Tier-0
D R
Tenant 1
Tier-1 D R
Tenant 2
Tier-1 D R
ESXi-
2 Tier-0
D R
Tenant 1
Tier-1
D R
Tenant 2
Tier-1 D R
• Tier0 and Tier1 routers are also instantiated on the hypervisors in order to prevent hair-
pinning
• Fully distributed architecture : as much routing as possible is performed upfront at the
source
15
Confidential │ ©2019 VMware, Inc.
100.64.224.0/31 100.64.224.2/31
100.64.224.1/31 100.64.224.3/31
100.64.224.0/31 100.64.224.2/31
100.64.224.1/31 100.64.224.3/31

16. Edge Node Architecture
16
Confidential │ ©2019 VMware, Inc.

17. Edge Node Architecture
Basics
•
• Edge Node needs to encapsulate/decapsulate
traffic to/from Compute Transport nodes.
-Overlay Transport Zone
Edge Node needs to send/receive traffic to/from
Physical infrastructure.
-Vlan Transport Zone
• Edge Node is a TN and has to be prepared
• Edge Node can belong to a single Overlay
Transport Zone and multiple VLAN
Transport Zone.
Uplink Profile- The Uplink Profile is a global
constructs that defines how traffic will leave a
Transport Node. The Uplink Profile is applied to a
Transport Node when it joins a Transport Zone.
-One Uplink Profile per Transport Node
-One Uplink Profile per N-VDS (default mode)
17
Confidential │ ©2019 VMware, Inc.

18. Edge Node Architecture
VM Form Factor Edge Node
Edge Node V M
•
•
4 vNIC VM
Leverages portgroups to
send/receive traffic
Supported on ESXi host only
Supported on VSS, VDS and
N-VDS*
Internal Architecture remains
the same as Baremetal Edge
Same N-VDS can carry both
Tunnel and Uplink traffic or
you could use a separate N-
VDS for Tunnel and Uplink
traffic
•
•
•
•
ESXi Host
vNIC1
Edge Node V M
vNIC2 vNIC3 vNIC4
Uplink1
N-VDS
Overlay
N-VDS
TEP-IP
TEP-IP
Uplink2
N-VDS
Edg e-M g m t
Vlan 72
Edg e-Transport
Vlan 75
Edg e-Uplink1
Vlan 76
Edg e-Uplink2
Vlan 77
VS S /VD S
Uplink2
Uplink1
18
Confidential │ ©2019 VMware, Inc.
Portgroups

19. Edge Node Architecture
VM Form Factor - Three N-VDS deployment
V D S
O verla
y N-
V D S
Ext-1
N -VD S
M g m t P G Transport PGExt1 PG Ext2 PG
Vlan 100 Vlan 200 Vlan 300 Vlan 400
vNIC2
fp-eth0
ESXi
Port Groups
VDS-Uplink1 VD S-Uplink2
P2
P1
Edg e
V M 1
TOR-Left
Mgmt VLAN : 100
Overlay VLAN : 200
Mgmt VLAN: 100
Overlay VLAN 200
External VLAN 1:300
Mgmt VLAN : 100
Overlay VLAN 200
External VLAN 2:400
TOR-Right
Ext-2
N -VD S
TEP-IP
Uplink1 Uplink1
vNIC3
fp-eth1
vNIC4
fp-eth2
Uplink1
vN IC1
eth0
Each N-VDS has its own Uplink i.e.
Uplink1 as defined by the transport
node profile below.
S a m e profile is used for all three N-
VDS.
Overlay N-VDS Uplink1 = vNIC2 of Edge
V M
Ext1 N-VDS Uplink1 = vNIC3 of Edge
V M
Ext2 N-VDS Uplink1 = vNIC4 of Edge
V M
VDS Uplinks
19
Confidential │ ©2019 VMware, Inc.
Fastpath interfaces
(fp-eth)

20. Edge Node Architecture
Edge VM configuration
20
Confidential │ ©2019 VMware, Inc.
Edge V M configuration
• Portgropus leveraged by Edge
VM can be access or trunk
PG should be configured as
Trunk when Edge VM sends
802.1Q tag.
-
- Vlan defined at the vlan
backed LS used to connect
Tier-0 to physical
Uplink profile has a transport
vlan defined

21. Edge Cluster
Basics and Single Edge Cluster
•
•
•
Edge Nodes are pooled in clusters for scale out and redundancy
Maximum of 10 Edge nodes in an Edge Cluster
There is total flexibility in assigning Logical routers to Edge Nodes and clusters
Design 1: All Edge nodes in a single Edge cluster.
- ECMP layer at Tier-0 uses all nodes (if uplinks are present)
- Simplicity
EN1 EN2 EN 3
Tier-0 Tier-0 Tier-0 Tier-0
EN 4
Edge Cluster1
Tier-1 Tier-1
Tier-1 Tier-1 Tier-1
21
Confidential │ ©2019 VMware, Inc.
Tier-1
Tier-1 Tier-1

22. High-Availability
22
Confidential │ ©2019 VMware, Inc.

23. Confidential │ ©2019 VMware, Inc. 23
High Availability
Active/Active and Active/Standby
Edge Cluster
NSX-T Edge nodes are pooled in edge-
cluster to provide scale out and High-
Availability for Services.
Gateway in Active/Active HA
mode
•
•
•
Scale out HA
ECMP
Stateless Services (Reflexive
NAT)
Gateway in Active/Standby HA
mode
Stateful Services
SNAT/DNAT
Load Balancer
Edge Firewall
DHCP server
VPN
•
•
•
•
•
Edge Cluster
Tier-1 Tier-1
Tier-0 Tier-0
Tier-1
Tier-1
Edge Node1 Edge Node2

24. Confidential │ ©2019 VMware, Inc. 24
High Availability
Failure Triggers
Failure Triggers
• BFD sessions on Management
and Tunnel interfaces of Edge
node are down.
• All GENEVE Tunnels down
• Northbound routing state is
Down (Applicable to Tier-0 SR
only)
Edge
Cluster
Active
Tier-0 S R S
Ttia
en
r-
d
0
by
S R
EN 1 EN 2
eBGP
Mgmt Network
Tunnel
Network
eBGP
eB GP
• TEP • TEP

25. Confidential │ ©2019 VMware, Inc.
Thank You