Data Center Networking - Trends and suggestions

1. www.huawei.com
www.huawei.com
2012/3/6
Data Center Networking
Trends and suggestions
Piotr Głaska
CCIE #15966

2. HUAWEI TECHNOLOGIES CO., LTD.
 Data Center Technology Analysis
 Suggestions
Contents

3. HUAWEI TECHNOLOGIES CO., LTD.
Security
The current data
center network is in
phase 2.
 Server sharing
 TRILL/802.1aq
 802.1Qbg/BR
 Network sharing
 FCOE/DCB
 Firewall
 IPS/IDS
 Anti-DDOS
 UTM
 Physical isolation
 VLAN
 VPN
Phase 1 Phase 2 Phase 3 Time
The network
focuses on security
and isolation, but
ignores sharing.
The cloud network balances
security, sharing, and isolation.
Cloud
Migration The network focuses
on sharing, but
ignores security and
isolation.
Isolation
SharingData center
network
Major Concerns on the Data Center Network
Page 3

4. HUAWEI TECHNOLOGIES CO., LTD.
WAN/Internet
L3
Physical server Physical server
vSwitch
VMVM
vSwitch
VMVM
Physical server Physical server
Migration
Traditional data center network: Small-scale Layer 2
network and large-scale Layer 3 network
Cloud data center network: Large-scale Layer 2
network and small-scale Layer 3 network
Access
Aggregation
Core
L2 switching L2 switching
WAN/Internet
Access
Aggregation
Core
L2 switching
L2
L3 routing L3 routingL3
L2
TRILL/
802.1aq
802.1Qbg/BR
Server Sharing Drives Network Changes
Virtualization-aware technology
(802.1Qbg/BR)
Policy control and migration
Large-scale Layer 2 network
technology (TRILL/802.1aq)
Online migration of VMs
Server
virtualization
Page 4

5. HUAWEI TECHNOLOGIES CO., LTD.
How loops are prevented?
Use a mechanism similar to IS-IS.
1
2
How multi-link bandwidth is used?
Use a mechanism similar to ECMP.
 TRILL/SPB requires network reconstruction including the control and forwarding planes.
 TRILL applies to large-scale Layer 2 networks where more than 5000 servers and over 250 racks are deployed in
a single service area.
 You are advised to use the clustering/stacking technologies to build Layer 2 networks for medium- and small-
scale data centers (less than 2000-3000 servers being deployed in a service area).
TOR TOR TOR TOR
Server
VM
VMM
VMVM
Server
VM
VMM
VMVM
Server
VM
VMM
VMVM
Server
VM
VMM
VMVM
Core
Switch
Core
Switch
Core
Switch
Large-scale Layer 2 network
technologies: TRILL/802.1aq
TRILL/SPB - Large-scale Layer 2 Network
 Switch control plane
 Use a mechanism similar to IS-IS. This
mechanism selects routes based on MAC
addresses and ensures a loop-free network.
 Use a mechanism similar to ECMP. This
mechanism implements load balancing on uplinks
 Switch forwarding plane
 The TRILL header is inserted between
the inner and outer Ethernet headers.
 The MAC-in-MAC or Q-in-Q mode is
used for 802.1aq packet encapsulation.
Page 5

6. HUAWEI TECHNOLOGIES CO., LTD.
TRILL/SPB Deployment Problems
LBL3 gateway FW
TRILL/SPB
One-armed
mode
Edge nodes must
support a large number
of MAC address entries.
TRILL/802.1aq deployment problems
FabricPath
TRILL
QFabric
Large-scale
Layer 2 network
technologies
Proprietary
Standards
802.1aq
Conflict
TRILL/SPB problems
Page 6
Problem Description
Technology
(1) Layer 3 forwarding
performance may
become the
bottleneck.
 The TRILL technology supports only Layer 2 forwarding;
therefore, a device is required to implement Layer 3
functions. The server gateway is also configured on the
device. Consequently, the Layer 3 gateway may
become the bottleneck.
 The Layer 3 gateway/load balancer/firewall is connected
in one-armed mode. The traffic may be looped back,
which wastes bandwidth.
(2) The TRILL
technology cannot
meet requirements
for multi-tenant
isolation.
 Although the TRILL technology scales the Layer 2
network, it still isolates services and users based on
VLANs (a maximum of 4096 VLANs).
(3) Edge nodes are
required to support a
large number of MAC
address entries.
 The TRILL technology scales the Layer 2 network;
therefore, edge nodes must support a large number of
MAC address entries. Assume that there are 10K
servers and the virtualization ratio is 1:50. Edge nodes
must support 500K (10K x 50) MAC address entries.
Standardizati
on and
commercial
use
(4) Vendors do not
conform to the TRILL
standard.
 Some vendors implement their own protocols based on
TRILL; therefore, the products of different vendors are
not interoperable.
(5) The TRILL
technology cannot
provide good OAM
capabilities.
 The OAM standard for the TRILL technology is not
mature, and cannot provide fault and performance
management and monitoring capabilities. Only one draft
defines the OAM standard.
(6) It is incompatible
with the FCoE/DCB
technology.
 The DCB technology is only applicable to the common
Ethernet.
(7) Rarely implemented  The TRILL technology is just promoted.

7. HUAWEI TECHNOLOGIES CO., LTD.
Software-based virtual
switch (VEB)
Network adapter-
based virtual switch
Network-based virtual switch
(VEPA/multi-channel)
 Advantage:
 Availability
 Common switches used
 Disadvantage:
 Low performance
 CPU resource occupied
 Advantage:
 High performance
 Common switches used
 Disadvantage:
 Complex management
 Network adapter
reconstruction and
interworking
 Advantage:
 High performance
 Hierarchical management
 Disadvantage:
 Physical switches need to be
customized.
Ethernet Virtual Bridge
 IEEE 802.1Qbg is supported by most mainstream vendors including HP, IBM, Huawei.
 IEEE 802.1BR is similar to Qbg and is promoted by Cisco.
Page 7

8. HUAWEI TECHNOLOGIES CO., LTD.
Problem Description
Technology
(1) Interface bandwidth is
wasted.
 The GE interface bandwidth is limited. When the GE server is configured with hardware-based
VEPA, traffic between two virtual machines on the GE server must pass through the TOR switch.
This wastes 50% bandwidth.
(2) VEPA deployment is
difficult after combining
VEPA and the software
firewall.
 When VEPA is combined the VM software firewall, the traffic needs to pass through the TOR
switch twice. The path where the traffic passes is difficult and the bandwidth is wasted. In
addition, the TOR switch must support security redirection. The VEPA implementation and
deployment are difficult.
(3) The cooperation
between two management
systems becomes difficult.
 VEPA involves integration of IT and network management and its deployment is difficult. VMs
involves IT management and VSwitches involves network management. The IT and network
management systems are required to cooperate.
Standardizati
on and
commercial
use
(4) There is conflict over
802.1Qbg and 802.1BR.
 There is conflict over the virtualization-aware technology. Cisco promotes 802.1BR and others
promote 802.1Qbg.
 The virtualization-aware technology involves many servers and vendors, and the draft is not
mature.
Hardware-based VEPA deployment problems
?
VM VM VMFirewall
Network adapter
Physical server
TOR switch
IT administrator
Network administrator
Security
redirection
VEPA Deployment Problems
Page 8

9. HUAWEI TECHNOLOGIES CO., LTD.
The LAN, SAN, and IPC are independent; therefore, cable layout is
complex.
Server data, storage, and management interfaces need to connect to network
adapters. The interface costs and power consumption are high.
The FCoE network provides three types of interfaces. This simplifies
cable layout and reduces costs and power consumption.
Ethernet
FC
IPC 10 GE
FCOE/iSCSI
Server interface integration
LAN
SAN
IPC
Bandwidth Delay No packet
loss
Data network
LAN
Storage
network SAN
IPC
√
√ √
√
√
√
The traditional Ethernet must be reconstructed so that all the packets are
forwarded in a short delay, meeting requirements of storage services.
DCB
FCoE
Network Sharing Drives FCoE/DCB
Page 9

10. HUAWEI TECHNOLOGIES CO., LTD.
FCoE/DCB Deployment Problems
Standard FCoE switch model
FC interfaceEth
FCoE interface
FCF/FCF in NPV
FCoE_LEP FCoE_LEP
DCB switch/FSB
Problem Description
Technology
(1) The industry chain
is required to
support FCoE.
 The industry chain must support FCoE, including server
vendors, network device vendors, and storage device
vendors. Connectivity and authentication will take a long
period of time.
(2) There are potential
security risks.
 Core storage services have potential risks.
(3) There is low
maintainability.
 FCoE involves convergence of the storage network and
the data network.
 It is difficult to maintain the network.
 Two maintenance teams need to be combined.
Standardization
and
commercial
use
(4) Some problems are
not resolved by FC-
BB-5.
 The FC domain ID is eight bits; therefore, only a
maximum of 239 TOR switches are allowed on the
network.
 The FSB/NPV model does not support local storage and
forwarding.
(5) It is incompatible
with the TRILL
technology.
 The DCB technology is only applicable to the common
Ethernet.
(6) There is no
success story.
 Not yet widely used
FCoE/DCB Deployment Problems
Standard extension:
 T11 FC Technical committee
 FC-BB-5: released
 FC-BB-6: under construction
 IEEE DCB (Data Center Bridging)
 DCB: released and being optimized.
Page 10

11. HUAWEI TECHNOLOGIES CO., LTD.
Data Center FCoE Standard Inconsistency
 FCoE standards include T11 FC-BB-5 and DCB. Vendor A and vendor B may use different FCoE network
configurations, causing interworking failures. In addition, scalability and migration are limited.
 The IETF, IEEE, and ITU want to integrate their own standards on the FCoE network. There are
uncertainties in the implementation.
FCoE
TRILL
DCB FC-BB-5
IEEE802.1aq/SPB
FC-BB-6
Page 11

12. HUAWEI TECHNOLOGIES CO., LTD.
Cloud Computing Data Center Technologies
Attention rate
Budding stage Overheating stage Disillusion stage Rejuvenation stage Maturity stage
Maturity rate
FCOE: FC-BB-6
DCB
IETF TRILL
IEEE802.1aq
> 4 K tenants
Stacking
10GE access
FCoE: FC-BB-5
IETF ARMD
IEEE 802.1BR
IEEE802.1Qbg
<4 K tenants
Network integration
Network convergence
Network virtualization
Enter the maturity stage
Buffer technology
GE access
Page 12

13. HUAWEI TECHNOLOGIES CO., LTD.
Contents
 Data Center Technology Analysis
 Suggestions

14. HUAWEI TECHNOLOGIES CO., LTD.
 Sharing
 In a service area, servers or storage devices can share data. It is recommended
that less than 100 server racks and less than 2000 servers be deployed in a
service area.
 If devices in service areas of the same security level have shared data, it is
recommended that shared data should be deployed in a specialized area.
 In service areas of different security levels, it is recommended that servers and
storage devices should not share data.
 Isolation
 Devices in service areas of different security levels need to be isolated at Layer 2
and communicate at Layer 3.
 Devices in the same service area can communicate at Layer 2 and process
special services using Layer 3.
 Security
 Security devices should use functions such as ACLs and firewalls to protect
security of Layer 3 services.
Suggestions for Three Major Concerns
Page 14

15. HUAWEI TECHNOLOGIES CO., LTD.
 Modular data center design method
 Use the modular design and new technologies with smooth expansion.
 TRILL/802.1aq
 If there are less than 2000 physical servers, to ensure security, isolation, and
sharing, use clustering/stacking technologies to build the data center network.
In this case, TRILL/SPB is not required.
 802.1Qbg/BR
 To meet requirements of GE servers and ensure security, the VEB mode is
preferred.
 FCoE/DCB
 FCoE and LFR technologies are used by server vendors, network device
vendors, and storage device vendors, therefore, the standardization process is
long. The NAS, IPSAN, or FC SAN is still preferred.
Data Center Network Design
Page 15

16. HUAWEI TECHNOLOGIES CO., LTD.
High-speed bus
Network
adapter
Management
CPU
(Tiny core)
CPU
(Tiny core)
Hard disk
Network
adapter
Network
adapter
Network
adapter
1
2
3
4
5
…
 Ideal data center: one super
computer
 CPU supporting hundreds of enterprise
applications such as Notes and ERP
 Large storage capacity, supporting data
storage of P level
ExtranetIntranet
Disaster recovery
network
CPU
(Tiny core)
CPU
(Tiny core)
 Utopia super computer
architecture
 Thousands of CPUs or CPU tiny cores
 Storage of Petabyte- level
 Many 100G egress network adapters
 Management
 Connection with high-speed buses
Utopia Data Center
Page 16

17. HUAWEI TECHNOLOGIES CO., LTD.
 Through verification of more than 60 years, the computing architecture is proved to be mature and extensible.
 A data center is divided into five areas by logical functions, which can be extended, deployed, and maintained easily.
Service
area 1
Service
area 2
Another service
area
Uniformoperation
andmaintenance
platform
Storage area
Core network
Intranet Extranet Internet
Disaster
recovery
network
Intranet Extranet Disaster recovery
network
Branch
Cooperative company
External user Remote disaster
recovery center
1
2
3
4
5
DMZ…
Monitoringmanagement
Processmanagement
Changemanagement
Oneportal
Logical Architecture
Page 17

18. HUAWEI TECHNOLOGIES CO., LTD.
SDH/VPN
Branch
SDH/VPN
Headquarters SDH/VPN
Cooperative company
internet
IPS
Firewall
Firewall
SDH/WDM
Disaster recovery center
RunAttentionFault
Remote
SP Present
Standby Power
Power
hp rp74xx
RunAttentionFault
Remote
SP Present
Standby Power
Power
hp rp74xx
RunAttentionFault
Remote
SP Present
Standby Power
Power
hp rp74xx
RunAttentionFault
Remote
SP Present
Standby Power
Power
hp rp74xx
RunAttentionFault
Remote
SP Present
Standby Power
Power
hp rp74xx
Service area 2
RunAttentionFault
Remote
SP Present
Standby Power
Power
hp rp74xx
RunAttentionFault
Remote
SP Present
Standby Power
Power
hp rp74xx
RunAttentionFault
Remote
SP Present
Standby Power
Power
hp rp74xx
RunAttentionFault
Remote
SP Present
Standby Power
Power
hp rp74xx
RunAttentionFault
Remote
SP Present
Standby Power
Power
hp rp74xx
Service area 1
RunAttentionFault
Remote
SP Present
Standby Power
Power
hp rp74xx
RunAttentionFault
Remote
SP Present
Standby Power
Power
hp rp74xx
RunAttentionFault
Remote
SP Present
Standby Power
Power
hp rp74xx
RunAttentionFault
Remote
SP Present
Standby Power
Power
hp rp74xx
RunAttentionFault
Remote
SP Present
Standby Power
Power
hp rp74xx
Another service area
FC switch
FC SAN Disk library
1
2
3
Operation,
maintenance, and
management area
4
5
Access
layer
Core
layer
Server
layer
Storage
layer
DMZ
IP SAN
Physical Architecture
Page 18

19. HUAWEI TECHNOLOGIES CO., LTD.
Requirements:
1. Medium- and small-scale data centers: There are less than 2000 servers.
2. Large-scale data centers: Each service area has less than 2000 servers and a small
volume of traffic is transmitted between service areas.
3. Servers are connected in dual-homing mode and work in redundancy mode.
4. There are requirements for VM services and VM migration.
LFR
Core switch
Aggregation switch
Service area 2 (2000 servers)Service area 1 (2000 servers) Service area N (2000 servers)
…Access switch
… … … … … …
1. The clustering/stacking technologies are used to build an LFR Ethernet network,
which improves network reliability.
2. Medium- and small-scale data centers use two-layer architecture (core layer + TOR).
3. Typical networking: Two core devices and up to 120 TOR devices can connect a
maximum of 2000-3000 GE servers.
4. Large-scale data centers use three-layer architecture (core layer + aggregation layer
+ TOR). A pair of core devices are added to transmit inter-area traffic.
Modular Data Center Network Solution
Page 19

20. HUAWEI TECHNOLOGIES CO., LTD.
Loop Loop
Internet WAN
Router
L1 L2&L3
BFD
/IP FRR
/VPN FRR
NIC
Teaming
End-to-end reliability
Cluster
LAG
Stack
+
+
A. LFR provides end-to-end reliability and
fast convergence of 200 ms, ensuring
non-stop service transmission.
B. The LFR technique simplifies network
architecture, provides fast
convergence, and makes full use of
bandwidth. (STP uses 50% bandwidth.)
C. Cluster+LAG+Stack technologies used
to build a LFR Ethernet networkLFR
Ethernet
Core/Aggregation
Access device
LFR - Loop Free Reliable Ethernet
High Reliability, Efficiency, and Easy Management
Page 20

21. HUAWEI TECHNOLOGIES CO., LTD.
Huawei and Data Center Standardization
Page 21
010101010
010101010
010101010
Donald Eastlake
 IETF TRILL standard co-chairman
 Research on large-scale Layer 2
networking on cloud computing
networks
Peter Ashwood-Smith
 IEEE 802.1aq standard contributor
 Research on large-scale Layer 2
networking on cloud computing
networks
Linda Dunbar
 IETF ARMD standard chairwoman
 Research on ARP performance on
cloud computing networks
Ben Mack-Crane
 IEEE 802.1Qbp standard chairman
 Research on load balancing on
cloud computing networks
Data center standards Data center industry chain
 Huawei performs
interworking tests with
upstream and downstream
industry chains.
 Involved vendors:
 Network device vendors
 NMS vendors
 Tester vendors
 Huawei and BT perform
testing on 802.1aq and
802.1ag OAM.
 BT test networking:
 Multiple Huawei S9300s
 90 simulation nodes
provided by
SPIRENT/SOLANA

22. Thank you
www.huawei.com
Huawei Enterprise A Better Way