Topic 15: Datacenter Design and Networking

15: Datacenter Design and Networking
Zubair Nabi
zubair.nabi@itu.edu.pk
April 21, 2013
Zubair Nabi 15: Datacenter Design and Networking April 21, 2013 1 / 27

Outline
1 Datacenter Topologies
2 Transport Protocols
3 Network Sharing
4 Wrapping Up

Outline
3 Network Sharing
4 Wrapping Up

Introduction
Datacenters are traditionally designed in the form of a 2/3-level tree

Introduction
Switching elements become more specialized and faster when we go
up the tree structure

Introduction
A three-level tree has a core switch at the root, aggregation switches
in the middle, and edge switches at the leaves of the tree

Introduction
Edge switches have a large number of 1Gbps ports and a small
number of 10Gbps ports

Introduction
The 1Gbps ports connect end-hosts while 10Gbps ports connect to
aggregation switches

Introduction
Aggregation and core switches have 10Gbps ports

Introduction
Aggregation and core switches have 10Gbps ports
Partitioning if switches up the tree go down

Oversubscription
Ideal value of 1:1 – All hosts may potentially communicate with others
at full bandwidth of their interface

Oversubscription
5:1 – Only 20% of the bandwidth is available (200Mbps)

Oversubscription
5:1 – Only 20% of the bandwidth is available (200Mbps)
Typical datacenter designs are oversubscribed by a factor of 2.5:1
(400Mbps) to 8:1 (125Mbps)

Fat-tree Topology
k-ary fat-tree has k pods

Fat-tree Topology
Each pod contains two layers of k/2 switches

Fat-tree Topology
Each k-port switch in the lower layer is directly connected to k/2 hosts

Fat-tree Topology
Each of the remaining k/2 ports is connected to k/2 of the k ports of the

Fat-tree Topology
(k/2)2
core switches

Fat-tree Topology
(k/2)2
core switches
Each core switch has one port connected to each of the k pods

Fat-tree Topology
(k/2)2
core switches
The ith port of any core switch is connected to pod i

Fat-tree Topology
(k/2)2
core switches
The ith port of any core switch is connected to pod i
A k-ary fat-tree supports k3
/4 hosts

DCell
Uses a recursively deﬁned structure to interconnect servers

DCell
Each server connects to different levels of DCells through multiple links

DCell
High-level DCells are built recursively from many low-level ones

DCell
Fault tolerant as there is no single point of failure

Structure
Uses servers with multiple network ports and mini-switches to
construct its recursive structure

Structure
DCell0 is the building block to construct larger DCells

Structure
Consists of n servers and a mini-switch

Structure

Structure
DCell1 constructed using n +1 DCell0s

Structure
DCell1 constructed using n +1 DCell0s
The same applies to DCellk

Outline
3 Network Sharing
4 Wrapping Up

TCP and UDP
TCP: Connection-oriented with reliability, ordering, and congestion
control

TCP and UDP
TCP: Connection-oriented with reliability, ordering, and congestion
control
UDP: Connectionless with no ordering, reliability, or congestion control

TCP and Datacenter Networks
Communication between different nodes is thought of as just opening a
TCP connection between them

Common sockets API

Common sockets API
But TCP was designed for a wide-area network

Common sockets API
Clearly, a datacenter is not a wide-area network

Common sockets API
Signiﬁcantly different bandwidth-delay product, round-trip time (RTT),
and retransmission timeout (RTO)

Common sockets API
For example, due to the low RTT, the congestion window for each ﬂow
is very small

Common sockets API
For example, due to the low RTT, the congestion window for each ﬂow
is very small
As a result, ﬂow recovery through TCP fast retransmit is impossible,
leading to poor net throughput

More problems for TCP
In production data centers, due to the widely-varying mix of
applications, congestion in the network can last from 10s to 100s of
seconds

seconds
In commodity switches the buffer pool is shared by all interfaces

seconds
If long ﬂows hog the memory, queues can build up for the short ﬂows

seconds
Many-to-one communication patterns can lead to TCP throughput
collapse or incast

seconds
collapse or incast
This can cause overall application throughput to decrease by up to 90%

seconds
collapse or incast
In virtualized environments, the time sharing of resources increases
the latency faced by the VMs

seconds
collapse or incast
In virtualized environments, the time sharing of resources increases
the latency faced by the VMs
This latency can be orders of magnitude higher than the RTT between
hosts inside a datacenter, leading to slow progress of TCP connections

Reaction
Some large-scale deployments have abandoned TCP altogether

Reaction
For instance, Facebook now uses a custom UDP transport

Reaction
It might be a “kitchen-sink” solution but it is sub-optimal in a datacenter
environment

Reaction
It might be a “kitchen-sink” solution but it is sub-optimal in a datacenter
environment
Over the years, a number of alternatives have been proposed

Datacenter TCP (DCTCP)
Uses Explicit Congestion Notiﬁcations (ECN) from switches to perform
active queue management-based congestion control

Switches set the congestion experienced ﬂag in packets whenever the
buffer occupancy exceeds a small threshold

DCTCP uses this information to reduce the size of the window based
on a fraction of the marked packets

DCTCP uses this information to reduce the size of the window based
on a fraction of the marked packets
Enables it to react quickly to queue build and avoid buffer pressure

Multipath TCP (MPTCP)
Establishes multiple subﬂows over different paths between a pair of
end-hosts

end-hosts
These subﬂows operate under a single TCP connection

end-hosts
The fraction of the total congestion window for each ﬂow is determined
by its speed

end-hosts
The fraction of the total congestion window for each ﬂow is determined
by its speed
Moves trafﬁc away from the most congested paths

tcpcrypt
Backwards compatible enhancement to TCP that aims to efﬁciently
and transparently provide encrypted communication to applications

tcpcrypt
Uses a custom key exchange protocol that leverages the TCP options
ﬁeld

tcpcrypt
ﬁeld
Like SSL, to reduce the cost of connection setup for short-lived ﬂows, it
enables cryptographic state from one TCP connection to bootstrap
subsequent ones

tcpcrypt
ﬁeld
Like SSL, to reduce the cost of connection setup for short-lived ﬂows, it
enables cryptographic state from one TCP connection to bootstrap
subsequent ones
Applications can also be made aware of the presence of tcpcrypt to
negate redundant encryption

Deadline-Driven Delivery (D3
)
Targets applications with distributed workﬂow and latency targets

)
Such applications associate a deadline with each network ﬂow and the
ﬂow is only useful if the deadline is met

)
Such applications associate a deadline with each network flow and the
flow is only useful if the deadline is met
Applications expose flow deadline and size information which is
exploited by end hosts to request rates from routers along the data path

Outline
3 Network Sharing
4 Wrapping Up

Introduction
Network resources are shared amongst the tenants, which can lead to
contention and other undesired behaviour

Introduction
Network performance isolation between tenants can be an important
tool for:
Minimizing disruption from legitimate tenants that run network-intensive
workloads

Introduction
tool for:
workloads
Protecting against malicious tenants that launch DoS attacks

Introduction
tool for:
workloads
Protecting against malicious tenants that launch DoS attacks
The standard methodology to ensure isolation is to use VLANs

Virtual LAN
Acts like an ordinary LAN but end-hosts do no necessarily have to be
physically connected to the same segment

Virtual LAN
Nodes are grouped together by the VLAN

Virtual LAN
Broadcasts can also be sent within the same VLAN

Virtual LAN
Broadcasts can also be sent within the same VLAN
VLAN membership information is inserted into Ethernet frames

Rate-limiting End-hosts
In Xen the network bandwidth available to each domU can be rate
limited

limited
Can be used to implement basic QoS

limited
Can be used to implement basic QoS
The virtual interface is simply rate-limited

Outline
3 Network Sharing
4 Wrapping Up

The End
In reverse order:
1 Cloud stacks be used to turn clusters and datacenters into private and
public clouds

The End
In reverse order:
public clouds
2 Virtualization of computation, storage, and networking can allow many
tenants to co-exist

The End
In reverse order:
public clouds
tenants to co-exist
3 Most data does not ﬁt the relational model and is more suited for
NoSQL stores

The End
In reverse order:
public clouds
tenants to co-exist
NoSQL stores
4 Data-intensive, task-parallel frameworks abstract away the details of
distribution, work allocation, sychronization, concurreny, and
communication; Perfect match for the cloud

The End
In reverse order:
public clouds
tenants to co-exist
NoSQL stores
4 Data-intensive, task-parallel frameworks abstract away the details of
distribution, work allocation, sychronization, concurreny, and
communication; Perfect match for the cloud
5 The future is Big Data and Cloud Computing!

References
1 Mohammad Al-Fares, Alexander Loukissas, and Amin Vahdat. 2008. A
scalable, commodity data center network architecture. In Proceedings
of the ACM SIGCOMM 2008 conference on Data communication
(SIGCOMM ’08). ACM, New York, NY, USA, 63-74.
2 Chuanxiong Guo, Haitao Wu, Kun Tan, Lei Shi, Yongguang Zhang, and
Songwu Lu. 2008. Dcell: a scalable and fault-tolerant network
structure for data centers. In Proceedings of the ACM SIGCOMM 2008
conference on Data communication (SIGCOMM ’08). ACM, New York,
NY, USA, 75-86.

Topic 15: Datacenter Design and Networking

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (19)

Similar to Topic 15: Datacenter Design and Networking

Similar to Topic 15: Datacenter Design and Networking (20)

More from Zubair Nabi

More from Zubair Nabi (14)

Recently uploaded

Recently uploaded (20)

Topic 15: Datacenter Design and Networking