a

1. Data Center Networking
Stanford CS144 Lecture 17
Philip Levis, 11/30/11

3. Low latencies: μs
High capacity: GigE, 10 GigE
Specialized traffic
Centrally managed

4. Topology
(picture courtesy of Al-Fares et al, “A Scalable, Commodity Data Center Network Architecture”)

5. Storage Workload
(picture courtesy of Phanishayee et al, “Measurement and Analysis of TCP Throughput Collapse in
Cluster-based Storage Systems”)

6. Query Workload
(picture courtesy of Alizadeh et al., “Data Center TCP (DCTCP)”)

7. Problems

8. Per-Pair Bandwidth
(picture courtesy of Al-Fares et al, “A Scalable, Commodity Data Center Network Architecture”)

9. Incast
(from Phanishayee et al, “Measurement and Analysis of TCP Throughput Collapse in Cluster-based Storage
Systems”)

10. Incast Details
(from Phanishayee et al, “Measurement and Analysis of TCP Throughput Collapse in Cluster-based Storage
Systems”)

11. Mixed traffic
• Low latency for short flows
• High burst tolerance (incast)
• High throughput for long flows

12. Recent Research
• New switching topology: Al-Fares et al.
• Fix TCP incast: Vasudevan et al.
• Data Center TCP: Alizadeh et al.

13. Per-Pair Bandwidth
(picture courtesy of Al-Fares et al, “A Scalable, Commodity Data Center Network Architecture”)

14. Fat Tree

15. Fat Tree
(k/2)2
k/2
k/2
k

16. Switching
Prefix Port
10.2.0.0/24 0
10.2.1.0/24 1
0.0.0.0/0 Suffix Port
0.0.0.2/8 2
0.0.0.3/8 3
TCAM
10.2.0.X
10.2.1.X
X.X.X.2
X.X.X.3
Encoder
Prefix Next Hop Port
00 10.2.0.1 0
01 10.2.1.1 1
10 10.4.1.1 2
11 10.4.1.2 3

17. Not Perfect
(k/2)2
k/2
k/2
k

18. Fat-Tree Status

19. Incast
• RTO = SRTT + (4 X RTTVAR)

20. Behavior
(from Phanishayee et al, “Measurement and Analysis of TCP Throughput Collapse in Cluster-based Storage
Systems”)

21. RFC 6298
(2.4) Whenever RTO is computed, if it is less than 1
second, then the RTO SHOULD be rounded up to 1 second.
- in practice, often 200ms
RFC 2581
The delayed ACK algorithm specified in [Bra89] SHOULD be
used by a TCP receiver. When used, a TCP receiver MUST NOT
excessively delay acknowledgments. Specifically, an ACK SHOULD
be generated for at least every second full-sized segment, and
MUST be generated within 500 ms of the arrival of the first
unacknowledged packet.
- in practice, often 40ms

22. Solutions
• Proposal 1: Adjust RTO (Vasudevan et al.)
• Proposal 2: DCTCP (Alizadeh et al.)

23. RTT

24. RTT 2

25. RTO
• Make RTOmin 200μs
• Timeout = (RTO + (rand(0.5) x RTO))

26. Improvement

27. Wide Area

28. DCTCP
• Three goals
• Low latency for short flows
• High burst tolerance (incast)
• High throughput for long flows
• Basic approach: keep switch queues short

29. Queue Length
• RTT measurements are noisy
• At high speeds, very small
• GigE: 10 packets is 120μs
• 10GigE: 10 paciets is 12μs
• Use ECN (explicit congestion notification)
• RFC 3168

30. Setting ECN
K
Set ECN bit

31. Monitoring α
• Per RTT, measure F, the fraction of packets
sent that had the ECN bit set
• DCTCP acks copy the ECN bit of the corresponding
data packets into ECN-Echo field
• Compute α, EWMA of F

32. Adjusting cwnd
• cwnd = cwnd x (1 - α/2)

33. DCTCP Caveat
“We stress that DCTCP is designed for the data
center environment. In this paper, we make no
claims about suitability of DCTCP for wide area
networks.”

34. Data Center Networks
• Very different than wide area Internet
• Tiny RTTs
• Different traffic patterns
• Single administrative domain
• Standards (e.g., IETF) much less important
• A lot of very novel network design