UDT

2,224 views

Published on

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,224
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
40
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

UDT

  1. 1. BREAKING THE DATA TRANSFER BOTTLENECK Yunhong GU [email_address] National Center for Data Mining University of Illinois at Chicago October 10, 2005 Updated on August 8, 2009 udt.sourceforge.net UDT: A High Performance Data Transport Protocol
  2. 2. Outline INTRODUCTION PROTOCOL DESIGN & IMPLEMENTATION CONGESTION CONTROL PERFORMANCE EVALUATION COMPOSABLE UDT CONCLUSIONS
  3. 3. >> INTRODUCTION PROTOCOL DESIGN & IMPLEMENTATION CONGESTION CONTROL PERFORMANCE EVALUATION COMPOSABLE UDT CONCLUSIONS
  4. 4. Motivations <ul><li>The widespread use of high-speed networks (1Gb/s, 10Gb/s, etc.) has enabled many new distributed data intensive applications </li></ul><ul><ul><li>Inexpensive fibers and advanced optical networking technologies (e.g., DWDM - Dense Wavelength Division Multiplexing) </li></ul></ul><ul><ul><li>10Gb/s is common in high speed network testbeds, 40 Gb/s is emerging </li></ul></ul><ul><li>Large volumetric datasets </li></ul><ul><ul><li>Satellite weather data </li></ul></ul><ul><ul><li>Astronomy observation </li></ul></ul><ul><ul><li>Network monitoring </li></ul></ul><ul><li>The Internet transport protocol (TCP) does NOT scale well as network bandwidth-delay product (BDP) increases </li></ul><ul><li>New transport protocol is needed! </li></ul>
  5. 5. Data Transport Protocol <ul><li>Functionalities </li></ul><ul><ul><li>Streaming, messaging </li></ul></ul><ul><ul><li>Reliability </li></ul></ul><ul><ul><li>Timeliness </li></ul></ul><ul><ul><li>Unicast vs. multicast </li></ul></ul><ul><li>Congestion control </li></ul><ul><ul><li>Efficiency </li></ul></ul><ul><ul><li>Fairness </li></ul></ul><ul><ul><li>Convergence </li></ul></ul><ul><ul><li>Distributedness </li></ul></ul>Physical Layer Applications Data link Layer Network Layer Transport Layer
  6. 6. TCP <ul><li>Reliable, data streaming, unicast </li></ul><ul><li>Congestion control </li></ul><ul><ul><li>Increase congestion window size ( cwnd ) one full sized packet per RTT </li></ul></ul><ul><ul><li>Halve the cwnd per loss event </li></ul></ul><ul><li>Poor efficiency in high bandwidth-delay product networks </li></ul><ul><li>Bias on flows with larger RTT </li></ul>½ Bandwidth * RTT
  7. 7. TCP Throughput (Mb/s) Throughput (Mb/s) Packet Loss Round Trip Time (ms) 0.01% 0.05% 0.1% 0.1% 0.5% 1000 800 600 400 200 1 10 100 200 400 1000 800 600 400 200 LAN US-EU US-ASIA US
  8. 8. Related Work <ul><li>TCP variants </li></ul><ul><ul><li>HighSpeed, Scalable, BiC, FAST, H-TCP, L-TCP </li></ul></ul><ul><li>Parallel TCP </li></ul><ul><ul><li>PSockets, GridFTP </li></ul></ul><ul><li>Rate-based reliable UDP </li></ul><ul><ul><li>RBUDP, Tsunami, FOBS, FRTP (based on SABUL), Hurricane (based on UDT) </li></ul></ul><ul><li>XCP </li></ul><ul><li>SABUL </li></ul>
  9. 9. Problems of Existing Work <ul><li>Hard to deploy </li></ul><ul><ul><li>TCP variants and XCP </li></ul></ul><ul><ul><li>Need modifications in OS kernel and/or routers </li></ul></ul><ul><li>Cannot be used in shared networks </li></ul><ul><ul><li>Most reliable UDP-based protocols </li></ul></ul><ul><li>Poor fairness </li></ul><ul><ul><li>Intra-protocol fairness </li></ul></ul><ul><ul><li>RTT fairness </li></ul></ul><ul><li>Manual parameter tuning </li></ul>
  10. 10. A New Protocol Throughput (Mb/s) Throughput (Mb/s) Packet Loss Round Trip Time (ms) 0.01% 0.05% 0.1% 0.1% 0.5% 1000 800 600 400 200 1 10 100 200 400 1000 800 600 400 200 LAN US-EU US-ASIA US
  11. 11. UDT (UDP-based Data Transfer Protocol) <ul><li>Application level, UDP-based </li></ul><ul><li>Similar functionalities to TCP </li></ul><ul><ul><li>Connection-oriented reliable duplex unicast data streaming </li></ul></ul><ul><li>New protocol design and implementation </li></ul><ul><li>New congestion control algorithm </li></ul><ul><li>Configurable congestion control framework </li></ul>
  12. 12. Objective & Non-objective <ul><li>Objective </li></ul><ul><ul><li>For distributed data intensive applications in high speed networks </li></ul></ul><ul><ul><li>A small number of flows share the abundant bandwidth </li></ul></ul><ul><ul><li>Efficient, fair, and friendly </li></ul></ul><ul><ul><li>Configurable </li></ul></ul><ul><ul><li>Easily deployable and usable </li></ul></ul><ul><li>Non-objective </li></ul><ul><ul><li>Replace TCP on the Internet </li></ul></ul>
  13. 13. UDT Project <ul><li>Open source (udt.sourceforge.net) </li></ul><ul><li>Design and implement the UDT protocol </li></ul><ul><li>Design the UDT congestion control algorithm </li></ul><ul><li>Evaluate experimentally the performance of UDT </li></ul><ul><li>Design and implement a configurable protocol framework based on UDT (Composable UDT) </li></ul>
  14. 14. >> PROTOCOL DESIGN & IMPLEMENTATION INTRODUCTION CONGESTION CONTROL PERFORMANCE EVALUATION COMPOSABLE UDT CONCLUSIONS
  15. 15. UDT Overview <ul><li>Two orthogonal elements </li></ul><ul><ul><li>The UDT protocol </li></ul></ul><ul><ul><li>The UDT congestion control algorithm </li></ul></ul><ul><li>Protocol design & implementation </li></ul><ul><ul><li>Functionality </li></ul></ul><ul><ul><li>Efficiency </li></ul></ul><ul><li>Congestion control algorithm </li></ul><ul><ul><li>Efficiency, fairness, friendliness, and stability </li></ul></ul>
  16. 16. UDT Overview UDP Socket API Applications TCP Socket API Applications Applications UDT UDT Socket
  17. 17. Functionality <ul><li>Reliability </li></ul><ul><ul><li>Packet-based sequencing </li></ul></ul><ul><ul><li>Acknowledgment and loss report from receiver </li></ul></ul><ul><ul><li>ACK sub-sequencing </li></ul></ul><ul><ul><li>Retransmission (based on loss report and timeout) </li></ul></ul><ul><li>Streaming and Messaging </li></ul><ul><ul><li>Buffer/memory management </li></ul></ul><ul><li>Connection maintenance </li></ul><ul><ul><li>Handshake, keep-alive message, teardown message </li></ul></ul><ul><li>Duplex </li></ul><ul><ul><li>Each UDT instance contains both a sender and a receiver </li></ul></ul>
  18. 18. Protocol Architecture UDP Channel Sender Sender Receiver  Sender Receiver  UDP Seq. No TS Payload ACK Seq. No NAK Loss List A B
  19. 19. Software Architecture CC API UDP Channel Sender Receiver Sender's Buffer Receiver's Buffer Sender's Loss List Receiver's Loss List Listener
  20. 20. Efficiency Consideration <ul><li>Less packets </li></ul><ul><ul><li>Timer-based acknowledging </li></ul></ul><ul><li>Less CPU time </li></ul><ul><ul><li>Reduce per packet processing time </li></ul></ul><ul><ul><li>Reduce memory copy </li></ul></ul><ul><ul><li>Reduce loss list processing time </li></ul></ul><ul><ul><li>Light ACK vs. regular ACK </li></ul></ul><ul><li>Parallel processing </li></ul><ul><ul><li>Threading architecture </li></ul></ul><ul><li>Less burst in processing </li></ul><ul><ul><li>Evenly distribute the processing time </li></ul></ul>
  21. 21. Application Programming Interface (API) <ul><li>Socket API </li></ul><ul><li>New API </li></ul><ul><ul><li>sendfile/recvfile: efficient file transfer </li></ul></ul><ul><ul><li>sendmsg/recvmsg: messaging with partial reliability </li></ul></ul><ul><ul><li>selectEx: a more efficient version of “select” </li></ul></ul><ul><li>Rendezvous Connect </li></ul><ul><ul><li>Firewall traversing </li></ul></ul>
  22. 22. >> CONGESTION CONTROL INTRODUCTION PROTOCOL DESIGN & IMPLEMENTATION PERFORMANCE EVALUATION COMPOSABLE UDT CONCLUSIONS
  23. 23. Overview <ul><li>Congestion control vs. flow control </li></ul><ul><ul><li>Congestion control: effectively utilize the network bandwidth </li></ul></ul><ul><ul><li>Flow control: prevent the receiver from being overwhelmed by incoming packets </li></ul></ul><ul><li>Window-based vs. rate-based </li></ul><ul><ul><li>Window-based: tune the maximum number of on-flight packets (TCP) </li></ul></ul><ul><ul><li>Rate-based: tune the inter-packet sending time (UDT) </li></ul></ul><ul><li>AIMD: additive increases multiplicative decreases </li></ul><ul><li>Feedback </li></ul><ul><ul><li>Packet loss (Most TCP variants, UDT) </li></ul></ul><ul><ul><li>Delay (Vegas, FAST) </li></ul></ul>
  24. 24. AIMD with Decreasing Increases <ul><li>AIMD </li></ul><ul><ul><li>x = x +  (x) , for every constant interval (e.g., RTT) </li></ul></ul><ul><ul><li>x = (1 -  ) x , when there is a packet loss event </li></ul></ul><ul><ul><li>where x is the packet sending rate. </li></ul></ul><ul><li>TCP </li></ul><ul><ul><li> (x)  1 , and the increase interval is RTT . </li></ul></ul><ul><ul><li> = 0.5 </li></ul></ul><ul><li>AIMD with Decreasing Increase </li></ul><ul><ul><li> (x) is non-increasing, and lim x->+   (x) = 0. </li></ul></ul>
  25. 25. AIMD with Decreasing Increases  ( x ) x AIMD (TCP NewReno) UDT HighSpeed TCP Scalable TCP
  26. 26. <ul><li>Increase </li></ul><ul><ul><li> (x) = f( B - x ) * c </li></ul></ul><ul><ul><li>where B is the link capacity (Bandwidth), c is a constant parameter </li></ul></ul><ul><li>Constant rate control interval ( SYN ), irrelevant to RTT </li></ul><ul><ul><li>SYN = 0.01 seconds </li></ul></ul><ul><li>Decrease </li></ul><ul><ul><li>Randomized decrease factor </li></ul></ul><ul><ul><li> = 1 – (8/9) n </li></ul></ul>UDT Control Algorithm  ( x ) x
  27. 27. The Increase Formula: an Example Bandwidth ( B ) = 10 Gbps, Packet size = 1500 bytes 0.00067 <0.1 9999.9+ 0.001 (0.1, 1] [9999, 9999.9) 0.01 (1, 10] [9990, 9999) 0.1 (10, 100] [9900, 9990) 1 (100, 1000] [9000, 9900) 10 (1000, 10000] [0, 9000) Increment (pkts/SYN) B - x (Mbps) x (Mbps)
  28. 28. Dealing with Packet Loss <ul><li>Loss synchronization </li></ul><ul><ul><li>Randomization method </li></ul></ul><ul><li>Non-congestion loss </li></ul><ul><ul><li>Do not decrease sending rate for the first packet loss </li></ul></ul><ul><li>Packet reordering </li></ul>M=5, N=2 M=8, N=3
  29. 29. Bandwidth Estimation <ul><li>Packet Pair </li></ul><ul><li>Filters </li></ul><ul><ul><li>Cross traffic </li></ul></ul><ul><ul><li>Interrupt Coalescence </li></ul></ul><ul><ul><li>Robust to estimation errors </li></ul></ul><ul><li>Randomized interval to send packet pair </li></ul>P2 P1 Packet Size / Space  Bottleneck Bandwidth P2 P1 P2 P1
  30. 30. >> PERFORMANCE EVALUATION INTRODUCTION PROTOCOL DESIGN & IMPLEMENTATION CONGESTION CONTROL COMPOSABLE UDT CONCLUSIONS
  31. 31. Performance Characteristics <ul><li>Efficiency </li></ul><ul><ul><li>Higher bandwidth utilization, less CPU usage </li></ul></ul><ul><li>Intra-protocol fairness </li></ul><ul><ul><li>Max-min fairness </li></ul></ul><ul><ul><li>Jain's fairness index </li></ul></ul><ul><li>TCP friendliness </li></ul><ul><ul><li>Bulk TCP flow vs Bulk UDT flow </li></ul></ul><ul><ul><li>Short-lived TCP flow (slow start phase) vs Bulk UDT flow </li></ul></ul><ul><li>Stability (oscillations) </li></ul><ul><ul><li>Stability index (standard deviation) </li></ul></ul>
  32. 32. Evaluation Strategies <ul><li>Simulations vs. experiments </li></ul><ul><ul><li>NS2 network simulator, NCDM teraflow testbed </li></ul></ul><ul><li>Setup </li></ul><ul><ul><li>Network topology, bandwidth, distance, queuing, Link error rate, etc. </li></ul></ul><ul><ul><li>Concurrency (number of parallel flows) </li></ul></ul><ul><li>Comparison (against TCP) </li></ul><ul><li>Real world applications </li></ul><ul><ul><li>SDSS data transfer, high performance mining of streaming data, etc. </li></ul></ul><ul><li>Independent evaluation </li></ul><ul><ul><li>SLAC, JGN2, UvA, Unipmn (Italy), etc. </li></ul></ul>
  33. 33. Efficiency, Fairness, & Stability Flow 1 Flow 2 Flow 3 Flow 4 0 100 200 300 400 500 600 700 Time (sec) 206.220.241.16 206.220.241.15 206.220.241.14 206.220.241.13 145.146.98.81 145.146.98.80 145.146.98.79 145.146.98.78 1Gb/s bandwidth, 106 ms RTT, StarLight, Chicago SARA, Amsterdam
  34. 34. Efficiency, Fairness, & Stability 0.04 0.02 0.04 0.16 0.08 0.11 0.11 Stability 1 1 0.999 0.998 0.999 0.999 1 Fairness 885 904 918 830 923 912 902 Efficiency 197 Flow 4 307 202 302 Flow 3 452 310 216 308 446 Flow 2 885 452 301 215 313 466 902 Flow 1
  35. 35. TCP Friendliness <ul><li>500 1MB TCP flows vs. 0 – 10 bulk UDT flows </li></ul><ul><li>1Gb/s between Chicago and Amsterdam </li></ul>0 1 2 3 4 5 6 7 8 9 10 20 30 40 50 60 70 80 Number of UDT flows TCP Throughput (Mb/s)
  36. 36. >> COMPOSABLE UDT INTRODUCTION PROTOCOL DESIGN & IMPLEMENTATION CONGESTION CONTROL PERFORMANCE EVALUATION CONCLUSIONS
  37. 37. Composable UDT - Objectives <ul><li>Easy implementation and deployment of new control algorithms </li></ul><ul><li>Easy evaluation of new control algorithms </li></ul><ul><li>Application awareness support and dynamic configuration </li></ul>
  38. 38. Composable UDT - Methodologies <ul><li>Packet sending control </li></ul><ul><ul><li>Window-based, rate-based, and hybrid </li></ul></ul><ul><li>Control event handling </li></ul><ul><ul><li>onACK, onLoss, onTimeout, onPktSent, onPktRecved, etc. </li></ul></ul><ul><li>Protocol parameters access </li></ul><ul><ul><li>RTT, loss rate, RTO, etc. </li></ul></ul><ul><li>Packet extension </li></ul><ul><ul><li>User-defined control packets </li></ul></ul>
  39. 39. Composable UDT - Evaluation <ul><li>Simplicity </li></ul><ul><ul><li>Can it be easily used? </li></ul></ul><ul><li>Expressiveness </li></ul><ul><ul><li>Can it be used to implement most control protocols? </li></ul></ul><ul><li>Similarity </li></ul><ul><ul><li>Can Composable UDT based implementations reproduce the performance of their native implementations? </li></ul></ul><ul><li>Overhead </li></ul><ul><ul><li>Will the overhead added by Composable UDT be too large? </li></ul></ul>
  40. 40. Simplicity & Expressiveness <ul><li>Eight event handlers, four protocol control functions, and one performance monitoring function. </li></ul><ul><li>Support a large variety of protocols </li></ul><ul><ul><li>Reliable UDT blast </li></ul></ul><ul><ul><li>TCP and its variants (both loss and delay based) </li></ul></ul><ul><ul><li>Group transport protocols </li></ul></ul>
  41. 41. Simplicity & Expressiveness CCC Base Congestion Control Class CTCP TCP NewReno CGTP Group Transport Protocol CUDPBlast Reliable UDP Blast CFAST FAST TCP CVegas TCP Vegas CScalable Scalable TCP CHS HighSpeed TCP CBiC BiC TCP CWestwood TCP Westwood 28 73 / +132-6 11 / +192-29 8 / +27-1 11 / +192-29 27 / +145-2 37 / +351-2
  42. 42. Similarity and Overhead <ul><li>Similarity </li></ul><ul><ul><li>How Composable UDT based implementations can simulate their native implementations </li></ul></ul><ul><li>CTCP vs. Linux TCP </li></ul><ul><li>CPU usage </li></ul><ul><ul><li>Sender: CTCP uses about 100% more times of CPU as Linux TCP </li></ul></ul><ul><ul><li>Receiver: CTCP uses about 20% more CPU than Linux TCP </li></ul></ul>Flow # Throughput Fairness Stability TCP CTCP TCP CTCP TCP CTCP 1 112 122 1 1 0.517 0.415 2 191 208 0.997 0.999 0.476 0.426 4 322 323 0.949 0.999 0.484 0.492 8 378 422 0.971 0.999 0.633 0.550 16 672 642 0.958 0.985 0.502 0.482 32 877 799 0.988 0.997 0.491 0.470 64 921 716 0.994 0.996 0.569 0.529
  43. 43. >> CONCLUSIONS INTRODUCTION PROTOCOL DESIGN & IMPLEMENTATION CONGESTION CONTROL PERFORMANCE EVALUATION COMPOSABLE UDT
  44. 44. Contributions <ul><li>A high performance data transport protocol and associated implementation </li></ul><ul><ul><li>The UDT protocol </li></ul></ul><ul><ul><li>Open source UDT library ( udt.sourceforge.net ) </li></ul></ul><ul><ul><li>User includes research institutes and industries </li></ul></ul><ul><li>An efficient and fair congestion control algorithm </li></ul><ul><ul><li>DAIMD & the UDT control algorithm </li></ul></ul><ul><ul><li>Packet loss handling techniques </li></ul></ul><ul><ul><li>Using bandwidth estimation technique in congestion control </li></ul></ul><ul><li>A configurable transport protocol framework </li></ul><ul><ul><li>Composable UDT </li></ul></ul>
  45. 45. Selected Publications <ul><li>Papers on the UDT Protocol </li></ul><ul><ul><li>UDT: UDP-based Data Transfer for High-Speed Wide Area Networks, Yunhong Gu and Robert L. Grossman, Computer Networks (Elsevier). Volume 51, Issue 7. May 2007. </li></ul></ul><ul><ul><li>Supporting Configurable Congestion Control in Data Transport Services , Yunhong Gu and Robert L. Grossman, SC 2005, Nov 12 - 18, Seattle, WA. </li></ul></ul><ul><ul><li>Experiences in Design and Implementation of a High Performance Transport Protocol , Yunhong Gu, Xinwei Hong, and Robert L. Grossman, SC 2004, Nov 6 - 12, Pittsburgh, PA. </li></ul></ul><ul><ul><li>An Analysis of AIMD Algorithms with Decreasing Increases , Yunhong Gu, Xinwei Hong and Robert L. Grossman, First Workshop on Networks for Grid Applications (Gridnets 2004), Oct. 29, San Jose, CA. </li></ul></ul><ul><li>Internet Draft </li></ul><ul><ul><li>UDT: A Transport Protocol for Data Intensive Applications , Yunhong Gu and Robert L. Grossman, draft-gg-udt-02.txt. </li></ul></ul>
  46. 46. Commercialization <ul><li>Baidu Hi Messenger </li></ul><ul><li>Maidsafe </li></ul><ul><li>Movie2Me by broadcasting center europe </li></ul><ul><li>NiFTy TV </li></ul><ul><li>Sterling File Accelerator (SFA) by Sterling Commerce </li></ul><ul><li>Tideworks </li></ul><ul><li>PowerFolder </li></ul><ul><li>GridFTP </li></ul><ul><li>etc. </li></ul>
  47. 47. Support and Services <ul><li>Online Forum </li></ul><ul><ul><li>https://sourceforge.net/forum/?group_id=115059 </li></ul></ul><ul><ul><li>Support both English and Chinese languages </li></ul></ul><ul><li>Consulting service </li></ul><ul><ul><li>Dedicated service to customers’ projects </li></ul></ul><ul><ul><li>Provided by UDT developers </li></ul></ul>
  48. 48. Achievements <ul><li>SC 2002 Bandwidth Challenge “Best Use of Emerging Network Infrastructure” Award </li></ul><ul><li>SC 2003 Bandwidth Challenge “Application Foundation” Award </li></ul><ul><li>SC 2004 Bandwidth Challenge “Best Replacement for FedEx / UDP Fairness” Award </li></ul><ul><li>SC 2006 Bandwidth Challenge Winner </li></ul><ul><li>SC 2008 Bandwidth Challenge Winner </li></ul>
  49. 49. Vision <ul><li>Short-term </li></ul><ul><ul><li>A practical solution to the distributed data intensive applications in high BDP environments </li></ul></ul><ul><li>Long-term </li></ul><ul><ul><li>Evolve with new technologies (open source & open standard) </li></ul></ul><ul><ul><li>More functionalities and support for more use scenarios </li></ul></ul><ul><ul><li>Network research platform (e.g., fast prototyping and evaluation of new control algorithms) </li></ul></ul>
  50. 50. The End <ul><li>Thank You! </li></ul><ul><li>Yunhong Gu, October 10, 2005 </li></ul><ul><li>Updated on August 8, 2009 </li></ul>

×