• Save
UDT
Upcoming SlideShare
Loading in...5
×
 

UDT

on

  • 1,755 views

This is the 2005 version of the UDT presentation

This is the 2005 version of the UDT presentation

Statistics

Views

Total Views
1,755
Views on SlideShare
1,738
Embed Views
17

Actions

Likes
3
Downloads
0
Comments
0

2 Embeds 17

http://www.linkedin.com 16
https://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

UDT UDT Presentation Transcript

  • BREAKING THE DATA TRANSFER BOTTLENECK Yunhong GU [email_address] Laboratory for Advanced Computing National Center for Data Mining University of Illinois at Chicago October 10, 2005 udt.sourceforge.net UDT: A High Performance Data Transport Protocol
  • Outline INTRODUCTION PROTOCOL DESIGN & IMPLEMENTATION CONGESTION CONTROL PERFORMANCE EVALUATION COMPOSABLE UDT CONCLUSIONS
  • >> INTRODUCTION PROTOCOL DESIGN & IMPLEMENTATION CONGESTION CONTROL PERFORMANCE EVALUATION COMPOSABLE UDT CONCLUSIONS
  • Motivations
    • The widespread use of high-speed networks (1Gb/s, 10Gb/s, etc.) has enabled many new distributed data intensive applications
      • Inexpensive fibers and advanced optical networking technologies (e.g., DWDM - Dense Wavelength Division Multiplexing)
      • 10Gb/s is common in high speed network testbeds, 40 Gb/s is emerging
    • Large volumetric datasets
      • Satellite weather data
      • Astronomy observation
      • Network monitoring
    • The Internet transport protocol (TCP) does NOT scale well as network bandwidth-delay product (BDP) increases
    • New transport protocol is needed!
  • Data Transport Protocol
    • Functionalities
      • Streaming, messaging
      • Reliability
      • Timeliness
      • Unicast vs. multicast
    • Congestion control
      • Efficiency
      • Fairness
      • Convergence
      • Distributedness
    Physical Layer Applications Data link Layer Network Layer Transport Layer
  • TCP
    • Reliable, data streaming, unicast
    • Congestion control
      • Increase congestion window size ( cwnd ) one full sized packet per RTT
      • Halve the cwnd per loss event
    • Poor efficiency in high bandwidth-delay product networks
    • Bias on flows with larger RTT
    ½ Bandwidth * RTT
  • TCP Throughput (Mb/s) Throughput (Mb/s) Packet Loss Round Trip Time (ms) 0.01% 0.05% 0.1% 0.1% 0.5% 1000 800 600 400 200 1 10 100 200 400 1000 800 600 400 200 LAN US-EU US-ASIA US
  • Related Work
    • TCP variants
      • HighSpeed, Scalable, BiC, FAST, H-TCP, L-TCP
    • Parallel TCP
      • PSockets, GridFTP
    • Rate-based reliable UDP
      • RBUDP, Tsunami, FOBS, FRTP (based on SABUL), Hurricane (based on UDT)
    • XCP
    • SABUL
  • Problems of Existing Work
    • Hard to deploy
      • TCP variants and XCP
      • Need modifications in OS kernel and/or routers
    • Cannot be used in shared networks
      • Most reliable UDP-based protocols
    • Poor fairness
      • Intra-protocol fairness
      • RTT fairness
    • Manual parameter tuning
  • A New Protocol Throughput (Mb/s) Throughput (Mb/s) Packet Loss Round Trip Time (ms) 0.01% 0.05% 0.1% 0.1% 0.5% 1000 800 600 400 200 1 10 100 200 400 1000 800 600 400 200 LAN US-EU US-ASIA US
  • UDT (UDP-based Data Transfer Protocol)
    • Application level, UDP-based
    • Similar functionalities to TCP
      • Connection-oriented reliable duplex unicast data streaming
    • New protocol design and implementation
    • New congestion control algorithm
    • Configurable congestion control framework
  • Objective & Non-objective
    • Objective
      • For distributed data intensive applications in high speed networks
      • A small number of flows share the abundant bandwidth
      • Efficient, fair, and friendly
      • Configurable
      • Easily deployable and usable
    • Non-objective
      • Replace TCP on the Internet
  • UDT Project
    • Open source (udt.sourceforge.net)
    • Design and implement the UDT protocol
    • Design the UDT congestion control algorithm
    • Evaluate experimentally the performance of UDT
    • Design and implement a configurable protocol framework based on UDT (Composable UDT)
  • >> PROTOCOL DESIGN & IMPLEMENTATION INTRODUCTION CONGESTION CONTROL PERFORMANCE EVALUATION COMPOSABLE UDT CONCLUSIONS
  • UDT Overview
    • Two orthogonal elements
      • The UDT protocol
      • The UDT congestion control algorithm
    • Protocol design & implementation
      • Functionality
      • Efficiency
    • Congestion control algorithm
      • Efficiency, fairness, friendliness, and stability
  • UDT Overview UDP Socket API Applications TCP Socket API Applications Applications UDT UDT Socket
  • Functionality
    • Reliability
      • Packet-based sequencing
      • Acknowledgment and loss report from receiver
      • ACK sub-sequencing
      • Retransmission (based on loss report and timeout)
    • Streaming
      • Buffer/memory management
    • Connection maintenance
      • Handshake, keep-alive message, teardown message
    • Duplex
      • Each UDT instance contains both a sender and a receiver
  • Protocol Architecture UDP Channel Sender Sender Receiver  Sender Receiver  UDP Seq. No TS Payload ACK Seq. No NAK Loss List A B
  • Software Architecture CC API UDP Channel Sender Receiver Sender's Buffer Receiver's Buffer Sender's Loss List Receiver's Loss List Listener
  • Efficiency Consideration
    • Less packets
      • Timer-based acknowledging
    • Less CPU time
      • Reduce per packet processing time
      • Reduce memory copy
      • Reduce loss list processing time
      • Light ACK vs. regular ACK
    • Parallel processing
      • Threading architecture
    • Less burst in processing
      • Evenly distribute the processing time
  • Application Programming Interface (API)
    • Socket API
    • New functionalities
      • sendfile/recvfile
      • Overlapped IO support
    • Transparent to existing applications
      • Recompilation needed
      • Certain limitations exist
    • XIO support (in Globus Toolkit 4.0)
    • Wrapper for other programming languages
      • Java, Python
  • >> CONGESTION CONTROL INTRODUCTION PROTOCOL DESIGN & IMPLEMENTATION PERFORMANCE EVALUATION COMPOSABLE UDT CONCLUSIONS
  • Overview
    • Congestion control vs. flow control
      • Congestion control: effectively utilize the network bandwidth
      • Flow control: prevent the receiver from being overwhelmed by incoming packets
    • Window-based vs. rate-based
      • Window-based: tune the maximum number of on-flight packets (TCP)
      • Rate-based: tune the inter-packet sending time (UDT)
    • AIMD: additive increases multiplicative decreases
    • Feedback
      • Packet loss (Most TCP variants, UDT)
      • Delay (Vegas, FAST)
  • AIMD with Decreasing Increases
    • AIMD
      • x = x +  (x) , for every constant interval (e.g., RTT)
      • x = (1 -  ) x , when there is a packet loss event
      • where x is the packet sending rate.
    • TCP
      •  (x)  1 , and the increase interval is RTT .
      •  = 0.5
    • AIMD with Decreasing Increase
      •  (x) is non-increasing, and lim x->+   (x) = 0.
  • AIMD with Decreasing Increases  ( x ) x AIMD (TCP NewReno) UDT HighSpeed TCP Scalable TCP
    • Increase
      •  (x) = f( B - x ) * c
      • where B is the link capacity (Bandwidth), c is a constant parameter
    • Constant rate control interval ( SYN ), irrelevant to RTT
      • SYN = 0.01 seconds
    • Decrease
      • Randomized decrease factor
      •  = 1 – (8/9) n
    UDT Control Algorithm  ( x ) x
  • The Increase Formula: an Example Bandwidth ( B ) = 10 Gbps, Packet size = 1500 bytes 0.00067 <0.1 9999.9+ 0.001 (0.1, 1] [9999, 9999.9) 0.01 (1, 10] [9990, 9999) 0.1 (10, 100] [9900, 9990) 1 (100, 1000] [9000, 9900) 10 (1000, 10000] [0, 9000) Increment (pkts/SYN) B - x (Mbps) x (Mbps)
  • Dealing with Packet Loss
    • Loss synchronization
      • Randomization method
    • Non-congestion loss
      • Do not decrease sending rate for the first packet loss
    • Packet reordering
    M=5, N=2 M=8, N=3
  • Bandwidth Estimation
    • Packet Pair
    • Filters
      • Cross traffic
      • Interrupt Coalescence
      • Robust to estimation errors
    • Randomized interval to send packet pair
    P2 P1 Packet Size / Space  Bottleneck Bandwidth P2 P1 P2 P1
  • >> PERFORMANCE EVALUATION INTRODUCTION PROTOCOL DESIGN & IMPLEMENTATION CONGESTION CONTROL COMPOSABLE UDT CONCLUSIONS
  • Performance Characteristics
    • Efficiency
      • Higher bandwidth utilization, less CPU usage
    • Intra-protocol fairness
      • Max-min fairness
      • Jain's fairness index
    • TCP friendliness
      • Bulk TCP flow vs Bulk UDT flow
      • Short-lived TCP flow (slow start phase) vs Bulk UDT flow
    • Stability (oscillations)
      • Stability index (standard deviation)
  • Evaluation Strategies
    • Simulations vs. experiments
      • NS2 network simulator, NCDM teraflow testbed
    • Setup
      • Network topology, bandwidth, distance, queuing, Link error rate, etc.
      • Concurrency (number of parallel flows)
    • Comparison (against TCP)
    • Real world applications
      • SDSS data transfer, high performance mining of streaming data, etc.
    • Independent evaluation
      • SLAC, JGN2, UvA, Unipmn (Italy), etc.
  • Efficiency, Fairness, & Stability Flow 1 Flow 2 Flow 3 Flow 4 0 100 200 300 400 500 600 700 Time (sec) 206.220.241.16 206.220.241.15 206.220.241.14 206.220.241.13 145.146.98.81 145.146.98.80 145.146.98.79 145.146.98.78 1Gb/s bandwidth, 106 ms RTT, StarLight, Chicago SARA, Amsterdam
  • Efficiency, Fairness, & Stability 0.04 0.02 0.04 0.16 0.08 0.11 0.11 Stability 1 1 0.999 0.998 0.999 0.999 1 Fairness 885 904 918 830 923 912 902 Efficiency 197 Flow 4 307 202 302 Flow 3 452 310 216 308 446 Flow 2 885 452 301 215 313 466 902 Flow 1
  • TCP Friendliness
    • 500 1MB TCP flows vs. 0 – 10 bulk UDT flows
    • 1Gb/s between Chicago and Amsterdam
    0 1 2 3 4 5 6 7 8 9 10 20 30 40 50 60 70 80 Number of UDT flows TCP Throughput (Mb/s)
  • >> COMPOSABLE UDT INTRODUCTION PROTOCOL DESIGN & IMPLEMENTATION CONGESTION CONTROL PERFORMANCE EVALUATION CONCLUSIONS
  • Composable UDT - Objectives
    • Easy implementation and deployment of new control algorithms
    • Easy evaluation of new control algorithms
    • Application awareness support and dynamic configuration
  • Composable UDT - Methodologies
    • Packet sending control
      • Window-based, rate-based, and hybrid
    • Control event handling
      • onACK, onLoss, onTimeout, onPktSent, onPktRecved, etc.
    • Protocol parameters access
      • RTT, loss rate, RTO, etc.
    • Packet extension
      • User-defined control packets
  • Composable UDT - Evaluation
    • Simplicity
      • Can it be easily used?
    • Expressiveness
      • Can it be used to implement most control protocols?
    • Similarity
      • Can Composable UDT based implementations reproduce the performance of their native implementations?
    • Overhead
      • Will the overhead added by Composable UDT be too large?
  • Simplicity & Expressiveness
    • Eight event handlers, four protocol control functions, and one performance monitoring function.
    • Support a large variety of protocols
      • Reliable UDT blast
      • TCP and its variants (both loss and delay based)
      • Group transport protocols
  • Simplicity & Expressiveness CCC Base Congestion Control Class CTCP TCP NewReno CGTP Group Transport Protocol CUDPBlast Reliable UDP Blast CFAST FAST TCP CVegas TCP Vegas CScalable Scalable TCP CHS HighSpeed TCP CBiC BiC TCP CWestwood TCP Westwood 28 73 / +132-6 11 / +192-29 8 / +27-1 11 / +192-29 27 / +145-2 37 / +351-2
  • Similarity and Overhead
    • Similarity
      • How Composable UDT based implementations can simulate their native implementations
    • CTCP vs. Linux TCP
    • CPU usage
      • Sender: CTCP uses about 100% more times of CPU as Linux TCP
      • Receiver: CTCP uses about 20% more CPU than Linux TCP
    Flow # Throughput Fairness Stability TCP CTCP TCP CTCP TCP CTCP 1 112 122 1 1 0.517 0.415 2 191 208 0.997 0.999 0.476 0.426 4 322 323 0.949 0.999 0.484 0.492 8 378 422 0.971 0.999 0.633 0.550 16 672 642 0.958 0.985 0.502 0.482 32 877 799 0.988 0.997 0.491 0.470 64 921 716 0.994 0.996 0.569 0.529
  • >> CONCLUSIONS INTRODUCTION PROTOCOL DESIGN & IMPLEMENTATION CONGESTION CONTROL PERFORMANCE EVALUATION COMPOSABLE UDT
  • Contributions
    • A high performance data transport protocol and associated implementation
      • The UDT protocol
      • Open source UDT library ( udt.sourceforge.net )
      • User includes ANL, ORNL, PNNL, etc.
    • An efficient and fair congestion control algorithm
      • DAIMD & the UDT control algorithm
      • Packet loss handling techniques
      • Using bandwidth estimation technique in congestion control
    • A configurable transport protocol framework
      • Composable UDT
  • Publications
    • Papers on the UDT Protocol
      • Supporting Configurable Congestion Control in Data Transport Services , Yunhong Gu and Robert L. Grossman, SC 2005, Nov 12 - 18, Seattle, WA.
      • Optimizing UDP-based Protocol Implementation , Yunhong Gu and Robert L. Grossman, PFLDNet 2005, Lyon, France, Feb. 2005.
      • Experiences in Design and Implementation of a High Performance Transport Protocol , Yunhong Gu, Xinwei Hong, and Robert L. Grossman, SC 2004, Nov 6 - 12, Pittsburgh, PA.
      • An Analysis of AIMD Algorithms with Decreasing Increases , Yunhong Gu, Xinwei Hong and Robert L. Grossman, First Workshop on Networks for Grid Applications (Gridnets 2004), Oct. 29, San Jose, CA.
      • SABUL: A Transport Protocol for Grid Computing , Yunhong Gu and Robert L. Grossman, Journal of Grid Computing, 2003, Volume 1, Issue 4, pp. 377-386.
    • Internet Draft
      • UDT: A Transport Protocol for Data Intensive Applications , Yunhong Gu and Robert L. Grossman, draft-gg-udt-01.txt.
  • Publications
    • Papers on Data Transfer Service using UDT
      • Experimental Studies of Data Transport and Data Access of Earth Science Data over Networks with High Bandwidth Delay Products , Robert Grossman, Yunhong Gu, Dave Hanley, Xinwei Hong and Parthasarathy Krishnaswamy, Computer Networks, Volume 46, Issue 3, Oct. 2004, pp. 411-421.
      • Teraflows over Gigabit WANs with UDT , Robert Grossman, Yunhong Gu, Xinwei Hong, Antony Antony, Johan Blom, Freek Dijkstra, and Cees de Laat,, Journal of Future Computer Systems, Vol. 21, Issue 4, pp. 501-513, April 2005.
      • The Photonic TeraStream: Enabling Next Generation Applications Through Intelligent Optical Networking at iGrid 2002 , J. Mambretti, J. Weinberger, J. Chen, E. Bacon, F. Yeh, D. Lillethun, R. Grossman, Y. Gu, M. Mazzuco,, Journal of Future Computer Systems, Volume 19, Number 6, pages 897-908.
      • Experimental Studies Using Photonic Data Services at IGrid 2002 , R. Grossman, Y. Gu, D. Hanley, X. Hong, D. Lillethun, J. Levera, J. Mambretti, M. Mazzucco, and J. Weinberger, Journal of Future Computer Systems, 2003, Volume 19, Number 6, pages 945-955.
  • Publications
    • Papers on Applications using UDT
      • Open DMIX: High Performance Web Services for Distributed Data Mining , R. Grossman, Y. Gu, C. Gupta, D. Hanley, X. Hong, and P. Krishnaswamy, 7th International Workshop on High Performance and Distributed Mining, .
      • Open DMIX - Data Integration and Exploration Services for Data Grids , R. Grossman, Y. Gu, D. Hanley, X. Hong, and G. Rao, First International Workshop on Knowledge Grid and Grid Intelligence (KGGI 2003).
      • Global Access to Large Distributed Data Sets using Photonic Data Services , R. Grossman, Y. Gu, D. Hanley, X. Hong, D. Lillethun, J. Levera, J. Mambretti, M. Mazzucco, and J. Weinberger, 20th IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies (MSST 2003), Los Alamitos, CA.
      • Data Webs for Earth Science Data , Asvin Ananthanarayan, Rajiv Balachandran, Yunhong Gu, Robert Grossman, Xinwei Hong, Jorge Levera, Marco Mazzucco, Parallel Computing, Volume 29, 2003, pages 1363-1379.
  • Achievements
    • SC 2002 Bandwidth Challenge “Best Use of Emerging Network Infrastructure” Award
    • SC 2003 Bandwidth Challenge “Application Foundation” Award
    • SC 2004 Bandwidth Challenge “Best Replacement for FedEx / UDP Fairness” Award
    • SC 2005 ?
      • Nov. 12 – 18, Seattle WA
      • High Performance Mining of Streaming Data using UDT
    • iGrid 2005
      • Exploring and mining remote data at 10Gb/s
  • Vision
    • Short-term
      • A practical solution to the distributed data intensive applications in high BDP environments
    • Long-term
      • Evolve with new technologies (open source & open standard)
      • More functionalities and support for more use scenarios
      • Network research platform (e.g., fast prototyping and evaluation of new control algorithms)
  • The End
    • Thank You!
    • Yunhong Gu, October 10, 2005