SlideShare a Scribd company logo
Application-Level Optimization of Big Data Transfers through Pipelining,
Parallelism and Concurrency
Abstract:
In end-to-end data transfers, there are several factors affecting the data transfer
throughput, such as the network characteristics (e.g., network bandwidth, round-
trip-time, background traffic); end-system characteristics (e.g., NIC capacity,
number of CPU cores and their clock rate, number of disk drives and their I/O
rate); and the dataset characteristics (e.g., average file size, dataset size, file size
distribution). Optimization of big data transfers over inter-cloud and intra-
cloud networks is a challenging task that requires joint-consideration of all of
these parameters. This optimization task becomes even more challenging when
transferring datasets comprised of heterogeneous file sizes (i.e., large files and
small files mixed). Previous work in this area only focuses on the end-system and
network characteristics however does not provide models regarding the dataset
characteristics. In this study, we analyze the effects of the three most important
transfer parameters that are used to enhance data transfer throughput:
pipelining,parallelism and concurrency. We provide models and guidelines to set
the best values for these parameters and present two different transfer
optimization algorithms that use the models developed. The tests conducted over
high-speed networking and cloud testbeds show that our algorithms outperform
the most popular data transfer tools like Globus Online and UDT in majority of the
cases.

More Related Content

Viewers also liked

PMSCS 657_Parallel and Distributed processing
PMSCS 657_Parallel and Distributed processingPMSCS 657_Parallel and Distributed processing
PMSCS 657_Parallel and Distributed processing
Md. Mashiur Rahman
 
Computer Architecture: A quantitative approach - Cap4 - Section 8
Computer Architecture: A quantitative approach - Cap4 - Section 8Computer Architecture: A quantitative approach - Cap4 - Section 8
Computer Architecture: A quantitative approach - Cap4 - Section 8
Marcelo Arbore
 
DATA SCIENCE Lesson 2 Parallelism Computing Data Processing Performance Measu...
DATA SCIENCE Lesson 2 Parallelism Computing Data Processing Performance Measu...DATA SCIENCE Lesson 2 Parallelism Computing Data Processing Performance Measu...
DATA SCIENCE Lesson 2 Parallelism Computing Data Processing Performance Measu...
Jean-Antoine Moreau
 
Instruction Level Parallelism Compiler optimization Techniques Anna Universit...
Instruction Level Parallelism Compiler optimization Techniques Anna Universit...Instruction Level Parallelism Compiler optimization Techniques Anna Universit...
Instruction Level Parallelism Compiler optimization Techniques Anna Universit...
Dr.K. Thirunadana Sikamani
 
INSTRUCTION LEVEL PARALLALISM
INSTRUCTION LEVEL PARALLALISMINSTRUCTION LEVEL PARALLALISM
INSTRUCTION LEVEL PARALLALISM
Kamran Ashraf
 
Instruction Level Parallelism (ILP) Limitations
Instruction Level Parallelism (ILP) LimitationsInstruction Level Parallelism (ILP) Limitations
Instruction Level Parallelism (ILP) Limitations
Jose Pinilla
 
Parallel Computing
Parallel ComputingParallel Computing
Parallel Computing
Ameya Waghmare
 

Viewers also liked (7)

PMSCS 657_Parallel and Distributed processing
PMSCS 657_Parallel and Distributed processingPMSCS 657_Parallel and Distributed processing
PMSCS 657_Parallel and Distributed processing
 
Computer Architecture: A quantitative approach - Cap4 - Section 8
Computer Architecture: A quantitative approach - Cap4 - Section 8Computer Architecture: A quantitative approach - Cap4 - Section 8
Computer Architecture: A quantitative approach - Cap4 - Section 8
 
DATA SCIENCE Lesson 2 Parallelism Computing Data Processing Performance Measu...
DATA SCIENCE Lesson 2 Parallelism Computing Data Processing Performance Measu...DATA SCIENCE Lesson 2 Parallelism Computing Data Processing Performance Measu...
DATA SCIENCE Lesson 2 Parallelism Computing Data Processing Performance Measu...
 
Instruction Level Parallelism Compiler optimization Techniques Anna Universit...
Instruction Level Parallelism Compiler optimization Techniques Anna Universit...Instruction Level Parallelism Compiler optimization Techniques Anna Universit...
Instruction Level Parallelism Compiler optimization Techniques Anna Universit...
 
INSTRUCTION LEVEL PARALLALISM
INSTRUCTION LEVEL PARALLALISMINSTRUCTION LEVEL PARALLALISM
INSTRUCTION LEVEL PARALLALISM
 
Instruction Level Parallelism (ILP) Limitations
Instruction Level Parallelism (ILP) LimitationsInstruction Level Parallelism (ILP) Limitations
Instruction Level Parallelism (ILP) Limitations
 
Parallel Computing
Parallel ComputingParallel Computing
Parallel Computing
 

Similar to Application level optimization of big data transfers through pipelining, parallelism and concurrency

In network aggregation techniques for wireless sensor networks - a survey
In network aggregation techniques for wireless sensor networks - a surveyIn network aggregation techniques for wireless sensor networks - a survey
In network aggregation techniques for wireless sensor networks - a survey
Gungi Achi
 
Traffic-aware adaptive server load balancing for softwaredefined networks
Traffic-aware adaptive server load balancing for softwaredefined networks Traffic-aware adaptive server load balancing for softwaredefined networks
Traffic-aware adaptive server load balancing for softwaredefined networks
IJECEIAES
 
Transfer reliability and congestion control strategies in opportunistic netwo...
Transfer reliability and congestion control strategies in opportunistic netwo...Transfer reliability and congestion control strategies in opportunistic netwo...
Transfer reliability and congestion control strategies in opportunistic netwo...
IEEEFINALYEARPROJECTS
 
JAVA 2013 IEEE NETWORKING PROJECT Transfer reliability and congestion control...
JAVA 2013 IEEE NETWORKING PROJECT Transfer reliability and congestion control...JAVA 2013 IEEE NETWORKING PROJECT Transfer reliability and congestion control...
JAVA 2013 IEEE NETWORKING PROJECT Transfer reliability and congestion control...
IEEEGLOBALSOFTTECHNOLOGIES
 
Enhancement of Single Moving Average Time Series Model Using Rough k-Means fo...
Enhancement of Single Moving Average Time Series Model Using Rough k-Means fo...Enhancement of Single Moving Average Time Series Model Using Rough k-Means fo...
Enhancement of Single Moving Average Time Series Model Using Rough k-Means fo...
IJERA Editor
 
Web based-distributed-sesnzer-using-service-oriented-architecture
Web based-distributed-sesnzer-using-service-oriented-architectureWeb based-distributed-sesnzer-using-service-oriented-architecture
Web based-distributed-sesnzer-using-service-oriented-architecture
Aidah Izzah Huriyah
 
Optimal configuration of network
Optimal configuration of networkOptimal configuration of network
Optimal configuration of network
jpstudcorner
 
Survey on Synchronizing File Operations Along with Storage Scalable Mechanism
Survey on Synchronizing File Operations Along with Storage Scalable MechanismSurvey on Synchronizing File Operations Along with Storage Scalable Mechanism
Survey on Synchronizing File Operations Along with Storage Scalable Mechanism
IRJET Journal
 
A New Architecture for Group Replication in Data Grid
A New Architecture for Group Replication in Data GridA New Architecture for Group Replication in Data Grid
A New Architecture for Group Replication in Data Grid
Editor IJCATR
 
Internet data mining 2006
Internet data mining   2006Internet data mining   2006
Internet data mining 2006
raj_vij
 
Dynamic control of coding for progressive packet arrivals in dtns
Dynamic control of coding for progressive packet arrivals in dtnsDynamic control of coding for progressive packet arrivals in dtns
Dynamic control of coding for progressive packet arrivals in dtns
JPINFOTECH JAYAPRAKASH
 
2005-03-17 Air Quality Cluster TechTrack
2005-03-17 Air Quality Cluster TechTrack2005-03-17 Air Quality Cluster TechTrack
2005-03-17 Air Quality Cluster TechTrack
Rudolf Husar
 
Ws Stuff
Ws StuffWs Stuff
Ws Stuff
Rudolf Husar
 
Cross Layer- Performance Enhancement Architecture (CL-PEA) for MANET
Cross Layer- Performance Enhancement Architecture (CL-PEA) for MANETCross Layer- Performance Enhancement Architecture (CL-PEA) for MANET
Cross Layer- Performance Enhancement Architecture (CL-PEA) for MANET
ijcncs
 
Workload-Aware Data Management in Shared-Nothing Distributed OLTP Databases
Workload-Aware Data Management in Shared-Nothing Distributed OLTP DatabasesWorkload-Aware Data Management in Shared-Nothing Distributed OLTP Databases
Workload-Aware Data Management in Shared-Nothing Distributed OLTP Databases
Joarder Kamal
 
Network-aware Data Management for High Throughput Flows Akamai, Cambridge, ...
Network-aware Data Management for High Throughput Flows   Akamai, Cambridge, ...Network-aware Data Management for High Throughput Flows   Akamai, Cambridge, ...
Network-aware Data Management for High Throughput Flows Akamai, Cambridge, ...
balmanme
 
Green wsn optimization of energy use
Green wsn  optimization of energy useGreen wsn  optimization of energy use
Green wsn optimization of energy use
ijfcstjournal
 
GREEN WSN- OPTIMIZATION OF ENERGY USE THROUGH REDUCTION IN COMMUNICATION WORK...
GREEN WSN- OPTIMIZATION OF ENERGY USE THROUGH REDUCTION IN COMMUNICATION WORK...GREEN WSN- OPTIMIZATION OF ENERGY USE THROUGH REDUCTION IN COMMUNICATION WORK...
GREEN WSN- OPTIMIZATION OF ENERGY USE THROUGH REDUCTION IN COMMUNICATION WORK...
ijfcstjournal
 
50120130406035
5012013040603550120130406035
50120130406035
IAEME Publication
 
A Platform for Large-Scale Grid Data Service on Dynamic High-Performance Netw...
A Platform for Large-Scale Grid Data Service on Dynamic High-Performance Netw...A Platform for Large-Scale Grid Data Service on Dynamic High-Performance Netw...
A Platform for Large-Scale Grid Data Service on Dynamic High-Performance Netw...
Tal Lavian Ph.D.
 

Similar to Application level optimization of big data transfers through pipelining, parallelism and concurrency (20)

In network aggregation techniques for wireless sensor networks - a survey
In network aggregation techniques for wireless sensor networks - a surveyIn network aggregation techniques for wireless sensor networks - a survey
In network aggregation techniques for wireless sensor networks - a survey
 
Traffic-aware adaptive server load balancing for softwaredefined networks
Traffic-aware adaptive server load balancing for softwaredefined networks Traffic-aware adaptive server load balancing for softwaredefined networks
Traffic-aware adaptive server load balancing for softwaredefined networks
 
Transfer reliability and congestion control strategies in opportunistic netwo...
Transfer reliability and congestion control strategies in opportunistic netwo...Transfer reliability and congestion control strategies in opportunistic netwo...
Transfer reliability and congestion control strategies in opportunistic netwo...
 
JAVA 2013 IEEE NETWORKING PROJECT Transfer reliability and congestion control...
JAVA 2013 IEEE NETWORKING PROJECT Transfer reliability and congestion control...JAVA 2013 IEEE NETWORKING PROJECT Transfer reliability and congestion control...
JAVA 2013 IEEE NETWORKING PROJECT Transfer reliability and congestion control...
 
Enhancement of Single Moving Average Time Series Model Using Rough k-Means fo...
Enhancement of Single Moving Average Time Series Model Using Rough k-Means fo...Enhancement of Single Moving Average Time Series Model Using Rough k-Means fo...
Enhancement of Single Moving Average Time Series Model Using Rough k-Means fo...
 
Web based-distributed-sesnzer-using-service-oriented-architecture
Web based-distributed-sesnzer-using-service-oriented-architectureWeb based-distributed-sesnzer-using-service-oriented-architecture
Web based-distributed-sesnzer-using-service-oriented-architecture
 
Optimal configuration of network
Optimal configuration of networkOptimal configuration of network
Optimal configuration of network
 
Survey on Synchronizing File Operations Along with Storage Scalable Mechanism
Survey on Synchronizing File Operations Along with Storage Scalable MechanismSurvey on Synchronizing File Operations Along with Storage Scalable Mechanism
Survey on Synchronizing File Operations Along with Storage Scalable Mechanism
 
A New Architecture for Group Replication in Data Grid
A New Architecture for Group Replication in Data GridA New Architecture for Group Replication in Data Grid
A New Architecture for Group Replication in Data Grid
 
Internet data mining 2006
Internet data mining   2006Internet data mining   2006
Internet data mining 2006
 
Dynamic control of coding for progressive packet arrivals in dtns
Dynamic control of coding for progressive packet arrivals in dtnsDynamic control of coding for progressive packet arrivals in dtns
Dynamic control of coding for progressive packet arrivals in dtns
 
2005-03-17 Air Quality Cluster TechTrack
2005-03-17 Air Quality Cluster TechTrack2005-03-17 Air Quality Cluster TechTrack
2005-03-17 Air Quality Cluster TechTrack
 
Ws Stuff
Ws StuffWs Stuff
Ws Stuff
 
Cross Layer- Performance Enhancement Architecture (CL-PEA) for MANET
Cross Layer- Performance Enhancement Architecture (CL-PEA) for MANETCross Layer- Performance Enhancement Architecture (CL-PEA) for MANET
Cross Layer- Performance Enhancement Architecture (CL-PEA) for MANET
 
Workload-Aware Data Management in Shared-Nothing Distributed OLTP Databases
Workload-Aware Data Management in Shared-Nothing Distributed OLTP DatabasesWorkload-Aware Data Management in Shared-Nothing Distributed OLTP Databases
Workload-Aware Data Management in Shared-Nothing Distributed OLTP Databases
 
Network-aware Data Management for High Throughput Flows Akamai, Cambridge, ...
Network-aware Data Management for High Throughput Flows   Akamai, Cambridge, ...Network-aware Data Management for High Throughput Flows   Akamai, Cambridge, ...
Network-aware Data Management for High Throughput Flows Akamai, Cambridge, ...
 
Green wsn optimization of energy use
Green wsn  optimization of energy useGreen wsn  optimization of energy use
Green wsn optimization of energy use
 
GREEN WSN- OPTIMIZATION OF ENERGY USE THROUGH REDUCTION IN COMMUNICATION WORK...
GREEN WSN- OPTIMIZATION OF ENERGY USE THROUGH REDUCTION IN COMMUNICATION WORK...GREEN WSN- OPTIMIZATION OF ENERGY USE THROUGH REDUCTION IN COMMUNICATION WORK...
GREEN WSN- OPTIMIZATION OF ENERGY USE THROUGH REDUCTION IN COMMUNICATION WORK...
 
50120130406035
5012013040603550120130406035
50120130406035
 
A Platform for Large-Scale Grid Data Service on Dynamic High-Performance Netw...
A Platform for Large-Scale Grid Data Service on Dynamic High-Performance Netw...A Platform for Large-Scale Grid Data Service on Dynamic High-Performance Netw...
A Platform for Large-Scale Grid Data Service on Dynamic High-Performance Netw...
 

More from ieeepondy

Demand aware network function placement
Demand aware network function placementDemand aware network function placement
Demand aware network function placement
ieeepondy
 
Service description in the nfv revolution trends, challenges and a way forward
Service description in the nfv revolution trends, challenges and a way forwardService description in the nfv revolution trends, challenges and a way forward
Service description in the nfv revolution trends, challenges and a way forward
ieeepondy
 
Secure optimization computation outsourcing in cloud computing a case study o...
Secure optimization computation outsourcing in cloud computing a case study o...Secure optimization computation outsourcing in cloud computing a case study o...
Secure optimization computation outsourcing in cloud computing a case study o...
ieeepondy
 
Spatial related traffic sign inspection for inventory purposes using mobile l...
Spatial related traffic sign inspection for inventory purposes using mobile l...Spatial related traffic sign inspection for inventory purposes using mobile l...
Spatial related traffic sign inspection for inventory purposes using mobile l...
ieeepondy
 
Standards for hybrid clouds
Standards for hybrid cloudsStandards for hybrid clouds
Standards for hybrid clouds
ieeepondy
 
Rfhoc a random forest approach to auto-tuning hadoop's configuration
Rfhoc a random forest approach to auto-tuning hadoop's configurationRfhoc a random forest approach to auto-tuning hadoop's configuration
Rfhoc a random forest approach to auto-tuning hadoop's configuration
ieeepondy
 
Resource and instance hour minimization for deadline constrained dag applicat...
Resource and instance hour minimization for deadline constrained dag applicat...Resource and instance hour minimization for deadline constrained dag applicat...
Resource and instance hour minimization for deadline constrained dag applicat...
ieeepondy
 
Reliable and confidential cloud storage with efficient data forwarding functi...
Reliable and confidential cloud storage with efficient data forwarding functi...Reliable and confidential cloud storage with efficient data forwarding functi...
Reliable and confidential cloud storage with efficient data forwarding functi...
ieeepondy
 
Rebuttal to “comments on ‘control cloud data access privilege and anonymity w...
Rebuttal to “comments on ‘control cloud data access privilege and anonymity w...Rebuttal to “comments on ‘control cloud data access privilege and anonymity w...
Rebuttal to “comments on ‘control cloud data access privilege and anonymity w...
ieeepondy
 
Scalable cloud–sensor architecture for the internet of things
Scalable cloud–sensor architecture for the internet of thingsScalable cloud–sensor architecture for the internet of things
Scalable cloud–sensor architecture for the internet of things
ieeepondy
 
Scalable algorithms for nearest neighbor joins on big trajectory data
Scalable algorithms for nearest neighbor joins on big trajectory dataScalable algorithms for nearest neighbor joins on big trajectory data
Scalable algorithms for nearest neighbor joins on big trajectory data
ieeepondy
 
Robust workload and energy management for sustainable data centers
Robust workload and energy management for sustainable data centersRobust workload and energy management for sustainable data centers
Robust workload and energy management for sustainable data centers
ieeepondy
 
Privacy preserving deep computation model on cloud for big data feature learning
Privacy preserving deep computation model on cloud for big data feature learningPrivacy preserving deep computation model on cloud for big data feature learning
Privacy preserving deep computation model on cloud for big data feature learning
ieeepondy
 
Pricing the cloud ieee projects, ieee projects chennai, ieee projects 2016,ie...
Pricing the cloud ieee projects, ieee projects chennai, ieee projects 2016,ie...Pricing the cloud ieee projects, ieee projects chennai, ieee projects 2016,ie...
Pricing the cloud ieee projects, ieee projects chennai, ieee projects 2016,ie...
ieeepondy
 
Protection of big data privacy
Protection of big data privacyProtection of big data privacy
Protection of big data privacy
ieeepondy
 
Power optimization with bler constraint for wireless fronthauls in c ran
Power optimization with bler constraint for wireless fronthauls in c ranPower optimization with bler constraint for wireless fronthauls in c ran
Power optimization with bler constraint for wireless fronthauls in c ran
ieeepondy
 
Performance aware cloud resource allocation via fitness-enabled auction
Performance aware cloud resource allocation via fitness-enabled auctionPerformance aware cloud resource allocation via fitness-enabled auction
Performance aware cloud resource allocation via fitness-enabled auction
ieeepondy
 
Performance limitations of a text search application running in cloud instances
Performance limitations of a text search application running in cloud instancesPerformance limitations of a text search application running in cloud instances
Performance limitations of a text search application running in cloud instances
ieeepondy
 
Performance analysis and optimal cooperative cluster size for randomly distri...
Performance analysis and optimal cooperative cluster size for randomly distri...Performance analysis and optimal cooperative cluster size for randomly distri...
Performance analysis and optimal cooperative cluster size for randomly distri...
ieeepondy
 
Predictive control for energy aware consolidation in cloud datacenters
Predictive control for energy aware consolidation in cloud datacentersPredictive control for energy aware consolidation in cloud datacenters
Predictive control for energy aware consolidation in cloud datacenters
ieeepondy
 

More from ieeepondy (20)

Demand aware network function placement
Demand aware network function placementDemand aware network function placement
Demand aware network function placement
 
Service description in the nfv revolution trends, challenges and a way forward
Service description in the nfv revolution trends, challenges and a way forwardService description in the nfv revolution trends, challenges and a way forward
Service description in the nfv revolution trends, challenges and a way forward
 
Secure optimization computation outsourcing in cloud computing a case study o...
Secure optimization computation outsourcing in cloud computing a case study o...Secure optimization computation outsourcing in cloud computing a case study o...
Secure optimization computation outsourcing in cloud computing a case study o...
 
Spatial related traffic sign inspection for inventory purposes using mobile l...
Spatial related traffic sign inspection for inventory purposes using mobile l...Spatial related traffic sign inspection for inventory purposes using mobile l...
Spatial related traffic sign inspection for inventory purposes using mobile l...
 
Standards for hybrid clouds
Standards for hybrid cloudsStandards for hybrid clouds
Standards for hybrid clouds
 
Rfhoc a random forest approach to auto-tuning hadoop's configuration
Rfhoc a random forest approach to auto-tuning hadoop's configurationRfhoc a random forest approach to auto-tuning hadoop's configuration
Rfhoc a random forest approach to auto-tuning hadoop's configuration
 
Resource and instance hour minimization for deadline constrained dag applicat...
Resource and instance hour minimization for deadline constrained dag applicat...Resource and instance hour minimization for deadline constrained dag applicat...
Resource and instance hour minimization for deadline constrained dag applicat...
 
Reliable and confidential cloud storage with efficient data forwarding functi...
Reliable and confidential cloud storage with efficient data forwarding functi...Reliable and confidential cloud storage with efficient data forwarding functi...
Reliable and confidential cloud storage with efficient data forwarding functi...
 
Rebuttal to “comments on ‘control cloud data access privilege and anonymity w...
Rebuttal to “comments on ‘control cloud data access privilege and anonymity w...Rebuttal to “comments on ‘control cloud data access privilege and anonymity w...
Rebuttal to “comments on ‘control cloud data access privilege and anonymity w...
 
Scalable cloud–sensor architecture for the internet of things
Scalable cloud–sensor architecture for the internet of thingsScalable cloud–sensor architecture for the internet of things
Scalable cloud–sensor architecture for the internet of things
 
Scalable algorithms for nearest neighbor joins on big trajectory data
Scalable algorithms for nearest neighbor joins on big trajectory dataScalable algorithms for nearest neighbor joins on big trajectory data
Scalable algorithms for nearest neighbor joins on big trajectory data
 
Robust workload and energy management for sustainable data centers
Robust workload and energy management for sustainable data centersRobust workload and energy management for sustainable data centers
Robust workload and energy management for sustainable data centers
 
Privacy preserving deep computation model on cloud for big data feature learning
Privacy preserving deep computation model on cloud for big data feature learningPrivacy preserving deep computation model on cloud for big data feature learning
Privacy preserving deep computation model on cloud for big data feature learning
 
Pricing the cloud ieee projects, ieee projects chennai, ieee projects 2016,ie...
Pricing the cloud ieee projects, ieee projects chennai, ieee projects 2016,ie...Pricing the cloud ieee projects, ieee projects chennai, ieee projects 2016,ie...
Pricing the cloud ieee projects, ieee projects chennai, ieee projects 2016,ie...
 
Protection of big data privacy
Protection of big data privacyProtection of big data privacy
Protection of big data privacy
 
Power optimization with bler constraint for wireless fronthauls in c ran
Power optimization with bler constraint for wireless fronthauls in c ranPower optimization with bler constraint for wireless fronthauls in c ran
Power optimization with bler constraint for wireless fronthauls in c ran
 
Performance aware cloud resource allocation via fitness-enabled auction
Performance aware cloud resource allocation via fitness-enabled auctionPerformance aware cloud resource allocation via fitness-enabled auction
Performance aware cloud resource allocation via fitness-enabled auction
 
Performance limitations of a text search application running in cloud instances
Performance limitations of a text search application running in cloud instancesPerformance limitations of a text search application running in cloud instances
Performance limitations of a text search application running in cloud instances
 
Performance analysis and optimal cooperative cluster size for randomly distri...
Performance analysis and optimal cooperative cluster size for randomly distri...Performance analysis and optimal cooperative cluster size for randomly distri...
Performance analysis and optimal cooperative cluster size for randomly distri...
 
Predictive control for energy aware consolidation in cloud datacenters
Predictive control for energy aware consolidation in cloud datacentersPredictive control for energy aware consolidation in cloud datacenters
Predictive control for energy aware consolidation in cloud datacenters
 

Recently uploaded

Eric Nizeyimana's document 2006 from gicumbi to ttc nyamata handball play
Eric Nizeyimana's document 2006 from gicumbi to ttc nyamata handball playEric Nizeyimana's document 2006 from gicumbi to ttc nyamata handball play
Eric Nizeyimana's document 2006 from gicumbi to ttc nyamata handball play
enizeyimana36
 
Textile Chemical Processing and Dyeing.pdf
Textile Chemical Processing and Dyeing.pdfTextile Chemical Processing and Dyeing.pdf
Textile Chemical Processing and Dyeing.pdf
NazakatAliKhoso2
 
学校原版美国波士顿大学毕业证学历学位证书原版一模一样
学校原版美国波士顿大学毕业证学历学位证书原版一模一样学校原版美国波士顿大学毕业证学历学位证书原版一模一样
学校原版美国波士顿大学毕业证学历学位证书原版一模一样
171ticu
 
Iron and Steel Technology Roadmap - Towards more sustainable steelmaking.pdf
Iron and Steel Technology Roadmap - Towards more sustainable steelmaking.pdfIron and Steel Technology Roadmap - Towards more sustainable steelmaking.pdf
Iron and Steel Technology Roadmap - Towards more sustainable steelmaking.pdf
RadiNasr
 
Recycled Concrete Aggregate in Construction Part III
Recycled Concrete Aggregate in Construction Part IIIRecycled Concrete Aggregate in Construction Part III
Recycled Concrete Aggregate in Construction Part III
Aditya Rajan Patra
 
BPV-GUI-01-Guide-for-ASME-Review-Teams-(General)-10-10-2023.pdf
BPV-GUI-01-Guide-for-ASME-Review-Teams-(General)-10-10-2023.pdfBPV-GUI-01-Guide-for-ASME-Review-Teams-(General)-10-10-2023.pdf
BPV-GUI-01-Guide-for-ASME-Review-Teams-(General)-10-10-2023.pdf
MIGUELANGEL966976
 
DEEP LEARNING FOR SMART GRID INTRUSION DETECTION: A HYBRID CNN-LSTM-BASED MODEL
DEEP LEARNING FOR SMART GRID INTRUSION DETECTION: A HYBRID CNN-LSTM-BASED MODELDEEP LEARNING FOR SMART GRID INTRUSION DETECTION: A HYBRID CNN-LSTM-BASED MODEL
DEEP LEARNING FOR SMART GRID INTRUSION DETECTION: A HYBRID CNN-LSTM-BASED MODEL
gerogepatton
 
Modelagem de um CSTR com reação endotermica.pdf
Modelagem de um CSTR com reação endotermica.pdfModelagem de um CSTR com reação endotermica.pdf
Modelagem de um CSTR com reação endotermica.pdf
camseq
 
ISPM 15 Heat Treated Wood Stamps and why your shipping must have one
ISPM 15 Heat Treated Wood Stamps and why your shipping must have oneISPM 15 Heat Treated Wood Stamps and why your shipping must have one
ISPM 15 Heat Treated Wood Stamps and why your shipping must have one
Las Vegas Warehouse
 
Casting-Defect-inSlab continuous casting.pdf
Casting-Defect-inSlab continuous casting.pdfCasting-Defect-inSlab continuous casting.pdf
Casting-Defect-inSlab continuous casting.pdf
zubairahmad848137
 
International Conference on NLP, Artificial Intelligence, Machine Learning an...
International Conference on NLP, Artificial Intelligence, Machine Learning an...International Conference on NLP, Artificial Intelligence, Machine Learning an...
International Conference on NLP, Artificial Intelligence, Machine Learning an...
gerogepatton
 
Presentation of IEEE Slovenia CIS (Computational Intelligence Society) Chapte...
Presentation of IEEE Slovenia CIS (Computational Intelligence Society) Chapte...Presentation of IEEE Slovenia CIS (Computational Intelligence Society) Chapte...
Presentation of IEEE Slovenia CIS (Computational Intelligence Society) Chapte...
University of Maribor
 
Unit-III-ELECTROCHEMICAL STORAGE DEVICES.ppt
Unit-III-ELECTROCHEMICAL STORAGE DEVICES.pptUnit-III-ELECTROCHEMICAL STORAGE DEVICES.ppt
Unit-III-ELECTROCHEMICAL STORAGE DEVICES.ppt
KrishnaveniKrishnara1
 
ACEP Magazine edition 4th launched on 05.06.2024
ACEP Magazine edition 4th launched on 05.06.2024ACEP Magazine edition 4th launched on 05.06.2024
ACEP Magazine edition 4th launched on 05.06.2024
Rahul
 
ML Based Model for NIDS MSc Updated Presentation.v2.pptx
ML Based Model for NIDS MSc Updated Presentation.v2.pptxML Based Model for NIDS MSc Updated Presentation.v2.pptx
ML Based Model for NIDS MSc Updated Presentation.v2.pptx
JamalHussainArman
 
TIME DIVISION MULTIPLEXING TECHNIQUE FOR COMMUNICATION SYSTEM
TIME DIVISION MULTIPLEXING TECHNIQUE FOR COMMUNICATION SYSTEMTIME DIVISION MULTIPLEXING TECHNIQUE FOR COMMUNICATION SYSTEM
TIME DIVISION MULTIPLEXING TECHNIQUE FOR COMMUNICATION SYSTEM
HODECEDSIET
 
22CYT12-Unit-V-E Waste and its Management.ppt
22CYT12-Unit-V-E Waste and its Management.ppt22CYT12-Unit-V-E Waste and its Management.ppt
22CYT12-Unit-V-E Waste and its Management.ppt
KrishnaveniKrishnara1
 
Redefining brain tumor segmentation: a cutting-edge convolutional neural netw...
Redefining brain tumor segmentation: a cutting-edge convolutional neural netw...Redefining brain tumor segmentation: a cutting-edge convolutional neural netw...
Redefining brain tumor segmentation: a cutting-edge convolutional neural netw...
IJECEIAES
 
KuberTENes Birthday Bash Guadalajara - K8sGPT first impressions
KuberTENes Birthday Bash Guadalajara - K8sGPT first impressionsKuberTENes Birthday Bash Guadalajara - K8sGPT first impressions
KuberTENes Birthday Bash Guadalajara - K8sGPT first impressions
Victor Morales
 
basic-wireline-operations-course-mahmoud-f-radwan.pdf
basic-wireline-operations-course-mahmoud-f-radwan.pdfbasic-wireline-operations-course-mahmoud-f-radwan.pdf
basic-wireline-operations-course-mahmoud-f-radwan.pdf
NidhalKahouli2
 

Recently uploaded (20)

Eric Nizeyimana's document 2006 from gicumbi to ttc nyamata handball play
Eric Nizeyimana's document 2006 from gicumbi to ttc nyamata handball playEric Nizeyimana's document 2006 from gicumbi to ttc nyamata handball play
Eric Nizeyimana's document 2006 from gicumbi to ttc nyamata handball play
 
Textile Chemical Processing and Dyeing.pdf
Textile Chemical Processing and Dyeing.pdfTextile Chemical Processing and Dyeing.pdf
Textile Chemical Processing and Dyeing.pdf
 
学校原版美国波士顿大学毕业证学历学位证书原版一模一样
学校原版美国波士顿大学毕业证学历学位证书原版一模一样学校原版美国波士顿大学毕业证学历学位证书原版一模一样
学校原版美国波士顿大学毕业证学历学位证书原版一模一样
 
Iron and Steel Technology Roadmap - Towards more sustainable steelmaking.pdf
Iron and Steel Technology Roadmap - Towards more sustainable steelmaking.pdfIron and Steel Technology Roadmap - Towards more sustainable steelmaking.pdf
Iron and Steel Technology Roadmap - Towards more sustainable steelmaking.pdf
 
Recycled Concrete Aggregate in Construction Part III
Recycled Concrete Aggregate in Construction Part IIIRecycled Concrete Aggregate in Construction Part III
Recycled Concrete Aggregate in Construction Part III
 
BPV-GUI-01-Guide-for-ASME-Review-Teams-(General)-10-10-2023.pdf
BPV-GUI-01-Guide-for-ASME-Review-Teams-(General)-10-10-2023.pdfBPV-GUI-01-Guide-for-ASME-Review-Teams-(General)-10-10-2023.pdf
BPV-GUI-01-Guide-for-ASME-Review-Teams-(General)-10-10-2023.pdf
 
DEEP LEARNING FOR SMART GRID INTRUSION DETECTION: A HYBRID CNN-LSTM-BASED MODEL
DEEP LEARNING FOR SMART GRID INTRUSION DETECTION: A HYBRID CNN-LSTM-BASED MODELDEEP LEARNING FOR SMART GRID INTRUSION DETECTION: A HYBRID CNN-LSTM-BASED MODEL
DEEP LEARNING FOR SMART GRID INTRUSION DETECTION: A HYBRID CNN-LSTM-BASED MODEL
 
Modelagem de um CSTR com reação endotermica.pdf
Modelagem de um CSTR com reação endotermica.pdfModelagem de um CSTR com reação endotermica.pdf
Modelagem de um CSTR com reação endotermica.pdf
 
ISPM 15 Heat Treated Wood Stamps and why your shipping must have one
ISPM 15 Heat Treated Wood Stamps and why your shipping must have oneISPM 15 Heat Treated Wood Stamps and why your shipping must have one
ISPM 15 Heat Treated Wood Stamps and why your shipping must have one
 
Casting-Defect-inSlab continuous casting.pdf
Casting-Defect-inSlab continuous casting.pdfCasting-Defect-inSlab continuous casting.pdf
Casting-Defect-inSlab continuous casting.pdf
 
International Conference on NLP, Artificial Intelligence, Machine Learning an...
International Conference on NLP, Artificial Intelligence, Machine Learning an...International Conference on NLP, Artificial Intelligence, Machine Learning an...
International Conference on NLP, Artificial Intelligence, Machine Learning an...
 
Presentation of IEEE Slovenia CIS (Computational Intelligence Society) Chapte...
Presentation of IEEE Slovenia CIS (Computational Intelligence Society) Chapte...Presentation of IEEE Slovenia CIS (Computational Intelligence Society) Chapte...
Presentation of IEEE Slovenia CIS (Computational Intelligence Society) Chapte...
 
Unit-III-ELECTROCHEMICAL STORAGE DEVICES.ppt
Unit-III-ELECTROCHEMICAL STORAGE DEVICES.pptUnit-III-ELECTROCHEMICAL STORAGE DEVICES.ppt
Unit-III-ELECTROCHEMICAL STORAGE DEVICES.ppt
 
ACEP Magazine edition 4th launched on 05.06.2024
ACEP Magazine edition 4th launched on 05.06.2024ACEP Magazine edition 4th launched on 05.06.2024
ACEP Magazine edition 4th launched on 05.06.2024
 
ML Based Model for NIDS MSc Updated Presentation.v2.pptx
ML Based Model for NIDS MSc Updated Presentation.v2.pptxML Based Model for NIDS MSc Updated Presentation.v2.pptx
ML Based Model for NIDS MSc Updated Presentation.v2.pptx
 
TIME DIVISION MULTIPLEXING TECHNIQUE FOR COMMUNICATION SYSTEM
TIME DIVISION MULTIPLEXING TECHNIQUE FOR COMMUNICATION SYSTEMTIME DIVISION MULTIPLEXING TECHNIQUE FOR COMMUNICATION SYSTEM
TIME DIVISION MULTIPLEXING TECHNIQUE FOR COMMUNICATION SYSTEM
 
22CYT12-Unit-V-E Waste and its Management.ppt
22CYT12-Unit-V-E Waste and its Management.ppt22CYT12-Unit-V-E Waste and its Management.ppt
22CYT12-Unit-V-E Waste and its Management.ppt
 
Redefining brain tumor segmentation: a cutting-edge convolutional neural netw...
Redefining brain tumor segmentation: a cutting-edge convolutional neural netw...Redefining brain tumor segmentation: a cutting-edge convolutional neural netw...
Redefining brain tumor segmentation: a cutting-edge convolutional neural netw...
 
KuberTENes Birthday Bash Guadalajara - K8sGPT first impressions
KuberTENes Birthday Bash Guadalajara - K8sGPT first impressionsKuberTENes Birthday Bash Guadalajara - K8sGPT first impressions
KuberTENes Birthday Bash Guadalajara - K8sGPT first impressions
 
basic-wireline-operations-course-mahmoud-f-radwan.pdf
basic-wireline-operations-course-mahmoud-f-radwan.pdfbasic-wireline-operations-course-mahmoud-f-radwan.pdf
basic-wireline-operations-course-mahmoud-f-radwan.pdf
 

Application level optimization of big data transfers through pipelining, parallelism and concurrency

  • 1. Application-Level Optimization of Big Data Transfers through Pipelining, Parallelism and Concurrency Abstract: In end-to-end data transfers, there are several factors affecting the data transfer throughput, such as the network characteristics (e.g., network bandwidth, round- trip-time, background traffic); end-system characteristics (e.g., NIC capacity, number of CPU cores and their clock rate, number of disk drives and their I/O rate); and the dataset characteristics (e.g., average file size, dataset size, file size distribution). Optimization of big data transfers over inter-cloud and intra- cloud networks is a challenging task that requires joint-consideration of all of these parameters. This optimization task becomes even more challenging when transferring datasets comprised of heterogeneous file sizes (i.e., large files and small files mixed). Previous work in this area only focuses on the end-system and network characteristics however does not provide models regarding the dataset characteristics. In this study, we analyze the effects of the three most important transfer parameters that are used to enhance data transfer throughput: pipelining,parallelism and concurrency. We provide models and guidelines to set the best values for these parameters and present two different transfer optimization algorithms that use the models developed. The tests conducted over high-speed networking and cloud testbeds show that our algorithms outperform the most popular data transfer tools like Globus Online and UDT in majority of the cases.