The document discusses improving the file transfer service (FTS3) at CERN. It describes two aspects: 1) selecting the best source site for file transfers from among multiple replicas by considering factors like throughput and success rate, not just pending files, and 2) maximizing throughput across the WLCG network by increasing TCP buffer sizes either through Linux auto-tuning or manual configuration. Evaluating different techniques for setting optimal TCP buffer sizes could help effectively utilize available network resources and increase FTS3 transfer speeds.
OPTIMIZING END-TO-END BIG DATA TRANSFERS OVER TERABITS NETWORK INFRASTRUCTURENexgen Technology
GET IEEE BIG DATA,JAVA ,DOTNET,ANDROID ,NS2,MATLAB,EMBEDED AT LOW COST WITH BEST QUALITY PLEASE CONTACT BELOW NUMBER
FOR MORE INFORMATION PLEASE FIND THE BELOW DETAILS:
Nexgen Technology
No :66,4th cross,Venkata nagar,
Near SBI ATM,
Puducherry.
Email Id: praveen@nexgenproject.com
Mobile: 9791938249
Telephone: 0413-2211159
www.nexgenproject.com
This document summarizes research on scheduling algorithms for loading streaming data into real-time data warehouses. The goal is to minimize data staleness over time. It describes how streaming warehouses continuously ingest incoming data streams to support time-critical analyses, unlike traditional warehouses which are periodically refreshed. It presents a model for temporal consistency and defines data staleness. It formulates the streaming warehouse update problem as a scheduling problem to minimize staleness and proves that any online, non-preemptive scheduling algorithm can achieve staleness within a constant factor of optimal if processors are sufficiently fast and no processor is idly waiting.
This document summarizes a research paper that proposes a method called flow-based slicing to preserve interflow packet order in multipath switching systems while avoiding congestion. Flow-based slicing works by splitting each network flow into smaller "slices" at intervals larger than a threshold, and balancing the load across paths at a finer granularity. This allows for uniform load sharing while maintaining packet ordering and minimizing out-of-order probability to less than 10-6. The method achieves good load balancing with low hardware overhead and timing complexity. Simulation results validate that when the slicing threshold is set to the smallest delay upper bound, flow-based slicing can optimally balance loads while keeping packet reordering negligible for internal speeds up to two times.
This document discusses dynamic adaptation techniques for optimizing data transfer performance over networks. It describes how the number of concurrent data transfer streams can be adjusted dynamically according to changing network conditions, without relying on historical measurements or external profiling. The proposed approach gradually increases the level of parallelism during a transfer to find a near-optimal number of streams based on instant throughput measurements, allowing it to adapt to varying environments and network utilization over time.
Achieving congestion diversity in multi hop wireless mesh networksieeeprojectschennai
This paper compares congestion-aware routing algorithms like Backpressure (BP), Enhanced Backpressure (E-BP), and Congestion Diversity Protocol (CDP) to a shortest path routing protocol (LQSR) on a wireless mesh network testbed. The results show that under moderate to heavy UDP traffic, CDP significantly outperforms LQSR in 80-90% of cases, while BP and E-BP often degrade performance compared to LQSR for both UDP and TCP traffic. Shortest path routing fails to utilize available path diversity and can increase congestion and delay, especially with significant UDP traffic loads.
The simple architecture with high forwarding capacity bandwidth is load balancing switch is build with two stage switch fabric to eliminate the mis sequencing problems and kept packets of the same flow in the order of load balanced two stage switch. Before load balancing two stages has perform with the Full
frame first algorithm and use centralized scheduler to eliminate those mis sequencing problemsand getting the 100% throughput result. There comes at some additional cost and needs costly online scheduling algorithms and need for a complex scheduler and quality being reduced. In this paper we introduce the two stage switch fabric with round robin scheduling algorithmis used to solve mis sequencing problems and two maximum weight matching algorithms Longest queue first(LQF) and Oldest cell first(OCF) to achieve 100% throughput results. Theoretical analysis and simulation results shows that using these algorithms, achieve all the possible outcome results in the switch.
This document is a thesis submitted by Robin Duda and Gustaf Nilstadius to KTH Royal Institute of Technology in Stockholm, Sweden, in 2016. The thesis evaluates different load balancing techniques for a distributed logging system. It compares the performance of a traditional load balancer using a push-based approach versus using messaging queues with a pull-based approach. The goal is to determine the most feasible method for load balancing in such a distributed logging system. Tests were conducted measuring the throughput of setups with different load balancers. The conclusion is that both messaging queues and traditional load balancing are equally feasible options for load balancing in a logging context.
This document proposes an adaptive algorithm called DyBBS that dynamically adjusts the batch size and execution parallelism in Spark Streaming to minimize end-to-end latency. The algorithm is based on two observations: 1) processing time increases monotonically with batch size, and 2) there is an optimal execution parallelism for a given batch size. DyBBS uses isotonic regression to learn and adapt batch size and parallelism as workload and conditions change. Experimental results show it significantly reduces latency compared to static configurations and other state-of-the-art approaches.
OPTIMIZING END-TO-END BIG DATA TRANSFERS OVER TERABITS NETWORK INFRASTRUCTURENexgen Technology
GET IEEE BIG DATA,JAVA ,DOTNET,ANDROID ,NS2,MATLAB,EMBEDED AT LOW COST WITH BEST QUALITY PLEASE CONTACT BELOW NUMBER
FOR MORE INFORMATION PLEASE FIND THE BELOW DETAILS:
Nexgen Technology
No :66,4th cross,Venkata nagar,
Near SBI ATM,
Puducherry.
Email Id: praveen@nexgenproject.com
Mobile: 9791938249
Telephone: 0413-2211159
www.nexgenproject.com
This document summarizes research on scheduling algorithms for loading streaming data into real-time data warehouses. The goal is to minimize data staleness over time. It describes how streaming warehouses continuously ingest incoming data streams to support time-critical analyses, unlike traditional warehouses which are periodically refreshed. It presents a model for temporal consistency and defines data staleness. It formulates the streaming warehouse update problem as a scheduling problem to minimize staleness and proves that any online, non-preemptive scheduling algorithm can achieve staleness within a constant factor of optimal if processors are sufficiently fast and no processor is idly waiting.
This document summarizes a research paper that proposes a method called flow-based slicing to preserve interflow packet order in multipath switching systems while avoiding congestion. Flow-based slicing works by splitting each network flow into smaller "slices" at intervals larger than a threshold, and balancing the load across paths at a finer granularity. This allows for uniform load sharing while maintaining packet ordering and minimizing out-of-order probability to less than 10-6. The method achieves good load balancing with low hardware overhead and timing complexity. Simulation results validate that when the slicing threshold is set to the smallest delay upper bound, flow-based slicing can optimally balance loads while keeping packet reordering negligible for internal speeds up to two times.
This document discusses dynamic adaptation techniques for optimizing data transfer performance over networks. It describes how the number of concurrent data transfer streams can be adjusted dynamically according to changing network conditions, without relying on historical measurements or external profiling. The proposed approach gradually increases the level of parallelism during a transfer to find a near-optimal number of streams based on instant throughput measurements, allowing it to adapt to varying environments and network utilization over time.
Achieving congestion diversity in multi hop wireless mesh networksieeeprojectschennai
This paper compares congestion-aware routing algorithms like Backpressure (BP), Enhanced Backpressure (E-BP), and Congestion Diversity Protocol (CDP) to a shortest path routing protocol (LQSR) on a wireless mesh network testbed. The results show that under moderate to heavy UDP traffic, CDP significantly outperforms LQSR in 80-90% of cases, while BP and E-BP often degrade performance compared to LQSR for both UDP and TCP traffic. Shortest path routing fails to utilize available path diversity and can increase congestion and delay, especially with significant UDP traffic loads.
The simple architecture with high forwarding capacity bandwidth is load balancing switch is build with two stage switch fabric to eliminate the mis sequencing problems and kept packets of the same flow in the order of load balanced two stage switch. Before load balancing two stages has perform with the Full
frame first algorithm and use centralized scheduler to eliminate those mis sequencing problemsand getting the 100% throughput result. There comes at some additional cost and needs costly online scheduling algorithms and need for a complex scheduler and quality being reduced. In this paper we introduce the two stage switch fabric with round robin scheduling algorithmis used to solve mis sequencing problems and two maximum weight matching algorithms Longest queue first(LQF) and Oldest cell first(OCF) to achieve 100% throughput results. Theoretical analysis and simulation results shows that using these algorithms, achieve all the possible outcome results in the switch.
This document is a thesis submitted by Robin Duda and Gustaf Nilstadius to KTH Royal Institute of Technology in Stockholm, Sweden, in 2016. The thesis evaluates different load balancing techniques for a distributed logging system. It compares the performance of a traditional load balancer using a push-based approach versus using messaging queues with a pull-based approach. The goal is to determine the most feasible method for load balancing in such a distributed logging system. Tests were conducted measuring the throughput of setups with different load balancers. The conclusion is that both messaging queues and traditional load balancing are equally feasible options for load balancing in a logging context.
This document proposes an adaptive algorithm called DyBBS that dynamically adjusts the batch size and execution parallelism in Spark Streaming to minimize end-to-end latency. The algorithm is based on two observations: 1) processing time increases monotonically with batch size, and 2) there is an optimal execution parallelism for a given batch size. DyBBS uses isotonic regression to learn and adapt batch size and parallelism as workload and conditions change. Experimental results show it significantly reduces latency compared to static configurations and other state-of-the-art approaches.
International Journal of Engineering Research and Applications (IJERA) is a team of researchers not publication services or private publications running the journals for monetary benefits, we are association of scientists and academia who focus only on supporting authors who want to publish their work. The articles published in our journal can be accessed online, all the articles will be archived for real time access.
Our journal system primarily aims to bring out the research talent and the works done by sciaentists, academia, engineers, practitioners, scholars, post graduate students of engineering and science. This journal aims to cover the scientific research in a broader sense and not publishing a niche area of research facilitating researchers from various verticals to publish their papers. It is also aimed to provide a platform for the researchers to publish in a shorter of time, enabling them to continue further All articles published are freely available to scientific researchers in the Government agencies,educators and the general public. We are taking serious efforts to promote our journal across the globe in various ways, we are sure that our journal will act as a scientific platform for all researchers to publish their works online.
Transport Layer Services : Multiplexing And DemultiplexingKeyur Vadodariya
This document discusses the transport layer of computer networks. It begins with introducing the group members and topic, which is the transport layer introduction, services, multiplexing and demultiplexing. Then it provides definitions of the transport layer, its functions and services. It describes how the transport layer provides process to process delivery, end-to-end connections, congestion control, data integrity, flow control, multiplexing and demultiplexing. It explains the differences between connectionless and connection-oriented multiplexing and demultiplexing. In the end, it lists some references.
To develop an indexing system which helps to build an Unsupervised Indexing for Big Data. With this indexing system one can search for data files not only based on keywords and file names but also with the closest meaningful data to your input content (clustering approach).
The document summarizes key topics related to transport layer protocols:
- It describes the services provided by the transport layer, including addressing, connection establishment and release, flow control, and multiplexing.
- It provides details on common transport protocols like TCP and UDP, including their packet headers, connection management, congestion control, and performance issues at high speeds.
- It also presents an example transport protocol and uses finite state machines to model its operation and connection management.
AN ENERGY EFFICIENT L2 CACHE ARCHITECTURE USING WAY TAG INFORMATION UNDER WR...Vijay Prime
This document proposes a new cache architecture called a way-tagged cache to improve energy efficiency in write-through caches. It does this by maintaining way tag information from the L2 cache in the L1 cache. During a write hit in the L1 cache, this allows the L2 cache to operate like a direct-mapped cache by only accessing one way. Simulation results show this approach reduces L2 cache energy by 65.4% on average with minimal area overhead and no performance impact.
Performance analysis of routing protocols and tcp variants under http and ftp...IJCNCJournal
MANET stands for mobile ad-hoc network that has multi-hop and dynamic nature, where each station changes its location frequently and automatically configures itself. In this paper, four routing protocols
that areOLSR,GRP,DSR, and AODV are discussed along with three TCP variants that are SACK, New Reno and Reno. The main focus of this paper is to study the impact
scalability, mobility and traffic loads on routing protocols and TCP variants. Thepaper results shows that the proactive protocols OLSR and GRP outperform the reactive protocols AODV and DSR with the same nodes size, nodes speed, and traffic load. On the other hand, the TCP variants research reveal the superiority of the TCP SACK variant over the other two variants in case of adapting to varying network size, while the TCP Reno variant acts more
robustly in varying mobility speeds and traffic loads.
Network-aware Data Management for Large Scale Distributed Applications, IBM R...balmanme
The document discusses network-aware data management for large-scale distributed applications. It provides an outline for a presentation on this topic, including discussing the performance of VSAN and VVOL storage in virtualized environments, the PetaShare distributed storage system and Stork data scheduler, data streaming in high-bandwidth networks, and several other related topics like network reservations and scheduling. The presenter's background and experience working on data transfer scheduling, distributed storage, and high-performance computing networks is also briefly summarized.
This document provides a survey of different void handling techniques that are suitable for vehicular ad hoc networks (VANETs). It begins with an introduction to geographic routing and the void problem in networks. Then it describes six categories of void handling techniques: planar graph based, face routing based, restricted direction based, hierarchical based, trajectory based, and cluster based. For each category, it briefly explains one or two representative techniques. Finally, it provides a comparative survey of the techniques based on important performance features.
Balman dissertation Copyright @ 2010 Mehmet Balmanbalmanme
This document discusses scheduling data transfer operations with advance reservation and provisioning. It proposes dividing time into windows where network bandwidth availability is stable. When a data transfer request is received, the scheduler checks all possible time windows to see if the request can fit within bandwidth constraints. If no window is available, it tries shifting existing transfers to earlier windows if they have less "desire" based on number of occupied time slots and order of the window. This allows requests to be scheduled in advance while minimizing disruption to existing transfers.
Multilevel priority packet scheduling scheme for wireless networksijdpsjournal
Scheduling different types of packets such as real
-
time and non
-
real time data packets in wireless links is
necessary to reduce energy consumption of the wireless device. Most of the existing packet schedulin
g
mechanism uses opportunistic transmission sche
duling, in which communication is postponed upto an
acceptable time deadline until the best expected channel conditions to transmit are found. This algo
rithm
incurs a large processing overhead and more energy consumption. In this paper we propose a Dynamic
Multilevel Queue Scheduling algorithm. In the proposed scheme, the ready queue is partitioned into t
hree
levels of priority queues. Real
-
time packets are placed into the highest priority queue and non
-
real time
data packets are placed into two other queue
s. We evaluate the performance of the proposed Dynamic
Multilevel Queue Scheduling scheme through simulations for real
-
time and non
-
real time data. Simulation
results illustrate that the Multilevel Priority packet scheduling scheme overcomes the convention
al methods
interms of average data waiting time and end
-
to
-
end delay
The transport layer provides end-to-end communication between processes on different machines. Two main transport protocols are TCP and UDP. TCP provides reliable, connection-oriented data transmission using acknowledgments and retransmissions. UDP provides simpler, connectionless transmission but without reliability. Both protocols use port numbers to identify processes and negotiate quality of service options during connection establishment.
Time series data monitoring at 99acres.comRavi Raj
The document describes the current single box setup for 99acres.com monitoring which includes Carbon, Whisper, and Graphite Web. Carbon receives metrics and flushes them to Whisper. Whisper is a flat-file database that stores each metric in a separate file. Graphite Web is a Django UI that queries Carbon and Whisper to return and graph metrics data. The proposed final approach adds a Carbon-Relay box and dedicated Graphite Web box for load balancing and fault tolerance across multiple Graphite storage nodes.
The transport layer provides end-to-end communication over a network by providing services such as connection-oriented communication, reliability, flow control, and multiplexing. It links the application layer to the network layer and performs functions like segmenting messages and establishing connections between endpoints. Common transport protocols are TCP, which provides connection-oriented and reliable data transfer, and UDP, which provides connectionless datagram delivery.
The document discusses network layer performance and congestion control. It covers key network layer performance metrics like delay, throughput and packet loss. It then discusses various sources of delay like transmission, propagation, processing and queuing delays. It also discusses throughput and packet loss. The second half of the document focuses on congestion control techniques including open-loop methods like retransmission policies and closed-loop methods like backpressure and explicit signaling.
This document provides a status update on Oak Ridge National Laboratory's evaluation of the Cray XT3 supercomputer. It describes the XT3 system architecture including its AMD Opteron processors, SeaStar interconnect, and lightweight operating system. It also summarizes performance results from microbenchmarks, kernels, and applications in areas like climate, biology, astrophysics, combustion, and fusion on up to 4,096 processors, demonstrating the XT3's competitive performance, interconnect bandwidth, and parallel efficiency.
Moolle fan-out control for scalable distributed data storesSungJu Cho
Many Online Social Networks horizontally partition data across data stores. This allows the addition of server nodes to increase capacity and throughput. For single key lookup queries such as computing a member's 1st degree connections, clients need to generate only one request to one data store. However, for multi key lookup queries such as computing a 2nd degree network, clients need to generate multiple requests to multiple data stores. The number of requests to fulfill the multi key lookup queries grows in relation to the number of partitions. Increasing the number of server nodes in order to increase capacity also increases the number of requests between the client and data stores. This may increase the latency of the query response time because of network congestion, tail-latency, and CPU bounding. Replication based partitioning strategies can reduce the number of requests in the multi key lookup queries. However, reducing the number of requests in a query can degrade the performance of certain queries where processing, computing, and filtering can be done by the data stores. A better system would provide the capability of controlling the number of requests in a query. This paper presents Moolle, a system of controlling the number of requests in queries to scalable distributed data stores. Moolle has been implemented in the LinkedIn distributed graph service that serves hundreds of thousands of social graph traversal queries per second. We believe that Moolle can be applied to other distributed systems that handle distributed data processing with a high volume of variable-sized requests.
This document summarizes key aspects of the transport layer:
- The transport layer provides logical communication between application processes running on different hosts and handles reliable data transfer.
- It provides both connection-oriented and connectionless services to the application layer. Quality of service parameters like throughput and delay can be negotiated.
- Transport layer protocols like TCP and UDP are described. TCP provides reliable byte-stream delivery using connections while UDP provides best-effort unreliable datagram delivery.
Transfer reliability and congestion control strategies in opportunistic netwo...IEEEFINALYEARPROJECTS
The document discusses transfer reliability and congestion control strategies in opportunistic networks. It begins by stating that opportunistic networks have unpredictable node contacts and rarely have complete end-to-end paths. It then discusses how modified TCP protocols are ineffective for these networks and they require different approaches than intermittently connected networks. The document surveys proposals for transfer reliability using hop-by-hop custody transfer and end-to-end receipts. It also categorizes storage congestion control based on single or multiple message copies. It identifies open research issues including replication management and drop policies for multiple copies.
Hw09 Hadoop Based Data Mining Platform For The Telecom IndustryCloudera, Inc.
The document summarizes a parallel data mining platform called BC-PDM developed by China Mobile Communication Corporation to address the challenges of analyzing their large scale telecom data. Key points:
- BC-PDM is based on Hadoop and designed to perform ETL and data mining algorithms in parallel to enable scalable analysis of datasets exceeding hundreds of terabytes.
- The platform implements various ETL operations and data mining algorithms using MapReduce. Initial experiments showed a 10-50x speedup over traditional solutions.
- Future work includes improving data security, migrating online systems to the platform, and enhancing the user interface.
This document summarizes a computer physics communications article about the conditions database system for the COMPASS experiment. The key points are:
1) COMPASS integrated a conditions database system to manage time-dependent detector condition, calibration, and geometry alignment information using software from CERN.
2) The conditions database consists of administration tools, a data handling library, and software to transfer data from detector controls to the database.
3) Performance tests on the COMPASS computing farm showed the conditions database system was able to efficiently manage the large volumes of time-dependent experimental data needed for the COMPASS experiment.
Grid computing involves distributing computing resources across a network to tackle large problems. The Worldwide LHC Computing Grid (WLCG) was established to support the Large Hadron Collider (LHC) experiment, which produces around 15 petabytes of data annually. The WLCG uses a four-tiered model, with raw data stored at Tier-0 (CERN), copies distributed to Tier-1 data centers, computational resources provided by Tier-2 centers, and Tier-3 facilities providing additional analysis capabilities. This distributed model has proven effective in supporting the first year of LHC data collection and analysis through globally shared computing resources.
International Journal of Engineering Research and Applications (IJERA) is a team of researchers not publication services or private publications running the journals for monetary benefits, we are association of scientists and academia who focus only on supporting authors who want to publish their work. The articles published in our journal can be accessed online, all the articles will be archived for real time access.
Our journal system primarily aims to bring out the research talent and the works done by sciaentists, academia, engineers, practitioners, scholars, post graduate students of engineering and science. This journal aims to cover the scientific research in a broader sense and not publishing a niche area of research facilitating researchers from various verticals to publish their papers. It is also aimed to provide a platform for the researchers to publish in a shorter of time, enabling them to continue further All articles published are freely available to scientific researchers in the Government agencies,educators and the general public. We are taking serious efforts to promote our journal across the globe in various ways, we are sure that our journal will act as a scientific platform for all researchers to publish their works online.
Transport Layer Services : Multiplexing And DemultiplexingKeyur Vadodariya
This document discusses the transport layer of computer networks. It begins with introducing the group members and topic, which is the transport layer introduction, services, multiplexing and demultiplexing. Then it provides definitions of the transport layer, its functions and services. It describes how the transport layer provides process to process delivery, end-to-end connections, congestion control, data integrity, flow control, multiplexing and demultiplexing. It explains the differences between connectionless and connection-oriented multiplexing and demultiplexing. In the end, it lists some references.
To develop an indexing system which helps to build an Unsupervised Indexing for Big Data. With this indexing system one can search for data files not only based on keywords and file names but also with the closest meaningful data to your input content (clustering approach).
The document summarizes key topics related to transport layer protocols:
- It describes the services provided by the transport layer, including addressing, connection establishment and release, flow control, and multiplexing.
- It provides details on common transport protocols like TCP and UDP, including their packet headers, connection management, congestion control, and performance issues at high speeds.
- It also presents an example transport protocol and uses finite state machines to model its operation and connection management.
AN ENERGY EFFICIENT L2 CACHE ARCHITECTURE USING WAY TAG INFORMATION UNDER WR...Vijay Prime
This document proposes a new cache architecture called a way-tagged cache to improve energy efficiency in write-through caches. It does this by maintaining way tag information from the L2 cache in the L1 cache. During a write hit in the L1 cache, this allows the L2 cache to operate like a direct-mapped cache by only accessing one way. Simulation results show this approach reduces L2 cache energy by 65.4% on average with minimal area overhead and no performance impact.
Performance analysis of routing protocols and tcp variants under http and ftp...IJCNCJournal
MANET stands for mobile ad-hoc network that has multi-hop and dynamic nature, where each station changes its location frequently and automatically configures itself. In this paper, four routing protocols
that areOLSR,GRP,DSR, and AODV are discussed along with three TCP variants that are SACK, New Reno and Reno. The main focus of this paper is to study the impact
scalability, mobility and traffic loads on routing protocols and TCP variants. Thepaper results shows that the proactive protocols OLSR and GRP outperform the reactive protocols AODV and DSR with the same nodes size, nodes speed, and traffic load. On the other hand, the TCP variants research reveal the superiority of the TCP SACK variant over the other two variants in case of adapting to varying network size, while the TCP Reno variant acts more
robustly in varying mobility speeds and traffic loads.
Network-aware Data Management for Large Scale Distributed Applications, IBM R...balmanme
The document discusses network-aware data management for large-scale distributed applications. It provides an outline for a presentation on this topic, including discussing the performance of VSAN and VVOL storage in virtualized environments, the PetaShare distributed storage system and Stork data scheduler, data streaming in high-bandwidth networks, and several other related topics like network reservations and scheduling. The presenter's background and experience working on data transfer scheduling, distributed storage, and high-performance computing networks is also briefly summarized.
This document provides a survey of different void handling techniques that are suitable for vehicular ad hoc networks (VANETs). It begins with an introduction to geographic routing and the void problem in networks. Then it describes six categories of void handling techniques: planar graph based, face routing based, restricted direction based, hierarchical based, trajectory based, and cluster based. For each category, it briefly explains one or two representative techniques. Finally, it provides a comparative survey of the techniques based on important performance features.
Balman dissertation Copyright @ 2010 Mehmet Balmanbalmanme
This document discusses scheduling data transfer operations with advance reservation and provisioning. It proposes dividing time into windows where network bandwidth availability is stable. When a data transfer request is received, the scheduler checks all possible time windows to see if the request can fit within bandwidth constraints. If no window is available, it tries shifting existing transfers to earlier windows if they have less "desire" based on number of occupied time slots and order of the window. This allows requests to be scheduled in advance while minimizing disruption to existing transfers.
Multilevel priority packet scheduling scheme for wireless networksijdpsjournal
Scheduling different types of packets such as real
-
time and non
-
real time data packets in wireless links is
necessary to reduce energy consumption of the wireless device. Most of the existing packet schedulin
g
mechanism uses opportunistic transmission sche
duling, in which communication is postponed upto an
acceptable time deadline until the best expected channel conditions to transmit are found. This algo
rithm
incurs a large processing overhead and more energy consumption. In this paper we propose a Dynamic
Multilevel Queue Scheduling algorithm. In the proposed scheme, the ready queue is partitioned into t
hree
levels of priority queues. Real
-
time packets are placed into the highest priority queue and non
-
real time
data packets are placed into two other queue
s. We evaluate the performance of the proposed Dynamic
Multilevel Queue Scheduling scheme through simulations for real
-
time and non
-
real time data. Simulation
results illustrate that the Multilevel Priority packet scheduling scheme overcomes the convention
al methods
interms of average data waiting time and end
-
to
-
end delay
The transport layer provides end-to-end communication between processes on different machines. Two main transport protocols are TCP and UDP. TCP provides reliable, connection-oriented data transmission using acknowledgments and retransmissions. UDP provides simpler, connectionless transmission but without reliability. Both protocols use port numbers to identify processes and negotiate quality of service options during connection establishment.
Time series data monitoring at 99acres.comRavi Raj
The document describes the current single box setup for 99acres.com monitoring which includes Carbon, Whisper, and Graphite Web. Carbon receives metrics and flushes them to Whisper. Whisper is a flat-file database that stores each metric in a separate file. Graphite Web is a Django UI that queries Carbon and Whisper to return and graph metrics data. The proposed final approach adds a Carbon-Relay box and dedicated Graphite Web box for load balancing and fault tolerance across multiple Graphite storage nodes.
The transport layer provides end-to-end communication over a network by providing services such as connection-oriented communication, reliability, flow control, and multiplexing. It links the application layer to the network layer and performs functions like segmenting messages and establishing connections between endpoints. Common transport protocols are TCP, which provides connection-oriented and reliable data transfer, and UDP, which provides connectionless datagram delivery.
The document discusses network layer performance and congestion control. It covers key network layer performance metrics like delay, throughput and packet loss. It then discusses various sources of delay like transmission, propagation, processing and queuing delays. It also discusses throughput and packet loss. The second half of the document focuses on congestion control techniques including open-loop methods like retransmission policies and closed-loop methods like backpressure and explicit signaling.
This document provides a status update on Oak Ridge National Laboratory's evaluation of the Cray XT3 supercomputer. It describes the XT3 system architecture including its AMD Opteron processors, SeaStar interconnect, and lightweight operating system. It also summarizes performance results from microbenchmarks, kernels, and applications in areas like climate, biology, astrophysics, combustion, and fusion on up to 4,096 processors, demonstrating the XT3's competitive performance, interconnect bandwidth, and parallel efficiency.
Moolle fan-out control for scalable distributed data storesSungJu Cho
Many Online Social Networks horizontally partition data across data stores. This allows the addition of server nodes to increase capacity and throughput. For single key lookup queries such as computing a member's 1st degree connections, clients need to generate only one request to one data store. However, for multi key lookup queries such as computing a 2nd degree network, clients need to generate multiple requests to multiple data stores. The number of requests to fulfill the multi key lookup queries grows in relation to the number of partitions. Increasing the number of server nodes in order to increase capacity also increases the number of requests between the client and data stores. This may increase the latency of the query response time because of network congestion, tail-latency, and CPU bounding. Replication based partitioning strategies can reduce the number of requests in the multi key lookup queries. However, reducing the number of requests in a query can degrade the performance of certain queries where processing, computing, and filtering can be done by the data stores. A better system would provide the capability of controlling the number of requests in a query. This paper presents Moolle, a system of controlling the number of requests in queries to scalable distributed data stores. Moolle has been implemented in the LinkedIn distributed graph service that serves hundreds of thousands of social graph traversal queries per second. We believe that Moolle can be applied to other distributed systems that handle distributed data processing with a high volume of variable-sized requests.
This document summarizes key aspects of the transport layer:
- The transport layer provides logical communication between application processes running on different hosts and handles reliable data transfer.
- It provides both connection-oriented and connectionless services to the application layer. Quality of service parameters like throughput and delay can be negotiated.
- Transport layer protocols like TCP and UDP are described. TCP provides reliable byte-stream delivery using connections while UDP provides best-effort unreliable datagram delivery.
Transfer reliability and congestion control strategies in opportunistic netwo...IEEEFINALYEARPROJECTS
The document discusses transfer reliability and congestion control strategies in opportunistic networks. It begins by stating that opportunistic networks have unpredictable node contacts and rarely have complete end-to-end paths. It then discusses how modified TCP protocols are ineffective for these networks and they require different approaches than intermittently connected networks. The document surveys proposals for transfer reliability using hop-by-hop custody transfer and end-to-end receipts. It also categorizes storage congestion control based on single or multiple message copies. It identifies open research issues including replication management and drop policies for multiple copies.
Hw09 Hadoop Based Data Mining Platform For The Telecom IndustryCloudera, Inc.
The document summarizes a parallel data mining platform called BC-PDM developed by China Mobile Communication Corporation to address the challenges of analyzing their large scale telecom data. Key points:
- BC-PDM is based on Hadoop and designed to perform ETL and data mining algorithms in parallel to enable scalable analysis of datasets exceeding hundreds of terabytes.
- The platform implements various ETL operations and data mining algorithms using MapReduce. Initial experiments showed a 10-50x speedup over traditional solutions.
- Future work includes improving data security, migrating online systems to the platform, and enhancing the user interface.
This document summarizes a computer physics communications article about the conditions database system for the COMPASS experiment. The key points are:
1) COMPASS integrated a conditions database system to manage time-dependent detector condition, calibration, and geometry alignment information using software from CERN.
2) The conditions database consists of administration tools, a data handling library, and software to transfer data from detector controls to the database.
3) Performance tests on the COMPASS computing farm showed the conditions database system was able to efficiently manage the large volumes of time-dependent experimental data needed for the COMPASS experiment.
Grid computing involves distributing computing resources across a network to tackle large problems. The Worldwide LHC Computing Grid (WLCG) was established to support the Large Hadron Collider (LHC) experiment, which produces around 15 petabytes of data annually. The WLCG uses a four-tiered model, with raw data stored at Tier-0 (CERN), copies distributed to Tier-1 data centers, computational resources provided by Tier-2 centers, and Tier-3 facilities providing additional analysis capabilities. This distributed model has proven effective in supporting the first year of LHC data collection and analysis through globally shared computing resources.
Analytical average throughput and delay estimations for LTESpiros Louvros
This document summarizes an article that appeared in a journal published by Elsevier. The article proposes an analytical model to estimate average throughput and packet transmission delay for uplink cell edge users in LTE networks. The model uses probability analysis and mathematical modeling to estimate transmission delay and throughput, providing cell planners with an analytical tool for evaluating uplink performance under different conditions. The model accounts for factors like scheduling decisions, resource allocation, channel conditions and buffering that impact transmission delay and throughput for cell edge users.
Discriminators for use in flow-based classificationDenis Zuev
This document describes data sets containing network flows that are characterized by discriminators (features) for use in flow-based classification. Each data set contains TCP flows sampled from different periods of a 24-hour network trace. The flows are characterized by features including port numbers, packet timing statistics, TCP header information, and Fourier transform frequency components of packet inter-arrival times. The data sets are provided to allow researchers to assess flow classification techniques.
A Kernel-Level Traffic Probe to Capture and Analyze Data Flows with Prioritiesidescitation
This paper describes the proposal of a priority flow
oriented design of the Ksensor architecture. Ksensor is a
multiprocessor traffic capture and analysis system for high
speed networks developed at kernel space. While the current
architecture permits the capture and analysis of data flows,
there are several scenarios where it does not perform
adequately to achieve this goal, for example, if a certain type
of traffic is more valuable than others. Thus, this work pursues
the design that allows Ksensor to provide data flow treatment
to a larger extent. This improvement will allow the new
architecture to provide more reliability in data flow capture
and processing.
Orchestrating bulk data transfers acrossnexgentech15
Nexgen Technology Address:
Nexgen Technology
No :66,4th cross,Venkata nagar,
Near SBI ATM,
Puducherry.
Email Id: praveen@nexgenproject.com.
www.nexgenproject.com
Mobile: 9751442511,9791938249
Telephone: 0413-2211159.
NEXGEN TECHNOLOGY as an efficient Software Training Center located at Pondicherry with IT Training on IEEE Projects in Android,IEEE IT B.Tech Student Projects, Android Projects Training with Placements Pondicherry, IEEE projects in pondicherry, final IEEE Projects in Pondicherry , MCA, BTech, BCA Projects in Pondicherry, Bulk IEEE PROJECTS IN Pondicherry.So far we have reached almost all engineering colleges located in Pondicherry and around 90km
Orchestrating Bulk Data Transfers across Geo-Distributed Datacentersnexgentechnology
bulk ieee projects in pondicherry,ieee projects in pondicherry,final year ieee projects in pondicherry
Nexgen Technology Address:
Nexgen Technology
No :66,4th cross,Venkata nagar,
Near SBI ATM,
Puducherry.
Email Id: praveen@nexgenproject.com.
www.nexgenproject.com
Mobile: 9751442511,9791938249
Telephone: 0413-2211159.
NEXGEN TECHNOLOGY as an efficient Software Training Center located at Pondicherry with IT Training on IEEE Projects in Android,IEEE IT B.Tech Student Projects, Android Projects Training with Placements Pondicherry, IEEE projects in pondicherry, final IEEE Projects in Pondicherry , MCA, BTech, BCA Projects in Pondicherry, Bulk IEEE PROJECTS IN Pondicherry.So far we have reached almost all engineering colleges located in Pondicherry and around 90km
ORCHESTRATING BULK DATA TRANSFERS ACROSS GEO-DISTRIBUTED DATACENTERSNexgen Technology
bulk ieee projects in pondicherry,ieee projects in pondicherry,final year ieee projects in pondicherry
Nexgen Technology Address:
Nexgen Technology
No :66,4th cross,Venkata nagar,
Near SBI ATM,
Puducherry.
Email Id: praveen@nexgenproject.com.
www.nexgenproject.com
Mobile: 9751442511,9791938249
Telephone: 0413-2211159.
NEXGEN TECHNOLOGY as an efficient Software Training Center located at Pondicherry with IT Training on IEEE Projects in Android,IEEE IT B.Tech Student Projects, Android Projects Training with Placements Pondicherry, IEEE projects in pondicherry, final IEEE Projects in Pondicherry , MCA, BTech, BCA Projects in Pondicherry, Bulk IEEE PROJECTS IN Pondicherry.So far we have reached almost all engineering colleges located in Pondicherry and around 90km
This document introduces a framework for monitoring and visualizing usage of Grid services. The framework collects usage data from sensors integrated with Grid services, stores the data in a database, and provides a portal with JSR168-compliant portlets to visualize the data. The framework has been implemented and integrated with the Globus Toolkit. Usage data is collected from services like GridFTP and stored in a standardized format. The portlets in the portal allow authorized users to query the database and view summaries of Grid usage over specified time periods in graphical or tabular formats.
The ALICE experiment at the LHC requires a data acquisition system capable of handling both frequent small events from proton-proton collisions and rare but large events from heavy ion collisions. The ALICE DAQ system uses over 300 front-end processors to collect data from detectors at up to 2.5 GB/s and store over 1 PB of data per year using a multi-tiered storage architecture. Regular data challenges since 1998 have tested the DAQ system and achieved event building rates over 1 GB/s.
This document summarizes early evaluation results of the Cray XT3 supercomputer installed at Oak Ridge National Laboratory. It describes the system architecture including the AMD Opteron processors, SeaStar interconnect, and Catamount lightweight operating system. It outlines the evaluation approach and benchmarks used, including micro-benchmarks, kernel benchmarks, and full application performance tests in areas like climate and fusion simulations. Initial results show scaling of applications to 4,096 processors and comparisons to other systems like the Cray X1, XD1, SGI Altix, IBM Blue Gene/L, and Earth Simulator.
The document discusses parallel computing platforms and trends in microprocessor architectures that enable implicit parallelism. It covers topics like pipelining, superscalar execution, limitations of memory performance, and how caches can improve effective memory latency. The key points are:
1) Microprocessor clock speeds have increased dramatically but limitations remain regarding memory latency and bandwidth. Parallelism addresses performance bottlenecks in processors, memory, and communication.
2) Techniques like pipelining and superscalar execution exploit implicit parallelism by executing multiple instructions concurrently, but dependencies and branch prediction limit performance gains.
3) Memory latency is often the bottleneck, but caches can reduce effective latency through data reuse and temporal locality.
This document discusses optimizations for TCP/IP networking performance on multicore systems. It describes several inefficiencies in the Linux kernel TCP/IP stack related to shared resources between cores, broken data locality, and per-packet processing overhead. It then introduces mTCP, a user-level TCP/IP stack that addresses these issues through a thread model with pairwise threading, batch packet processing from I/O to applications, and a BSD-like socket API. mTCP achieves a 2.35x performance improvement over the kernel TCP/IP stack on a web server workload.
A flexible phasor data concentrator designsunder fou
This document proposes a flexible phasor data concentrator system called FIPS to address issues with existing synchrophasor systems. FIPS would receive, store, and share synchrophasor data efficiently using open-source software technologies. It describes a flat file database to store synchrophasor data in an ordered fashion for fast retrieval. FIPS would provide a robust foundation for applications using real-time and stored synchrophasor data.
A flexible phasor data concentrator designsunder fou
This document proposes a flexible phasor data concentrator system called FIPS to address issues with existing synchrophasor systems. FIPS would receive, store, and share synchrophasor data efficiently using open-source software technologies. It describes a flat file database to store synchrophasor data in an ordered fashion for fast retrieval. FIPS would provide a robust foundation for applications using real-time and stored synchrophasor data.
ANALYSIS OF LINK STATE RESOURCE RESERVATION PROTOCOL FOR CONGESTION MANAGEMEN...ijgca
With the wide spread of WiFi hotspots, concentrated traffic workload on Smart Web (SW) can slow down the network performance. This paper presents a congestion management strategy considering real time activities in today’s smart web. With the SW context, cooperative packet recovery using resource reservation procedure for TCP flows was adapted for mitigating packet losses. This is to maintain data consistency between various access points of smart web hotspot. Using a real world scenario, it was confirmed that generic TCP cannot handle traffic congestion in a SW hotspot network. With TCP in scalable workload environments, continuous packet drops at the event of congestion remains obvious. This is unacceptable for mission critical domains. An enhanced Link State Resource Reservation Protocol (LSRSVP) which serves as dynamic feedback mechanism in smart web hotspots is presented. The contextual behaviour was contrasted with the generic TCP model. For the LS-RSVP, a simulation experiment for TCP connection between servers at the remote core layer and the access layer was carried out while using selected benchmark metrics. From the results, under realistic workloads, a steady-state throughput response was achieved by TCP LS-RSVP to about 3650Bits/secs compared with generic TCP plots in a previous study. Considering network service availability, this was found to be dependent on fault-tolerance of the hotspot network. From study, a high peak threshold of 0.009 (i.e. 90%) was observed. This shows fairly acceptable service availability behaviour compared with the existing TCP schemes. For packet drop effects, an analysis on the network behaviour with respect to the LS-RSVP yielded a drop response of about 0.000106 bits/sec which is much lower compared with the case with generic TCP with over 0.38 bits/sec. The latency profile of average FTP download response was found to be 0.030secs, but with that of FTP upload response, this yielded about 0.028 sec. The results from the study demonstrate efficiency and optimality for realistic loads in Smart web contexts.
Analysis of Link State Resource Reservation Protocol for Congestion Managemen...ijgca
With the wide spread of WiFi hotspots, concentrated traffic workload on Smart Web (SW) can slow down
the network performance. This paper presents a congestion management strategy considering real time
activities in today’s smart web. With the SW context, cooperative packet recovery using resource
reservation procedure for TCP flows was adapted for mitigating packet losses. This is to maintain data
consistency between various access points of smart web hotspot. Using a real world scenario, it was
confirmed that generic TCP cannot handle traffic congestion in a SW hotspot network. With TCP in
scalable workload environments, continuous packet drops at the event of congestion remains obvious. This
is unacceptable for mission critical domains. An enhanced Link State Resource Reservation Protocol (LSRSVP)
which serves as dynamic feedback mechanism in smart web hotspots is presented. The contextual
behaviour was contrasted with the generic TCP model. For the LS-RSVP, a simulation experiment for TCP
connection between servers at the remote core layer and the access layer was carried out while using
selected benchmark metrics. From the results, under realistic workloads, a steady-state throughput
response was achieved by TCP LS-RSVP to about 3650Bits/secs compared with generic TCP plots in a
previous study. Considering network service availability, this was found to be dependent on fault-tolerance
of the hotspot network. From study, a high peak threshold of 0.009 (i.e. 90%) was observed. This shows
fairly acceptable service availability behaviour compared with the existing TCP schemes. For packet drop
effects, an analysis on the network behaviour with respect to the LS-RSVP yielded a drop response of about
0.000106 bits/sec which is much lower compared with the case with generic TCP with over 0.38 bits/sec.
The latency profile of average FTP download response was found to be 0.030secs, but with that of FTP
upload response, this yielded about 0.028 sec. The results from the study demonstrate efficiency and
optimality for realistic loads in Smart web contexts.
The document provides an overview of parallel computing concepts including:
1) Implicit parallelism in microprocessor architectures has led to techniques like pipelining and superscalar execution to better utilize increasing transistor budgets, though dependencies limit parallelism.
2) Memory latency and bandwidth bottlenecks have shifted performance limitations to the memory system, though caches can improve effective latency through higher hit rates.
3) Communication costs, including startup time, per-hop latency, and per-word transfer time, are a major overhead in parallel programs that use techniques like message passing, packet routing, and cut-through routing to reduce communication costs.
Delivering Application-Layer Traffic Optimization (ALTO) Services based on ...Danny Alex Lachos Perez
Application-Layer Traffic Optimization (ALTO) is an IETF standardized protocol that provides abstract network topology and cost maps in addition to endpoint information services that can be consumed by applications in order to become network-aware and take optimized decisions regarding traffic flows. In this work, we propose a public service based on the ALTO specification using public routing information available at the Brazilian Internet eXchange Points (IXPs). Our ALTO server prototype takes the acronym of AaaS (ALTO-as-a-Service) and is based on over 2.5GB of real BGP data from the 25 Brazilian IX.br public IXPs. We evaluate our proposal in terms of functional behaviour and performance via proof of concept experiments which point to the potential benefits of applications being able to take smart endpoint selection decisions when consuming the developer-friendly ALTO APIs.
This document discusses improving congestion control in data communication networks using queuing theory models. It begins by defining congestion control and distinguishing it from flow control. It then discusses various techniques for congestion detection and avoidance, including using warning bits, choke packets, and traffic shaping algorithms like leaky bucket and token bucket. The document goes on to present queuing theory models like the pure birth-death model that can be used to model congestion control. It describes developing a computational software using these queuing theory concepts to calculate performance parameters and help network engineers with planning. In conclusion, it states that queuing theory models can effectively control congestion in data networks by enabling optimized network planning.
1. Improving File Transfer Service ▬ FTS3
September 2015
Author:
Hamza Zafar
Email: hamza.zafar@cern.ch
Supervisor(s):
Oliver Keeble
Alejandro Alvarez
CERN openlab Summer Student Report 2015
2. CERN openlab Summer Student Report 2015
Project Specification
The Experiments at CERN generate colossal amount of data. The data centre stores it and sends it
around the world for analysis. In the first run of LHC, 30 petabytes of data was produced
annually, larger amounts of data are expected to be produced during the second run of LHC [1].
To store and process this data, CERN relies on a grid infrastructure known as WLCG (Worldwide
LHC Computing Grid), which consists of 170 collaborating computing centres in 42 countries.
One of the major challenges at CERN is to globally replicate and distribute the data coming
colliders across the WLCG infrastructure. To address this problem, a file transfer service (FTS3)
is developed at CERN for bulk transfers of physics data. In this manner, real-time LHC data is
not only distributed and replicated across WLCG, but also made available to a community of
~8000 physicist around the globe.
Improving the file transfer service projects is geared towards effectively utilizing the available
networks resources as well as introducing new algorithms in FTS3 scheduler to make intelligent
decisions for scheduling file transfers.
3. CERN openlab Summer Student Report 2015
Abstract
This project deals with two aspects of improving the file transfer service – FTS3. The first one is
the selection of best source site for file transfers. Since files are replicated at different sites, the
selection of the best source site based on the networks throughput and success rate can have a
major impact on FTS3. The second one is maximizing the file throughput across WLCG network
by increasing the TCP buffer sizes. TCP is the only transport layer protocol used widely for data
transfers; it was originally designed with focus on reliability and long-term fairness. In high
bandwidth networks, the system administrators have to manually optimize/tune the TCP
configurations. Some of these configurations have a major impact on throughput. TCP buffer size
is one such setting, which sets a limit on TCP congestion window size. With the release of Linux
Kernel 2.6, a new feature “Linux TCP Auto-Tuning” was introduced, which selects the optimal
TCP buffer sizes based on system resource usage. Another way to increase the TCP buffer size is
to use setsocketopts system call. Since FTS3 implements gridFTP protocol, it gives us the
flexibility to set TCP buffer sizes manually. This project evaluates the pros and cons of different
techniques for setting TCP buffer sizes.
Keywords: FTS3, TCP , Linux TCP Auto-Tuning
4. CERN openlab Summer Student Report 2015
Table of Contents
Contents
1 Introduction ..............................................................................................................6
1.1 WLCG Architecture ...................................................................................................... 6
1.2 Use Cases of FTS3...................................................................................................... 6
1.3 FTS3 @ CERN ............................................................................................................ 7
2 Improving the FTS3 Scheduler.................................................................................7
2.1 Current Scheduling Behaviour for multiple replicas..................................................... 7
2.2 New Scheduling Behaviour for multiple replicas.......................................................... 8
2.3 Caching Database Queries........................................................................................ 10
3 Effect of TCP configurations on throughput............................................................10
3.1 TCP Optimal Buffer Size............................................................................................ 11
3.2 Performance Evaluation of FTS3 transfers with and without Linux auto-tuning........ 12
4 Bibliography ...........................................................................................................15
5. CERN openlab Summer Student Report 2015
List of Figures
Figure 1: Tiers in WLCG ................................................................................................................ 6
Figure 2: FTS3 workflow................................................................................................................ 7
Figure 3: Transfer request specifying multiple replicas.................................................................. 7
Figure 4: List of scheduling algorithms with their selection criteria............................................... 8
Figure 5: Transfer request specifying selection strategy................................................................. 8
Figure 6: Activity priorities for ATLAS ......................................................................................... 9
Figure 7: FTS3 caching layer ........................................................................................................ 10
Figure 8: Effect of large buffer sizes on throughput ..................................................................... 12
Figure 9: Graph for file transfer when Linux auto-tuning is in action .......................................... 12
Figure 10: Graph for file transfer when manually setting a 16MB Buffer (32 MB allocated)...... 13
Figure 11: Graph for file transfers when Linux auto-tuning is in action....................................... 14
Figure 12: Graph for file transfers when manually setting a 32MB Buffer (64 MB allocated) .... 14
6. CERN openlab Summer Student Report 2015
6 | P a g e
1 Introduction
FTS3 [2] is one of the projects critical for the data management at CERN [3]. It is the major
service for distributing the majority of LHC [4] data across WLCG [5] infrastructure. It provides
reliable bulk transfers of files from one WLCG site to another, while allowing participating sites
to control the network resource usage. FTS3 is a mature service, running for more than 2 years at
CERN.
1.1 WLCG Architecture
WLCG stands for WorldWide LHC Computing Grid; it is a collaboration of more than 170
computing centres in 42 countries. The mission of WLCG is to store, analyse and replicate LHC
data. Figure 1 shows the architecture of WLCG, it consist of three tiers 0, 1 and 2. These tiers are
made up of several computing centres. Tier 0 is the CERN’s datacentre, which is connected to 13
tier 1 sites with 10Gbps links. Tier 2 sites are connected using 1Gbps links. FTS3 service plays a
vital role in moving data across this complex mesh of computing centres.
1.2 Use Cases of FTS3
An individual or small team can access the web interface to FTS to schedule transfers
between storage systems. They can browse the contents of the storage, invoke and
manage transfers, and leave FTS to do the rest.
A team's data manager can use the FTS command line interface to schedule bulk transfers
between storage systems.
A data manager can install an FTS service for local users. The service is equipped with
advanced monitoring and debugging capabilities which enable her to give support to her
users.
Maintainers of frameworks which provide higher level functionality can delegate
responsibility for transfer management to FTS by integrating it using the various
programming interfaces available, including a REST API. The users thus continue to use
a familiar interface while profiting from the power of FTS transfer management.
Figure 1: Tiers in WLCG
7. CERN openlab Summer Student Report 2015
7 | P a g e
1.3 FTS3 @ CERN
Four major experiments at CERN are ATLAS, CMS, LHC(b) and ALICE. The first three
experiments use FTS for file transfer purposes. The figure 2 shows the general flow for using
FTS3 service. The clients --- CMS, ATLAS and LHC(b) --- send file transfer requests to FTS3,
gridFTP [6] protocol is used by FTS3 to initiate third party transfers on storage endpoints. FTS3
supervises the transfers between the storage endpoints and finally archives the job status. fts-
transfer-status command provided by FTS3 command line interface can be used by to inquire
about the job status. Alternatively, clients can also use the FTS3 web interface for transfer
management and monitoring. On average, FTS3 transfers 15 petabytes of data per month.
2 Improving the FTS3 Scheduler
To handle the incoming transfer jobs, FTS3 maintains a separate queue for each link between two
WLCG endpoints. Files are usually replicated across different sites in WLCG. If a client --- Atlas,
CMS, LHC(b) --- submits request for transferring a file with multiple replicas, then FTS3
scheduler is responsible for selecting the best source site. The figure 3 shows a transfer request
specifying source sites for multiple replicas.
2.1 Current Scheduling Behaviour for multiple replicas
For each site, FTS3 database maintains the count of pending files in the queue. It also contains
information about the throughput and success rate from previous transfers. In order to choose the
best replica, FTS3 scheduler queries the number of pending files. The site with the minimum
number of pending files in the queue is chosen as a source site. This approach is not efficient
because the factors like throughput and success rate are completely ignored.
Figure 2: FTS3 workflow
Figure 3: Transfer request specifying multiple replicas
8. CERN openlab Summer Student Report 2015
8 | P a g e
2.2 New Scheduling Behaviour for multiple replicas
We have developed a number of algorithms to address the short comings of FTS3 scheduling
decisions for multiple replicas. In addition to the number of pending files in the queue, the new
algorithms also consider the throughput and success rate when making a scheduling decision. The
algorithms are listed in table below.
Figure 4: List of scheduling algorithms with their selection criteria
FTS3 clients can now mention their selection strategy in the transfer request. In this way clients
can control the behaviour of FTS3 scheduler. The figure 5 shows a transfer request specifying the
selection strategy.
Algorithm Selection Criteria
orderly Selects the first site mentioned in transfer request
queue / auto Selects the source site with the least number of pending files
success Selects the source site with highest success rate
throughput Selects the source site with highest total throughput
file-throughput Selects the source site with highest per file throughput
pending-data Selects the source site with the minimum amount of pending data
waiting-time Selects the source site with the earliest waiting time
waiting-time-with-error Selects the source site with the earliest waiting time with error
duration Selects the source site with the earliest finish time
Figure 5: Transfer request specifying selection strategy
9. CERN openlab Summer Student Report 2015
9 | P a g e
- Calculation of Pending Data:
FTS3 maintains the priority configuration for client's activities. Figure 6 shows the
configurations for Atlas. If a newly submitted job has higher priority than the jobs
waiting in queue, then it takes precedence and the transfer is started immediately.
Therefore, in order to calculate the amount of pending data in queue, we aggregate the
amount of data for all jobs with priorities greater or equal the priority of newly submitted
job.
- Calculation of waiting time:
Waiting time is defined as the time a transfer request spends in the FTS3 queue. The
formula for calculating waiting time is as follow:
𝑊𝑎𝑖𝑡𝑖𝑛𝑔 𝑇𝑖𝑚𝑒 =
𝑃𝑒𝑛𝑑𝑖𝑛𝑔 𝐷𝑎𝑡𝑎
𝑇ℎ𝑟𝑜𝑢𝑔ℎ𝑝𝑢𝑡
- Calculation of waiting time with error:
Error is the predicted amount of data that should be resent in case of transfer failures. It
is as follow:
𝐹𝑎𝑖𝑙𝑢𝑟𝑒 𝑅𝑎𝑡𝑒 = 100 − 𝑆𝑢𝑐𝑐𝑒𝑠𝑠 𝑅𝑎𝑡𝑒
𝐸𝑟𝑟𝑜𝑟 = 𝑃𝑒𝑛𝑑𝑖𝑛𝑔 𝐷𝑎𝑡𝑎 ∗
𝐹𝑎𝑖𝑙𝑢𝑟𝑒 𝑅𝑎𝑡𝑒
100
𝑊𝑎𝑖𝑡𝑖𝑛𝑔 𝑇𝑖𝑚𝑒 𝑊𝑖𝑡ℎ 𝐸𝑟𝑟𝑜𝑟 =
𝑃𝑒𝑛𝑑𝑖𝑛𝑔 𝐷𝑎𝑡𝑎 + 𝐸𝑟𝑟𝑜𝑟
𝑇ℎ𝑟𝑜𝑢𝑔ℎ𝑝𝑢𝑡
Figure 6: Activity priorities for ATLAS
10. CERN openlab Summer Student Report 2015
10 | P a g e
- Calculation of Finish Time:
The algorithm named “duration” ranks the source sites based on total finish time. Finish
time is defined as the time it takes to complete the file transfer; it also includes the
waiting time in FTS3 queue.
𝐹𝑖𝑛𝑖𝑠ℎ 𝑇𝑖𝑚𝑒 = 𝑊𝑎𝑖𝑡𝑖𝑛𝑔 𝑇𝑖𝑚𝑒 𝑊𝑖𝑡ℎ 𝐸𝑟𝑟𝑜𝑟 +
𝑆𝑢𝑏𝑚𝑖𝑡𝑡𝑒𝑑 𝐹𝑖𝑙𝑒 𝑆𝑖𝑧𝑒
𝑃𝑒𝑟 𝐹𝑖𝑙𝑒 𝑇ℎ𝑟𝑜𝑢𝑔ℎ𝑝𝑢𝑡
2.3 Caching Database Queries
The new FTS3 scheduler depends on the number of pending files; throughput and success rate
information stored in FTS3 database, so it makes a lot of queries and eventually increases the
load on database. To overcome this problem, we have implemented a caching layer between
FTS3 scheduler and database. In order to maximize the performance we maintain a separate
cache for each thread. A cache entry expiration timer is configured for each cache entry. Default
time for cache entry expiration is 5 minutes. We also ended up with situations when cache had a
large number of expired entries. Therefore, to free the memory, we added a cache clean-up timer.
When the cache clean-up timer expires, FTS3 caching module deletes all the expired entries from
cache. It should be noted here that the maximum memory the cache could consume is estimated
to be less than a megabyte. The default time for cache clean-up timer is 30 minutes. In the future,
distributed memory caching e.g memcached [7] and Redis cache [8] can also be integrated in the
caching layer.
3 Effect of TCP configurations on throughput
TCP is the only transport layer protocol used widely for data transfers; it was originally designed
with focus on reliability and long-term fairness. In high bandwidth networks, the system
administrators have to manually optimize/tune the TCP configurations. Some of these
configurations have a major impact on throughput. TCP buffer size is one such setting, which
limits the size of TCP congestion window.
Since computer centres in WLCG infrastructure are linked with high bandwidth and high latency
networks. We are focused on effectively utilizing the available network resources. Our end goal is
Figure 7: FTS3 caching layer
11. CERN openlab Summer Student Report 2015
11 | P a g e
to increase the throughput for FTS3 file transfers. In this section we compare and contrast impact
of different TCP tuning methods on the throughput of FTS3 file transfers.
3.1 TCP Optimal Buffer Size
TCP maintains a “congestion window” to determine how many packets can be sent at one time.
This implies that larger the size of congestion window, higher the throughput. The kernel
enforces a limit on the maximum size of TCP congestion window. By default, on most Linux
distributions, the maximum limit is 4MB, which is still very small for high bandwidth links.
System admins can edit the /etc/sysctl.conf file change the default settings. The optimal TCP
buffer size can be calculated using the following formula
𝑂𝑝𝑡𝑖𝑚𝑎𝑙 𝐵𝑢𝑓𝑓𝑒𝑟 𝑆𝑖𝑧𝑒 = 2 ∗ 𝐵𝑎𝑛𝑑𝑤𝑖𝑑𝑡ℎ ∗ 𝑑𝑒𝑙𝑎𝑦
We can use the ping command to calculate the delay between two endpoints. Since ping
command returns the Round Trip Time (RTT), the above formula can be reduced to:
𝑂𝑝𝑡𝑖𝑚𝑎𝑙 𝐵𝑢𝑓𝑓𝑒𝑟 𝑆𝑖𝑧𝑒 = 𝐵𝑎𝑛𝑑𝑤𝑖𝑑𝑡ℎ ∗ 𝑅𝑇𝑇
For example, if the ping time is 200ms and the network bandwidth is 1Gbps, then the optimal
buffer is 25.6MB which is way larger than the default settings.
𝑂𝑝𝑡𝑖𝑚𝑎𝑙 𝐵𝑢𝑓𝑓𝑒𝑟 𝑆𝑖𝑧𝑒 =
1 𝐺𝑏𝑖𝑡
8 𝑏𝑖𝑡𝑠
∗ 0.200𝑠𝑒𝑐 = 25.6 𝑀𝐵
System admins have to add/modify the following setting in /etc/sysctl.conf file:
# increase max memory for sockets to 32MB
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
# increase Linux autotuning TCP buffer limit to 32MB
net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.tcp_wmem = 4096 65536 33554432
Now there are two ways to make use of the increased TCP buffer sizes:
- Linux TCP Auto-Tuning:
Linux TCP auto-tuning automatically adjusts the socket buffer sizes to balance the TCP
performance and system’s memory usage. TCP auto-tuning is enabled by default in
Linux release after version 2.6.6 and 2.4.16. For Linux auto-tuning the maximum send
and receive buffer limit is specified by net.ipv4.tcp_wmem and net.ipv4.tcp_rmem
parameters respectively.
- Manually setting the socket buffer sizes:
Application programmers can mention the socket buffer sizes using the setsocketopts
system call. net.core.wmem_max and net.core.rmem_max parameters impose an upper
limit on the amount of memory requested for send and receive buffers respectively. The
Linux kernel allocates double the amount of memory requested. It should be noted here
that setting the socket buffer sizes manually disables the Linux auto-tuning.
12. CERN openlab Summer Student Report 2015
12 | P a g e
3.2 Performance Evaluation of FTS3 transfers with and without Linux
auto-tuning
During our initial testing, we transferred files from CERN to Australia. Our calculations for the
optimal buffer sizes suggested a 37 MB buffer, whereas the configured maximum TCP buffer
size on our system was 4MB. Therefore, we increased the maximum buffer limit to 37 MB and
calculated the throughput. The Figure 8 shows a plot of throughput with system configured with a
4MB and a 37 MB TCP buffer. It should be noted here that the buffer sizes are not passed to fts-
transfer-submit (buffer sizes are not set using setsocketopt system call) command, in fact TCP
auto-tuning is taking care of socket memory allocation. It is evident that increasing the default
limits on maximum TCP buffer sizes, the congestion window can open more, which results in
higher throughput.
Figure 8: Effect of large buffer sizes on throughput
Now the question arises, should FTS3 rely on Linux auto-tuning or the users should pass the
buffer size to fts-transfer-submit command?
To answer this question, we transferred files from CERN to Tokyo. The number of streams was
set to 1. The endpoints at both sites were configured with 32 MB optimal buffer size. Receiver’s
advertised window and CNWD sizes were recorded from tcpdump and Linux ss utility
respectively. Figure 9 shows the results when Linux auto-tuning is in action and Figure 10 shows
the results by passing the buffer size to fts-transfer-submit command. When we manually set the
buffer sizes, Linux allocates twice the amount requested. Therefore, for a fair comparison with
auto-tuning, we request half the amount of memory available for Linux auto tuning i.e 32MB.
0
20
40
60
1
17
33
49
65
81
97
113
129
145
161
177
193
209
225
241
257
273
289
305
321
337
353
369
385
401
417
433
449
Throughput
(MB/sec)
Time (seconds)
4 MB Buffer 37MB Buffer
0
20
40
60
80
100
1 3 5 7 9 111315171921232527293133353739414345474951535557596163
DataMB
Time(seconds)
Rcv-window Throughput CWND
Figure 9: Graph for file transfer when Linux auto-tuning is in action
13. CERN openlab Summer Student Report 2015
13 | P a g e
0
20
40
60
80
100
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59
DataMB
Time (seconds)
Rcv-window Throughput CWND
With Linux auto-tuning the transfer time is 63 seconds, whereas the transfer time by manually
setting the buffer sizes is 59 seconds. Since both transfers reach the same maximum throughput,
we can conclude that manually setting the buffer sizes has no advantage over Linux auto-tuning.
Auto-Tuning is a much safer option to use as compared to manually setting the buffer sizes
because it can dynamically resize the TCP buffers based on network performance and system
resource usage.
We now shift our focus on comparing the effect of using multiple streams (with and without auto-
tuning). We transferred files from CERN to Tokyo with multiple numbers of streams. Figure 11
shows a graph when Linux auto-tuning is in action. Figure 12 shows the graph of transfers when
we are manually setting buffer sizes to 32MB. If we compare the file transfer with 1 stream (auto-
tuning vs manually setting buffer), we achieve higher throughput for manually setting buffer.
This is due to the fact that auto-tuning can increase the CNWD up to 32MB whereas when setting
the buffer size to 32MB , kernel allocates 64MB, then CNWD can increase up to 64MB. For 2
and 4 number of streams the throughput is almost the same (with and without auto-tuning). It is
also evident from the graphs that increasing the number of streams fills the pipe more quickly.
Figure 10: Graph for file transfer when manually setting a 16MB Buffer (32 MB allocated)
0
50
100
150
1
5
9
13
17
21
25
29
33
37
41
45
49
53
57
61
65
69
73
77
81
85
89
93
97
101
105
109
ThroughputMB/sec
Time(seconds)
1-Stream 2-Streams 4-Streams
14. CERN openlab Summer Student Report 2015
14 | P a g e
Figure 11: Graph for file transfers when Linux auto-tuning is in action
Figure 12: Graph for file transfers when manually setting a 32MB Buffer (64 MB allocated)
With this work we conclude that the operating system’s configured maximum buffer sizes are too
small for WLCG’s high bandwidth network, the kernel enforced limits on TCP buffer sizes
should be increased. Since, FTS3 supports 3rd
party file transfers, there is currently no mechanism
to get the RTTs by remotely pinging two storage endpoints, and hence there is no possibility of
calculating the bandwidth delay product, unless we have the access to the storage endpoint. In the
future, we would be able to get the RTT information from WLCG perfSONAR project [9]. We
have also seen that there is no difference on throughput whether we use Linux auto-tuning or set
the buffer sizes explicitly. All we have to do is to increase the maximum TCP buffer size on
storage endpoints and let the Linux auto-tuning decide optimal buffer sizes.
Future work includes the support of distributed memory caching techniques for FTS3 caching
layer and the performance evaluation of single file transfer with multiple streams vs. multiple file
transfers with single stream.
0
50
100
150
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88
ThroughputMB/sec
Time (seconds)
1-Stream 2-Stream 4-Stream
15. CERN openlab Summer Student Report 2015
15 | P a g e
4 Bibliography
[1] LHC Season 2. [Online]. http://home.web.cern.ch/about/updates/2015/06/lhc-season-2-first-
physics-13-tev-start-tomorrow
[2] M Salichos, M K Simon, O Keeble A A Ayllon, "FTS - New Data Movement Service for
WLCG".
[3] CERN. [Online]. http://home.web.cern.ch/
[4] Large Hadron Collider. [Online]. http://home.web.cern.ch/topics/large-hadron-collider
[5] WorldWide LHC Computing Grid. [Online]. wlcg.web.cern.ch
[6] gridFTP. [Online]. https://en.wikipedia.org/wiki/GridFTP
[7] Memchached. [Online]. http://memcached.org/
[8] Redis. [Online]. http://redis.io/
[9] WLCG perSONAR. [Online]. http://maddash.aglt2.org/maddash-webui/