1. The Trans-Pacific Grid Datafarm testbed provides 70 terabytes of disk capacity and 13 gigabytes per second of disk I/O performance across clusters in Japan, the US, and Thailand.
2. Using the GNET-1 network testbed device, the Trans-Pacific Grid Datafarm achieved stable transfer rates of up to 3.79 gigabits per second during a file replication experiment between Japan and the US, near the theoretical maximum of 3.9 gigabits per second.
3. Precise pacing of network traffic flows using inter-frame gap controls on the GNET-1 device allowed for high-speed, lossless utilization of long-haul trans-Pacific network links.
High Performance Cyberinfrastructure Enabling Data-Driven Science in the Biom...Larry Smarr
11.04.06
Joint Presentation
UCSD School of Medicine Research Council
Larry Smarr, Calit2 & Phil Papadopoulos, SDSC/Calit2
Title: High Performance Cyberinfrastructure Enabling Data-Driven Science in the Biomedical Sciences
Using Photonics to Prototype the Research Campus Infrastructure of the Future...Larry Smarr
08.02.21
Presentation
Philip Papadopoulos, Larry Smarr, Joseph Ford, Shaya Fainman, and Brian Dunne
University of California, San Diego
Title: Using Photonics to Prototype the Research Campus Infrastructure of the Future: The UCSD Quartzite Project
La Jolla, CA
High Performance Cyberinfrastructure Enabling Data-Driven Science in the Biom...Larry Smarr
11.04.06
Joint Presentation
UCSD School of Medicine Research Council
Larry Smarr, Calit2 & Phil Papadopoulos, SDSC/Calit2
Title: High Performance Cyberinfrastructure Enabling Data-Driven Science in the Biomedical Sciences
Using Photonics to Prototype the Research Campus Infrastructure of the Future...Larry Smarr
08.02.21
Presentation
Philip Papadopoulos, Larry Smarr, Joseph Ford, Shaya Fainman, and Brian Dunne
University of California, San Diego
Title: Using Photonics to Prototype the Research Campus Infrastructure of the Future: The UCSD Quartzite Project
La Jolla, CA
In this video from the 2017 Argonne Training Program on Extreme-Scale Computing, Pavan Balaji from Argonne presents an overview of system interconnects for HPC.
Watch the video: https://wp.me/p3RLHQ-hA4
Learn more: https://extremecomputingtraining.anl.gov/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
High Performance Cyberinfrastructure Enables Data-Driven Science in the Globa...Larry Smarr
10.10.28
Invited Speaker
Grand Challenges in Data-Intensive Discovery Conference
San Diego Supercomputer Center, UC San Diego
Title: High Performance Cyberinfrastructure Enables Data-Driven Science in the Globally Networked World
La Jolla, CA
Marco Cattaneo "Event data processing in LHCb"Yandex
Семинар «Использование современных информационных технологий для решения современных задач физики частиц» в московском офисе Яндекса, 3 июля 2012
Marco Cattaneo, CERN
Scale-out AI Training on Massive Core System from HPC to Fabric-based SOCinside-BigData.com
In this video from Arm HPC Asia 2019, Fu Li from Quantum Cloud presents: Scale out AI Training on Massive Core System from HPC to Fabric based SOC.
"The purpose of these workshops has been to bring together the leading Arm vendors, end users and open source development community to discuss the latest products, developments and open source software support in HPC on Arm."
Learn more: https://www.linaro.org/events/workshop/arm-hpc-asia-2019/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Review of CERN's objectives and how the computing infrastructure is evolving to address the challenges at scale using community supported software such as Puppet and OpenStack.
40 Powers of 10 - Simulating the Universe with the DiRAC HPC Facilityinside-BigData.com
In this deck from the Swiss HPC Conference, Mark Wilkinson presents: 40 Powers of 10 - Simulating the Universe with the DiRAC HPC Facility.
"DiRAC is the integrated supercomputing facility for theoretical modeling and HPC-based research in particle physics, and astrophysics, cosmology, and nuclear physics, all areas in which the UK is world-leading. DiRAC provides a variety of compute resources, matching machine architecture to the algorithm design and requirements of the research problems to be solved. As a single federated Facility, DiRAC allows more effective and efficient use of computing resources, supporting the delivery of the science programs across the STFC research communities. It provides a common training and consultation framework and, crucially, provides critical mass and a coordinating structure for both small- and large-scale cross-discipline science projects, the technical support needed to run and develop a distributed HPC service, and a pool of expertise to support knowledge transfer and industrial partnership projects. The on-going development and sharing of best-practice for the delivery of productive, national HPC services with DiRAC enables STFC researchers to produce world-leading science across the entire STFC science theory program."
Watch the video: https://wp.me/p3RLHQ-k94
Learn more: https://dirac.ac.uk/
and
http://hpcadvisorycouncil.com/events/2019/swiss-workshop/agenda.php
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
How to Terminate the GLIF by Building a Campus Big Data Freeway SystemLarry Smarr
12.10.11
Keynote Lecture
12th Annual Global LambdaGrid Workshop
Title: How to Terminate the GLIF by Building a Campus Big Data Freeway System
Chicago, IL
ASCI Terascale Simulation Requirements and DeploymentsGlenn K. Lockwood
Presented at the Oak Ridge Interconnects Workshop in 1999. A fun historical perpsective on where the HPC industry in 1999 thought we would be going forward into the petascale industry.
In this video from the 2017 Argonne Training Program on Extreme-Scale Computing, Pavan Balaji from Argonne presents an overview of system interconnects for HPC.
Watch the video: https://wp.me/p3RLHQ-hA4
Learn more: https://extremecomputingtraining.anl.gov/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
High Performance Cyberinfrastructure Enables Data-Driven Science in the Globa...Larry Smarr
10.10.28
Invited Speaker
Grand Challenges in Data-Intensive Discovery Conference
San Diego Supercomputer Center, UC San Diego
Title: High Performance Cyberinfrastructure Enables Data-Driven Science in the Globally Networked World
La Jolla, CA
Marco Cattaneo "Event data processing in LHCb"Yandex
Семинар «Использование современных информационных технологий для решения современных задач физики частиц» в московском офисе Яндекса, 3 июля 2012
Marco Cattaneo, CERN
Scale-out AI Training on Massive Core System from HPC to Fabric-based SOCinside-BigData.com
In this video from Arm HPC Asia 2019, Fu Li from Quantum Cloud presents: Scale out AI Training on Massive Core System from HPC to Fabric based SOC.
"The purpose of these workshops has been to bring together the leading Arm vendors, end users and open source development community to discuss the latest products, developments and open source software support in HPC on Arm."
Learn more: https://www.linaro.org/events/workshop/arm-hpc-asia-2019/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Review of CERN's objectives and how the computing infrastructure is evolving to address the challenges at scale using community supported software such as Puppet and OpenStack.
40 Powers of 10 - Simulating the Universe with the DiRAC HPC Facilityinside-BigData.com
In this deck from the Swiss HPC Conference, Mark Wilkinson presents: 40 Powers of 10 - Simulating the Universe with the DiRAC HPC Facility.
"DiRAC is the integrated supercomputing facility for theoretical modeling and HPC-based research in particle physics, and astrophysics, cosmology, and nuclear physics, all areas in which the UK is world-leading. DiRAC provides a variety of compute resources, matching machine architecture to the algorithm design and requirements of the research problems to be solved. As a single federated Facility, DiRAC allows more effective and efficient use of computing resources, supporting the delivery of the science programs across the STFC research communities. It provides a common training and consultation framework and, crucially, provides critical mass and a coordinating structure for both small- and large-scale cross-discipline science projects, the technical support needed to run and develop a distributed HPC service, and a pool of expertise to support knowledge transfer and industrial partnership projects. The on-going development and sharing of best-practice for the delivery of productive, national HPC services with DiRAC enables STFC researchers to produce world-leading science across the entire STFC science theory program."
Watch the video: https://wp.me/p3RLHQ-k94
Learn more: https://dirac.ac.uk/
and
http://hpcadvisorycouncil.com/events/2019/swiss-workshop/agenda.php
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
How to Terminate the GLIF by Building a Campus Big Data Freeway SystemLarry Smarr
12.10.11
Keynote Lecture
12th Annual Global LambdaGrid Workshop
Title: How to Terminate the GLIF by Building a Campus Big Data Freeway System
Chicago, IL
ASCI Terascale Simulation Requirements and DeploymentsGlenn K. Lockwood
Presented at the Oak Ridge Interconnects Workshop in 1999. A fun historical perpsective on where the HPC industry in 1999 thought we would be going forward into the petascale industry.
The Missing Link: Dedicated End-to-End 10Gbps Optical Lightpaths for Clusters...Larry Smarr
11.05.24
Invited Keynote Presentation
11th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing
Title: The Missing Link: Dedicated End-to-End 10Gbps Optical Lightpaths for Clusters, Grids, and Clouds
Newport Beach, CA
'Omics in extreme Environments (Lightweight bioinformatics)Joe Parker
Presentation on lightweight bioinformatics (Raspi / cloud computing) for real-time field-based analyses.
Presented at iEOS2015, St. Andrews, 3-6th July 2015.
Project StarGate An End-to-End 10Gbps HPC to User Cyberinfrastructure ANL * C...Larry Smarr
09.11.03
Report to the
Dept. of Energy Advanced Scientific Computing Advisory Committee
Title: Project StarGate An End-to-End 10Gbps HPC to User Cyberinfrastructure ANL * Calit2 * LBNL * NICS * ORNL * SDSC
Oak Ridge, TN
Seven years ago at LCA, Van Jacobsen introduced the concept of net channels but since then the concept of user mode networking has not hit the mainstream. There are several different user mode networking environments: Intel DPDK, BSD netmap, and Solarflare OpenOnload. Each of these provides higher performance than standard Linux kernel networking; but also creates new problems. This talk will explore the issues created by user space networking including performance, internal architecture, security and licensing.
Alexis Dacquay – is CCIE with over 10 years experience in the networking industry. He has in the past been designing, deploying, and supporting some large corporate LAN/WAN networks. He has in the last 4 years specialised in high performance datacenter networking to satisfy the needs of cloud providers, web2.0, big data, HPC, HFT, and any other enterprise for which high performing network is critical to their business. Originally from Bretagne, privately a huge fan of polish cuisine.
Topic of Presentation: Handling high-bandwidth-consumption applications in a modern DC design
Language: English
Abstract: Modern Data Centre requires proper handling of high-bandwidth consuming applications, like BigData or IP Storage. To achieve this, next generation Ethernet speeds of 25, 50 and 100Gbps are being pursued. We are to show _why_ these new Ethernet speeds are vital from technology standpoint and _how_ to cope with the those sparkling new requirements by networking hardware enablements. We are to share ethernet switches’ design considerations, with the biggest emphasis put on the importance of big buffers and how they accommodate this bursty traffic. Throughout the presentation we will additionally elaborate on the evolution of variety of modern applications, and how we can handle those with the properly designed hardware, software, and Data Centre itself.
Analyzing Data Movements and Identifying Techniques for Next-generation Networksbalmanme
Jan 28th, 2013 - 10:00 am
UC Davis
Title: Analyzing Data Movements and Identifying Techniques for Next-generation Networks
Abstract: Large bandwidth provided by today’s networks requires careful evaluation in order to eliminate system overheads and to bring anticipated high performance to the application layer. As a part of the Advance Network Initiative (ANI) project, we have conducted a large number of experiments in the initial evaluation of the 100Gbps network prototype.
We needed intense fine-tuning, both in network and application layers, to take advantage of the higher network capacity. Instead of explicit improvements in every application as we keep changing the underlying link technology, we require novel data movement mechanisms and abstract layers for end-to-end processing of data. Based on our experience in 100Gbps network, we have developed an experimental prototype, called MemzNet: Memory-mapped Zero-copy Network Channel. MemzNet def ines new data access methods in which applications map memory blocks for remote data, in contrast to the send/receive semantics. In one of the early demonstrations of 100Gbps network applications, we used the initial implementation of MemzNet that takes the approach of aggregating files into blocks and providing dynamic data channel management. We observed that MemzNet showed better results in terms of performance and efficiency,
than the current state-of-the-art file-centric data transfer tools for the transfer of climate datasets with many small files. In this talk, I will mainly describe our experience in 100Gbps tests and present results from the 100Gbps demonstration. I will briefly explain the ANI testbed environment and highlight future research plans.
Bio: Mehmet Balman is a researcher working as a computer engineer in the Computational Research Division at Lawrence Berkeley National Laboratory. His recent work
particularly deals with efficient data transfer mechanisms, high-performance network protocols, bandwidth reservation, network virtualization, scheduling and resource management for large-scale applications. He received his doctoral degree in computer science from Louisiana State University (LSU) in 2010. He has several years of industrial experience as system administrator and R&D specialist, at various software companies before joining LSU. He also worked as a summer intern in Los Alamos National Laboratory.
An End-to-End Campus-Scale High Performance Cyberinfrastructure for Data-Inte...Larry Smarr
12.04.19
The Annual Robert Stewart Distinguished Lecture
Iowa State University
Title: An End-to-End Campus-Scale High Performance Cyberinfrastructure for Data-Intensive Research
Ames, IA
Slides from the High Performance Cloud Computing tutorial at Supercomputing 2011 in Seattle. Additional materials available from: cloudsupercomputing.net.
PostgreSQL is very differently architected and presents none
of these problems. All PostgreSQL operations are multi-versioned using
Multi-Version Concurrency Control (MVCC). As a result, common
operations such as re-indexing, adding or dropping columns, and
recreating views can be performed online and without excessive locking,
Lesson 9 - Resisting Temptation Along the Way.pptxCelso Napoleon
Lesson 9 - Resisting Temptation Along the Way
SBs – Sunday Bible School
Adult Bible Lessons 2nd quarter 2024 CPAD
MAGAZINE: THE CAREER THAT IS PROPOSED TO US: The Path of Salvation, Holiness and Perseverance to Reach Heaven
Commentator: Pastor Osiel Gomes
Presentation: Missionary Celso Napoleon
Renewed in Grace
HANUMAN STORIES: TIMELESS TEACHINGS FOR TODAY’S WORLDLearnyoga
Hanuman Stories: Timeless Teachings for Today’s World" delves into the inspiring tales of Hanuman, highlighting lessons of devotion, strength, and selfless service that resonate in modern life. These stories illustrate how Hanuman's unwavering faith and courage can guide us through challenges and foster resilience. Through these timeless narratives, readers can find profound wisdom to apply in their daily lives.
The PBHP DYC ~ Reflections on The Dhamma (English).pptxOH TEIK BIN
A PowerPoint Presentation based on the Dhamma Reflections for the PBHP DYC for the years 1993 – 2012. To motivate and inspire DYC members to keep on practicing the Dhamma and to do the meritorious deed of Dhammaduta work.
The texts are in English.
For the Video with audio narration, comments and texts in English, please check out the Link:
https://www.youtube.com/watch?v=zF2g_43NEa0
What Should be the Christian View of Anime?Joe Muraguri
We will learn what Anime is and see what a Christian should consider before watching anime movies? We will also learn a little bit of Shintoism religion and hentai (the craze of internet pornography today).
The Chakra System in our body - A Portal to Interdimensional Consciousness.pptxBharat Technology
each chakra is studied in greater detail, several steps have been included to
strengthen your personal intention to open each chakra more fully. These are designed
to draw forth the highest benefit for your spiritual growth.
Exploring the Mindfulness Understanding Its Benefits.pptxMartaLoveguard
Slide 1: Title: Exploring the Mindfulness: Understanding Its Benefits
Slide 2: Introduction to Mindfulness
Mindfulness, defined as the conscious, non-judgmental observation of the present moment, has deep roots in Buddhist meditation practice but has gained significant popularity in the Western world in recent years. In today's society, filled with distractions and constant stimuli, mindfulness offers a valuable tool for regaining inner peace and reconnecting with our true selves. By cultivating mindfulness, we can develop a heightened awareness of our thoughts, feelings, and surroundings, leading to a greater sense of clarity and presence in our daily lives.
Slide 3: Benefits of Mindfulness for Mental Well-being
Practicing mindfulness can help reduce stress and anxiety levels, improving overall quality of life.
Mindfulness increases awareness of our emotions and teaches us to manage them better, leading to improved mood.
Regular mindfulness practice can improve our ability to concentrate and focus our attention on the present moment.
Slide 4: Benefits of Mindfulness for Physical Health
Research has shown that practicing mindfulness can contribute to lowering blood pressure, which is beneficial for heart health.
Regular meditation and mindfulness practice can strengthen the immune system, aiding the body in fighting infections.
Mindfulness may help reduce the risk of chronic diseases such as type 2 diabetes and obesity by reducing stress and improving overall lifestyle habits.
Slide 5: Impact of Mindfulness on Relationships
Mindfulness can help us better understand others and improve communication, leading to healthier relationships.
By focusing on the present moment and being fully attentive, mindfulness helps build stronger and more authentic connections with others.
Mindfulness teaches us how to be present for others in difficult times, leading to increased compassion and understanding.
Slide 6: Mindfulness Techniques and Practices
Focusing on the breath and mindful breathing can be a simple way to enter a state of mindfulness.
Body scan meditation involves focusing on different parts of the body, paying attention to any sensations and feelings.
Practicing mindful walking and eating involves consciously focusing on each step or bite, with full attention to sensory experiences.
Slide 7: Incorporating Mindfulness into Daily Life
You can practice mindfulness in everyday activities such as washing dishes or taking a walk in the park.
Adding mindfulness practice to daily routines can help increase awareness and presence.
Mindfulness helps us become more aware of our needs and better manage our time, leading to balance and harmony in life.
Slide 8: Summary: Embracing Mindfulness for Full Living
Mindfulness can bring numerous benefits for physical and mental health.
Regular mindfulness practice can help achieve a fuller and more satisfying life.
Mindfulness has the power to change our perspective and way of perceiving the world, leading to deeper se
The Good News, newsletter for June 2024 is hereNoHo FUMC
Our monthly newsletter is available to read online. We hope you will join us each Sunday in person for our worship service. Make sure to subscribe and follow us on YouTube and social media.
The Book of Joshua is the sixth book in the Hebrew Bible and the Old Testament, and is the first book of the Deuteronomistic history, the story of Israel from the conquest of Canaan to the Babylonian exile.
In Jude 17-23 Jude shifts from piling up examples of false teachers from the Old Testament to a series of practical exhortations that flow from apostolic instruction. He preserves for us what may well have been part of the apostolic catechism for the first generation of Christ-followers. In these instructions Jude exhorts the believer to deal with 3 different groups of people: scoffers who are "devoid of the Spirit", believers who have come under the influence of scoffers and believers who are so entrenched in false teaching that they need rescue and pose some real spiritual risk for the rescuer. In all of this Jude emphasizes Jesus' call to rescue straying sheep, leaving the 99 safely behind and pursuing the 1.
Evangelization in the footsteps of Saint Vincent de Paul
Gfarm Fs Tatebe Tip2004
1. Joint Techs Workshop, TIP 2004
Jan 28, 2004
Honolulu, Hawaii
Trans-Pacific Grid Datafarm
Osamu Tatebe
Grid Technology Research Center, AIST
On behalf of the Grid Datafarm Project
National Institute of Advanced Industrial Science and Technology
2. Key points of this talk
Trans-pacific Grid file system and testbed
70 TBytes disk capacity, 13 GB/sec disk I/O performance
Trans-pacific file replication [SC2003 Bandwidth Challenge]
1.5TB data transferred in an hour
Multiple high-speed Trans-Pacific networks;
APAN/TransPAC (2.4 Gbps OC48 POS, 500 Mbps OC-12
ATM), SuperSINET (2.4 Gbps x 2, 1 Gbps available)
6,000 miles
stable 3.79 Gbps out of theoretical peak 3.9 Gbps (97%)
using 11 node pairs (MTU 6000B)
We won the "Distributed Infrastructure" award!
National Institute of Advanced Industrial Science and Technology
3. [Background] Petascale Data Intensive
Computing
High Energy Physics
CERN LHC, KEK Belle
~MB/collision,
100 collisions/sec Detector for
LHCb experiment
~PB/year
2000 physicists, 35 countries
Detector for
ALICE experiment
Astronomical Data Analysis
data analysis of whole the data
TB~PB/year/telescope
SUBARU telescope
10 GB/night, 3 TB/year
National Institute of Advanced Industrial Science and Technology
4. [Background 2] Large-scale File Sharing
P2P – exclusive and special-purpose approach
Napster, Gnutella, Freenet, . . .
Grid technology – file transfer, metadata management
GridFTP, Replica Location Service
Storage Resource Broker (SRB)
Large-scale file system – general approach
Legion, Avaki [Grid, no replica management]
Grid Datafarm [Grid]
Farsite, OceanStore [P2P]
AFS, DFS, . . .
National Institute of Advanced Industrial Science and Technology
5. Goal and feature of Grid Datafarm
Goal
Dependable data sharing among multiple organizations
High-speed data access, High-speed data processing
Grid Datafarm
Grid File System – Global dependable virtual file system
Integrates CPU + storage
Parallel & distributed data processing
Features
Secured based on Grid Security Infrastructure
Scalable depending on data size and usage scenarios
Data location transparent data access
Automatic and transparent replica access for fault tolerance
High-performance data access and processing by accessing multiple
dispersed storages in parallel (file affinity scheduling)
National Institute of Advanced Industrial Science and Technology
6. Grid Datafarm (1): Gfarm file system -
World-wide virtual file system [CCGrid 2002]
Transparent access to dispersed file data in a Grid
POSIX I/O APIs, and native Gfarm APIs for extended file
view semantics and replications
Map from virtual directory tree to physical file
Automatic and transparent replica access for fault
tolerance and access-concentration avoidance
Virtual Directory /grid
Tree File system metadata
ggf jp
aist gtrc file1 file2
mapping
file1 file2 file3 file4
File replica creation
Gfarm File System
National Institute of Advanced Industrial Science and Technology
7. Grid Datafarm (2): High-performance data access
and processing support [CCGrid 2002]
World-wide parallel and distributed processing
Aggregate of files = superfile
Data processing of superfiles = parallel and
distributed data processing of member files
Local file view (SPMD parallel file access)
File-affinity scheduling (“Owner-computes”)
World-wide
Virtual CPU Parallel &
distributed
processing
Grid File System Astronomic archival data
365 parallel analysis
in a year (superfile)
National Institute of Advanced Industrial Science and Technology
8. Transfer technology in long fat networks
Bandwidth and latency between US and Japan
1 10 Gbps, 150 300 msec in RTT
TCP acceleration
Adjustment of congestion window
Multiple TCP connections
HighSpeed TCP、Scalable TCP、FAST TCP
XCP (not TCP)
UDP based acceleration
Tsunami、UDT、RBUDP、atou、. . .
Bandwidth prediction without packet loss
National Institute of Advanced Industrial Science and Technology
9. Multiple TCP streams sometimes
considered harmful . . .
Multiple TCP streams achieve good bandwidth, but
excessively congest the network. In fact would
“shoot oneself in the foot”.
APAN/TransPAC LA-Tokyo (2.4Gbps)
2800
Too much
High oscillation
2600 congestion
Not stable! 2400
2200
2000
Bandwidth (Mbp
1800
1600 TxTotal
TxBW0
1400
TxBW1
1200
TxBW2
1000
Compensate 800
each other 600
400
200
0
375.5 Too much 377
376 376.5 network flow
377.5 378
Time (seconds)
[10 msec average]
Need to limit bandwidth appropriately
National Institute of Advanced Industrial Science and Technology
10. A programmable network testbed device
GNET-1
Large high-speed
memory blocks
Programmable hardware
network testbed
WAN emulation
- latency, bandwidth,
packet loss, jitter, . . .
Precise measurement
- bandwidth in 100 usec
- latency, jitter between 2 GNET-1
General purpose, very flexible!
National Institute of Advanced Industrial Science and Technology
11. IFG-based pace control by GNET-1
Shaping by GNET-1 (700Mbps x 3 @ APAN LA-Tokyo(2.4Gbps))
Shaping by GNET-1 (700Mbps x 3 @ APAN LA-Tokyo(2.4Gbps)) Shaping by GNET-1 (700Mbps x 3 @ APAN LA-Tokyo(2.4Gbps))
1000
1000 1000
900
900 900
800
800 800
700
700 700
Bandwidth (Mb
Bandwidth (Mbp
Bandwidth (Mbp
600
600 600
RxBW0 TxBW0 TxBW0
500 TxBW1 TxBW1
RxBW1 500 500
400 TxBW2 TxBW2
400 400
300 300 300
200 200 200
100 100 100
0 0 0
245.5 246 246.5 247 245.5 246 246.5 247 245.5 246 246.5 247
Time (Second) Time (Second) Time (Second)
GNET-1 Bottleneck
1 Gbps
700 Mbps 700 Mbps
(enable flow control)
NO PACKET LOSS!
GNET-1 provides
Precise traffic pacing at any data rate by
changing IFG (Inter-Frame Gap)
Packet loss free network using large input buffer
(16MB)
National Institute of Advanced Industrial Science and Technology
12. Summary of technologies for performance
improvement
[Disk I/O performance] Grid Datafarm – A Grid file system with high-
performance data-intensive computing support
A world-wide virtual file system that federates local file systems of
multiple clusters
It provides scalable disk I/O performance for file replication via high-
speed network links and large-scale data-intensive applications
Trans-Pacific Grid Datafarm testbed
5 clusters in Japan, 3 clusters in US, and 1 cluster in Thailand, provides 70
TBytes disk capacity, 13 GB/sec disk I/O performance
It supports file replication for fault tolerance and access-concentration
avoidance
[World-wide high-speed network efficient utilization] GNET-1 – a gigabit
network testbed device
Provides IFG-based precise rate-controlled flow at any rate
Enables stable and efficient Trans-Pacific network use of HighSpeed
TCP
National Institute of Advanced Industrial Science and Technology
13. Trans-Pacific Grid Datafarm testbed:
Network and cluster configuration
SuperSINET Trans-Pacific thoretical peak 3.9 Gbps Indiana
Gfarm disk capacity 70 TBytes Univ
Titech
disk read/write 13 GB/sec
147 nodes
16 TBytes 10G SuperSINET
4 GB/sec SC2003
Univ 2.4G
Tsukuba NII New 2.4G Phoenix
10 nodes 10G York
1 TBytes 2.4G(1G)
300 MB/sec 10G [950 Mbps]
Abilene
Abilene
KEK [500 Mbps]
7 nodes 1G OC-12 ATM
3.7 TBytes 622M Chicago
200 MB/sec
APAN 1G 10G
Maffin Tokyo XP
APAN/TransPAC
1G 1G
32 nodes
AIST 5G 2.4G Los Angeles
[2.34 Gbps] 23.3 TBytes
10G
Tsukuba SDSC 2 GB/sec
16 nodes 16 nodes Kasetsart
WAN
11.7 TBytes 11.7 TBytes Univ,
1 GB/sec Thailand
1 GB/sec
National Institute of Advanced Industrial Science and Technology
14. Scientific Data for Bandwidth Challenge
Trans-Pacific File Replication of scientific data
For transparent, high-performance, and fault-tolerant access
Astronomical Object Survey on Grid Datafarm [HPC Challenge participant]
World-wide data analysis on whole the archive
652 GBytes data observed by SUBARU telescope
N. Yamamoto (AIST)
Large configuration data from Lattice QCD
Three sets of hundreds of gluon field configurations on a 24^3*48 4-D
space-time lattice (3 sets x 364.5 MB x 800 = 854.3 GB)
Generated by the CP-PACS parallel computer at Center for
Computational Physics, Univ. of Tsukuba (300Gflops x years of CPU
time) [Univ Tsukuba Booth]
National Institute of Advanced Industrial Science and Technology
15. Network bandwidth in APAN/TransPAC
LA route
PC RTT: 141 ms PC
switch switch
PC 3G 3G PC
10G 2.4G
PC FC10 Juniper PC
switch router switch
PC E600 M20 PC
PC switch LA Tokyo switch PC
GNET-1 Stable transfer rate of 2.3 Gbps
2
[Gbps]
1
No pacing Pacing in 2.3 Gbps
(900 + 900 + 500)
National Institute of Advanced Industrial Science and Technology
16. APAN/TransPAC LA route (1)
National Institute of Advanced Industrial Science and Technology
17. APAN/TransPAC LA route (2)
National Institute of Advanced Industrial Science and Technology
18. APAN/TransPAC LA route (3)
National Institute of Advanced Industrial Science and Technology
19. File replication between Japan and US
(network configuration)
PC RTT: 141 ms PC
switch switch
PC 10G PC
3G 3G
PC PC
switch LA Tokyo switch
PC 2.4G PC
PC router PC
Abilene
Abilene
switch Juniper switch
PC FC10 PC
M20
E600
PC PC
switch router switch
PC 500M 1G PC
1G Chicago RTT: 250 ms
PC PC
PC switch 1G router router switch PC
PC 2.4G 1G PC
(1G) NYC RTT: 285 ms
Phoenix GNET-1 Tokyo,
Tsukuba
National Institute of Advanced Industrial Science and Technology
20. File replication performance between Japan
and US (total)
National Institute of Advanced Industrial Science and Technology
21. APAN/TransPAC Chicago
Pacing at 500 Mbps, quite stable
National Institute of Advanced Industrial Science and Technology
22. APAN/TransPAC LA (1)
After re-pacing from 800 to 780 Mbps, quite stable
National Institute of Advanced Industrial Science and Technology
23. APAN/TransPAC LA (2)
After re-pacing of LA (1), quite stable
National Institute of Advanced Industrial Science and Technology
24. APAN/TransPAC LA (3)
After re-pacing of LA (1), quite stable
National Institute of Advanced Industrial Science and Technology
25. SuperSINET NYC
Re-pacing from 930 to 950 Mbps
National Institute of Advanced Industrial Science and Technology
26. Summary
Efficient use around the peak rate in long fat networks
IFG-based precise pacing within packet loss free bandwidth with GNET-1
-> packet loss free network
Stable network flow even with HighSpeed TCP
Disk I/O performance improvement
Parallel disk access using Gfarm
Trans-pacific file replication performance: 3.79Gbps out of theoretical peak 3.9
Gbps (97%) using 11 node pairs (MTU 6000B)
1.5TB data was transferred in an hour
Linux 2.4 kernel problem during file replication (transfer)
Network transfer stopped in a few minutes when flushing buffer cache to disk
Linux kernel bug?
Defensive solution: set very short interval for buffer cache flush
This limits file transfer rate to 400 Mbps for one node pair
Successful Trans-pacific scale data analysis
. . . Scalability problem of LDAP server for a metadata server
Further improvement needed
National Institute of Advanced Industrial Science and Technology
27. Future work
Standardization effort with GGF Grid File System WG
Foster (world-wide) storage sharing and integration
dependable data sharing, high-performance data access
among several organizations
Application area
High energy physics experiment
Astronomic data analysis
Bioinformatics, . . .
Dependable data processing in eGovernment and
eCommerce
Other applications that needs dependable file sharing
among several organizations
National Institute of Advanced Industrial Science and Technology
28. Special thanks to
Hirotaka Ogawa, Yuetsu Kodama, Tomohiro Kudoh, Satoshi
Sekiguchi (AIST), Satoshi Matsuoka, Kento Aida (Titech),
Taisuke Boku, Mitsuhisa Sato (Univ Tsukuba),
Youhei Morita (KEK), Yoshinori Kitatsuji (APAN Tokyo XP),
Jim Williams, John Hicks (TransPAC/Indiana Univ)
Eguchi Hisashi (Maffin), Kazunori Konishi, Jin Tanaka,
Yoshitaka Hattori (APAN), Jun Matsukata (NII), Chris Robb
(Abilene)
Tsukuba WAN NOC team, APAN NOC team, NII SuperSINET
NOC team
Force10 Networks
PRAGMA, ApGrid, SDSC, Indiana University, Kasetsart
University
National Institute of Advanced Industrial Science and Technology