Larry Smarr, founding director of Calit2 (now Distinguished Professor Emeritus at the University of California San Diego) and the first director of NCSA, is one of the seminal figures in the U.S. supercomputing community. What began as a personal drive, shared by others, to spur the creation of supercomputers in the U.S. for scientific use, later expanded into a drive to link those supercomputers with high-speed optical networks, and blossomed into the notion of building a distributed, high-performance computing infrastructure – replete with compute, storage and management capabilities – available broadly to the science community.
Why Researchers are Using Advanced NetworksLarry Smarr
07.07.03
Remote Talk from Calit2 to:
Building KAREN Communities for Collaboration Forum
KIWI Advanced Research and Education Network
University of Auckland, Auckland City, New Zealand
Title: Why Researchers are Using Advanced Networks
La Jolla, CA
From the Shared Internet to Personal Lightwaves: How the OptIPuter is Transfo...Larry Smarr
08.04.03
Invited Talk
Cyberinfrastructure Colloquium
Clemson University
Title: From the Shared Internet to Personal Lightwaves: How the OptIPuter is Transforming Scientific Research
Clemson, SC
The Jump to Light Speed - Data Intensive Earth Sciences are Leading the Way t...Larry Smarr
05.06.14
Keynote to the 15th Federation of Earth Science Information Partners Assembly Meeting: Linking Data and Information to Decision Makers
Title: The Jump to Light Speed - Data Intensive Earth Sciences are Leading the Way to the International LambdaGrid
San Diego, CA
The OptiPuter, Quartzite, and Starlight Projects: A Campus to Global-Scale Te...Larry Smarr
05.03.09
Invited Talk
Optical Fiber Communication Conference (OFC2005)
Title: The OptiPuter, Quartzite, and Starlight Projects: A Campus to Global-Scale Testbed for Optical Technologies Enabling LambdaGrid Computing
Anaheim, CA
The Singularity: Toward a Post-Human RealityLarry Smarr
06.02.13
Talk to UCSD's Sixth College
Honor's Course on Kurzweil's The Singularity is Near
Title: The Singularity: Toward a Post-Human Reality
La Jolla, CA
Remote Telepresence for Exploring Virtual WorldsLarry Smarr
08.01.26
Foundational Talk
Virtual World and Immersive Environments
NASA Ames
Title: Remote Telepresence for Exploring Virtual Worlds
Mountain View, CA
Why Researchers are Using Advanced NetworksLarry Smarr
07.07.03
Remote Talk from Calit2 to:
Building KAREN Communities for Collaboration Forum
KIWI Advanced Research and Education Network
University of Auckland, Auckland City, New Zealand
Title: Why Researchers are Using Advanced Networks
La Jolla, CA
From the Shared Internet to Personal Lightwaves: How the OptIPuter is Transfo...Larry Smarr
08.04.03
Invited Talk
Cyberinfrastructure Colloquium
Clemson University
Title: From the Shared Internet to Personal Lightwaves: How the OptIPuter is Transforming Scientific Research
Clemson, SC
The Jump to Light Speed - Data Intensive Earth Sciences are Leading the Way t...Larry Smarr
05.06.14
Keynote to the 15th Federation of Earth Science Information Partners Assembly Meeting: Linking Data and Information to Decision Makers
Title: The Jump to Light Speed - Data Intensive Earth Sciences are Leading the Way to the International LambdaGrid
San Diego, CA
The OptiPuter, Quartzite, and Starlight Projects: A Campus to Global-Scale Te...Larry Smarr
05.03.09
Invited Talk
Optical Fiber Communication Conference (OFC2005)
Title: The OptiPuter, Quartzite, and Starlight Projects: A Campus to Global-Scale Testbed for Optical Technologies Enabling LambdaGrid Computing
Anaheim, CA
The Singularity: Toward a Post-Human RealityLarry Smarr
06.02.13
Talk to UCSD's Sixth College
Honor's Course on Kurzweil's The Singularity is Near
Title: The Singularity: Toward a Post-Human Reality
La Jolla, CA
Remote Telepresence for Exploring Virtual WorldsLarry Smarr
08.01.26
Foundational Talk
Virtual World and Immersive Environments
NASA Ames
Title: Remote Telepresence for Exploring Virtual Worlds
Mountain View, CA
08.04.14
Invited Talk
National Astrobiology Institute Executive Council Meeting
Astrobiology Science Conference 2008
Santa Clara Convention Center
Title: High Performance Collaboration
Santa Clara, CA
OptIPuter-A High Performance SOA LambdaGrid Enabling Scientific ApplicationsLarry Smarr
07.03.21
IEEE Computer Society Tsutomu Kanai Award Keynote
At the Joint Meeting of the: 8th International Symposium on Autonomous Decentralized Systems
2nd International Workshop on Ad Hoc, Sensor and P2P Networks
11th IEEE International Workshop on Future Trends of Distributed Computing Systems
Title: OptIPuter-A High Performance SOA LambdaGrid Enabling Scientific Applications
Sedona, AZ
Toward a Global Interactive Earth Observing CyberinfrastructureLarry Smarr
05.01.12
Invited Talk to the 21st International Conference on Interactive Information Processing Systems (IIPS) for Meteorology, Oceanography, and Hydrology Held at the 85th AMS Annual Meeting
Title: Toward a Global Interactive Earth Observing Cyberinfrastructure
San Diego, CA
Global Telepresence in Support of Global Public HealthLarry Smarr
08.04.17
Briefing
University of California School of Global Health
All Campuses Planning Committee
Calit2@UCSD
Title: Global Telepresence in Support of Global Public Health
La Jolla, CA
08.09.19
Invited Lecture to the Green IT Workshop
Canada-California Strategic Innovation Partnership
Title: Toward Greener Cyberinfrastructure
Palo Alto, CA
The Importance of Large-Scale Computer Science Research EffortsLarry Smarr
05.10.20
Talk at Public Seminar on Large-Scale NSF Research Efforts for the Future Computer Museum
Title: The Importance of Large-Scale Computer Science Research Efforts
Mountain View, CA
High Performance Cyberinfrastructure is Needed to Enable Data-Intensive Scien...Larry Smarr
11.03.28
Remote Luncheon Presentation from Calit2@UCSD
National Science Board
Expert Panel Discussion on Data Policies
National Science Foundation
Title: High Performance Cyberinfrastructure is Needed to Enable Data-Intensive Science and Engineering
Arlington, Virginia
Cyberinfrastructure for Advanced Marine Microbial Ecology Research and Analys...Larry Smarr
06.07.31
Invited Talk
CONNECT Investment Community Meeting
Calit2@UCSD
Title: Cyberinfrastructure for Advanced Marine Microbial Ecology Research and Analysis (CAMERA)
La Jolla, CA
The Energy Efficient Cyberinfrastructure in Slowing Climate ChangeLarry Smarr
10.04.28
Invited Speaker
Community Alliance for Distributed Energy Resources
Scripps Forum, UCSD
Title: The Energy Efficient Cyberinfrastructure in Slowing Climate Change
La Jolla, CA
08.04.14
Invited Talk
National Astrobiology Institute Executive Council Meeting
Astrobiology Science Conference 2008
Santa Clara Convention Center
Title: High Performance Collaboration
Santa Clara, CA
OptIPuter-A High Performance SOA LambdaGrid Enabling Scientific ApplicationsLarry Smarr
07.03.21
IEEE Computer Society Tsutomu Kanai Award Keynote
At the Joint Meeting of the: 8th International Symposium on Autonomous Decentralized Systems
2nd International Workshop on Ad Hoc, Sensor and P2P Networks
11th IEEE International Workshop on Future Trends of Distributed Computing Systems
Title: OptIPuter-A High Performance SOA LambdaGrid Enabling Scientific Applications
Sedona, AZ
Toward a Global Interactive Earth Observing CyberinfrastructureLarry Smarr
05.01.12
Invited Talk to the 21st International Conference on Interactive Information Processing Systems (IIPS) for Meteorology, Oceanography, and Hydrology Held at the 85th AMS Annual Meeting
Title: Toward a Global Interactive Earth Observing Cyberinfrastructure
San Diego, CA
Global Telepresence in Support of Global Public HealthLarry Smarr
08.04.17
Briefing
University of California School of Global Health
All Campuses Planning Committee
Calit2@UCSD
Title: Global Telepresence in Support of Global Public Health
La Jolla, CA
08.09.19
Invited Lecture to the Green IT Workshop
Canada-California Strategic Innovation Partnership
Title: Toward Greener Cyberinfrastructure
Palo Alto, CA
The Importance of Large-Scale Computer Science Research EffortsLarry Smarr
05.10.20
Talk at Public Seminar on Large-Scale NSF Research Efforts for the Future Computer Museum
Title: The Importance of Large-Scale Computer Science Research Efforts
Mountain View, CA
High Performance Cyberinfrastructure is Needed to Enable Data-Intensive Scien...Larry Smarr
11.03.28
Remote Luncheon Presentation from Calit2@UCSD
National Science Board
Expert Panel Discussion on Data Policies
National Science Foundation
Title: High Performance Cyberinfrastructure is Needed to Enable Data-Intensive Science and Engineering
Arlington, Virginia
Cyberinfrastructure for Advanced Marine Microbial Ecology Research and Analys...Larry Smarr
06.07.31
Invited Talk
CONNECT Investment Community Meeting
Calit2@UCSD
Title: Cyberinfrastructure for Advanced Marine Microbial Ecology Research and Analysis (CAMERA)
La Jolla, CA
The Energy Efficient Cyberinfrastructure in Slowing Climate ChangeLarry Smarr
10.04.28
Invited Speaker
Community Alliance for Distributed Energy Resources
Scripps Forum, UCSD
Title: The Energy Efficient Cyberinfrastructure in Slowing Climate Change
La Jolla, CA
Larry Smarr, Founding Director of the California Institute for Telecommunications and Information Technology (Calit2), shares his presentation delivered at Venture Summit Friday, July 12, 2013
Supercomputers and Supernetworks are Transforming ResearchLarry Smarr
09.03.25
Invited Talk
Computing Research that Changed the World: Reflections and Perspectives
Title: Supercomputers and Supernetworks are Transforming Research
Washington, DC
This lesson provides basic knowledge in website design and development. It will help you to understand the internet and the web as two different but inseparable components of website.
Montana State, Research Networking and the Outcomes from the First National R...Jerry Sheehan
Presentation at Educause 17 with our Partner Cisco on Research networking, covers our campus experience and the first National Research Platform Workshop findings
The Academic and R&D Sectors' Current and Future Broadband and Fiber Access N...Larry Smarr
05.02.23
Invited Access Grid Talk
MSCMC FORUM Series
Examining the National Vision for Global Peace and Prosperity
Title: The Academic and R&D Sectors' Current and Future Broadband and Fiber Access Needs for US Global Competitiveness
Arlington, VA
Metacomputer Architecture of the Global LambdaGridLarry Smarr
06.01.13
Invited Talk
Department of Computer Science
Donald Bren School of Information and Computer Sciences
Title: Metacomputer Architecture of the Global LambdaGrid
Irvine, CA
Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...John Andrews
SlideShare Description for "Chatty Kathy - UNC Bootcamp Final Project Presentation"
Title: Chatty Kathy: Enhancing Physical Activity Among Older Adults
Description:
Discover how Chatty Kathy, an innovative project developed at the UNC Bootcamp, aims to tackle the challenge of low physical activity among older adults. Our AI-driven solution uses peer interaction to boost and sustain exercise levels, significantly improving health outcomes. This presentation covers our problem statement, the rationale behind Chatty Kathy, synthetic data and persona creation, model performance metrics, a visual demonstration of the project, and potential future developments. Join us for an insightful Q&A session to explore the potential of this groundbreaking project.
Project Team: Jay Requarth, Jana Avery, John Andrews, Dr. Dick Davis II, Nee Buntoum, Nam Yeongjin & Mat Nicholas
Show drafts
volume_up
Empowering the Data Analytics Ecosystem: A Laser Focus on Value
The data analytics ecosystem thrives when every component functions at its peak, unlocking the true potential of data. Here's a laser focus on key areas for an empowered ecosystem:
1. Democratize Access, Not Data:
Granular Access Controls: Provide users with self-service tools tailored to their specific needs, preventing data overload and misuse.
Data Catalogs: Implement robust data catalogs for easy discovery and understanding of available data sources.
2. Foster Collaboration with Clear Roles:
Data Mesh Architecture: Break down data silos by creating a distributed data ownership model with clear ownership and responsibilities.
Collaborative Workspaces: Utilize interactive platforms where data scientists, analysts, and domain experts can work seamlessly together.
3. Leverage Advanced Analytics Strategically:
AI-powered Automation: Automate repetitive tasks like data cleaning and feature engineering, freeing up data talent for higher-level analysis.
Right-Tool Selection: Strategically choose the most effective advanced analytics techniques (e.g., AI, ML) based on specific business problems.
4. Prioritize Data Quality with Automation:
Automated Data Validation: Implement automated data quality checks to identify and rectify errors at the source, minimizing downstream issues.
Data Lineage Tracking: Track the flow of data throughout the ecosystem, ensuring transparency and facilitating root cause analysis for errors.
5. Cultivate a Data-Driven Mindset:
Metrics-Driven Performance Management: Align KPIs and performance metrics with data-driven insights to ensure actionable decision making.
Data Storytelling Workshops: Equip stakeholders with the skills to translate complex data findings into compelling narratives that drive action.
Benefits of a Precise Ecosystem:
Sharpened Focus: Precise access and clear roles ensure everyone works with the most relevant data, maximizing efficiency.
Actionable Insights: Strategic analytics and automated quality checks lead to more reliable and actionable data insights.
Continuous Improvement: Data-driven performance management fosters a culture of learning and continuous improvement.
Sustainable Growth: Empowered by data, organizations can make informed decisions to drive sustainable growth and innovation.
By focusing on these precise actions, organizations can create an empowered data analytics ecosystem that delivers real value by driving data-driven decisions and maximizing the return on their data investment.
Opendatabay - Open Data Marketplace.pptxOpendatabay
Opendatabay.com unlocks the power of data for everyone. Open Data Marketplace fosters a collaborative hub for data enthusiasts to explore, share, and contribute to a vast collection of datasets.
First ever open hub for data enthusiasts to collaborate and innovate. A platform to explore, share, and contribute to a vast collection of datasets. Through robust quality control and innovative technologies like blockchain verification, opendatabay ensures the authenticity and reliability of datasets, empowering users to make data-driven decisions with confidence. Leverage cutting-edge AI technologies to enhance the data exploration, analysis, and discovery experience.
From intelligent search and recommendations to automated data productisation and quotation, Opendatabay AI-driven features streamline the data workflow. Finding the data you need shouldn't be a complex. Opendatabay simplifies the data acquisition process with an intuitive interface and robust search tools. Effortlessly explore, discover, and access the data you need, allowing you to focus on extracting valuable insights. Opendatabay breaks new ground with a dedicated, AI-generated, synthetic datasets.
Leverage these privacy-preserving datasets for training and testing AI models without compromising sensitive information. Opendatabay prioritizes transparency by providing detailed metadata, provenance information, and usage guidelines for each dataset, ensuring users have a comprehensive understanding of the data they're working with. By leveraging a powerful combination of distributed ledger technology and rigorous third-party audits Opendatabay ensures the authenticity and reliability of every dataset. Security is at the core of Opendatabay. Marketplace implements stringent security measures, including encryption, access controls, and regular vulnerability assessments, to safeguard your data and protect your privacy.
Levelwise PageRank with Loop-Based Dead End Handling Strategy : SHORT REPORT ...Subhajit Sahu
Abstract — Levelwise PageRank is an alternative method of PageRank computation which decomposes the input graph into a directed acyclic block-graph of strongly connected components, and processes them in topological order, one level at a time. This enables calculation for ranks in a distributed fashion without per-iteration communication, unlike the standard method where all vertices are processed in each iteration. It however comes with a precondition of the absence of dead ends in the input graph. Here, the native non-distributed performance of Levelwise PageRank was compared against Monolithic PageRank on a CPU as well as a GPU. To ensure a fair comparison, Monolithic PageRank was also performed on a graph where vertices were split by components. Results indicate that Levelwise PageRank is about as fast as Monolithic PageRank on the CPU, but quite a bit slower on the GPU. Slowdown on the GPU is likely caused by a large submission of small workloads, and expected to be non-issue when the computation is performed on massive graphs.
Adjusting primitives for graph : SHORT REPORT / NOTESSubhajit Sahu
Graph algorithms, like PageRank Compressed Sparse Row (CSR) is an adjacency-list based graph representation that is
Multiply with different modes (map)
1. Performance of sequential execution based vs OpenMP based vector multiply.
2. Comparing various launch configs for CUDA based vector multiply.
Sum with different storage types (reduce)
1. Performance of vector element sum using float vs bfloat16 as the storage type.
Sum with different modes (reduce)
1. Performance of sequential execution based vs OpenMP based vector element sum.
2. Performance of memcpy vs in-place based CUDA based vector element sum.
3. Comparing various launch configs for CUDA based vector element sum (memcpy).
4. Comparing various launch configs for CUDA based vector element sum (in-place).
Sum with in-place strategies of CUDA mode (reduce)
1. Comparing various launch configs for CUDA based vector element sum (in-place).
Techniques to optimize the pagerank algorithm usually fall in two categories. One is to try reducing the work per iteration, and the other is to try reducing the number of iterations. These goals are often at odds with one another. Skipping computation on vertices which have already converged has the potential to save iteration time. Skipping in-identical vertices, with the same in-links, helps reduce duplicate computations and thus could help reduce iteration time. Road networks often have chains which can be short-circuited before pagerank computation to improve performance. Final ranks of chain nodes can be easily calculated. This could reduce both the iteration time, and the number of iterations. If a graph has no dangling nodes, pagerank of each strongly connected component can be computed in topological order. This could help reduce the iteration time, no. of iterations, and also enable multi-iteration concurrency in pagerank computation. The combination of all of the above methods is the STICD algorithm. [sticd] For dynamic graphs, unchanged components whose ranks are unaffected can be skipped altogether.
Explore our comprehensive data analysis project presentation on predicting product ad campaign performance. Learn how data-driven insights can optimize your marketing strategies and enhance campaign effectiveness. Perfect for professionals and students looking to understand the power of data analysis in advertising. for more details visit: https://bostoninstituteofanalytics.org/data-science-and-artificial-intelligence/
SC21: Larry Smarr on The Rise of Supernetwork Data Intensive Computing
1. “The Rise of Supernetwork
Data Intensive Computing”
Invited Remote Lecture to SC21
The International Conference for High Performance
Computing, Networking, Storage, and Analysis
St. Louis, Missouri
November 18, 2021
Dr. Larry Smarr
Founding Director Emeritus, California Institute for Telecommunications and Information Technology;
Distinguished Professor Emeritus, Dept. of Computer Science and Engineering
Jacobs School of Engineering, UCSD
http://lsmarr.calit2.net
2. Abstract
Over the last 35 years, a fundamental architectural transformation in high performance data-intensive
computing has occurred, driven by the rise of optical fiber Supernetworks connecting the globe.
Ironically, this cyberinfrastructure revolution has been led by supercomputer centers, which then became
SuperNodes in this distributed system. I will review key moments, including the birth of the NSF
Supercomputer Centers and NSFnet, the gigabit testbeds, the NSF PACI program, the emergence of
Internet2 and the Regional Optical Networks, all eventually enabling, through a series of NSF grants, the
National and Global Research Platforms. Over this same period a similar cyberinfrastructure architecture
allowed the commercial clouds to develop, which are now interconnected with this academic distributed
system. Critical to this transformation has been the continual exponential rise of data and a new
generation of distributed applications utilizing this connected digital fabric. Throughout this period, the
role of the US Federal Government has been essential, anchored by the 1991 High-Performance
Computing Act, which established the Networking and Information Technology Research and
Development (NITRD) Program. Particularly important to the initiation of this distributed computing
paradigm shift was the continued visionary leadership of Representative, then Senator, then Vice
President Al Gore in the 1990s.
3. 1975-1985: My Early Research was on Computational Astrophysics
Before There Were National Academic Supercomputer Centers
I Spent a Decade Supercomputing at LLNL (with Jim Wilson) and
Then at The Max Planck Institute for Physics and Astrophysics (with Mike Norman and Karl-Heinz Winkler)
Gas Accretion Onto a Black Hole
With Wilson and Hawley
1982
Cosmic Jets Emerging From Galactic Centers
With Norman and Winkler
1981
Gravitational Radiation From Black Hole Collisions
With Eppley
1978
4. 1982-1983: Documenting The Unmet Supercomputing Needs
of a Broad Range of Disciplines Led to the NCSA Proposal to NSF
1982 1983
http://lsmarr.calit2.net/supercomputer_famine_1982.pdf http://lsmarr.calit2.net/Black_Proposal.pdf
1984: NSF Creates Office of Advanced Scientific Computing (John Connolly, Director)
Issues National Competition for Supercomputer Centers
5. 1985: NSF Adopted a DOE High-Performance Computing Model
For Two of the New NSF Supercomputer Centers
NCSA Was Modeled on LLNL SDSC Was Modeled on MFEnet
1985
6. SuperNetworks Have Co-Evolved
with Supercomputers For 35 Years
“We ought to consider
a national initiative
to build interstate highways
for information
with a fiber optics network
connecting the
major computational centers
in this country”
-Senator Al Gore
“The University of Illinois
will be experimenting with
fiber optic
"information flow pipes,"
which promise to be able
to reach
billions of bits per second.””
-NCSA Director
Larry Smarr
http://lsmarr.calit2.net/hrg-1985-tec-0068_from_1_to_806_s.pdf
1985
7. Remote Interactive Visual Supercomputing End-to-End Prototype:
Using Analog Communications to Prototype the Fiber Optic Future
“We’re using satellite technology…
to demonstrate
what It might be like to have
high-speed fiber-optic links
between advanced computers
in two different
geographic locations.”
Illinois
Boston
SIGGRAPH 1989
“What we really have to do is eliminate distance
between individuals who want to interact
with other people and computers.”
― Larry Smarr, Director, NCSA
www.youtube.com/watch?v=C3d_6lw8_0M
-Al Gore, Senator
Chair, US Senate
Subcommittee
Cray 2 Driven by
Sun Workstation
AT&T & Sun
Telepresenc
e
8. 1991: Networking Information Technology Research and Development (NITRD)
• NITRID Was Enacted in 1991 by Congress
Through the High-Performance Computing and Communication Act
• Brought Multiple Federal Agencies Together
to Plan and Coordinate Frontier Computing, Networking, Software, and Data
• Bill Was Sponsored and Driven by Senator Al Gore
December 2, 2021
9. The Bandwidth and Number of Endpoints
on NSFNET Grew Rapidly
Visualization of Inbound Traffic on the NSFNET T1 Backbone
(September 1991) by NCSA’s Donna Cox and Robert Patterson;
Data Collected by Merit Network, Inc.
1994
1991
10. • The First National 155 Mbps Research Network
– Inter-Connected Telco Networks Via IP/ATM With:
– Supercomputer Centers
– Virtual Reality Research Locations, and
– Applications Development Sites
– Into the San Diego Convention Center
– 65 Science Projects
• I-Way Featured:
– Networked Visualization Applications
– Large-Scale Immersive Displays
– I-Soft Programming Environment
– Led to the Globus Project
Supercomputing ’95:
I-WAY: A Model for Distributed Collaborative Computing
For details see:
“Overview of the I-WAY: Wide Area Visual Supercomputing”
DeFanti, Foster, Papka, Stevens, Kuhfuss
www.globus.org/sites/default/files/iway_overview.pdf SC95 Chair Sid Karin
SC95 Program Chair, Larry Smarr
11. 1990-1996 CNRI’s Gigabit Testbeds
Demonstrated Host I/O Was the Distributed Computing Bottleneck
“Host I/O proved to be
the Achilles' heel
of gigabit networking –
whereas LAN and WAN technologies were
operated in the gigabit regime, many
obstacles impeded
achieving gigabit flows
into and out of
the host computers
used in the testbeds.”
--Final Report
The Gigabit Testbed Initiative
December 1996
Corporation for
National Research Initiatives (CNRI)
Robert Kahn
CNRI Chairman, CEO & President
12. NSF’s PACI Program was Built on the vBNS
to Prototype America’s 21st Century Information Infrastructure
PACI National Technology Grid
Testbed
National Computational Science
1997
vBNS
led to
Key Role
of Miron Livny
& Condor
13. The 25 Years From the National Techology Grid
To the National Research Platform
From I-WAY to the National Technology Grid, CACM, 40, 51 (1997)
Rick Stevens, Paul Woodward, Tom DeFanti, and Charlie Catlett
14. Dave Bader Created the First Linux COTS Supercluster -Roadrunner-
on the National Technology Grid, with the Support of NCSA and NSF
NCSA Director Larry Smarr (left),
UNM President William Gordon,
and U.S. Sen. Pete Domenici
Turn on the Roadrunner Supercomputer
in April 1999
1999
National Computational Science
15. Illinois’s I-WIRE and Indiana’s I-LIGHT Dark Fiber Networks
Inspired Many Other State and Regional Optical
Source: Larry Smarr, Rick Stevens, Tom DeFanti, Charlie Catlett
1999
Today California’s
CENIC R&E
Backbone Includes
~ 8,000 Miles of
CENIC-Owned and
Managed Fiber
16. 1999: The President’s Information Technology Advisory Committee (PITAC) Report
Led to Funding NSF’s Information Technology Research (ITR) for National Priorities Program
Meeting with Vice President Gore in the White House
To Present Our PITAC Report
PITAC
Co-Chairs:
Ken Kennedy
Bill Joy
17. The OptIPuter
Exploits a New World
in Which
the Central Architectural Element
is Optical Networking,
Not Computers
to Support
Data-Intensive Scientific Research
and Collaboration
OptIPuter
NSF ITR Grant
$13.5M
PI Smarr,
Co-PIs DeFanti,
Papadopoulos, Ellisman
2002-2009
2002-2009: The NSF OptIPuter ITR Grant-
Can We Make Wide-Area Bandwidth Equal to Cluster Backplane Speeds?
18. Integrated “OptIPlatform” Cyberinfrastructure System:
A 10Gbps Lightpath Cloud
National LambdaRail
Campus
Optical
Switch
Data Repositories & Clusters
HPC
HD/4k Video Images
HD/4k Video Cams
End User
OptIPortal
10G
Lightpath
HD/4k Telepresence
Instruments
LS 2009
Slide
19. David Abramson Led OptIPuter Global Workflows and
UCSD/Monash Univ. Co-Mentoring of Undergrads and Graduate Students
First OptIPortal/Kepler
Remote Microscopy Link Feb 2009
Monash U.
UCSD
Monash U.
20. 2010-2020:
NSF Adopted a DOE High-Performance Networking Model
Science
DMZ
Data Transfer
Nodes
(DTN/FIONA)
Network
Architecture
(zero friction)
Performance
Monitoring
(perfSONAR)
ScienceDMZ Coined in 2010 by ESnet
http://fasterdata.es.net/science-dmz/
Slide Adapted From Inder Monga, ESnet
DOE
NSF
NSF Campus Cyberinfrastructure Program
2012-2020
Has Made Over 340 Awards:
Across 50 States and Territories
Slide Adapted From Kevin Thompson, NSF
21. 2013-2015: UCSD as a Laboratory for a “Big Data” 10-100 Gbps ScienceDMZ
NSF-Funded Campus CI Grants: Prism@UCSD and CHeruB
Prism@UCSD, Phil Papadopoulos, SDSC, Calit2, PI (2013-15)
CHERuB, Mike Norman, SDSC PI
CHERuB
22. (GDC)
2015 Vision: The Pacific Research Platform Will Connect Science DMZs
Creating a Regional End-to-End Science-Driven Community Cyberinfrastructure
NSF CC*DNI Grant
$6.3M 10/2015-10/2020
In Year 6 Now, Year 7 is Funded
Source: John Hess, CENIC
Supercomputer
Centers
23. PRP Website Has All Details Needed to Get Started
https://pacificresearchplatform.org/
24. 2015-2021: UCSD Designs PRP Data Transfer Nodes (DTNs) --
Flash I/O Network Appliances (FIONAs)
FIONAs Solved the the Gigabit Testbed Disk-to-Disk Data Transfer Problem
at Near Full Speed on Best-Effort 10G, 40G and 100G
FIONAs Designed by UCSD’s Phil Papadopoulos, John Graham,
Joe Keefe, and Tom DeFanti
Up to 192 TB Rotating Storage
www.pacificresearchplatform.org
Today’s
Roadrunner!
25. 2018/2019: PRP Game Changer!
Using Google’s Kubernetes to Orchestrate Containers Across the PRP
User
Applications
Containers
Clouds
26. PRP’s Nautilus Hypercluster Adopted Kubernetes
to Orchestrate Software Containers and Manage Distributed Storage
“Kubernetes with Rook/Ceph Allows Us to Manage Petabytes of
Distributed Storage and GPUs for Data Science,
While We Measure and Monitor Network Use.”
--John Graham, Calit2/QI UC San Diego
Kubernetes (K8s) is an open-source system for
automating deployment, scaling, and
management of containerized applications.
27. 2017-2020: NSF CHASE-CI Grant Adds a Machine Learning Layer
Built on Top of the Pacific Research Platform
Caltech
UCB
UCI UCR
UCSD
UCSC
Stanford
MSU
UCM
SDSU
NSF Grant for High Speed “Cloud” of 256 GPUs
For 30 ML Faculty & Their Students at 10 Campuses
for Training AI Algorithms on Big Data
PI: Larry Smarr
Co-PIs:
• Tajana Rosing
• Ken Kreutz-Delgado
• Ilkay Altintas
• Tom DeFanti
28. Original
PRP
CENIC/PW Link
2018-2021: Toward the National Research Platform (TNRP) -
Using CENIC & Internet2 to Connect Quilt Regional R&E Networks
“Towards
The NRP”
3-Year Grant
Funded
by NSF
$2.5M
October 2018
Award #1826967
PI Smarr
Co-PIs Altintas,
Papadopoulos,
Wuerthwein,
Rosing
29. Rotating Storage
4000 TB
PRP’s Nautilus is a Multi-Institution Hypercluster
Connected by Optical Networks
184 FIONAs on 25 Partner Campuses
Networked Together at 10-100Gbps
32. PRP’s Nautilus is Centered in SoCal
FIONAs
UCSD &
SDSU
UCI
Caltech
UCSB
UCR
CSUSB
33. We Measure Disk-to-Disk Throughput with 10GB File Transfer
4 Times Per Day in Both Directions for All PRP Sites
January 29,
2016
From Start of Monitoring 12 DTNs
to 24 DTNs Connected at 10-40G
in 1 ½ Years
July 21, 2017
Source: John Graham, Calit2
34. Operational Metrics: Containerized Trace Route Tool Allows Realtime Visualization
of Status of PRP Network Links on a National and Global Scale
Source: Dima Mishin, SDSC
9/16/2019
Guam
Univ. Queensland
Australia
LIGO
UK
Netherlands
Korea
36. Director:
F. Martin Ralph
Big Data Collaboration with:
Scott Sellers, PhD CHRS; Postdoc CW3E
PRP Accelerates Data-Intensive Workflow on Atmospheric Water in the West
Between NASA MERRA Archive, UC San Diego, and UC Irvine
Director:
Soroosh Sorooshian
Complete Workflow Time:
19.2 days🡪52 Minutes!
See Paper by Sellars, et al., IEEE eScience (2019)
http://lsmarr.calit2.net/sellars_accelerating_image_segmentation.pdf
37. The New Pacific Research Platform Video
Highlights 3 Different Applications Out of 600 Nautilus Namespace Projects
Pacific Research Platform Video:
www.thequilt.net/campus-cyberinfrastructure-program-resource/
www.pacificresearchplatform.org
38. Co-Existence of Interactive and
Non-Interactive Computing on PRP
GPU Simulations Needed to Improve Ice Model.
=> Results in Significant Improvement
in Pointing Resolution for Multi-Messenger Astrophysics
NSF Large-Scale Observatories Are Using PRP and OSG
as a Cohesive, Federated, National-Scale Research Data Infrastructure
NSF’s IceCube & LIGO Both See Nautilus
as Just Another OSG Resource
IceCube Used Up to Half of
PRP’s 500 GPUs in 2020!
39. UC President Napolitano's Research Catalyst Award to
UC San Diego (Tom Levy), UC Berkeley (Benjamin Porter), UC Merced (Nicola Lercari) and UCLA (Willeke Wendrich)
PRP Links At-Risk Cultural Heritage and Archaeology Datasets
to Virtual Reality Systems at Multiple Campuses
48 Megapixel CAVEkiosk
UCSD Library
48 Megapixel CAVEkiosk
UCB CITRIS Tech Museum
24 Megapixel CAVEkiosk
UCM Library
40. Once a Wildfire is Spotted, PRP Brings High-Resolution Weather Data
to Fire Modeling Workflows in WIFIRE
Real-Time
Meteorological Sensors
Weather Forecast
Landscape data
WIFIRE Firemap
Fire Perimeter
Work Flow
PRP
Source: Ilkay Altintas, SDSC
41. Community Building Though Inclusion and Diversity
• Grants
– 3 Female co-PIs
– 1 Hispanic co-PI
• Campuses
– 8 Minority-Serving Institutions in PRP/CHASE-CI
• Workshops
– NRPII Workshop Steering Committee 80% Female
– Multiple MSI, EPSCoR Focused Workshops
Jackson State University
PRP MSI Workshop
Presenting
FIONettes
42. 2021-2024 NRP Future I: Proposed Extension of Nautilus
CHASE-CI ENS, Tom DeFanti PI (NSF Award # 2120019)
CHASE-CI ABR, Larry Smarr PI (NSF Award # 2100237)
$2.8M
43. 2021-2026 NRP Future II: PRP Federates with SDSC’s EXPANSE
Using CHASE-CI Developed Composable Systems
~$20M over 5 Years
PI Mike Norman, SDSC
44. 2021-2026 NRP Future III: PRP Federates with
NSF-Funded Prototype National Research Platform
NSF Award OAC #2112167 (June 2021) [$5M Over 5 Years]
PI Frank Wuerthwein (UCSD, SDSC)
Co-PIs Tajana Rosing (UCSD), Thomas DeFanti (UCSD), Mahidhar Tatineni (SDSC), Derek Weitzel (UNL)
45. PRP/TNRP/CHASE-CI Support and Community:
• US National Science Foundation (NSF) awards to UCSD, NU, and SDSC
⮚ CNS-1456638, CNS-1730158, ACI-1540112, ACI-1541349, & OAC-1826967
⮚ OAC 1450871 (NU) and OAC-1659169 (SDSU)
• UC Office of the President, Calit2 and Calit2’s UCSD Qualcomm Institute
• San Diego Supercomputer Center and UCSD’s Research IT and Instructional IT
• Partner Campuses: UCB, UCSC, UCI, UCR, UCLA, USC, UCD, UCSB, SDSU, Caltech, NU,
UWash UChicago, UIC, UHM, CSUSB, HPWREN, UMo, MSU, NYU, UNeb, UNC,UIUC,
UTA/Texas Advanced Computing Center, FIU, KISTI, UVA, AIST
• CENIC, Pacific Wave/PNWGP, StarLight/MREN, The Quilt, Kinber, Great Plains Network,
NYSERNet, LEARN, Open Science Grid, Internet2, DOE ESnet, NCAR/UCAR & Wyoming
Supercomputing Center, AWS, Google, Microsoft, Cisco