“The Pacific Research Platform
Two Years In”
Welcome and Overview Talk
to the Pacific Research Platform “PRPv2” Workshop 2017
University of California, San Diego
February 21, 2017
Dr. Larry Smarr
Director, California Institute for Telecommunications and Information Technology
Harry E. Gruber Professor,
Dept. of Computer Science and Engineering
Jacobs School of Engineering, UCSD
http://lsmarr.calit2.net
1
DOE ESnet’s Science DMZ: A Scalable Network
Design Model for Optimizing Science Data Transfers
• A Science DMZ integrates 4 key concepts into a unified whole:
– A network architecture designed for high-performance applications,
with the science network distinct from the general-purpose network
– The use of dedicated systems as data transfer nodes (DTNs)
– Performance measurement and network testing systems that are
regularly used to characterize and troubleshoot the network
– Security policies and enforcement mechanisms that are tailored for
high performance science environments
http://fasterdata.es.net/science-dmz/
Science DMZ
Coined 2010
The DOE ESnet Science DMZ and the NSF “Campus Bridging” Taskforce Report Formed the Basis
for the NSF Campus Cyberinfrastructure Network Infrastructure and Engineering (CC-NIE) Program
FIONAs and FIONettes – Flash I/O Network Appliances – Are DTNs:
Linux PCs Optimized for DMZs over Distance
FIONAs Are
Science DMZ Data Transfer Nodes (DTNs) &
Also Compute/Visualization/ML Nodes
Phil Papadopoulos & Tom DeFanti
Joe Keefe & John Graham
FIONAS—40G, $8,000
FIONette—1G, $1,000
The Pacific Research Platform’s Second Year:
a Working End-to-End Science-Driven DMZ-Connector
FIONAs as
Uniform DTN End Points
NSF CC*DNI Grant
$5M 10/2015-10/2020
PI: Larry Smarr, UC San Diego Calit2
Co-Pis:
• Camille Crittenden, UC Berkeley CITRIS,
• Tom DeFanti, UC San Diego Calit2,
• Philip Papadopoulos, UCSD SDSC,
• Frank Wuerthwein, UCSD Physics and SDSC
PRP Continues to Expand Rapidly While Increasing Connectivity:
One Year of Progress – 12 Sites to 20 Sites
January 29, 2016 December 15, 2016
Connected 20 DMZs at 10G and 40G,
demonstrating disk-to-disk GridFTP
at ~7.5G and 12.5G respectively,
and 900Mb at 1G
Many PRP Science Engagement Workshops
Were Held in 2016
• PRP Workshop Held in Collaboration with UC-Wide Research IT
– May 1, 2016
– 45 Attendees
– Ten UC Campuses
See Talk by Camille Crittenden and
Tom DeFanti
PRP’s First 1.5 Years:
Connecting Campus Application Teams and Devices
100 Gbps FIONA at UCSC Connects the UCSC Hyades Cluster
to the NERSC Supercomputer at LBNL
Supporting UCSC Remote Access
to Large Data Subsets
of the Dark Energy Spectroscopic Instrument (DESI)
and AGORA Galaxy Simulation Data
Produced at NERSC.
250 images per night
800GB per night
See Talk by Shawfeng Dong
UCSC Feb 7, 2017
40G FIONAs
20x40G PRP-connected
WAVE@UC San Diego
PRP Will Enable
Distributed Virtual Reality
PRP
MerWAVE @UC Merced
See Talk by Jeff Weekley
PRP Will Link the Laboratories of
the Pacific Earthquake Engineering Research Center
http://peer.berkeley.edu/
The Second FIONette was Deployed at the PEER Facility at UC Berkeley,
and its Performance is Being Monitored
John Graham Installing FIONette at PEER Feb 10, 2017
SIO Researchers Beginning to Use
Big Data Networks (Prism, IDI, PRP)
Jules Jaffe - Microscope Off Scripps Pier Frank Vernon - Expansion of HPWREN
Dan Cayan, Mike Dettinger
Regional Downscaling of Climate Models
Scott Sellars, Marty Ralph
Center for Western Weather and Water Extremes
PRP Backbone Sets Stage for 2017 Expansion
of HPWREN, Connected to CENIC, into Orange and Riverside Counties
• PRP CENIC 100G Link
UCSD to SDSU
– DTN FIONAs Endpoints
– Data Redundancy
– Disaster Recovery
– High Availability
– Network Redundancy
• Anchor to CENIC at UCI
– PRP FIONA Connects to
CalREN-HPR Network
– Data Replication Site
• Potential Future UCR
CENIC Anchor
UCR
UCI
UCSD
SDSU
Source: Frank Vernon,
Greg Hidley, UCSD
Pacific
City
Neptune
Canada
45°N
47°30’N
130°W 127°30’W
N
Seattle
GigaPOP
Portland
Possible PRP 2017 Expansion to Include NSF’s Ocean Observatory Initiative
Fiber Optic SensorNets on Seafloor Off Washington
To PRP via
Pacific Wave
Sea Bottom
Electro-optical Cable:
8,000 Volts
10 Gbps Optics
Slide Courtesy, John Delaney, UWash John Delaney Visiting UCSD’s SIO
For Three Months in 2017
Axial Volcano
140 Scientific Instruments
Being There - Remote Live High Definition Video
of Deep Sea Hydrothermal Vents
http://novae.ocean.washington.edu/story/Ashes_CAMHD_Liv
Mushroom Hydrothermal Vent
on Axial Seamount
1 Mile Below Sea Level
Picture Created
From 40 HD Frames
14 Minutes Live HD Video
On-Line Every 3 Hours
15 feet
Slide Courtesy, John Delaney, UWash
The Future of Supercomputing Will Blend Traditional HPC and Data Analytics
Integrating Non-von Neumann Architectures
“High Performance Computing Will Evolve
Towards a Hybrid Model,
Integrating Emerging Non-von Neumann Architectures,
with Huge Potential in Pattern Recognition,
Streaming Data Analysis,
and Unpredictable New Applications.”
Horst Simon, Deputy Director,
U.S. Department of Energy’s
Lawrence Berkeley National Laboratory
Calit2’s Qualcomm Institute Has Established a Pattern Recognition Lab
For Machine Learning on non-von Neumann Processors
Source: Dr. Dharmendra Modha
Founding Director, IBM Cognitive Computing Group
August 8, 2014
UCSD ECE Professor Ken Kreutz-Delgado Brings
the IBM TrueNorth Chip
to Start Calit2’s Qualcomm Institute
Pattern Recognition Laboratory
September 16, 2015
New Brain-Inspired Non-von Neumann Processors Are Emerging:
KnuEdge Has Provided Processor to Calit2’s PRL
www.tomshardware.com/news/knuedge-announces-knuverse-and-knupath,31981.html
www.calit2.net/newsroom/release.php?id=2704
“KnuEdge and Calit2
have worked together
since the early days of
the KnuEdge LambdaFabric
processor, when key
personnel and technology
from UC San Diego
provided the genesis for
the first processor design.”
www.calit2.net/newsroom/release.php?id=2726
June 6, 2016
Proposed Cognitive Hardware and Software Ecosystem
On the Pacific Research Platform
• Working With 30 CSE Machine Learning Researchers
– Goal is 320 Game GPUs in 32-40 FIONAs at 10 PRP Campuses
– PRP Couples FIONAs with GPUs into a Condor-Managed Cloud
• PRP Access to Emerging Processors
– IBM TrueNorth, KnuEdge, FPGA, and Qualcomm Snapdragon
• Software Including a Wide Range of Open ML Algorithms
• Metrics for Performance of Processors and Algorithms
Source: Tom DeFanti, Calit2
Multiple Proposals Under Review
FIONA with 8-Game GPUs
Expanding to National Research Platform and Global Research Platform
Via CENIC/Pacific Wave, Internet2, and International Links
PRP’s Current
International
Partners
PRP Timeline
• PRPv1
– A Routed Layer 3 Architecture
– Tested, Measured, Optimized, With Multi-Domain Science Data
– Bring Many Of Our Science Teams Up
– Each Community Thus Will Have Its Own Certificate-Based Access
To its Specific Federated Data Infrastructure
• PRPv2
– Incorporating SDN/SDX, AutoGOLE / NSI
– Advanced IPv6-Only Version with Robust Security Features
– e.g. Trusted Platform Module Hardware and SDN/SDX Software
– Support Rates up to 100Gb/s in Bursts and Streams
– Develop Means to Operate a Shared Federation of Caches
– Cooperating Research Groups
Our Support:
• US National Science Foundation (NSF) awards CNS 0821155 and
CNS-1338192, CNS-1456638, ACI-1540112, and ACI-1541349
• University of California Office of the President CIO
• UCSD Chancellor’s Integrated Digital Infrastructure Program
• UCSD Next Generation Networking initiative
• Calit2 and Calit2 Qualcomm Institute
• CENIC, PacificWave and StarLight
• DOE ESnet

The Pacific Research Platform Two Years In

  • 1.
    “The Pacific ResearchPlatform Two Years In” Welcome and Overview Talk to the Pacific Research Platform “PRPv2” Workshop 2017 University of California, San Diego February 21, 2017 Dr. Larry Smarr Director, California Institute for Telecommunications and Information Technology Harry E. Gruber Professor, Dept. of Computer Science and Engineering Jacobs School of Engineering, UCSD http://lsmarr.calit2.net 1
  • 2.
    DOE ESnet’s ScienceDMZ: A Scalable Network Design Model for Optimizing Science Data Transfers • A Science DMZ integrates 4 key concepts into a unified whole: – A network architecture designed for high-performance applications, with the science network distinct from the general-purpose network – The use of dedicated systems as data transfer nodes (DTNs) – Performance measurement and network testing systems that are regularly used to characterize and troubleshoot the network – Security policies and enforcement mechanisms that are tailored for high performance science environments http://fasterdata.es.net/science-dmz/ Science DMZ Coined 2010 The DOE ESnet Science DMZ and the NSF “Campus Bridging” Taskforce Report Formed the Basis for the NSF Campus Cyberinfrastructure Network Infrastructure and Engineering (CC-NIE) Program
  • 3.
    FIONAs and FIONettes– Flash I/O Network Appliances – Are DTNs: Linux PCs Optimized for DMZs over Distance FIONAs Are Science DMZ Data Transfer Nodes (DTNs) & Also Compute/Visualization/ML Nodes Phil Papadopoulos & Tom DeFanti Joe Keefe & John Graham FIONAS—40G, $8,000 FIONette—1G, $1,000
  • 4.
    The Pacific ResearchPlatform’s Second Year: a Working End-to-End Science-Driven DMZ-Connector FIONAs as Uniform DTN End Points NSF CC*DNI Grant $5M 10/2015-10/2020 PI: Larry Smarr, UC San Diego Calit2 Co-Pis: • Camille Crittenden, UC Berkeley CITRIS, • Tom DeFanti, UC San Diego Calit2, • Philip Papadopoulos, UCSD SDSC, • Frank Wuerthwein, UCSD Physics and SDSC
  • 5.
    PRP Continues toExpand Rapidly While Increasing Connectivity: One Year of Progress – 12 Sites to 20 Sites January 29, 2016 December 15, 2016 Connected 20 DMZs at 10G and 40G, demonstrating disk-to-disk GridFTP at ~7.5G and 12.5G respectively, and 900Mb at 1G
  • 6.
    Many PRP ScienceEngagement Workshops Were Held in 2016 • PRP Workshop Held in Collaboration with UC-Wide Research IT – May 1, 2016 – 45 Attendees – Ten UC Campuses See Talk by Camille Crittenden and Tom DeFanti
  • 7.
    PRP’s First 1.5Years: Connecting Campus Application Teams and Devices
  • 8.
    100 Gbps FIONAat UCSC Connects the UCSC Hyades Cluster to the NERSC Supercomputer at LBNL Supporting UCSC Remote Access to Large Data Subsets of the Dark Energy Spectroscopic Instrument (DESI) and AGORA Galaxy Simulation Data Produced at NERSC. 250 images per night 800GB per night See Talk by Shawfeng Dong UCSC Feb 7, 2017
  • 9.
    40G FIONAs 20x40G PRP-connected WAVE@UCSan Diego PRP Will Enable Distributed Virtual Reality PRP MerWAVE @UC Merced See Talk by Jeff Weekley
  • 10.
    PRP Will Linkthe Laboratories of the Pacific Earthquake Engineering Research Center http://peer.berkeley.edu/
  • 11.
    The Second FIONettewas Deployed at the PEER Facility at UC Berkeley, and its Performance is Being Monitored John Graham Installing FIONette at PEER Feb 10, 2017
  • 12.
    SIO Researchers Beginningto Use Big Data Networks (Prism, IDI, PRP) Jules Jaffe - Microscope Off Scripps Pier Frank Vernon - Expansion of HPWREN Dan Cayan, Mike Dettinger Regional Downscaling of Climate Models Scott Sellars, Marty Ralph Center for Western Weather and Water Extremes
  • 13.
    PRP Backbone SetsStage for 2017 Expansion of HPWREN, Connected to CENIC, into Orange and Riverside Counties • PRP CENIC 100G Link UCSD to SDSU – DTN FIONAs Endpoints – Data Redundancy – Disaster Recovery – High Availability – Network Redundancy • Anchor to CENIC at UCI – PRP FIONA Connects to CalREN-HPR Network – Data Replication Site • Potential Future UCR CENIC Anchor UCR UCI UCSD SDSU Source: Frank Vernon, Greg Hidley, UCSD
  • 14.
    Pacific City Neptune Canada 45°N 47°30’N 130°W 127°30’W N Seattle GigaPOP Portland Possible PRP2017 Expansion to Include NSF’s Ocean Observatory Initiative Fiber Optic SensorNets on Seafloor Off Washington To PRP via Pacific Wave Sea Bottom Electro-optical Cable: 8,000 Volts 10 Gbps Optics Slide Courtesy, John Delaney, UWash John Delaney Visiting UCSD’s SIO For Three Months in 2017 Axial Volcano 140 Scientific Instruments
  • 15.
    Being There -Remote Live High Definition Video of Deep Sea Hydrothermal Vents http://novae.ocean.washington.edu/story/Ashes_CAMHD_Liv Mushroom Hydrothermal Vent on Axial Seamount 1 Mile Below Sea Level Picture Created From 40 HD Frames 14 Minutes Live HD Video On-Line Every 3 Hours 15 feet Slide Courtesy, John Delaney, UWash
  • 16.
    The Future ofSupercomputing Will Blend Traditional HPC and Data Analytics Integrating Non-von Neumann Architectures “High Performance Computing Will Evolve Towards a Hybrid Model, Integrating Emerging Non-von Neumann Architectures, with Huge Potential in Pattern Recognition, Streaming Data Analysis, and Unpredictable New Applications.” Horst Simon, Deputy Director, U.S. Department of Energy’s Lawrence Berkeley National Laboratory
  • 17.
    Calit2’s Qualcomm InstituteHas Established a Pattern Recognition Lab For Machine Learning on non-von Neumann Processors Source: Dr. Dharmendra Modha Founding Director, IBM Cognitive Computing Group August 8, 2014 UCSD ECE Professor Ken Kreutz-Delgado Brings the IBM TrueNorth Chip to Start Calit2’s Qualcomm Institute Pattern Recognition Laboratory September 16, 2015
  • 18.
    New Brain-Inspired Non-vonNeumann Processors Are Emerging: KnuEdge Has Provided Processor to Calit2’s PRL www.tomshardware.com/news/knuedge-announces-knuverse-and-knupath,31981.html www.calit2.net/newsroom/release.php?id=2704 “KnuEdge and Calit2 have worked together since the early days of the KnuEdge LambdaFabric processor, when key personnel and technology from UC San Diego provided the genesis for the first processor design.” www.calit2.net/newsroom/release.php?id=2726 June 6, 2016
  • 19.
    Proposed Cognitive Hardwareand Software Ecosystem On the Pacific Research Platform • Working With 30 CSE Machine Learning Researchers – Goal is 320 Game GPUs in 32-40 FIONAs at 10 PRP Campuses – PRP Couples FIONAs with GPUs into a Condor-Managed Cloud • PRP Access to Emerging Processors – IBM TrueNorth, KnuEdge, FPGA, and Qualcomm Snapdragon • Software Including a Wide Range of Open ML Algorithms • Metrics for Performance of Processors and Algorithms Source: Tom DeFanti, Calit2 Multiple Proposals Under Review FIONA with 8-Game GPUs
  • 20.
    Expanding to NationalResearch Platform and Global Research Platform Via CENIC/Pacific Wave, Internet2, and International Links PRP’s Current International Partners
  • 21.
    PRP Timeline • PRPv1 –A Routed Layer 3 Architecture – Tested, Measured, Optimized, With Multi-Domain Science Data – Bring Many Of Our Science Teams Up – Each Community Thus Will Have Its Own Certificate-Based Access To its Specific Federated Data Infrastructure • PRPv2 – Incorporating SDN/SDX, AutoGOLE / NSI – Advanced IPv6-Only Version with Robust Security Features – e.g. Trusted Platform Module Hardware and SDN/SDX Software – Support Rates up to 100Gb/s in Bursts and Streams – Develop Means to Operate a Shared Federation of Caches – Cooperating Research Groups
  • 22.
    Our Support: • USNational Science Foundation (NSF) awards CNS 0821155 and CNS-1338192, CNS-1456638, ACI-1540112, and ACI-1541349 • University of California Office of the President CIO • UCSD Chancellor’s Integrated Digital Infrastructure Program • UCSD Next Generation Networking initiative • Calit2 and Calit2 Qualcomm Institute • CENIC, PacificWave and StarLight • DOE ESnet