“Coupling Australia’s Researchers
 to the Global Innovation Economy”
                Fourth Lecture in the
Australian American Leadership Dialogue Scholar Tour
                Swinburne University
                 Hawthorn, Australia
                   October 8, 2008
                            Dr. Larry Smarr
     Director, California Institute for Telecommunications and
                       Information Technology
                     Harry E. Gruber Professor,
           Dept. of Computer Science and Engineering
               Jacobs School of Engineering, UCSD
The Large Hadron Collider
Uses a Global Fiber Infrastructure To Connect Its Users




• The grid relies on optical fiber networks to distribute data from
  CERN to 11 major computer centers in Europe, North America,
  and Asia
• The grid is capable of routinely processing 250,000 jobs a day
• The data flow will be ~6 Gigabits/sec or 15 million gigabytes a
  year for 10 to 15 years
EXPReS-Oz eVLBI Using 1 Gbps Lightpaths
                    October 2007




                   Image Credit: Paul Boven,




                Data Streamed at 256-512 Mbps
                      Image created by Paul Boven, JIVE
Satellite image: Blue Marble Next Generation, courtesy of NASA Visible Earth
Next Great Planetary Instrument:
The Square Kilometer Array Requires Dedicated Fiber

                             www.skatelescope.org




          Transfers Of
        1 TByte Images
          World-wide
        Will Be Needed
         Every Minute!
Dedicated 10Gbps Lightpaths Tie Together
         State and Regional Fiber Infrastructure




                                                         Interconnects
                                                           Two Dozen
                                                       State and Regional
 Internet2 Dynamic                                      Optical Networks
   Circuit Network
Under Development




                      NLR 40 x 10Gb Wavelengths
                     Expanding with Darkstrand to 80
Global Lambda Integrated Facility
1 to 10G Dedicated Lambda Infrastructure




            Interconnects Global
     Public Research Innovation Centers
  Source: Maxine Brown, UIC and Robert Patterson, NCSA
AARNet Provides the National and Global Bandwidth
          Required Between Campuses

            25 Gbps to US
            60 Gbps Brisbrane - Sydney - Melbourne
            30 Gbps Melbourne - Adelaide
            10 Gbps Adelaide - Perth
The OptIPuter Project: High Resolution Collaboratory
Over Dedicated Optical Channels to Global Science Data
                                                                            Scalable
                                                                           Adaptive
                                                                           Graphics
                                                                          Environment
                                                                            (SAGE)




                                                                                Now in
                                                                               Sixth and
                                                                               Final Year
                                                                                  Picture
                                                                                  Source:
                                                                                   Mark
                                                                                 Ellisman,
                                                                                 David Lee,
                                                                                Jason Leigh
         Calit2 (UCSD, UCI), SDSC, and UIC Leads—Larry Smarr PI
       Univ. Partners: NCSA, USC, SDSU, NW, TA&M, UvA, SARA, KISTI, AIST
        Industry: IBM, Sun, Telcordia, Chiaro, Calient, Glimmerglass, Lucent
My OptIPortalTM – Affordable
    Termination Device for the OptIPuter Global Backplane
•    20 Dual CPU Nodes, 20 24” Monitors, ~$50,000
•    1/4 Teraflop, 5 Terabyte Storage, 45 Mega Pixels--Nice PC!
•    Scalable Adaptive Graphics Environment ( SAGE) Jason Leigh, EVL-UIC




                                  Source: Phil Papadopoulos SDSC, Calit2
Students Learn Case Studies
in the Context of Diverse Medical Evidence



                   UIC Anatomy Class




      electronic visualization laboratory, university of illinois at chicago
CoreWall:
           Use of OptIPortal in Geosciences
Using High Resolution Core Images to Study                                          Before
 Paleogeology, Learning about the History
    of The Planet to Better Understand
        Causes of Global Warming

5 Deployed In Antarctica
   www.corewall.org                            After




           electronic visualization laboratory, university of illinois at chicago
Group Analysis of
Global Change Supercomputer Simulations

                         Latest Atmospheric Data
                         is Displayed for Classes,
                       Research Meetings, and Lunch
                                Gatherings-
                          A Truly Communal Wall

         Before
                                                After




              Source: U of Michigan
         Atmospheric Sciences Department
AARNet Pioneered Uncompressed HD VTC with
UWashington Research Channel--Supercomputing 2004




          Canberra               Pittsburgh
OptIPlanet Collaboratory Integrating HD with OptIPortals
           Between Calit2 and U Washington
Photo Credit: Alan Decker                                             Feb. 29, 2008

                                          Ginger
                                        Armbrust’s
                                         Diatoms:
                                       Micrographs,
                                      Chromosomes,
                                         Genetic
                                        Assembly

    iHDTV: 1500 Mbits/sec Calit2 to
    UW Research Channel Over NLR




                                                  UW’s Research Channel
                                                    Michael Wellings
Two New Calit2 Buildings Provide
New Laboratories for “Living in the Future”
               •   “Convergence” Laboratory Facilities
                   – Nanotech, BioMEMS, Chips, Radio, Photonics
                   – Virtual Reality, Digital Cinema, HDTV, Gaming
               •   Over 1000 Researchers in Two Buildings
                   – Linked via Dedicated Optical Networks




UC Irvine
                  www.calit2.net
            Preparing for a World in Which
               Distance is Eliminated…
The Calit2 OptIPortals at UCSD and UCI
              Are Now a Gbit/s HD Collaboratory
                                           NASA Ames Visit Feb. 29, 2008




                                                       HiPerVerse:
                                                    First ½ Gigapixel
                                                       Distributed
                                                       OptIPortal-
                                                        124 Tiles
                                                     Sept. 15, 2008

                                    Calit2@ UCI wall




Calit2@ UCSD wall

            UCSD cluster: 15 x Quad core Dell XPS with Dual nVIDIA 5600s
            UCI cluster: 25 x Dual Core Apple G5
EVL’s SAGE OptIPortal VisualCasting
              Multi-Site OptIPuter Collaboratory
                                       CENIC CalREN-XD Workshop Sept. 15, 2008
EVL-UI Chicago
                 At Supercomputing 2008 Austin, Texas
                           November, 2008                             Streaming 4k

                    SC08 Bandwidth Challenge Entry


                  On site:                              Remote:

        SARA (Amsterdam)
U Michigan
                                                  U of Michigan
        GIST / KISTI (Korea)                         UIC/EVL
        Osaka Univ. (Japan)                     U of Queensland
        Masaryk Univ. (CZ),                Russian Academy of Science
               Calit2
                 Requires 10 Gbps Lightpath to Each Site


                 Source: Jason Leigh, Luc Renambot, EVL, UI Chicago
CENIC’s New “Hybrid Network” - Traditional Routed IP
 and the New Switched Ethernet and Optical Services




 ~ $14M
Invested
    in
Upgrade




             Now
           Campuses
            Need to
            Upgrade


                      Source: Jim Dolgonas, CENIC
The “Golden Spike” UCSD Experimental Optical Core:
     Ready to Couple Users to CENIC L1, L2, L3 Services
                                       Quartzite Communications
  To 10GigE cluster                    Goals by 2008:
                                                  Core Year 3
   node interfaces
                                       >= 60 endpoints at 10 GigE                CENIC L1, L2
                                       >= 30 Packet switched Wavelength
                                               Quartzite         Selective
                                                                                   Services
                                                 Corewavelengths
       .....


                                                                  Switch
                                       >= 30 Switched
                                       >= 400 Connected endpoints
                                                                Lucent                   To 10GigE cluster
                                                                                        node interfaces and
                                                                                             other switches


To cluster nodes
                   .....
                                                                  Glimmerglass
                                          Approximately 0.5 Tbps                           .....
                                                                                                   To cluster nodes


            GigE Switch with
           Dual 10GigE Upliks
                                           Arrive at the “Optical”
                                                             Production
                                                               OOO
                                                               Switch
To cluster nodes
                                         Center of Hybrid Campus
                                               32 10GigE

                   .....
                                                    Switch                          GigE Switch with
                                                                                   Dual 10GigE Upliks
                                                                       Force10
                                                 ...


                                        To             Packet Switch             CalREN-HPR
            GigE Switch with
           Dual 10GigE Upliks           other                                     Research
                                        nodes
                                                                                    Cloud
    GigE
                           Funded by
 10GigE
                            NSF MRI                                              Campus Research
  4 GigE
  4 pair fiber               Grant                                                   Cloud
                                                       Cisco 6509
                                                       Juniper T320

                                                  OptIPuter Border Router
                                 Source: Phil Papadopoulos, SDSC/Calit2
                                      (Quartzite PI, OptIPuter co-PI)
Use Campus Investment in Fiber and Networks
    to Physically Connect Campus Resources



                HPC System
Cluster Condo

                 PetaScale
                Data Analysis                UCSD Storage
                   Facility

UC Grid Pilot
                   Digital
                 Collections      Research
                  Manager          Cluster            OptIPortal
  Research
                10Gbps
 Instrument

                Source:Phil Papadopoulos,
                       SDSC/Calit2
AARNet 10Gbps Access Product is Here!!!

• HD and Other High Bandwidth Applications Combined
  with “Big Research” Pushing Large Data Sets Means
  1 Gbps is No Longer Adequate for All Users
• AARNet Helps Connect Campus Users or Remote Instruments
• Will Permit Researchers to Exchange Large Amounts of
  Data within Australia, and Internationally via SXTransPORT




                          © 2008, AARNet Pty Ltd       21
                Slide From Chris Hancock, CEO AARNet
OptIPortals
                 Are Being Adopted Globally
                 U Melbourne



  AIST-Japan     Osaka U-Japan            KISTI-Korea         CNIC-China


   NCHC-Taiwan
                 Russian                                  U Queensland
                 Academy
                 Sciences
                 Moscow             SARA- Netherlands Brno-Czech Republic




And Today
 Monash!
    EVL@UIC          Calit2@UCSD           Calit2@UCI   CICESE, Mexico



                                   CSIRO Discovery Center Canberra
Green
                                                    Initiative:

                                                    Can Optical
                                                  Fiber Replace
                                                   Airline Travel
                                                  for Continuing
                                                  Collaborations
                                                         ?




Source: Maxine Brown, OptIPuter Project Manager
“Using the Link to Build the Link”
 Calit2 and Univ. Melbourne Technology Teams
Jan 15, 2008




     No Calit2 Person Physically Flew to Australia to Bring This Up!
            www.calit2.net/newsroom/release.php?id=1219
UM Professor Graeme Jackson Planning
  Brain Surgery for Severe Epilepsy




                                       Jan 15, 2008
    www.calit2.net/newsroom/release.php?id=1219
Smarr American Australian Leadership Dialogue
OptIPlanet Collaboratory Lecture Tour October 2008
AARNet National Network
                          •   Oct 2—University of Adelaide
                          •   Oct 6—Univ of Western Australia
                          •   Oct 8—Monash Univ.; Swinburne
                              Univ.
                          •   Oct 9—Univ. of Melbourne
                          •   Oct 10—Univ. of Queensland
                          •   Oct 13—Univ. of Technology
                              Sydney
                          •   Oct 14—Univ. of New South Wales
                          •   Oct 15—ANU; AARNet;
                              Leadership Dialogue Scholar
                              Oration, Canberra
                          •   Oct 16—CSIRO, Canberra
                          •   Oct 17—Sydney Univ.
AARNet’s “EN4R” –
Experimental Network For Researchers

                                   • For Researchers

                                   • Free Access for
                                     up to 12 months

                                   • 2 Circuits Reserved
                                     for EN4R on Each
                                     Optical Backbone
                                     Segment

                                   • Access to North
                                     America via.
                                     SXTransPORT
                                                  27
        Source: Chris Hancock, AARNet
“To ensure a competitive
  economy for the 21st century,
   the Australian Government
   should set a goal of making
     Australia the pre-eminent
    location to attract the best
 researchers and be a preferred
partner for international research
  institutions, businesses and
      national governments.”
HD Talk
                to Monash University from Calit2




July 30, 2008

July 31, 2008
21st Century Australian Information Infrastructure:
    Joining the Global Data-Intensive Collaboratory
• All Data-Intensive Australian:
  – Researchers
  – Scientific Instruments
  – Data Repositories
• Should Have Best-of-Breed End-End Connectivity
• Today, that Means 10Gbps Lightpaths
• This Requires a Spirited Partnership:
  –   Federal
  –   State
  –   Universities and CSIRO
  –   AARnet
             The Mutuality Principle at Work!

Coupling Australia’s Researchers to the Global Innovation Economy

  • 1.
    “Coupling Australia’s Researchers to the Global Innovation Economy” Fourth Lecture in the Australian American Leadership Dialogue Scholar Tour Swinburne University Hawthorn, Australia October 8, 2008 Dr. Larry Smarr Director, California Institute for Telecommunications and Information Technology Harry E. Gruber Professor, Dept. of Computer Science and Engineering Jacobs School of Engineering, UCSD
  • 2.
    The Large HadronCollider Uses a Global Fiber Infrastructure To Connect Its Users • The grid relies on optical fiber networks to distribute data from CERN to 11 major computer centers in Europe, North America, and Asia • The grid is capable of routinely processing 250,000 jobs a day • The data flow will be ~6 Gigabits/sec or 15 million gigabytes a year for 10 to 15 years
  • 3.
    EXPReS-Oz eVLBI Using1 Gbps Lightpaths October 2007 Image Credit: Paul Boven, Data Streamed at 256-512 Mbps Image created by Paul Boven, JIVE Satellite image: Blue Marble Next Generation, courtesy of NASA Visible Earth
  • 4.
    Next Great PlanetaryInstrument: The Square Kilometer Array Requires Dedicated Fiber www.skatelescope.org Transfers Of 1 TByte Images World-wide Will Be Needed Every Minute!
  • 5.
    Dedicated 10Gbps LightpathsTie Together State and Regional Fiber Infrastructure Interconnects Two Dozen State and Regional Internet2 Dynamic Optical Networks Circuit Network Under Development NLR 40 x 10Gb Wavelengths Expanding with Darkstrand to 80
  • 6.
    Global Lambda IntegratedFacility 1 to 10G Dedicated Lambda Infrastructure Interconnects Global Public Research Innovation Centers Source: Maxine Brown, UIC and Robert Patterson, NCSA
  • 7.
    AARNet Provides theNational and Global Bandwidth Required Between Campuses 25 Gbps to US 60 Gbps Brisbrane - Sydney - Melbourne 30 Gbps Melbourne - Adelaide 10 Gbps Adelaide - Perth
  • 8.
    The OptIPuter Project:High Resolution Collaboratory Over Dedicated Optical Channels to Global Science Data Scalable Adaptive Graphics Environment (SAGE) Now in Sixth and Final Year Picture Source: Mark Ellisman, David Lee, Jason Leigh Calit2 (UCSD, UCI), SDSC, and UIC Leads—Larry Smarr PI Univ. Partners: NCSA, USC, SDSU, NW, TA&M, UvA, SARA, KISTI, AIST Industry: IBM, Sun, Telcordia, Chiaro, Calient, Glimmerglass, Lucent
  • 9.
    My OptIPortalTM –Affordable Termination Device for the OptIPuter Global Backplane • 20 Dual CPU Nodes, 20 24” Monitors, ~$50,000 • 1/4 Teraflop, 5 Terabyte Storage, 45 Mega Pixels--Nice PC! • Scalable Adaptive Graphics Environment ( SAGE) Jason Leigh, EVL-UIC Source: Phil Papadopoulos SDSC, Calit2
  • 10.
    Students Learn CaseStudies in the Context of Diverse Medical Evidence UIC Anatomy Class electronic visualization laboratory, university of illinois at chicago
  • 11.
    CoreWall: Use of OptIPortal in Geosciences Using High Resolution Core Images to Study Before Paleogeology, Learning about the History of The Planet to Better Understand Causes of Global Warming 5 Deployed In Antarctica www.corewall.org After electronic visualization laboratory, university of illinois at chicago
  • 12.
    Group Analysis of GlobalChange Supercomputer Simulations Latest Atmospheric Data is Displayed for Classes, Research Meetings, and Lunch Gatherings- A Truly Communal Wall Before After Source: U of Michigan Atmospheric Sciences Department
  • 13.
    AARNet Pioneered UncompressedHD VTC with UWashington Research Channel--Supercomputing 2004 Canberra Pittsburgh
  • 14.
    OptIPlanet Collaboratory IntegratingHD with OptIPortals Between Calit2 and U Washington Photo Credit: Alan Decker Feb. 29, 2008 Ginger Armbrust’s Diatoms: Micrographs, Chromosomes, Genetic Assembly iHDTV: 1500 Mbits/sec Calit2 to UW Research Channel Over NLR UW’s Research Channel Michael Wellings
  • 15.
    Two New Calit2Buildings Provide New Laboratories for “Living in the Future” • “Convergence” Laboratory Facilities – Nanotech, BioMEMS, Chips, Radio, Photonics – Virtual Reality, Digital Cinema, HDTV, Gaming • Over 1000 Researchers in Two Buildings – Linked via Dedicated Optical Networks UC Irvine www.calit2.net Preparing for a World in Which Distance is Eliminated…
  • 16.
    The Calit2 OptIPortalsat UCSD and UCI Are Now a Gbit/s HD Collaboratory NASA Ames Visit Feb. 29, 2008 HiPerVerse: First ½ Gigapixel Distributed OptIPortal- 124 Tiles Sept. 15, 2008 Calit2@ UCI wall Calit2@ UCSD wall UCSD cluster: 15 x Quad core Dell XPS with Dual nVIDIA 5600s UCI cluster: 25 x Dual Core Apple G5
  • 17.
    EVL’s SAGE OptIPortalVisualCasting Multi-Site OptIPuter Collaboratory CENIC CalREN-XD Workshop Sept. 15, 2008 EVL-UI Chicago At Supercomputing 2008 Austin, Texas November, 2008 Streaming 4k SC08 Bandwidth Challenge Entry On site: Remote: SARA (Amsterdam) U Michigan U of Michigan GIST / KISTI (Korea) UIC/EVL Osaka Univ. (Japan) U of Queensland Masaryk Univ. (CZ), Russian Academy of Science Calit2 Requires 10 Gbps Lightpath to Each Site Source: Jason Leigh, Luc Renambot, EVL, UI Chicago
  • 18.
    CENIC’s New “HybridNetwork” - Traditional Routed IP and the New Switched Ethernet and Optical Services ~ $14M Invested in Upgrade Now Campuses Need to Upgrade Source: Jim Dolgonas, CENIC
  • 19.
    The “Golden Spike”UCSD Experimental Optical Core: Ready to Couple Users to CENIC L1, L2, L3 Services Quartzite Communications To 10GigE cluster Goals by 2008: Core Year 3 node interfaces >= 60 endpoints at 10 GigE CENIC L1, L2 >= 30 Packet switched Wavelength Quartzite Selective Services Corewavelengths ..... Switch >= 30 Switched >= 400 Connected endpoints Lucent To 10GigE cluster node interfaces and other switches To cluster nodes ..... Glimmerglass Approximately 0.5 Tbps ..... To cluster nodes GigE Switch with Dual 10GigE Upliks Arrive at the “Optical” Production OOO Switch To cluster nodes Center of Hybrid Campus 32 10GigE ..... Switch GigE Switch with Dual 10GigE Upliks Force10 ... To Packet Switch CalREN-HPR GigE Switch with Dual 10GigE Upliks other Research nodes Cloud GigE Funded by 10GigE NSF MRI Campus Research 4 GigE 4 pair fiber Grant Cloud Cisco 6509 Juniper T320 OptIPuter Border Router Source: Phil Papadopoulos, SDSC/Calit2 (Quartzite PI, OptIPuter co-PI)
  • 20.
    Use Campus Investmentin Fiber and Networks to Physically Connect Campus Resources HPC System Cluster Condo PetaScale Data Analysis UCSD Storage Facility UC Grid Pilot Digital Collections Research Manager Cluster OptIPortal Research 10Gbps Instrument Source:Phil Papadopoulos, SDSC/Calit2
  • 21.
    AARNet 10Gbps AccessProduct is Here!!! • HD and Other High Bandwidth Applications Combined with “Big Research” Pushing Large Data Sets Means 1 Gbps is No Longer Adequate for All Users • AARNet Helps Connect Campus Users or Remote Instruments • Will Permit Researchers to Exchange Large Amounts of Data within Australia, and Internationally via SXTransPORT © 2008, AARNet Pty Ltd 21 Slide From Chris Hancock, CEO AARNet
  • 22.
    OptIPortals Are Being Adopted Globally U Melbourne AIST-Japan Osaka U-Japan KISTI-Korea CNIC-China NCHC-Taiwan Russian U Queensland Academy Sciences Moscow SARA- Netherlands Brno-Czech Republic And Today Monash! EVL@UIC Calit2@UCSD Calit2@UCI CICESE, Mexico CSIRO Discovery Center Canberra
  • 23.
    Green Initiative: Can Optical Fiber Replace Airline Travel for Continuing Collaborations ? Source: Maxine Brown, OptIPuter Project Manager
  • 24.
    “Using the Linkto Build the Link” Calit2 and Univ. Melbourne Technology Teams Jan 15, 2008 No Calit2 Person Physically Flew to Australia to Bring This Up! www.calit2.net/newsroom/release.php?id=1219
  • 25.
    UM Professor GraemeJackson Planning Brain Surgery for Severe Epilepsy Jan 15, 2008 www.calit2.net/newsroom/release.php?id=1219
  • 26.
    Smarr American AustralianLeadership Dialogue OptIPlanet Collaboratory Lecture Tour October 2008 AARNet National Network • Oct 2—University of Adelaide • Oct 6—Univ of Western Australia • Oct 8—Monash Univ.; Swinburne Univ. • Oct 9—Univ. of Melbourne • Oct 10—Univ. of Queensland • Oct 13—Univ. of Technology Sydney • Oct 14—Univ. of New South Wales • Oct 15—ANU; AARNet; Leadership Dialogue Scholar Oration, Canberra • Oct 16—CSIRO, Canberra • Oct 17—Sydney Univ.
  • 27.
    AARNet’s “EN4R” – ExperimentalNetwork For Researchers • For Researchers • Free Access for up to 12 months • 2 Circuits Reserved for EN4R on Each Optical Backbone Segment • Access to North America via. SXTransPORT 27 Source: Chris Hancock, AARNet
  • 28.
    “To ensure acompetitive economy for the 21st century, the Australian Government should set a goal of making Australia the pre-eminent location to attract the best researchers and be a preferred partner for international research institutions, businesses and national governments.”
  • 29.
    HD Talk to Monash University from Calit2 July 30, 2008 July 31, 2008
  • 30.
    21st Century AustralianInformation Infrastructure: Joining the Global Data-Intensive Collaboratory • All Data-Intensive Australian: – Researchers – Scientific Instruments – Data Repositories • Should Have Best-of-Breed End-End Connectivity • Today, that Means 10Gbps Lightpaths • This Requires a Spirited Partnership: – Federal – State – Universities and CSIRO – AARnet The Mutuality Principle at Work!