How Global-Scale Personal Lightwaves are Transforming Scientific Research

Slide 1: How Global-Scale Personal Lighwaves are Transforming Scientific Research Distinguished Lecturer Technology for a Changing World Series Baskin School of Engineering UC Santa Cruz March 22, 2007 Dr. Larry Smarr Director, California Institute for Telecommunications and Information Technology Harry E. Gruber Professor, Dept. of Computer Science and Engineering Jacobs School of Engineering, UCSD

Slide 2: Abstract During the last few years, a radical restructuring of optical networks supporting e-Science projects is beginning to occur around the world. U.S. universities are beginning to acquire access to high-bandwidth lightwaves (termed “lambdas”) on fiber optics through the National LambdaRail and the Global Lambda Integrated Facility. These user-controlled 1- or 10- Gbps lambdas are providing direct access to global data repositories, scientific instruments, and computational resources from the researchers’ Linux clusters in their campus laboratories. These dedicated connections have a number of significant advantages over shared Internet connections, including high bandwidth, controlled performance (no jitter), lower cost-per-unit bandwidth, and security. These lambdas enable the Grid program to be completed, in that they add the network elements to the compute and storage elements which can be discovered, reserved, and integrated by the Grid middleware to form global LambdaGrids. I will describe how LambdaGrids enable new capabilities in medical imaging, Earth sciences, interactive ocean observatories, and marine microbial metagenomics.

Slide 3: Great to be Back in Slug Land! Source: Benjamin Smarr, UCSC ‘04

Slide 4: Two New Calit2 Buildings Provide New Laboratories for “Living in the Future” • Over 1000 Researchers in Two Buildings – Linked via Dedicated Optical Networks – International Conferences and Testbeds • New Laboratories – Nanotechnology – Virtual Reality, Digital Cinema UC San Diego UC Irvine Preparing for a World in Which Distance is Eliminated…

Slide 5: Calit2--A Systems Approach to the Future of the Internet and its Transformation of Our Society Calit2 Has Assembled a Complex Social Network of Over 350 UC San Diego & UC Irvine Faculty Working in Multidisciplinary Teams With Staff, Students, Industry, and the Community Integrating Technology Consumers and Producers Into “Living Laboratories” www.calit2.net

Slide 6: Calit2 is Experimenting with Open Reconfigurable Work Spaces to Enhance Collaboration Photos by John Durant; Barbara Haynor, Calit2

Slide 7: Calit2 Materials Nano3 Facility and Devices Laboratory: “Nano3”–NanoScience, CALIT2.UCSD NanoEngineering, NanoMedicine 10,000 sq. feet State-of-the-Art Materials and Devices Laboratory Similar Clean Rooms at UCI Source: Bernd Fruhberger, Calit2

Slide 8: Calit2 “Lives in the Future” By Building Systems of Emerging Disruptive Technologies Co-Evolution of Personal Automobile and Highway/Petroleum Infrastructure Technologies Diffuse Into Society Following an S-Curve Calit2 Works Here { Source: Harry Dent, The Great Boom Ahead

Slide 9: The Calit2@UCSD Building is Designed for Prototyping Extremely High Bandwidth Applications 1.8 Million Feet of Cat6 Ethernet Cabling 24 Fiber Pairs to Each Lab UCSD has one 10G CENIC Over 10,000 Connection for Individual ~30,000 Users 1 Gbps Drops in the Building ~10G per Person 150 Fiber Strands to Building; Experimental Roof Radio Antenna Farm Photo: Tim Beach, Calit2 Ubiquitous WiFi

Slide 10: Dedicated Optical Channels Makes High Performance Cyberinfrastructure Possible (WDM) 10 Gbps per User ~ 200x Shared Internet Throughput c* f Source: Steve Wallach, Chiaro Networks “Lambdas” Parallel Lambdas are Driving Optical Networking The Way Parallel Processors Drove 1990s Computing

Slide 11: e-Science Data Intensive Science Will Require LambdaGrid Cyberinfrastructure

Slide 12: High Energy and Nuclear Physics A Terabit/s WAN by 2013! Source: Harvey Newman, Caltech

Slide 13: Gigabit Fibers on the Ocean Floor -- Controlling Sensors and HDTV Cameras Remotely www.neptune.washington.edu LOOKING: (Laboratory for the Ocean Observatory Knowledge Integration Grid) http://lookingtosea.ucsd.edu/ • Goal: – Prototype Cyberinfrastructure for NSF’s Ocean Research Interactive Observatory Networks (ORION) Building on OptIPuter • LOOKING NSF ITR with PIs: – John Orcutt & Larry Smarr - UCSD LOOKING is – John Delaney & Ed Lazowska –UW Driven By – Mark Abbott – OSU NEPTUNE CI • Collaborators at: Requirements – MBARI, WHOI, NCSA, UIC, CalPoly, UVic, CANARIE, Microsoft, NEPTUNE- Canarie Making Management of Gigabit Flows Routine

Slide 14: First Remote Interactive High Definition Video Exploration of Deep Sea Vents Canadian-U.S. Collaboration Source John Delaney & Deborah Kelley, UWash

Slide 15: High Definition Still Frame of Hydrothermal Vent Ecology 2.3 Km Deep 1 cm. Source: John Delaney and Research Channel, U Washington White Filamentous Bacteria on 'Pill Bug' Outer Carapace

Slide 16: Calit2 Has Become a Global Hub for Optical Connections Between University Research Centers at 10Gbps Maxine Brown, Tom DeFanti, Co-Chairs iGrid 2005 TH E GL OBAL LAMBDA INTEGRATED FACILITY www.igrid2005.org September 26-30, 2005 Calit2 @ University of California, San Diego California Institute for Telecommunications and Information Technology 21 Countries Driving 50 Demonstrations 1 or 10Gbps to Calit2@UCSD Building Sept 2005

Slide 17: iGrid Lambda Digital Cinema Streaming Services: Telepresence Meeting in Calit2 Digital Cinema Auditorium Lays Technical Basis for Global Keio University Digital President Anzai Cinema Sony UCSD NTT Chancellor Fox SGI

Slide 18: Emerging CineGrid Infrastructure Tokyo Cisco is building two 10 GigE \"Cisco Waves” on NLR on the West Coast and switches Seattle for access points in San Diego, Los Angeles, Sunnyvale, & Seattle for CineGrid Toronto Europe Chicago CENIC is making available persistent 1 GigE access ports in San Diego, Los Angeles, Sunnyvale Sunnyvale, & San Francisco for CineGrid and the fiber for 2x10GigE between UCSD and LA Los Angeles CalIT2 Via GLIF, CineGrid extends to Japan via Seattle & San Diego Chicago; to Canada via Seattle & Chicago; to Europe via Chicago & Amsterdam. CineGrid Cisco 6506 Further extension likely to China, Korea, 10GigE Cisco NLR Wave Singapore, India, New Zealand, Australia, others. 1& 10 GigE CENIC Waves IEEAF Wave via PNWGP/TLEX CAVEwave (CENIC and NLR via PNWGP JGN2 CA*net4

Slide 19: The Synergy of Digital Art and Science Visualization of JPL Simulation of Monterey Bay 4k Resolution Source: Donna Cox, Robert Patterson, NCSA Funded by NSF LOOKING Grant

Slide 20: National Lambda Rail (NLR) and TeraGrid Provides Cyberinfrastructure Backbone for U.S. Researchers Seattle NSF’s TeraGrid Has 4 x 10Gb International Lambda Backbone Collaborators Portland Boise UC-TeraGrid Ogden/ UIC/NW-Starlight Salt Lake City Cleveland Chicago New York City San Francisco Denver Pittsburgh Washington, DC Kansas City Raleigh Albuquerque Tulsa Los Angeles Atlanta San Diego Phoenix Dallas Baton Rouge Las Cruces / Links Two El Paso Jacksonville Dozen State and Pensacola Regional Optical Houston NLR Is to San Antonio Networks Merge With Internet2 NLR 4 x 10Gb Lambdas Initially Capable of 40 x 10Gb wavelengths at Buildout

Slide 21: The OptIPuter Project – Creating High Resolution Portals Over Dedicated Optical Channels to Global Science Data • NSF Large Information Technology Research Proposal – Calit2 (UCSD, UCI) and UIC Lead Campuses—Larry Smarr PI – Partnering Campuses: SDSC, USC, SDSU, NCSA, NW, TA&M, UvA, SARA, NASA Goddard, KISTI, AIST, CRC(Canada), CICESE (Mexico) • Engaged Industrial Partners: – IBM, Sun, Telcordia, Chiaro, Calient, Glimmerglass, Lucent • $13.5 Million Over Five Years—Now In the Fifth Year NIH Biomedical Informatics Research Network NSF EarthScope and ORION Go Slugs!

Slide 22: OptIPuter Software Architecture--a Service-Oriented Architecture Integrating Lambdas Into the Grid Distributed Applications/ Web Services Source: Andrew Chien, UCSD Visualization Telescience SAGE JuxtaView Data Services LambdaRAM Vol-a-Tile Distributed Virtual Computer (DVC) API DVC Configuration DVC Runtime Library DVC Services DVC Job DVC Scheduling Communication DVC Core Services Resource Namespace Security High Speed Storage Identify/Acquire Management Management Communication Services Globus PIN/PDC GRAM GSI XIO RobuStore Discovery and Control GTP XCP UDT I Lambdas P CEP LambdaStream RBUDP

Slide 23: OptIPuter / OptIPortal Demonstration of SAGE Applications MagicCarpet Bitplayer Streaming Blue Marble Streaming animation dataset from San Diego of tornado simulation to EVL using UDP. using UDP. 6.7Gbps 516 Mbps ~ 9 Gbps in Total. SAGE Can Simultaneously Support These Applications Without Decreasing Their Performance SVC JuxtaView Locally streaming Locally streaming the aerial HD camera live photography of downtown video using UDP. Chicago using TCP. 538Mbps 850 Mbps Source: Xi Wang, UIC/EVL

Slide 24: My OptIPortalTM – Affordable Termination Device for the OptIPuter Global Backplane • 20 Dual CPU Nodes, 20 24” Monitors, ~$50,000 • 1/4 Teraflop, 5 Terabyte Storage, 45 Mega Pixels--Nice PC! • Scalable Adaptive Graphics Environment ( SAGE) Jason Leigh, EVL-UIC Source: Phil Papadopoulos SDSC, Calit2

Slide 25: Showing your Science at Meetings-- The Portable Mini-Mac Wall ANL’s Rick Stevens Studying Deep Sea Vent Ecology at Supercomputing ‘06

Slide 26: Paul Gilna Ex. Dir. PI Larry Smarr Announced January 17, 2006 $24.5M Over Seven Years

Slide 27: Most of Evolutionary Time Was in the Microbial World Slug You is Are Here Here Tree of Life Derived from 16S rRNA Sequences Source: Carl Woese, et al

Slide 28: Marine Genome Sequencing Project – Measuring the Genetic Diversity of Ocean Microbes Need Ocean Data Sorcerer II Data Will Double Number of Proteins in GenBank!

Slide 29: The First Science Results of Have Been Published from the Global Ocean Sampling Expedition March 2007

Slide 30: GOS Analysis -- Protein Families in Nature Have Been Poorly Explored Thus Far • Novel Sequence Similarity Clustering Process Predicts Proteins and Groups Related Sequences Into Clusters (Families) • GOS Proteins Increase Size / Diversity of Many Protein Families • 1,700 Novel GOS-Only Clusters Identified (>20 per Cluster) – 10% of 17,000 Clusters NCBI_nr GOS + NCBI_nr + Ensembl + TIGR Gene Indices + Prokaryotic Genomes Source: Shibu Yooseph, Granger Sutton, --JCVI

Slide 31: The Calit2 CAMERA Microbial Metagenomics Server is Open to the Community PLOS Biology March 2007

Slide 32: Calit2’s Direct Access Core Architecture Will Create Next Generation Metagenomics Server Sargasso Sea Data Sorcerer II Expedition Dedicated (GOS) Compute Farm Traditional User (1000s of CPUs) JGI Community W E B PORTAL Sequencing Project + Web Services Moore Marine Data- Request 10 GigE Microbial Project Base Fabric Response Farm NASA and NOAA Local Satellite Data Environment Flat File Community Microbial Direct Server Web Metagenomics Data Access Farm Lambda (other service) Cnxns Local Cluster TeraGrid: Cyberinfrastructure Backplane (scheduled activities, e.g. all by all comparison) (10,000s of CPUs) Source: Phil Papadopoulos, SDSC, Calit2

Slide 33: Calit2 CAMERA Production Compute and Storage Complex 512 Processors ~5 Teraflops ~ 200 Terabytes Storage

Slide 34: The Calit2 CAMERA Metagenomics Site is Now Active http://camera.calit2.net/

Slide 35: CAMERA is Already in Use Worldwide • Users from over 200 Institutions in 30 Countries – > 500 Research Scientists, Postdocs, and Students – 1/3 From Outside U.S. • North & South America, Europe, and the South Pacific – Including Australia, Brazil, Canada, France, Germany, Israel, Japan, Mexico, the Netherlands, Spain, Sweden, Switzerland, and the U.K.

Slide 36: Interactive Exploration of Marine Genomes Using 100 Million Pixels Ginger Armburst (UW), Terry Gaasterland (UCSD SIO)

Slide 37: Use of Tiled Display Wall OptIPortal to Interactively View Microbial Genome Acidobacteria bacterium Ellin345 Soil Bacterium 5.6 Mb

Slide 38: Use of Tiled Display Wall OptIPortal to Interactively View Microbial Genome Source: Raj Singh, UCSD

Slide 39: Use of Tiled Display Wall OptIPortal to Interactively View Microbial Genome Source: Raj Singh, UCSD

Slide 40: Calit2 is Now OptIPuter Connecting Remote OptIPortal Moore-Funded Microbial Researchers via NLR UW OptIPortals NW! UIC EVL MIT JCVI UCI SIO UCSD SDSU OptIPortal CICESE CAMERA Servers

Slide 41: How Do You Get From Your Lab to the National LambdaRail? “Research is being stalled by ‘information overload,’ Mr. Bement said, because data from digital instruments are piling up far faster than researchers can study. In particular, he said, campus networks need to be improved. High-speed data lines crossing the nation are the equivalent of six-lane superhighways, he said. But networks at colleges and universities are not so capable. “Those massive conduits are reduced to two-lane roads at most college and university campuses,” he said. Improving cyberinfrastructure, he said, “will transform the capabilities of campus-based scientists.” -- Arden Bement, the director of the National Science Foundation www.ctwatch.org

Slide 42: 2007

Slide 43: OptIPuter@UCI is Up and Working ONS 15540 WDM at UCI campus MPOE (CPL) 10 GE DWDM Network Line 1 GE DWDM Network Line Tustin CENIC CalREN POP UCSD Optiputer Calit2 Building Wave-2: layer-2 GE. Network 67.58.33.0/25 using 11- Floor 4 Catalyst 6500 126 at UCI. GTWY is .1 Engineering Gateway Building, SPDS Kim Jitter Floor 3 Catalyst 6500 Measurements Lab E1127 Wave-1: layer-2 GE Catalyst 3750 in Los 67.58.21.128/25 UCI using 1st floor IDF Angeles 141-254. GTWY .128 Floor 2 Catalyst 6500 Catalyst 3750 in NACS Machine ESMF HIPerWall UCInet Room (Optiputer) Catalyst 6500, Beckman Laser Institute Bldg. 1st floor MDF Catalyst 3750 in CSI Berns’ Lab-- Remote Microscopy 10 GE Created 09-27-2005 by Garrett Hildebrand Wave 1 1GE Modified 02-28-2009 by Smarr/Hildebrand Wave 2 1GE

Slide 44: Calit2/SDSC Proposal to Create a UC Cyberinfrastructure of OptIPuter “On-Ramps” to TeraGrid Resources OptIPuter + CalREN-XD + TeraGrid = “OptiGrid” UC Davis UC Berkeley UC San Francisco UC Merced UC Santa Cruz UC Los Angeles UC Santa Barbara UC Riverside UC Irvine Creating a Critical Mass of End Users UC San Diego on a Secure LambdaGrid Source: Fran Berman, SDSC , Larry Smarr, Calit2

Slide 45: Great Opportunity to Bring Gigabit Fiber to Monterey Bay Research & Education Institutions