Quantum Leap in Open Source Collaboration

    Bridging the gap between campus infrastructures




© Ton van Alebeek




   Harold Teunissen et al.
   April 2012 - #I2SMM12
Cyberinfra in the Netherlands

• All ICT activities for Higher Education and Research in the
  Netherlands are brought under one umbrella




                  National Research &       Commercial ICT       Scientific Computing &   Shared Professional and
                  Education Network       Products & Services            Storage          Educational Services




Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA                                                2
Big Data — when size does matter




                                                   • More is not less...
                                                        - Large scale and global research
                                                        - Collaboration in Virtual Organizations
                                                        - Shared resources
                                                        - High dependence on ICT



Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA                         3
Collaborations and Partnerships

• Virtual Research Team
                                                                           Virtual Laboratory

     - Focussed on doing research
                                                                             Virtual Project
     - Small scale
     - Temporary and elastic                                                 Virtual Colla-
                                                                               boration

     - Little ICT awareness
• “Virtual” Infrastructure Provider                                   Virtual Breeding Environment



     - Focussed on providing infrastructure for specific discipline
     - Well-organized with grands and budgets
     - Longer term collaborations and better ICT awareness
     - Not virtual (e.g. CLARIN, Lifewatch, LHC, Project Bamboo)


• Virtual Collaboration* = People + Groups + Resources +
  Multi-Disciplines + Services + Policies + Funding + ...

Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA                                  4
Researchers knocking on our door




       Radio Astronomy — Pulsar               Climate Modeling for Scientist and   CineGRID — 4K+ Video Distribution
               Research                                Decision Makers                         Testbed




  Centralized Imaging for Large Scale        Jungle Computing and Multi-Model      Next Generation DNA Sequencing
     Population Imaging Studies                   Multi-Kernel Simulations



Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA                                          5
e-VBLI relies on intl. networks




Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA   6
Genomics in the Netherlands

• DNA reads of 750 individuals, 300 TB data
• Creating a “reference genome” and looking at variations
  in individuals to find cause of diseases
• Hospitals generate data, and carry out the 1st analysis


• Next Generation Networking for Next Generation
  Sequencing!




Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA   7
Multi-Disciplinary Collaboration


                                                                                     ƒ




                                       Virtual Collaboration A
                                                           Virtual Collaboration B




Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA                      8
Multi-Disciplinary Collaboration


                                                                      ƒ




                                            Enabled by




Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA       8
e-Research without limits


                                                                                  ƒ




                                                                      Network C

                                   Network A
                                                      Network B




                                          Open Lightpath
                                            Exchange


Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA                   9
e-Research without limits


                                                                      ƒ




                                            Enabled by




Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA       9
Circle of Pain...



                                                       Resources &
                                                        Services



                                                                        Access
                                              Authorization             Control




                                    Virtual              Relationship             Users
                                 Collaboration




Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA                       10
Multi-Domain Service Delivery

    Key words — trust, mutual understanding and standardization!


      Domain I                                                                          Domain VII


                                                   Virtual Collaboration A
                                                                                             S3




          Domain II                Domain III        Domain IV               Domain V     Domain VI



Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA                            11
Multi-Domain Service Delivery

    Key words — trust, mutual understanding and standardization!


      Domain I                                                                                Domain VII


                                                   Virtual Collaboration A
                                                                                                   S3

                                                                         What happens if if
                                                                          What happens a
                                                                         domain is removed,
                                                                          domain is removed
                                                                         added added?
                                                                            or or modified?




          Domain II                Domain III        Domain IV               Domain V           Domain VI



Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA                                  11
OpenConext

• Provides the glue and lubrication middleware to make
  campus, cloud based services and resources usable
  for collaboration for both inter campus and virtual
  collaboration scenarios
• OpenConext offers the Identity and Group
  Infrastructure
• Offer the platform software for large, virtual
  collaboration-type collaboration to run-you-own


• OpenConext allows you to create your own
  collaboration environment that organizes your
  applications, services and resources



Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA   12
Core Components

• Federated Identity Management — SAML
• Groups Management — Grouper
• Social Network 'Portal' technology — OpenSocial
• Collaboration Services and Resources
     - Document Sharing
                                                                      Virtual Organization
     - Video Collaboration
     - Learning Systems
     - Data Storage
     - Data Mining
     - Workspaces                                                                            9000
                                                                                             CALORIES
     - Workflows
     - Et cetera                                                                                 100%
                                                                                             SATISFACTION




Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA                                  13
Eco-system and supply chain

       Collaboration Portal                                                    Research Data Storage




                                                                           ƒ




                                                                                     DNA Sequencer


     Bandwidth on demand
                                                     Virtual
                                                           Collaboration



        09      12      14



Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA                              14
Dynamic Network Service 1.0



              NOC
           Engineer                   Users                      Application
                                                                               • User Management
                      Admin GUI                BoD GUI          Legacy API     • Resource Management
                                                                               • Topology & Discovery
        Network Management System                                              • Path finding
                                                                               • Scheduling & Reservation
                                                                               • Network Element
                                                                                Provisioning & Association




Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA                                         15
Resource Reservation 101

                                              Bandwidth on Demand
                                            APRIL


                                            25
                                          8am-10pm




                                                                                   ƒ


                                                                       Network C

                                  Network A
                                                           Network B




                                         Open Lightpath Exchange




Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA                    16
Resource Reservation 101

                                              Bandwidth on Demand
                                            APRIL                    APRIL


                                            25
                                          8am-10pm
                                                                     26
                                                                   10am-12pm




                                                                                           ƒ


                                                                               Network C

                                  Network A
                                                           Network B




                                         Open Lightpath Exchange




Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA                            16
Resource Reservation 101

                                              Bandwidth on Demand
                                            APRIL                    APRIL         APRIL


                                            25
                                          8am-10pm
                                                                     26
                                                                   10am-12pm
                                                                                    26
                                                                                12.10pm-3pm




                                                                                              ƒ


                                                                               Network C

                                  Network A
                                                           Network B




                                         Open Lightpath Exchange




Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA                               16
Dynamic Network Service 2.0


• Group Management                                 NOC
• Resource Management                           Engineer                        Users   Applications

                                                                      BoD GUI             NSI
                                                       Admin GUI
                                                                                                NSI    Inter Domain   NSI

                                       NOC            Bandwidth on Demand Service
                                    Engineer                             CIENA NMS


                                          Admin GUI          Network Management
• Topology & Discovery
• Path finding                                                      System
• Scheduling & Reservation
• Network Element
  Provisioning & Association




Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA                                             17
New SURFnet7 and BoD

• All Institutes are upgraded to a MultiService Port
• Allows for On Demand Lightpath Services
• Provide connectivity towards Cloud providers and
  Netherlight
• Offers fixed lightpath Quality of Service (bandwidth,
  latency, availability, et cetera)
• Separation of lightpaths through VLAN tagging


                                               10 Gbps MultiService Port

                     Lightpath         Lightpath         On Demand Lightpath   Available


                        2 Gbps            2 Gbps                      4 Gbps    2 Gbps




Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA                        18
Fusion between users and services

                                                                                                  Virtual Collaboration


                 BoD Admin Group
                                                               Bandwidth on
                                         BoD Physical Ports
                                                              Demand Service
                                                                                                                        Admin


                 Physical Resource        Physical Resource       Physical Resource
                     Group X                  Group Y                 Group Z
  Ports are
allocated to
 Institutes...




                                                                                                                          Users



                 Institute’s Admin            Institute’s         Institute’s Admin
                                                Admin                                     Virtual Resource Group         Virtual
                                                                                                                       User Group


                   Institute X              Institute Y             Institute Z




                                                                                                                       ...and can be
                                                                                                                     combined into a

                                     +                        +                       =                              Virtual Resource
                                                                                                                   Group and then used
                                                                                                                   by the collaboration




Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA                                                                 19
Architecture




Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA   20
Simplified view

                                             SAML2      ADFS   TBD         Identity Provider

                                                     SAML2                    Federation
                        Web Portal                             Grouper      Group Management
                          Gadget                Conext               TBD   Attribute Management
                                                Engine
                        OpenSocial                              XACML       Policy Management
                        Container          OpenSocial
                                           Rest
                                                                JANUS         Service Registry

                       OpenSocial Rest          SAML2 SP XACML                 SAML2 Metadata
                           OAuth                                                                      NSI
                                         Bandwidth on Demand Service                    CIENA NMS



                                         Network Management System                          NGE



                                                   NGE / Network
                                                                                   * Developed by SURFnet




Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA                                         21
BoD Service — Work in Progress




Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA   22
Virtual Ports Assignment

                                  Bandwidth on Demand




Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA   23
Virtual Ports Overview


                               Bandwidth on Demand




Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA   24
Reservations

                                Bandwidth on Demand




Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA   25
Physical Ports


                                  Bandwidth on Demand




Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA   26
BoD uptake in the Netherlands

                                             Institute                     # of sites

                                   Genome of the Netherlands                12 (2 Intl)

                                   Population Imaging Studies                   3
                             Visual Analysis of Flooding Scenarios              3
                                 Connecting climate model data                  3
                      Distributed Multi-Model / Multi-Kernel Simulations        3
                                            Proteomics                          4
                                            CineGrid NL                         4
                                          Pulsar Research                       3
                                          FOM Rijnhuizen                        2
                                   Visualization of climate data                2
                                              Visionair                         2
                                               DAS-4                            5
                                             NEXPReS                        5+ (intl)




Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA                       27
© ESA.




         harold.teunissen@surfnet.nl   haroldteunissen

Quantum Leap in Open Source Collaboration

  • 1.
    Quantum Leap inOpen Source Collaboration Bridging the gap between campus infrastructures © Ton van Alebeek Harold Teunissen et al. April 2012 - #I2SMM12
  • 2.
    Cyberinfra in theNetherlands • All ICT activities for Higher Education and Research in the Netherlands are brought under one umbrella National Research & Commercial ICT Scientific Computing & Shared Professional and Education Network Products & Services Storage Educational Services Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA 2
  • 3.
    Big Data —when size does matter • More is not less... - Large scale and global research - Collaboration in Virtual Organizations - Shared resources - High dependence on ICT Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA 3
  • 4.
    Collaborations and Partnerships •Virtual Research Team Virtual Laboratory - Focussed on doing research Virtual Project - Small scale - Temporary and elastic Virtual Colla- boration - Little ICT awareness • “Virtual” Infrastructure Provider Virtual Breeding Environment - Focussed on providing infrastructure for specific discipline - Well-organized with grands and budgets - Longer term collaborations and better ICT awareness - Not virtual (e.g. CLARIN, Lifewatch, LHC, Project Bamboo) • Virtual Collaboration* = People + Groups + Resources + Multi-Disciplines + Services + Policies + Funding + ... Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA 4
  • 5.
    Researchers knocking onour door Radio Astronomy — Pulsar Climate Modeling for Scientist and CineGRID — 4K+ Video Distribution Research Decision Makers Testbed Centralized Imaging for Large Scale Jungle Computing and Multi-Model Next Generation DNA Sequencing Population Imaging Studies Multi-Kernel Simulations Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA 5
  • 6.
    e-VBLI relies onintl. networks Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA 6
  • 7.
    Genomics in theNetherlands • DNA reads of 750 individuals, 300 TB data • Creating a “reference genome” and looking at variations in individuals to find cause of diseases • Hospitals generate data, and carry out the 1st analysis • Next Generation Networking for Next Generation Sequencing! Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA 7
  • 8.
    Multi-Disciplinary Collaboration ƒ Virtual Collaboration A Virtual Collaboration B Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA 8
  • 9.
    Multi-Disciplinary Collaboration ƒ Enabled by Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA 8
  • 10.
    e-Research without limits ƒ Network C Network A Network B Open Lightpath Exchange Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA 9
  • 11.
    e-Research without limits ƒ Enabled by Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA 9
  • 12.
    Circle of Pain... Resources & Services Access Authorization Control Virtual Relationship Users Collaboration Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA 10
  • 13.
    Multi-Domain Service Delivery Key words — trust, mutual understanding and standardization! Domain I Domain VII Virtual Collaboration A S3 Domain II Domain III Domain IV Domain V Domain VI Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA 11
  • 14.
    Multi-Domain Service Delivery Key words — trust, mutual understanding and standardization! Domain I Domain VII Virtual Collaboration A S3 What happens if if What happens a domain is removed, domain is removed added added? or or modified? Domain II Domain III Domain IV Domain V Domain VI Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA 11
  • 15.
    OpenConext • Provides theglue and lubrication middleware to make campus, cloud based services and resources usable for collaboration for both inter campus and virtual collaboration scenarios • OpenConext offers the Identity and Group Infrastructure • Offer the platform software for large, virtual collaboration-type collaboration to run-you-own • OpenConext allows you to create your own collaboration environment that organizes your applications, services and resources Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA 12
  • 16.
    Core Components • FederatedIdentity Management — SAML • Groups Management — Grouper • Social Network 'Portal' technology — OpenSocial • Collaboration Services and Resources - Document Sharing Virtual Organization - Video Collaboration - Learning Systems - Data Storage - Data Mining - Workspaces 9000 CALORIES - Workflows - Et cetera 100% SATISFACTION Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA 13
  • 17.
    Eco-system and supplychain Collaboration Portal Research Data Storage ƒ DNA Sequencer Bandwidth on demand Virtual Collaboration 09 12 14 Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA 14
  • 18.
    Dynamic Network Service1.0 NOC Engineer Users Application • User Management Admin GUI BoD GUI Legacy API • Resource Management • Topology & Discovery Network Management System • Path finding • Scheduling & Reservation • Network Element Provisioning & Association Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA 15
  • 19.
    Resource Reservation 101 Bandwidth on Demand APRIL 25 8am-10pm ƒ Network C Network A Network B Open Lightpath Exchange Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA 16
  • 20.
    Resource Reservation 101 Bandwidth on Demand APRIL APRIL 25 8am-10pm 26 10am-12pm ƒ Network C Network A Network B Open Lightpath Exchange Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA 16
  • 21.
    Resource Reservation 101 Bandwidth on Demand APRIL APRIL APRIL 25 8am-10pm 26 10am-12pm 26 12.10pm-3pm ƒ Network C Network A Network B Open Lightpath Exchange Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA 16
  • 22.
    Dynamic Network Service2.0 • Group Management NOC • Resource Management Engineer Users Applications BoD GUI NSI Admin GUI NSI Inter Domain NSI NOC Bandwidth on Demand Service Engineer CIENA NMS Admin GUI Network Management • Topology & Discovery • Path finding System • Scheduling & Reservation • Network Element Provisioning & Association Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA 17
  • 23.
    New SURFnet7 andBoD • All Institutes are upgraded to a MultiService Port • Allows for On Demand Lightpath Services • Provide connectivity towards Cloud providers and Netherlight • Offers fixed lightpath Quality of Service (bandwidth, latency, availability, et cetera) • Separation of lightpaths through VLAN tagging 10 Gbps MultiService Port Lightpath Lightpath On Demand Lightpath Available 2 Gbps 2 Gbps 4 Gbps 2 Gbps Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA 18
  • 24.
    Fusion between usersand services Virtual Collaboration BoD Admin Group Bandwidth on BoD Physical Ports Demand Service Admin Physical Resource Physical Resource Physical Resource Group X Group Y Group Z Ports are allocated to Institutes... Users Institute’s Admin Institute’s Institute’s Admin Admin Virtual Resource Group Virtual User Group Institute X Institute Y Institute Z ...and can be combined into a + + = Virtual Resource Group and then used by the collaboration Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA 19
  • 25.
    Architecture Quantum Leap inOpen Source Collaboration - I2SMM12 - Arlington, VA 20
  • 26.
    Simplified view SAML2 ADFS TBD Identity Provider SAML2 Federation Web Portal Grouper Group Management Gadget Conext TBD Attribute Management Engine OpenSocial XACML Policy Management Container OpenSocial Rest JANUS Service Registry OpenSocial Rest SAML2 SP XACML SAML2 Metadata OAuth NSI Bandwidth on Demand Service CIENA NMS Network Management System NGE NGE / Network * Developed by SURFnet Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA 21
  • 27.
    BoD Service —Work in Progress Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA 22
  • 28.
    Virtual Ports Assignment Bandwidth on Demand Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA 23
  • 29.
    Virtual Ports Overview Bandwidth on Demand Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA 24
  • 30.
    Reservations Bandwidth on Demand Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA 25
  • 31.
    Physical Ports Bandwidth on Demand Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA 26
  • 32.
    BoD uptake inthe Netherlands Institute # of sites Genome of the Netherlands 12 (2 Intl) Population Imaging Studies 3 Visual Analysis of Flooding Scenarios 3 Connecting climate model data 3 Distributed Multi-Model / Multi-Kernel Simulations 3 Proteomics 4 CineGrid NL 4 Pulsar Research 3 FOM Rijnhuizen 2 Visualization of climate data 2 Visionair 2 DAS-4 5 NEXPReS 5+ (intl) Quantum Leap in Open Source Collaboration - I2SMM12 - Arlington, VA 27
  • 33.
    © ESA. harold.teunissen@surfnet.nl haroldteunissen