Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
globus onlineCampus Bridging Made Easyvia Globus ServicesIan Foster, Rajkumar Kettimuthu, Stuart Martin,Steve Tuecke: Chic...
Campus bridging“the seamlessly integrated use of cyberinfrastructure operated by a scientist or engineer with other cyberi...
“Use of data resources from campus on                 XSEDE, or from XSEDE at a campus”*    •  Researchers often use a ran...
Two distinct groups of stakeholders1) Individuals (researchers, educators, students)  – Easy installation of access layer ...
Two distinct groups of stakeholders1) Individuals (researchers, educators, students)  – Easy installation of access layer ...
as          Globus Transfer: Data movement a                                         ServiceReliable file transfer. - Fire...
XSEDE-aware              www.globusonline.org
www.globusonline.org
Globus Connect                                                  Globus          (1) Globus ConnectUser           (2) User ...
Two distinct groups of stakeholders1) Individuals (researchers, educators, students)  – Easy installation of access layer ...
Two distinct groups of stakeholders1) Individuals (researchers, educators, students)  – Easy installation of access layer ...
GridFTP security configuration, old way     Installation     •  Download, untar, configure, make     Security configuratio...
Globus Connect Multi-User     •  What is GCMU?           •  Multi-user version of Globus Connect           •  Packages a G...
GCMU makes deploy and config trivial     Make GridFTP deployment trivial     •  GridFTP transfers can be achieved “instant...
Globus Transfer / GCMU Interactionwww.globustoolkit.org                             www.globusonline.org
Globus Transfer / GCMU Interactionwww.globustoolkit.org                             www.globusonline.org
OAuth protocol to protect passwords•  Site passwords flow through Globus Online     •  Globus Online does not store passwo...
Globus Connect Multi User with                               OAuth (coming soon)                          Step 1          ...
Two distinct groups of stakeholders     1) Individuals (researchers, educators, students)                 – Easy installat...
GCMU deployments (as of April 2012)www.globustoolkit.org            21              www.globusonline.org
GCMU endpoints and userswww.globustoolkit.org                          www.globusonline.org
GCMU – Bytes transferredwww.globustoolkit.org                          www.globusonline.org
GC userswww.globustoolkit.org          www.globusonline.org
GC – Bytes transferredwww.globustoolkit.org                        www.globusonline.org
Campus bridging at CU-Boulder     •  Janus Supercomputer               - 16,416 Westmere cores, 2GB memory per core       ...
Campus bridging at CU-Boulder (contd)     •  Globus Transfer and “manual tuning”               •        CLI transfer with ...
CU-Boulder  Data transferred from colorado#gridftp       122.5 TB  Data transferred to colorado#gridftp         21.6 TB  P...
GridFTP at Michigan   Single MyProxy Server for Campus         •  Users: PAM+Kerberos+LDAP         •  Built from GCMU   Mu...
Many small userswww.globustoolkit.org                  www.globusonline.org
Campus bridging at UMichigan  •  UMichigan has five Globus Transfer endpoints  •  Two endpoints at College of Engineering ...
Two distinct groups of stakeholders     1) Individuals (researchers, educators, students)                 – Easy installat...
Globus and Campus Bridging •  Globus Transfer – simple file transfer service            •     SaaS methods for easy fire-a...
Campus Bridging with Globus Services
Upcoming SlideShare
Loading in …5
×

Campus Bridging with Globus Services

1,059 views

Published on

Talk given at XSEDE 2012 conference in Chicago. The highlight were Dan Milroy and Brock Palen's presentations on experiences at Colorado and Michigan.

Paper is at https://www.globusonline.org/files/2012/07/XSEDE12-Globus-Campus-Bridging.pdf

As science becomes more computation and data intensive, computing needs often exceed campus capacity. Thus we see a desire to scale from the local environment to other campuses, to national cyberinfrastructure providers such as XSEDE, and/or to cloud providers—in other words, to “bridge” to the wider world. But given the realities of limited resources, time, and expertise, campus bridging methods must be exceedingly easy to use: as easy, for example, as are Netflix and Amazon movie streaming services. We report here on experiences with a service called Globus Online, which seeks to do for campus bridging what Netflix and Amazon do for movies: that is, use powerful cloud-hosted services and simple, intuitive web interfaces to make it “so easy that your grandparent can do it.” Specifically, we describe Globus Transfer, which addresses the important campus bridging use case of moving or synchronizing data across institutional boundaries. We describe how Globus Transfer achieves both ease of use for researchers and ease of administration for campus IT staff. We provide technical details on the Globus solution; quantitative data on usage by more than 25 early adopter campuses; and experience reports from two early adopters, the University of Michigan and the University of Colorado Boulder.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Campus Bridging with Globus Services

  1. 1. globus onlineCampus Bridging Made Easyvia Globus ServicesIan Foster, Rajkumar Kettimuthu, Stuart Martin,Steve Tuecke: Chicago and ArgonneThomas Hauser, Daniel Milroy, Jazcek Braden: ColoradoBrock Palen: Michigan www.globusonline.org
  2. 2. Campus bridging“the seamlessly integrated use of cyberinfrastructure operated by a scientist or engineer with other cyberinfrastructure on the scientist’s campus, at other campuses, and at the regional, national, and international levels as if they were proximate to the scientist” -- NSF Advisory Committee for Cyberinfrastructure Task Force on Campus Bridging Final Report, March 2011. www.globusonline.org
  3. 3. “Use of data resources from campus on XSEDE, or from XSEDE at a campus”* •  Researchers often use a range of resources and must move data among them •  Desktop, campus clusters, remote instruments, national computing facilities, commercial clouds, … •  Researcher desktops and campus clusters often lack sophisticated data movement tools •  Transient network and system failures have to be dealt with •  Each resource has its own security domain •  Firewalls and other problems often get in the way too www.globusonline.org*Campus Bridging Use Cases, XSEDE Project, 2012.
  4. 4. Two distinct groups of stakeholders1) Individuals (researchers, educators, students) – Easy installation of access layer interface – Intuitive GUI for file transfer – No interruptions for transient failures – Transfer efficiency2) System administrators – Easy integration of a campus resource into campus and national cyberinfrastructure – Easy management in terms of adding users, tracking usage, etc. www.globusonline.org
  5. 5. Two distinct groups of stakeholders1) Individuals (researchers, educators, students) – Easy installation of access layer interface – Intuitive GUI for file transfer – No interruptions for transient failures – Transfer efficiency2) System administrators – Easy integration of a campus resource into campus and national cyberinfrastructure – Easy management in terms of adding users, tracking usage, etc. www.globusonline.org
  6. 6. as Globus Transfer: Data movement a ServiceReliable file transfer. - Fire-and-forget - Automatic fault recovery - High performance - Across security domainsNo IT required. - Intuitive Web 2.0 interface - No client software install - New features available automatically - Consolidated support and troubleshooting Works with existing GridFTP servers; also Globus Connect www.globusonline.org
  7. 7. XSEDE-aware www.globusonline.org
  8. 8. www.globusonline.org
  9. 9. Globus Connect Globus (1) Globus ConnectUser (2) User makes request Online client registers with to Globus Online: e.g.,"transfer Globus Online data from MyDesktop to SiteA" (3) Globus Online forwards requests to Globus Connect GridFTP Globus server Connect "SiteA" (4) Globus Connect establishes data channel connection to SiteA and transfers data "MyDesktop" www.globusonline.org
  10. 10. Two distinct groups of stakeholders1) Individuals (researchers, educators, students) – Easy installation of access layer interface ✔ – Intuitive GUI for file transfer ✔ – No interruptions for transient failures ✔ – Transfer efficiency ✔2) System administrators – Easy integration of a campus resource into campus and national cyberinfrastructure – Easy management in terms of adding users, tracking usage, etc. www.globusonline.org
  11. 11. Two distinct groups of stakeholders1) Individuals (researchers, educators, students) – Easy installation of access layer interface – Intuitive GUI for file transfer – No interruptions for transient failures – Transfer efficiency2) System administrators – Easy integration of a campus resource into campus and national cyberinfrastructure – Easy management in terms of adding users, tracking usage, etc. www.globusonline.org
  12. 12. GridFTP security configuration, old way Installation •  Download, untar, configure, make Security configuration (server admins) •  Obtain and install X.509 host certificate from well-known CA •  Configure trust roots Security configuration (users) •  Obtain and install user certificate from well-known CA •  Configure trust roots Setup authorization (both users and admins) •  DN to local username mapping in gridmap file •  /DC=org/DC=doegrids/OU=People/CN=Rajkumar Kettimuthu 227852 rajk Too complex for many users and small labswww.globustoolkit.org 13 www.globusonline.org
  13. 13. Globus Connect Multi-User •  What is GCMU? •  Multi-user version of Globus Connect •  Packages a GridFTP server and MyProxy CA, pre-configured for use with Globus Online •  Why GCMU? •  Create transfer endpoints in minutes •  Avoid complex GridFTP install •  Avoid frequent sources of user and administrator error •  To download: https://www.globusonline.org/gcmu/ “We used GCMU to form a campus- “As a resource admin, Ive wide GSI authentication service found GCMU an exceedingly spanning multiple servers. Now my useful tool.... With GCMU, users have a fast, easy way to get setting up a GridFTP server their data wherever it needs to go, and handling authentication for and the setup process was trivial." multiple users is easy." --University of Michigan --Oak Ridge National Labwww.globustoolkit.org www.globusonline.org
  14. 14. GCMU makes deploy and config trivial Make GridFTP deployment trivial •  GridFTP transfers can be achieved “instantly” even by non-experts Automate the process of configuring security •  Avoid the need for any end-user or system administrator involvement in security configuration Reduce burden on both users and administrators •  Eliminate frequent sources of errors in GridFTP configuration and use.www.globustoolkit.org 15 www.globusonline.org
  15. 15. Globus Transfer / GCMU Interactionwww.globustoolkit.org www.globusonline.org
  16. 16. Globus Transfer / GCMU Interactionwww.globustoolkit.org www.globusonline.org
  17. 17. OAuth protocol to protect passwords•  Site passwords flow through Globus Online •  Globus Online does not store passwords •  Just pass along to MyProxy servers at site •  Still a security concern for some sites•  OAuth •  Sites run an OAuth server •  Users enter username and password only on a site’s webpage •  Globus Online gets an X.509 credential via Oauth protocolwww.globustoolkit.org www.globusonline.org
  18. 18. Globus Connect Multi User with OAuth (coming soon) Step 1 Access Endpoint Globus Online (Hosted Service) Step 2 Step 7 Step 8 Transfer Step 3 Redirect request Username password certificate certificate Step 4 GCMU Username Step 11 GridFTP MyProxy password GridFTP certificate Server Oauth Online CA Authentication Server Server certificate & Data Transfer Remote Cluster PAMCampus Cluster Step 6 / User’s PC Step 5 Step 9 Step 10 Authorization Username Access files certifficate password Local Authentication System Local (LDAP, RADIUS, Kerberos etc) Storagewww.globustoolkit.org www.globusonline.org
  19. 19. Two distinct groups of stakeholders 1) Individuals (researchers, educators, students) – Easy installation of access layer interface – Intuitive GUI for file transfer – No interruptions for transient failures – Transfer efficiency 2) System administrators – Easy integration of a campus resource into ✔ campus and national cyberinfrastructure – Easy management in terms of adding users, ✔ tracking usage, etc.www.globustoolkit.org www.globusonline.org
  20. 20. GCMU deployments (as of April 2012)www.globustoolkit.org 21 www.globusonline.org
  21. 21. GCMU endpoints and userswww.globustoolkit.org www.globusonline.org
  22. 22. GCMU – Bytes transferredwww.globustoolkit.org www.globusonline.org
  23. 23. GC userswww.globustoolkit.org www.globusonline.org
  24. 24. GC – Bytes transferredwww.globustoolkit.org www.globusonline.org
  25. 25. Campus bridging at CU-Boulder •  Janus Supercomputer - 16,416 Westmere cores, 2GB memory per core - Four Dell PowerEdge R710s as GridFTP servers - Dedicated 10Gb ethernet per node - RC network: “private VLANs” •  Globus Online endpoints - colorado#gridftp 122 TB transferred from 22 TB transferred to - colorado#jila, colorado#nsidc --data-interface <vlan>www.globustoolkit.org www.globusonline.org
  26. 26. Campus bridging at CU-Boulder (contd) •  Globus Transfer and “manual tuning” •  CLI transfer with -cc 4 -p 4 -pp 4 •  In “external” transfers, we noticed 44% increase in transfer rate for default packets and 26% for MTU 9000 •  Problem with jumbo frames •  Path MTU discovery and ICMP filtering •  Probably the issue- reverting to default packets solved the problem •  Determined to be the issue with JILA transferswww.globustoolkit.org www.globusonline.org
  27. 27. CU-Boulder Data transferred from colorado#gridftp 122.5 TB Data transferred to colorado#gridftp 21.6 TB Peak transfer rate between distinct endpoints 2.9 Gb/s Peak transfer rate to/from Janus (disk) 5.9 Gb/s Peak transfer rate to/from Janus (memory) 9.5 Gb/swww.globustoolkit.org www.globusonline.org
  28. 28. GridFTP at Michigan Single MyProxy Server for Campus •  Users: PAM+Kerberos+LDAP •  Built from GCMU Multiple GridFTP Servers •  Not all under umich# •  Offer documentation and help to setup endpoints •  Built from GCMU http://cac.engin.umich.edu/resources/loginnodes/globus.htmlwww.globustoolkit.org www.globusonline.org
  29. 29. Many small userswww.globustoolkit.org www.globusonline.org
  30. 30. Campus bridging at UMichigan •  UMichigan has five Globus Transfer endpoints •  Two endpoints at College of Engineering HPC systems •  The other three endpoints at other departments Data transferred from umich#nyx 9.8 TB Data transferred to umich#nyx 10.4 TB Data transferred from umich#flux 20.4 TB Data transferred to umich#flux 6.5 TBwww.globustoolkit.org www.globusonline.org
  31. 31. Two distinct groups of stakeholders 1) Individuals (researchers, educators, students) – Easy installation of access layer interface – Intuitive GUI for file transfer – No interruptions for transient failures – Transfer efficiency 2) System administrators – Easy integration of a campus resource into campus and national cyberinfrastructure – Easy management in terms of adding users, tracking usage, etc.www.globustoolkit.org www.globusonline.org
  32. 32. Globus and Campus Bridging •  Globus Transfer – simple file transfer service •  SaaS methods for easy fire-and-forget transfers, high performance, automatic fault recovery •  Web 2.0; integrated knowledge of XSEDE resources •  (Leverages Globus Nexus – identity management; sign in from federated identity systems such as InCommon and from OpenID providers such as Google) •  Globus Connect – one click GridFTP for desktops •  Globus Connect Multi User (GCMU) – easy-to- install GridFTP and security package •  Globus Storage – user-managed storage [soon]www.globustoolkit.org www.globusonline.org

×