Presented at the GlobusWorld Tour West, a virtual workshop on September 15, 2021.

1. Making Storage Systems
Accessible via Globus
Vas Vasiliadis
vas@uchicago.edu
September 15, 2021

2. Hybrid SaaS Architecture
DATA
Channel
CONTROL
Channel
Source Destination
Subscriber owned
and administered
storage system
Globus
“connector”
software
No data relay or
staging via Globus
cloud service
Subscriber
Control
Domain
Globus
Control
Domain
Single, globally accessible
multi-tenant service

3. Globus Connect Server
3
• Makes your storage accessible via Globus
• Software/tools installed and managed by sysadmin
docs.globus.org/globus-connect-server-installation-guide/
Local system users
Local Storage System
(HPC cluster, NAS, …)
Globus
Connect
Server
DTN
• Default access for
all local accounts
• Native packaging
Linux: DEB, RPM

4. Creating a Globus endpoint
Globus Connect Server v5
(GCSv5) should be used
for all new endpoint
installations

5. GCSv5 improvements
• Standards based web authorization (OAuth2, OIDC)
• Modular configuration
• Multiple distinct access policies on a single endpoint
• Simplified multi-DTN endpoint config/management
• Direct browser up/download, with full access control
• Guest collections, with fine-grained access control
• Interoperability with endpoints running older versions

6. Globus Connect Server v5 Architecture

7. GCS management conceptual architecture
7
Data Transfer Node
GCS Command
Line Interface
GridFTP
Server
Globus
Transfer
Service
GCS
management
requests
Globus
Auth
Service
GCS Manager authorize request
using client ID/secret
GCS Manager endpoint:
abc.abc.data.globus.org
Register a Globus Connect Server at
developers.globus.org get GCS
client ID, secret
Define Globus
Transfer
resources
(gateways,
collections, …)

8. Available through a Globus subscription
GCSv5 installation/configuration summary
1. Register a Globus Connect Server with Globus Auth
2. Install GCS packages on data transfer node (DTN)
3. Set up the endpoint and add node(s)
4. Create a POSIX storage gateway
5. Create a mapped collection
6. Associate endpoint with a subscription
7. Create a guest collection
8. Optional: Add other storage systems to the endpoint

9. 1. Register GCS and get credentials
developers.globus.org

10. 2. Install Globus Connect Server v5 packages
$ curl -LOs
http://downloads.globus.org/toolkit/gt6/stable/installers/repo/deb/globus-
toolkit-repo_latest_all.deb
$ dpkg -i globus-toolkit-repo_latest_all.deb
$ sed -i /etc/apt/sources.list.d/globus-toolkit-6-stable*.list
> -e 's/^# deb /deb /’
$ sed -i /etc/apt/sources.list.d/globus-connect-server-stable*.list
> -e 's/^# deb /deb /’
$ apt-key add /usr/share/globus-toolkit-repo/RPM-GPG-KEY-Globus
$ apt-get update
$ apt-get --assume-yes install globus-connect-server54

11. 3. Set up endpoint and add node
$ globus-connect-server endpoint setup
> "Endpoint Display Name"
> --organization "Globus Demonstration"
> --client-id 4321dddd-af72-4c4b-9533-a0f4055dd321
> --owner vas@globusdemo.org
$ globus-connect-server node setup
> --client-id 4321dddd-af72-4c4b-9533-a0f4055dd321
Note: endpoint setup command generates deployment-key.json
Use this file when setting up additional data transfer nodes

12. Our setup so far
Run globus-connect-server node setup
to set up additional data transfer nodes
Copy deployment-key.json
from original DTN

13. 4. Create a storage gateway
$ globus-connect-server storage-gateway create posix
> "Storage Gateway Display Name"
> --domain globusdemo.org
> --authentication-timeout-mins 720
Allowed
authentication
domain(s)
Duration of user session
when accessing collections
via this storage gateway

14. Storage Gateways define a set of access policies
• Authentication for local account-holders
– Which identity domain(s) are acceptable?
– How are identities mapped from domain(s) to local accounts?
• Policy scope
– Which parts of the storage system are accessible via Globus?
– Which local accounts does this policy allow (or deny)?
• High Assurance settings
• MFA requirements

15. Picking identity domains
• User must present identity from one of the configured
domains
– On access attempts, linked identities will be scanned for a match
– If no identity from the required domain(s), will be asked to link one
• Identity domains may include…
– …any organization in Globus federated list
– …your institution’s identity provider trusted by Globus
– …a local OpenID Connect (OIDC) server using your PAM stack

16. Mapping identities to local accounts
• Default: Strip identity domain (everything after “@”)
– e.g. userX@globusdemo.org maps to local account userX
– Best for campus identities w/synchronized local accounts
• Use --identity-mapping option on storage gateway
– Specify expression in a JSON document
– Execute a custom script
docs.globus.org/globus-connect-server/v5.4/identity-mapping-guide/

17. Our setup so far…

18. 5. Create a mapped collection
$ globus-connect-server collection create
> f77ff456-1f18-41d3-94a7-f3fd8858ea4d
> /
> "Collection Display Name"
Collections are rooted at the specified base path
Specifying "/" as the base path sets the collection root to the local
user’s home directory (this was the default in GCS v4)
Storage gateway ID
Collection base path

19. Common Collection configuration options
• Restrict access: local users, local groups
• Allow guest collections à enables sharing
• Restrict sharing: paths, local users, local groups
• Enable HTTPS access

20. Our setup so far…

21. Accessing mapped
collections
25

23. Alternative authentication flow
(if not using Globus trusted IdP)

24. 6. Associate endpoint with a subscription
$ globus-connect-server endpoint set-subscription-id DEFAULT
$ globus-connect-server endpoint set-subscription-id aa767d89-6237-dec3-
94a7-f3fd8858ea4d
Can also be set via the web app Endpoints page
app.globus.org/endpoints (search for endpoint name)

25. Subscriptions and Endpoint Roles
• Subscription(s) configured for your institution
• Multiple Subscription Managers per subscription
• Subscription Manager ties endpoint to subscription
– Results in a “managed” endpoint
• Assign additional roles for endpoint management
– Administrator, Manager, Monitor

26. Be identity-, role-, and permission-aware
• Default: Only endpoint owner can configure endpoint
• Delegate administrator role to other sysadmins
– Best practice: Delegate to a Globus group, not individuals
• Check identity using the session command
• Check resource permissions on storage gateways and
collections with --include-private-policies option
docs.globus.org/globus-connect-server/v5.4/reference/role/

27. 7. Create a guest collection
• Created by user, not endpoint administrator
• Grants access to specific Globus users without a
mapped local account
• “Guest” users have same (or more limited)
permissions as the guest collection creator
– Access logs show access by the collection creator*
• Guest collection’s root is relative to the mapped
collection’s base path
* High Assurance collections log guest user identities to enable auditing

28. 8. Add other storage systems to the endpoint
• Update your GCS packages
• Add the appropriate storage gateway
– Non-POSIX systems require add-on connector subscription(s)
• Gateway configuration options vary by connector
– e.g., specify bucket name(s) for AWS S3
• Collection authentication options vary by connector
– e.g., provide user access key and secret key for AWS S3

29. When you really need a “re-do”…
• Proper clean-up—both on your system and in the
Globus service—is important!
• Execute these commands in the specified order:
o globus-connect-server node cleanup
o globus-connect-server endpoint cleanup
• Delete the GCS registration at developers.globus.org
• Don’t use the same Client ID for another endpoint!

30. GCSv5 resources – please consult these first
• Quickstart Guide
docs.globus.org/globus-connect-server/v5.4/quickstart
• GCS Command Line Reference
docs.globus.org/globus-connect-server/v5.4/reference
• Video walkthrough of an installation
www.youtube.com/watch?v=8ILtsSRiML8

31. General Resources
• Access the service: app.globus.org
• Documentation: docs.globus.org/globus-connect-server
• Engage: discuss@globus.org
• Subscribe: globus.org/subscriptions
• Need help? support@globus.org
• Follow us: @globus