Globus for System Administrators

Globus
Globus for System Administrators
Vas Vasiliadis
vas@uchicago.edu
June 22, 2022
Agenda
• Globus Connect Server v5 Overview
• Deploying a Globus endpoint and creating collections
• Common GCS configuration options
• Customizing identity mapping
• Monitoring and managing user activity
2
Our focus in this session
Globus Connect Server
4
• Makes your storage accessible via Globus
• Software/tools installed and managed by sysadmin
docs.globus.org/globus-connect-server
Local system users
Local Storage System
(HPC cluster, NAS, …)
Globus
Connect
Server
DTN
• Default access for
all local accounts
• Native packaging
Linux: DEB, RPM
Globus Connect
Server v5 Overview
5
Globus Connect Server v5 Architecture
Globus Connect Server version 5
7
GCS management conceptual architecture
8
Data Transfer Node
GCS Command
Line Interface
GridFTP
Server
Globus
Transfer
Service
GCS
management
requests
Globus
Auth
Service
GCS Manager authorize request
using client ID/secret
GCS Manager endpoint:
abc.abc.data.globus.org
Register a Globus Connect Server at
developers.globus.org get GCS
client ID, secret
Define Globus
resources
(gateways,
collections, …)
GCS v5 install
walkthrough
9
docs.globus.org/globus-connect-server
docs.globus.org/globus-connect-server/v5.4/quickstart
Tutorial materials and other handy links
bit.ly/gw-tut
Fixin’ to get ready
• Yes, you must have a chat with OpSec, …
• Requires traffic flow on ports
–443 (control channel) – must be publicly routable
–50000-51000 (data channel) – can be on private net
Requires a Globus subscription
GCSv5 installation/configuration summary
1. Register a Globus Connect Server with Globus Auth
2. Install GCS packages on data transfer node (DTN)
3. Set up the endpoint and add node(s)
4. Create a POSIX storage gateway
5. Create a mapped collection
6. Associate endpoint with a subscription
7. Create a guest collection
8. Enable browser down/upload (HTTPS access)
9. Add other storage systems to the endpoint
GCS registration
13
Register GCS and get credentials
• Navigate to developers.globus.org and log in
• (Optional) Create a project
• Add a new Globus Connect Server
• Generate a client secret
• Save the client ID and secret
1. Register GCS and get credentials
developers.globus.org
2. Install Globus Connect Server v5 packages
$ curl -LOs http://downloads.globus.org/globus-connect-
server/stable/installers/repo/deb/globus-repo_latest_all.deb
$ dpkg -i globus-repo_latest_all.deb
$ apt-key add /usr/share/globus-repo/RPM-GPG-KEY-Globus
$ apt-get update
$ apt-get --assume-yes install globus-connect-server54
Already done on EC2 instances.
Endpoint creation and
node setup
17
3. Set up endpoint and add node
$ globus-connect-server endpoint setup 
> "My Endpoint" 
> --organization "My Organization" 
> --client-id 4321dddd-af72-4c4b-9533-a0f4055dd321 
> --contact-email me@uchicago.edu 
> --owner me@globusid.org
$ globus-connect-server node setup 
> --client-id 4321dddd-af72-4c4b-9533-a0f4055dd321
Note: endpoint setup command generates deployment-key.json
Use this file when setting up additional data transfer nodes
Set up endpoint and add a DTN
• Access server: ssh adminN@ornlN.globusdemo.org
• Switch to root: sudo su
• Run: globus-connect-server endpoint setup ...
– Ensure --owner is the identity you used to register the GCS
• Run: globus-connect-server node setup ...
• Run: systemctl restart apache2
• Display endpoint details:
– globus-connect-server login localhost
– globus-connect-server endpoint show Cheatsheet
bit.ly/gw-tut
Our setup so far
Run globus-connect-server node setup
to set up additional data transfer nodes
Copy deployment-key.json
from original DTN
Storage Gateways define a set of access policies
• Authentication for local account-holders
– Which identity domain(s) are acceptable?
– How are identities mapped from domain(s) to local accounts?
• Policy scope
– Which parts of the storage system are accessible via Globus?
– Which local accounts does this policy allow (or deny)?
• High Assurance settings
• MFA requirements
Authentication for local account-holders
• Primary access (via a mapped collection) requires an
account on the host system*
• Two-part authentication configuration:
1. Pick one or more identity domains
2. Configure the method to map the authenticated identity to an
account on your system
* You may allow primary users to share with others who don’t have accounts on your system
Picking identity domains
• User must present identity from one of the configured
domains
– On access attempts, linked identities will be scanned for a match
– If no identity from the required domain(s), will be asked to link one
• Identity domains may include…
– …any organization in Globus federated list (incl. ornl.gov)
– …your institution’s identity provider trusted by Globus
– …a local OpenID Connect (OIDC) server using your PAM stack
Mapping identities to local accounts
• Default: Strip identity domain (everything after “@”)
– e.g. userX@globusdemo.org maps to local account userX
– Best for campus identities w/synchronized local accounts
• Use --identity-mapping option on storage gateway
– Specify expression in a JSON document
– Execute a custom script
docs.globus.org/globus-connect-server/v5.4/identity-mapping-guide/
Create a POSIX
storage gateway
25
Creating a storage gateway
• Our storage gateway will access a POSIX system
– This is the only type permitted without a subscription
• It will allow access to users with credentials from the
globusid.org (or ornl.gov) domain
• Reauthentication will be required every 90 minutes
Cheatsheet
bit.ly/gw-tut
4. Create a storage gateway
$ globus-connect-server storage-gateway create posix 
> "My Storage Gateway" 
> --domain globusid.org 
> --authentication-timeout-mins 90
Allowed authentication
domain
Duration of user session
when accessing collections
via this storage gateway
Our setup so far…
Create a mapped
collection on the
POSIX gateway
29
Creating a collection
• Our collection will use the default identity mapping
• It will be “rooted” at the user’s home directory
• Access will require authentication with an identity
from the globusid.org (or ornl.gov) domain
Cheatsheet
bit.ly/gw-tut
5. Create a mapped collection
$ globus-connect-server collection create 
> f77ff456-1f18-41d3-94a7-f3fd8858ea4d 
> / 
> "My Mapped Collection"
Collections are rooted at the specified base path
Specifying "/" as the base path sets the collection root to the local
user’s home directory
Storage gateway ID
Collection base path
Our setup so far…
Understanding
access to mapped
collections
33
Globus for System Administrators
Alternative authentication flow
(if not using Globus trusted IdP)
Access our mapped
collection
36
We are using the default identity mapping, so…
• Create a local user account with the same name as
your globusid.org (or ornl.org) identity
– e.g., for me@globusid.org create local account “me”
– e.g., for me@ornl.gov create local account “me”
• adduser --disabled-password --gecos 'me’ me
• Access your mapped collection via the web app…
• …and move some files into it!
Cheatsheet
bit.ly/gw-tut
Common Collection configuration options
• Restrict access: local users, local groups
• Restrict sharing: paths, local users, local groups
• Allow guest collections à enables sharing
• Enable HTTPS access
• Force data channel encryption
Local account restrictions
• Note: These only apply to mapped collections
• A storage gateway’s allowed identity domains and
identity mapping method determine the universe of local
accounts that may access the mapped collection
• You can further narrow the access universe using…
--user-allow
--user-deny
--posix-group-allow (POSIX storage gateways only)
--posix-group-deny (POSIX storage gateways only)
Path restrictions
• Always use the narrowest base path possible for your storage
gateway(s) and collection(s)
– Storage gateway base specifies where collections may be created
– Collection base specifies the base directory for the collection
• POSIX storage gateway
– Use --restrict_paths to specify narrower read, read/write, or none
access for specific paths
– You provide a JSON doc that lists paths for each permission type
– Note: These are absolute paths on the host system
• Collection: specify narrowest base path that satisfies the need
Restrict collection
access to filesystem
41
Setting path restrictions
• A new storage gateway will limit access to /home
– NB: No change to local permissions, only visibility via Globus
• We specify the path restrictions in paths.json
– This file is in your admin user’s home directory
• Run: storage-gateway create command with the
--restrict-paths option
• Create a new POSIX mapped collection
Cheatsheet
bit.ly/gw-tut
Create a restricted storage gateway, collection
$ globus-connect-server storage-gateway create posix 
> "My Storage Gateway - Restricted" 
> --domain globusid.org 
> --authentication-timeout-mins 90 
> --restrict-paths file:/home/adminN/paths.json
$ globus-connect-server collection create 
> 3926bf02-6bc3-11e7-a9c6-22000bf2d287 
> / 
> "My Mapped Collection – Restricted"
Fully qualified filename containing
rule(s) for restricting access to
specific filesystem paths
Revisit your mapped collections
• Your will need to authenticate on your new (restricted
access) collection, and consent
• Note the access behavior differences between the two
mapped collections
Subscriptions and Endpoint Roles
• Subscription(s) configured for your institution
• Multiple Subscription Managers per subscription
• Subscription Manager ties endpoint to subscription
– Results in a “managed” endpoint
• Assign additional roles for endpoint management
– Administrator, Manager, Monitor
Associate the
endpoint with a
subscription
46
6. Associate endpoint with a subscription
• Subscription managers can enable subscription
features on an endpoint
• If you are not the subscription manager, just send
your endpoint ID to your subscription manager and
ask them to add it.
Make your endpoint “Managed”
• Option A: Put your endpoint ID in the spreadsheet
and Greg will make it managed
• Option B: Run globus-connect-server endpoint
set-subscription-id
• Confirm: globus-connect-server endpoint show
Cheatsheet
bit.ly/gw-tut
6. Associate endpoint with a subscription
$ globus-connect-server endpoint set-subscription-id DEFAULT
$ globus-connect-server endpoint set-subscription-id 
> 39299902-6bc3-aa56-a9c6-22000bf2d287
Subscription managers can also set this via the web app Console
page: app.globus.org/console (look under the Endpoints tab)
Your identity may already be a
subscription manager on a
subscription
Be identity-, role-, and permission-aware
• Default: Only endpoint owner can configure an endpoint
• Delegate administrator role to other sysadmins
– Best practice: Delegate to a Globus group, not individuals
• Check identity using the session command
• Check resource permissions on storage gateways and
collections with --include-private-policies option
docs.globus.org/globus-connect-server/v5.4/reference/role/
7. Create a guest collection
• Created by user, not endpoint administrator
• Grants access to specific Globus users without a
mapped local account
• “Guest” users have the same (or more limited)
permissions as the guest collection creator
– Access logs show access by the collection creator*
• Guest collection’s root is relative to the mapped
collection’s base path
* High Assurance collections log guest user identities to enable auditing
Sharing restrictions
• Guest collections may be created in any directory accessible by
the collection, by any authorized local account
• You can restrict the authorized accounts…
o --sharing-user-allow
--sharing-user-deny
o --posix-sharing-group-allow
o --posix-sharing-group-deny
• …and sharing paths…
o --sharing-restrict-paths (specify JSON PathRestrictions)
• You can also set policies for specific user/path combinations
o $ globus-connect-server sharing-policy create ...
Create and access a
guest collection
53
Create and access a guest collection
• Enable creation of guest collections
• Run: globus-connect-server collection update
• Access the mapped collection; create /projects
• Create a guest collection on the /projects directory
• Grant read access to the “Tutorial Users” group
• Authenticate and browse guest collection
8. Enable web browser upload/download
• Authorized users can upload,
download files via a browser
• Must have permissions to the
collection
– Collection configuration governs
access
– Web server is a different
application (separate
authentication)
Enable/disable file
download/upload via
browser
56
Enable HTTPS access
• Run: globus-connect-server collection update
• Access your mapped collection
• Upload a file from your laptop (and download it too!)
Cheatsheet
bit.ly/gw-tut
Using the
management console
58
Things to do with the management console
• Monitor current transfers on your endpoints
– See what’s going on at the transfer request level
– Much better than watching individual file transfers
• Pause (and later resume) a transfer in progress
– Sends a notice to the transfer owner
• Set a pause rule for current and future transfers
– Ideal for maintenance mode
– Notifies transfer owners,
– Tasks resume when endpoint is un-paused
docs.globus.org/management-console-guide/
Resources
• GCSv5 Guides: docs.globus.org/globus-connect-server/
• Migration: docs.globus.org/globus-connect-
server/migrating-to-v5.4/
• Globus support: support@globus.org
60
1 of 60

Recommended

Introduction to Globus for System Administrators by
Introduction to Globus for System AdministratorsIntroduction to Globus for System Administrators
Introduction to Globus for System AdministratorsGlobus
146 views59 slides
Introduction to Globus for System Administrators by
Introduction to Globus for System AdministratorsIntroduction to Globus for System Administrators
Introduction to Globus for System AdministratorsGlobus
27 views57 slides
Introduction to Globus for System Administrators by
Introduction to Globus for System AdministratorsIntroduction to Globus for System Administrators
Introduction to Globus for System AdministratorsGlobus
94 views54 slides
Connecting Your System to Globus (APS Workshop) by
Connecting Your System to Globus (APS Workshop)Connecting Your System to Globus (APS Workshop)
Connecting Your System to Globus (APS Workshop)Globus
137 views67 slides
Making Storage Systems Accessible via Globus (GlobusWorld Tour West) by
Making Storage Systems Accessible via Globus (GlobusWorld Tour West)Making Storage Systems Accessible via Globus (GlobusWorld Tour West)
Making Storage Systems Accessible via Globus (GlobusWorld Tour West)Globus
128 views31 slides
GlobusWorld 2021 Tutorial: Globus for System Administrators by
GlobusWorld 2021 Tutorial: Globus for System AdministratorsGlobusWorld 2021 Tutorial: Globus for System Administrators
GlobusWorld 2021 Tutorial: Globus for System AdministratorsGlobus
141 views31 slides

More Related Content

Similar to Globus for System Administrators

Introduction to Globus for System Administrators (GlobusWorld Tour - UMich) by
Introduction to Globus for System Administrators (GlobusWorld Tour - UMich)Introduction to Globus for System Administrators (GlobusWorld Tour - UMich)
Introduction to Globus for System Administrators (GlobusWorld Tour - UMich)Globus
357 views59 slides
Tutorial: Introduction to Globus for System Administrators by
Tutorial: Introduction to Globus for System AdministratorsTutorial: Introduction to Globus for System Administrators
Tutorial: Introduction to Globus for System AdministratorsGlobus
465 views51 slides
Globus for System Administrators by
Globus for System AdministratorsGlobus for System Administrators
Globus for System AdministratorsGlobus
269 views33 slides
Globus Command Line Interface (APS Workshop) by
Globus Command Line Interface (APS Workshop)Globus Command Line Interface (APS Workshop)
Globus Command Line Interface (APS Workshop)Globus
100 views25 slides
Globus for System Administrators (GlobusWorld Tour - UCSD) by
Globus for System Administrators (GlobusWorld Tour - UCSD)Globus for System Administrators (GlobusWorld Tour - UCSD)
Globus for System Administrators (GlobusWorld Tour - UCSD)Globus
110 views55 slides
Globus Endpoint Administration (GlobusWorld Tour - STFC) by
Globus Endpoint Administration (GlobusWorld Tour - STFC)Globus Endpoint Administration (GlobusWorld Tour - STFC)
Globus Endpoint Administration (GlobusWorld Tour - STFC)Globus
230 views54 slides

Similar to Globus for System Administrators(20)

Introduction to Globus for System Administrators (GlobusWorld Tour - UMich) by Globus
Introduction to Globus for System Administrators (GlobusWorld Tour - UMich)Introduction to Globus for System Administrators (GlobusWorld Tour - UMich)
Introduction to Globus for System Administrators (GlobusWorld Tour - UMich)
Globus 357 views
Tutorial: Introduction to Globus for System Administrators by Globus
Tutorial: Introduction to Globus for System AdministratorsTutorial: Introduction to Globus for System Administrators
Tutorial: Introduction to Globus for System Administrators
Globus 465 views
Globus for System Administrators by Globus
Globus for System AdministratorsGlobus for System Administrators
Globus for System Administrators
Globus 269 views
Globus Command Line Interface (APS Workshop) by Globus
Globus Command Line Interface (APS Workshop)Globus Command Line Interface (APS Workshop)
Globus Command Line Interface (APS Workshop)
Globus 100 views
Globus for System Administrators (GlobusWorld Tour - UCSD) by Globus
Globus for System Administrators (GlobusWorld Tour - UCSD)Globus for System Administrators (GlobusWorld Tour - UCSD)
Globus for System Administrators (GlobusWorld Tour - UCSD)
Globus 110 views
Globus Endpoint Administration (GlobusWorld Tour - STFC) by Globus
Globus Endpoint Administration (GlobusWorld Tour - STFC)Globus Endpoint Administration (GlobusWorld Tour - STFC)
Globus Endpoint Administration (GlobusWorld Tour - STFC)
Globus 230 views
Globus Endpoint Setup and Configuration - XSEDE14 Tutorial by Globus
Globus Endpoint Setup and Configuration - XSEDE14 TutorialGlobus Endpoint Setup and Configuration - XSEDE14 Tutorial
Globus Endpoint Setup and Configuration - XSEDE14 Tutorial
Globus 1.2K views
Globus for System Administrators (CHPC 2019 - South Africa) by Globus
Globus for System Administrators (CHPC 2019 - South Africa)Globus for System Administrators (CHPC 2019 - South Africa)
Globus for System Administrators (CHPC 2019 - South Africa)
Globus 180 views
Globus for System Administrators (GlobusWorld Tour - Columbia University) by Globus
Globus for System Administrators (GlobusWorld Tour - Columbia University)Globus for System Administrators (GlobusWorld Tour - Columbia University)
Globus for System Administrators (GlobusWorld Tour - Columbia University)
Globus 84 views
Automating Research Data Flows and Introduction to the Globus Platform by Globus
Automating Research Data Flows and Introduction to the Globus PlatformAutomating Research Data Flows and Introduction to the Globus Platform
Automating Research Data Flows and Introduction to the Globus Platform
Globus 50 views
Globus Connect Server v5 Q&A Briefing by Globus
Globus Connect Server v5 Q&A BriefingGlobus Connect Server v5 Q&A Briefing
Globus Connect Server v5 Q&A Briefing
Globus 214 views
Automating Research Data Flows and an Introduction to the Globus Platform by Globus
Automating Research Data Flows and an Introduction to the Globus PlatformAutomating Research Data Flows and an Introduction to the Globus Platform
Automating Research Data Flows and an Introduction to the Globus Platform
Globus 132 views
GlobusWorld 2021 Tutorial: The Globus CLI, Platform and SDK by Globus
GlobusWorld 2021 Tutorial: The Globus CLI, Platform and SDKGlobusWorld 2021 Tutorial: The Globus CLI, Platform and SDK
GlobusWorld 2021 Tutorial: The Globus CLI, Platform and SDK
Globus 177 views
Working with Globus Platform Services and Portals by Globus
Working with Globus Platform Services and PortalsWorking with Globus Platform Services and Portals
Working with Globus Platform Services and Portals
Globus 28 views
Data Publication and Discovery with Globus by Globus
Data Publication and Discovery with GlobusData Publication and Discovery with Globus
Data Publication and Discovery with Globus
Globus 266 views
Globus Connect Server 5.1 Webinar by Globus
Globus Connect Server 5.1 WebinarGlobus Connect Server 5.1 Webinar
Globus Connect Server 5.1 Webinar
Globus 509 views
Simple Data Automation with Globus (GlobusWorld Tour West) by Globus
Simple Data Automation with Globus (GlobusWorld Tour West)Simple Data Automation with Globus (GlobusWorld Tour West)
Simple Data Automation with Globus (GlobusWorld Tour West)
Globus 102 views
[Devconf.cz][2017] Understanding OpenShift Security Context Constraints by Alessandro Arrichiello
[Devconf.cz][2017] Understanding OpenShift Security Context Constraints[Devconf.cz][2017] Understanding OpenShift Security Context Constraints
[Devconf.cz][2017] Understanding OpenShift Security Context Constraints
Globus Endpoint Migration and Advanced Administration Topics by Globus
Globus Endpoint Migration and Advanced Administration TopicsGlobus Endpoint Migration and Advanced Administration Topics
Globus Endpoint Migration and Advanced Administration Topics
Globus 55 views
Advanced Globus System Administration by Globus
Advanced Globus System AdministrationAdvanced Globus System Administration
Advanced Globus System Administration
Globus 21 views

More from Globus

Introduction to Data Transfer and Sharing for Researchers by
Introduction to Data Transfer and Sharing for ResearchersIntroduction to Data Transfer and Sharing for Researchers
Introduction to Data Transfer and Sharing for ResearchersGlobus
4 views33 slides
Introduction to the Globus Platform for Developers by
Introduction to the Globus Platform for DevelopersIntroduction to the Globus Platform for Developers
Introduction to the Globus Platform for DevelopersGlobus
4 views28 slides
Introduction to the Command Line Interface (CLI) by
Introduction to the Command Line Interface (CLI)Introduction to the Command Line Interface (CLI)
Introduction to the Command Line Interface (CLI)Globus
13 views12 slides
Automating Research Data with Globus Flows and Compute by
Automating Research Data with Globus Flows and ComputeAutomating Research Data with Globus Flows and Compute
Automating Research Data with Globus Flows and ComputeGlobus
9 views60 slides
Advanced Globus System Administration by
Advanced Globus System AdministrationAdvanced Globus System Administration
Advanced Globus System AdministrationGlobus
26 views29 slides
Introduction to Globus for New Users by
Introduction to Globus for New UsersIntroduction to Globus for New Users
Introduction to Globus for New UsersGlobus
55 views26 slides

More from Globus (20)

Introduction to Data Transfer and Sharing for Researchers by Globus
Introduction to Data Transfer and Sharing for ResearchersIntroduction to Data Transfer and Sharing for Researchers
Introduction to Data Transfer and Sharing for Researchers
Globus 4 views
Introduction to the Globus Platform for Developers by Globus
Introduction to the Globus Platform for DevelopersIntroduction to the Globus Platform for Developers
Introduction to the Globus Platform for Developers
Globus 4 views
Introduction to the Command Line Interface (CLI) by Globus
Introduction to the Command Line Interface (CLI)Introduction to the Command Line Interface (CLI)
Introduction to the Command Line Interface (CLI)
Globus 13 views
Automating Research Data with Globus Flows and Compute by Globus
Automating Research Data with Globus Flows and ComputeAutomating Research Data with Globus Flows and Compute
Automating Research Data with Globus Flows and Compute
Globus 9 views
Advanced Globus System Administration by Globus
Advanced Globus System AdministrationAdvanced Globus System Administration
Advanced Globus System Administration
Globus 26 views
Introduction to Globus for New Users by Globus
Introduction to Globus for New UsersIntroduction to Globus for New Users
Introduction to Globus for New Users
Globus 55 views
Globus Automation by Globus
Globus AutomationGlobus Automation
Globus Automation
Globus 23 views
Introduction to Globus by Globus
Introduction to GlobusIntroduction to Globus
Introduction to Globus
Globus 43 views
Working with Globus Platform Services by Globus
Working with Globus Platform ServicesWorking with Globus Platform Services
Working with Globus Platform Services
Globus 42 views
Advanced Globus System Administration by Globus
Advanced Globus System AdministrationAdvanced Globus System Administration
Advanced Globus System Administration
Globus 29 views
Using Globus to Streamline Research at Scale by Globus
Using Globus to Streamline Research at ScaleUsing Globus to Streamline Research at Scale
Using Globus to Streamline Research at Scale
Globus 30 views
Introduction to Globus for Researchers by Globus
Introduction to Globus for ResearchersIntroduction to Globus for Researchers
Introduction to Globus for Researchers
Globus 89 views
Introduction to Globus for New Users by Globus
Introduction to Globus for New UsersIntroduction to Globus for New Users
Introduction to Globus for New Users
Globus 58 views
Building Data Portals and Science Gateways with Globus by Globus
Building Data Portals and Science Gateways with GlobusBuilding Data Portals and Science Gateways with Globus
Building Data Portals and Science Gateways with Globus
Globus 133 views
Automating Research Data Management with Globus by Globus
Automating Research Data Management with GlobusAutomating Research Data Management with Globus
Automating Research Data Management with Globus
Globus 250 views
Moemoea nui Aotearoa: Challenges and Strategies in Data Lifecycle Management ... by Globus
Moemoea nui Aotearoa: Challenges and Strategies in Data Lifecycle Management ...Moemoea nui Aotearoa: Challenges and Strategies in Data Lifecycle Management ...
Moemoea nui Aotearoa: Challenges and Strategies in Data Lifecycle Management ...
Globus 151 views
Integrating Globus into the Tapis API by Globus
Integrating Globus into the Tapis APIIntegrating Globus into the Tapis API
Integrating Globus into the Tapis API
Globus 116 views
Getting Started with Globus for Developers by Globus
Getting Started with Globus for DevelopersGetting Started with Globus for Developers
Getting Started with Globus for Developers
Globus 111 views
iRODS Globus Connector by Globus
iRODS Globus ConnectoriRODS Globus Connector
iRODS Globus Connector
Globus 85 views
Shaping the Future: To Globus Compute and Beyond! by Globus
Shaping the Future: To Globus Compute and Beyond!Shaping the Future: To Globus Compute and Beyond!
Shaping the Future: To Globus Compute and Beyond!
Globus 116 views

Recently uploaded

Software evolution understanding: Automatic extraction of software identifier... by
Software evolution understanding: Automatic extraction of software identifier...Software evolution understanding: Automatic extraction of software identifier...
Software evolution understanding: Automatic extraction of software identifier...Ra'Fat Al-Msie'deen
9 views33 slides
HarshithAkkapelli_Presentation.pdf by
HarshithAkkapelli_Presentation.pdfHarshithAkkapelli_Presentation.pdf
HarshithAkkapelli_Presentation.pdfharshithakkapelli
11 views16 slides
Navigating container technology for enhanced security by Niklas Saari by
Navigating container technology for enhanced security by Niklas SaariNavigating container technology for enhanced security by Niklas Saari
Navigating container technology for enhanced security by Niklas SaariMetosin Oy
14 views34 slides
Dapr Unleashed: Accelerating Microservice Development by
Dapr Unleashed: Accelerating Microservice DevelopmentDapr Unleashed: Accelerating Microservice Development
Dapr Unleashed: Accelerating Microservice DevelopmentMiroslav Janeski
10 views29 slides
Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated... by
Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated...Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated...
Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated...TomHalpin9
6 views29 slides
Myths and Facts About Hospice Care: Busting Common Misconceptions by
Myths and Facts About Hospice Care: Busting Common MisconceptionsMyths and Facts About Hospice Care: Busting Common Misconceptions
Myths and Facts About Hospice Care: Busting Common MisconceptionsCare Coordinations
6 views1 slide

Recently uploaded(20)

Software evolution understanding: Automatic extraction of software identifier... by Ra'Fat Al-Msie'deen
Software evolution understanding: Automatic extraction of software identifier...Software evolution understanding: Automatic extraction of software identifier...
Software evolution understanding: Automatic extraction of software identifier...
Navigating container technology for enhanced security by Niklas Saari by Metosin Oy
Navigating container technology for enhanced security by Niklas SaariNavigating container technology for enhanced security by Niklas Saari
Navigating container technology for enhanced security by Niklas Saari
Metosin Oy14 views
Dapr Unleashed: Accelerating Microservice Development by Miroslav Janeski
Dapr Unleashed: Accelerating Microservice DevelopmentDapr Unleashed: Accelerating Microservice Development
Dapr Unleashed: Accelerating Microservice Development
Miroslav Janeski10 views
Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated... by TomHalpin9
Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated...Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated...
Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated...
TomHalpin96 views
Myths and Facts About Hospice Care: Busting Common Misconceptions by Care Coordinations
Myths and Facts About Hospice Care: Busting Common MisconceptionsMyths and Facts About Hospice Care: Busting Common Misconceptions
Myths and Facts About Hospice Care: Busting Common Misconceptions
Advanced API Mocking Techniques by Dimpy Adhikary
Advanced API Mocking TechniquesAdvanced API Mocking Techniques
Advanced API Mocking Techniques
Dimpy Adhikary19 views
Unlocking the Power of AI in Product Management - A Comprehensive Guide for P... by NimaTorabi2
Unlocking the Power of AI in Product Management - A Comprehensive Guide for P...Unlocking the Power of AI in Product Management - A Comprehensive Guide for P...
Unlocking the Power of AI in Product Management - A Comprehensive Guide for P...
NimaTorabi212 views
DSD-INT 2023 The Danube Hazardous Substances Model - Kovacs by Deltares
DSD-INT 2023 The Danube Hazardous Substances Model - KovacsDSD-INT 2023 The Danube Hazardous Substances Model - Kovacs
DSD-INT 2023 The Danube Hazardous Substances Model - Kovacs
Deltares10 views
AI and Ml presentation .pptx by FayazAli87
AI and Ml presentation .pptxAI and Ml presentation .pptx
AI and Ml presentation .pptx
FayazAli8712 views
SUGCON ANZ Presentation V2.1 Final.pptx by Jack Spektor
SUGCON ANZ Presentation V2.1 Final.pptxSUGCON ANZ Presentation V2.1 Final.pptx
SUGCON ANZ Presentation V2.1 Final.pptx
Jack Spektor23 views
DSD-INT 2023 European Digital Twin Ocean and Delft3D FM - Dols by Deltares
DSD-INT 2023 European Digital Twin Ocean and Delft3D FM - DolsDSD-INT 2023 European Digital Twin Ocean and Delft3D FM - Dols
DSD-INT 2023 European Digital Twin Ocean and Delft3D FM - Dols
Deltares9 views
Ports-and-Adapters Architecture for Embedded HMI by Burkhard Stubert
Ports-and-Adapters Architecture for Embedded HMIPorts-and-Adapters Architecture for Embedded HMI
Ports-and-Adapters Architecture for Embedded HMI
Burkhard Stubert21 views
Sprint 226 by ManageIQ
Sprint 226Sprint 226
Sprint 226
ManageIQ5 views
360 graden fabriek by info33492
360 graden fabriek360 graden fabriek
360 graden fabriek
info33492122 views
Team Transformation Tactics for Holistic Testing and Quality (Japan Symposium... by Lisi Hocke
Team Transformation Tactics for Holistic Testing and Quality (Japan Symposium...Team Transformation Tactics for Holistic Testing and Quality (Japan Symposium...
Team Transformation Tactics for Holistic Testing and Quality (Japan Symposium...
Lisi Hocke35 views
DSD-INT 2023 3D hydrodynamic modelling of microplastic transport in lakes - J... by Deltares
DSD-INT 2023 3D hydrodynamic modelling of microplastic transport in lakes - J...DSD-INT 2023 3D hydrodynamic modelling of microplastic transport in lakes - J...
DSD-INT 2023 3D hydrodynamic modelling of microplastic transport in lakes - J...
Deltares12 views
FIMA 2023 Neo4j & FS - Entity Resolution.pptx by Neo4j
FIMA 2023 Neo4j & FS - Entity Resolution.pptxFIMA 2023 Neo4j & FS - Entity Resolution.pptx
FIMA 2023 Neo4j & FS - Entity Resolution.pptx
Neo4j8 views

Globus for System Administrators

  • 1. Globus for System Administrators Vas Vasiliadis vas@uchicago.edu June 22, 2022
  • 2. Agenda • Globus Connect Server v5 Overview • Deploying a Globus endpoint and creating collections • Common GCS configuration options • Customizing identity mapping • Monitoring and managing user activity 2
  • 3. Our focus in this session
  • 4. Globus Connect Server 4 • Makes your storage accessible via Globus • Software/tools installed and managed by sysadmin docs.globus.org/globus-connect-server Local system users Local Storage System (HPC cluster, NAS, …) Globus Connect Server DTN • Default access for all local accounts • Native packaging Linux: DEB, RPM
  • 6. Globus Connect Server v5 Architecture
  • 7. Globus Connect Server version 5 7
  • 8. GCS management conceptual architecture 8 Data Transfer Node GCS Command Line Interface GridFTP Server Globus Transfer Service GCS management requests Globus Auth Service GCS Manager authorize request using client ID/secret GCS Manager endpoint: abc.abc.data.globus.org Register a Globus Connect Server at developers.globus.org get GCS client ID, secret Define Globus resources (gateways, collections, …)
  • 10. Tutorial materials and other handy links bit.ly/gw-tut
  • 11. Fixin’ to get ready • Yes, you must have a chat with OpSec, … • Requires traffic flow on ports –443 (control channel) – must be publicly routable –50000-51000 (data channel) – can be on private net
  • 12. Requires a Globus subscription GCSv5 installation/configuration summary 1. Register a Globus Connect Server with Globus Auth 2. Install GCS packages on data transfer node (DTN) 3. Set up the endpoint and add node(s) 4. Create a POSIX storage gateway 5. Create a mapped collection 6. Associate endpoint with a subscription 7. Create a guest collection 8. Enable browser down/upload (HTTPS access) 9. Add other storage systems to the endpoint
  • 14. Register GCS and get credentials • Navigate to developers.globus.org and log in • (Optional) Create a project • Add a new Globus Connect Server • Generate a client secret • Save the client ID and secret
  • 15. 1. Register GCS and get credentials developers.globus.org
  • 16. 2. Install Globus Connect Server v5 packages $ curl -LOs http://downloads.globus.org/globus-connect- server/stable/installers/repo/deb/globus-repo_latest_all.deb $ dpkg -i globus-repo_latest_all.deb $ apt-key add /usr/share/globus-repo/RPM-GPG-KEY-Globus $ apt-get update $ apt-get --assume-yes install globus-connect-server54 Already done on EC2 instances.
  • 18. 3. Set up endpoint and add node $ globus-connect-server endpoint setup > "My Endpoint" > --organization "My Organization" > --client-id 4321dddd-af72-4c4b-9533-a0f4055dd321 > --contact-email me@uchicago.edu > --owner me@globusid.org $ globus-connect-server node setup > --client-id 4321dddd-af72-4c4b-9533-a0f4055dd321 Note: endpoint setup command generates deployment-key.json Use this file when setting up additional data transfer nodes
  • 19. Set up endpoint and add a DTN • Access server: ssh adminN@ornlN.globusdemo.org • Switch to root: sudo su • Run: globus-connect-server endpoint setup ... – Ensure --owner is the identity you used to register the GCS • Run: globus-connect-server node setup ... • Run: systemctl restart apache2 • Display endpoint details: – globus-connect-server login localhost – globus-connect-server endpoint show Cheatsheet bit.ly/gw-tut
  • 20. Our setup so far Run globus-connect-server node setup to set up additional data transfer nodes Copy deployment-key.json from original DTN
  • 21. Storage Gateways define a set of access policies • Authentication for local account-holders – Which identity domain(s) are acceptable? – How are identities mapped from domain(s) to local accounts? • Policy scope – Which parts of the storage system are accessible via Globus? – Which local accounts does this policy allow (or deny)? • High Assurance settings • MFA requirements
  • 22. Authentication for local account-holders • Primary access (via a mapped collection) requires an account on the host system* • Two-part authentication configuration: 1. Pick one or more identity domains 2. Configure the method to map the authenticated identity to an account on your system * You may allow primary users to share with others who don’t have accounts on your system
  • 23. Picking identity domains • User must present identity from one of the configured domains – On access attempts, linked identities will be scanned for a match – If no identity from the required domain(s), will be asked to link one • Identity domains may include… – …any organization in Globus federated list (incl. ornl.gov) – …your institution’s identity provider trusted by Globus – …a local OpenID Connect (OIDC) server using your PAM stack
  • 24. Mapping identities to local accounts • Default: Strip identity domain (everything after “@”) – e.g. userX@globusdemo.org maps to local account userX – Best for campus identities w/synchronized local accounts • Use --identity-mapping option on storage gateway – Specify expression in a JSON document – Execute a custom script docs.globus.org/globus-connect-server/v5.4/identity-mapping-guide/
  • 26. Creating a storage gateway • Our storage gateway will access a POSIX system – This is the only type permitted without a subscription • It will allow access to users with credentials from the globusid.org (or ornl.gov) domain • Reauthentication will be required every 90 minutes Cheatsheet bit.ly/gw-tut
  • 27. 4. Create a storage gateway $ globus-connect-server storage-gateway create posix > "My Storage Gateway" > --domain globusid.org > --authentication-timeout-mins 90 Allowed authentication domain Duration of user session when accessing collections via this storage gateway
  • 28. Our setup so far…
  • 29. Create a mapped collection on the POSIX gateway 29
  • 30. Creating a collection • Our collection will use the default identity mapping • It will be “rooted” at the user’s home directory • Access will require authentication with an identity from the globusid.org (or ornl.gov) domain Cheatsheet bit.ly/gw-tut
  • 31. 5. Create a mapped collection $ globus-connect-server collection create > f77ff456-1f18-41d3-94a7-f3fd8858ea4d > / > "My Mapped Collection" Collections are rooted at the specified base path Specifying "/" as the base path sets the collection root to the local user’s home directory Storage gateway ID Collection base path
  • 32. Our setup so far…
  • 35. Alternative authentication flow (if not using Globus trusted IdP)
  • 37. We are using the default identity mapping, so… • Create a local user account with the same name as your globusid.org (or ornl.org) identity – e.g., for me@globusid.org create local account “me” – e.g., for me@ornl.gov create local account “me” • adduser --disabled-password --gecos 'me’ me • Access your mapped collection via the web app… • …and move some files into it! Cheatsheet bit.ly/gw-tut
  • 38. Common Collection configuration options • Restrict access: local users, local groups • Restrict sharing: paths, local users, local groups • Allow guest collections à enables sharing • Enable HTTPS access • Force data channel encryption
  • 39. Local account restrictions • Note: These only apply to mapped collections • A storage gateway’s allowed identity domains and identity mapping method determine the universe of local accounts that may access the mapped collection • You can further narrow the access universe using… --user-allow --user-deny --posix-group-allow (POSIX storage gateways only) --posix-group-deny (POSIX storage gateways only)
  • 40. Path restrictions • Always use the narrowest base path possible for your storage gateway(s) and collection(s) – Storage gateway base specifies where collections may be created – Collection base specifies the base directory for the collection • POSIX storage gateway – Use --restrict_paths to specify narrower read, read/write, or none access for specific paths – You provide a JSON doc that lists paths for each permission type – Note: These are absolute paths on the host system • Collection: specify narrowest base path that satisfies the need
  • 42. Setting path restrictions • A new storage gateway will limit access to /home – NB: No change to local permissions, only visibility via Globus • We specify the path restrictions in paths.json – This file is in your admin user’s home directory • Run: storage-gateway create command with the --restrict-paths option • Create a new POSIX mapped collection Cheatsheet bit.ly/gw-tut
  • 43. Create a restricted storage gateway, collection $ globus-connect-server storage-gateway create posix > "My Storage Gateway - Restricted" > --domain globusid.org > --authentication-timeout-mins 90 > --restrict-paths file:/home/adminN/paths.json $ globus-connect-server collection create > 3926bf02-6bc3-11e7-a9c6-22000bf2d287 > / > "My Mapped Collection – Restricted" Fully qualified filename containing rule(s) for restricting access to specific filesystem paths
  • 44. Revisit your mapped collections • Your will need to authenticate on your new (restricted access) collection, and consent • Note the access behavior differences between the two mapped collections
  • 45. Subscriptions and Endpoint Roles • Subscription(s) configured for your institution • Multiple Subscription Managers per subscription • Subscription Manager ties endpoint to subscription – Results in a “managed” endpoint • Assign additional roles for endpoint management – Administrator, Manager, Monitor
  • 46. Associate the endpoint with a subscription 46
  • 47. 6. Associate endpoint with a subscription • Subscription managers can enable subscription features on an endpoint • If you are not the subscription manager, just send your endpoint ID to your subscription manager and ask them to add it.
  • 48. Make your endpoint “Managed” • Option A: Put your endpoint ID in the spreadsheet and Greg will make it managed • Option B: Run globus-connect-server endpoint set-subscription-id • Confirm: globus-connect-server endpoint show Cheatsheet bit.ly/gw-tut
  • 49. 6. Associate endpoint with a subscription $ globus-connect-server endpoint set-subscription-id DEFAULT $ globus-connect-server endpoint set-subscription-id > 39299902-6bc3-aa56-a9c6-22000bf2d287 Subscription managers can also set this via the web app Console page: app.globus.org/console (look under the Endpoints tab) Your identity may already be a subscription manager on a subscription
  • 50. Be identity-, role-, and permission-aware • Default: Only endpoint owner can configure an endpoint • Delegate administrator role to other sysadmins – Best practice: Delegate to a Globus group, not individuals • Check identity using the session command • Check resource permissions on storage gateways and collections with --include-private-policies option docs.globus.org/globus-connect-server/v5.4/reference/role/
  • 51. 7. Create a guest collection • Created by user, not endpoint administrator • Grants access to specific Globus users without a mapped local account • “Guest” users have the same (or more limited) permissions as the guest collection creator – Access logs show access by the collection creator* • Guest collection’s root is relative to the mapped collection’s base path * High Assurance collections log guest user identities to enable auditing
  • 52. Sharing restrictions • Guest collections may be created in any directory accessible by the collection, by any authorized local account • You can restrict the authorized accounts… o --sharing-user-allow --sharing-user-deny o --posix-sharing-group-allow o --posix-sharing-group-deny • …and sharing paths… o --sharing-restrict-paths (specify JSON PathRestrictions) • You can also set policies for specific user/path combinations o $ globus-connect-server sharing-policy create ...
  • 53. Create and access a guest collection 53
  • 54. Create and access a guest collection • Enable creation of guest collections • Run: globus-connect-server collection update • Access the mapped collection; create /projects • Create a guest collection on the /projects directory • Grant read access to the “Tutorial Users” group • Authenticate and browse guest collection
  • 55. 8. Enable web browser upload/download • Authorized users can upload, download files via a browser • Must have permissions to the collection – Collection configuration governs access – Web server is a different application (separate authentication)
  • 57. Enable HTTPS access • Run: globus-connect-server collection update • Access your mapped collection • Upload a file from your laptop (and download it too!) Cheatsheet bit.ly/gw-tut
  • 59. Things to do with the management console • Monitor current transfers on your endpoints – See what’s going on at the transfer request level – Much better than watching individual file transfers • Pause (and later resume) a transfer in progress – Sends a notice to the transfer owner • Set a pause rule for current and future transfers – Ideal for maintenance mode – Notifies transfer owners, – Tasks resume when endpoint is un-paused docs.globus.org/management-console-guide/
  • 60. Resources • GCSv5 Guides: docs.globus.org/globus-connect-server/ • Migration: docs.globus.org/globus-connect- server/migrating-to-v5.4/ • Globus support: support@globus.org 60