SlideShare a Scribd company logo
1 of 55
Download to read offline
Vas Vasiliadis
vas@uchicago.edu
February 27, 2024
Globus for System Administrators
To participate see: bit.ly/gw-tut-rpi
Follow instructions under
Globus for System Administrators
2
Our focus in this session
• Makes your storage accessible via Globus
• Software/tools installed and managed by sysadmin
• Native packaging Linux: DEB, RPM
docs.globus.org/globus-connect-server
Agenda
• Globus Connect Server Overview
• Deploying a Globus endpoint and creating collections
• Common GCS configuration options
• Monitoring and managing user activity
5
Globus Connect
Server Overview
6
Globus Connect Server Components
GCS management conceptual architecture
8
Data Transfer Node
GCS Command
Line Interface
GridFTP
Server
Globus
Transfer
Service
GCS
management
requests
Globus
Auth
Service
GCS Manager authorize request
using client ID/secret
GCS Manager endpoint:
abc.abc.data.globus.org
Installation register GCS client with Globus
Auth; first-timers will need to create a
Project to contain the registration
Define Globus
resources
(gateways,
collections, …)
GCS installation
walkthrough
9
docs.globus.org/globus-connect-server
Key prerequisite: network accessibility
• Yes, you must have a chat with OpSec, NetOps, …
• Control channel: must be on publicly routable IP
– Default: port 443; configurable
– Inbound and outbound traffic from all
– Can restrict to CIDR block, but severely limits functionality
• Data channel: can be on private network
– Default: 50000-51000
– Configurable, but strongly advise against it
• But you already have Science DMZ, right? ;-)
Requires a Globus subscription
GCSv5 installation/configuration summary
1. Install GCS packages on data transfer node (DTN)
2. Set up the endpoint and add node(s)
3. Create a POSIX storage gateway
4. Create a mapped collection
5. Associate endpoint with a subscription
6. Create a guest collection
7. Enable browser down/upload (HTTPS access)
8. Add other storage systems to the endpoint
1. Install Globus Connect Server v5 packages
$ curl -LOs http://downloads.globus.org/globus-connect-
server/stable/installers/repo/deb/globus-repo_latest_all.deb
$ dpkg -i globus-repo_latest_all.deb
$ apt-key add /usr/share/globus-repo/RPM-GPG-KEY-Globus
$ apt-get update
$ apt-get --assume-yes install globus-connect-server54
Already done on your EC2 instances.
Endpoint creation and
node setup
13
2. Set up endpoint and add node
$ globus-connect-server endpoint setup 
> "My Endpoint" 
> --organization "My Organization" 
> --contact-email me@uchicago.edu 
> --owner me@uchicago.edu
$ sudo globus-connect-server node setup
Note: endpoint setup command generates deployment-key.json
Use this file when setting up additional data transfer nodes
Identity must be known to
Globus Auth; log in and
confirm prior to endpoint setup
Set up endpoint and add a DTN
• Access server: ssh adminN@rpiN.globusdemo.org
• Switch to root: sudo su
• Run: globus-connect-server endpoint setup ...
– Ensure --owner is the identity you used to register the GCS
• Run: globus-connect-server node setup ...
• Run: systemctl restart apache2
• Display endpoint details:
– globus-connect-server login localhost
– globus-connect-server endpoint show Cheatsheet
bit.ly/gw-tut-rpi
GCS clients and projects
• Running the endpoint setup command creates a Globus Auth
client
• If this is your first GCS installation, a project will also be
automatically created
• Projects are mostly an organizing construct, necessary for
access control over clients and their secrets
• View projects and Globus Auth clients: Settings à Developers
• If you have existing endpoints/client applications registered
with Globus, endpoint setup will ask for a project ID
Our setup so far
Run globus-connect-server node setup
to set up additional data transfer nodes
Copy deployment-key.json
from original DTN
Storage Gateways define a set of access policies
• Authentication for local account-holders
– Which identity domain(s) are acceptable?
– How are identities mapped from domain(s) to local accounts?
• Policy scope
– Which parts of the storage system are accessible via Globus?
– Which local accounts does this policy allow (or deny)?
• High Assurance settings
• MFA requirements
Authentication for local account-holders
• Primary access (via a mapped collection) requires an
account on the host system*
• Two-part authentication configuration:
1. Pick one or more identity domains
2. Configure the method to map the authenticated identity to an
account on your system
* You may allow primary users to share with others who don’t have accounts on your system
Picking identity domains
• User must present identity from one of the configured
domains
– On access attempts, linked identities will be scanned for a match
– If no identity from the required domain(s), will be asked to link one
• Identity domains may include…
– …any organization in Globus federated list
– …your institution’s identity provider trusted by Globus
– …a local OpenID Connect (OIDC) server using your PAM stack
Mapping identities to local accounts
• Default: Strip identity domain (everything after “@”)
– e.g. userX@uchicago.edu maps to local account userX
– Best for campus identities w/synchronized local accounts
• Use --identity-mapping option on storage gateway
– Specify expression in a JSON document
– Execute a custom script
docs.globus.org/globus-connect-server/v5.4/identity-mapping-guide/
Create a POSIX
storage gateway
22
Creating a storage gateway
• Our storage gateway will access a POSIX system
– This is the only type permitted without a subscription
• It will allow access to users with credentials from the
uchicago.edu (subsctitute your own) domain
• Reauthentication will be required every 90 minutes
Cheatsheet
bit.ly/gw-tut-rpi
3. Create a storage gateway
$ globus-connect-server storage-gateway create posix 
> "My Storage Gateway" 
> --domain uchicago.edu 
> --authentication-timeout-mins 90
Allowed authentication
domain
Duration of user session
when accessing collections
via this storage gateway
Our setup so far…
/ (user home)
Create a mapped
collection on the
POSIX gateway
26
Creating a collection
• Our collection will use the default identity mapping
• It will be “rooted” at the user’s home directory
• Access will require authentication with an identity
from the uchicago.edu (substitute your own) domain
Cheatsheet
bit.ly/gw-tut-rpi
4. Create a mapped collection
$ globus-connect-server collection create 
> f77ff456-1f18-41d3-94a7-f3fd8858ea4d 
> / 
> "My Mapped Collection"
Collections are rooted at the specified base path
Specifying "/" as the base path sets the collection root to the local
user’s home directory
Storage gateway ID
Collection base path
Our setup so far…
Understanding
access to mapped
collections
30
Access our mapped
collection
33
We are using the default identity mapping, so…
• Create a local user account with the same name as
your globusid.org (or other IdP) identity
– e.g., for me@uchicago.edu create local account “me”
– e.g., for me2@orcid.org create local account “me2”
• adduser --disabled-password --gecos 'me' me
• Access your mapped collection via the web app…
• …and move some files, if you like
Cheatsheet
bit.ly/gw-tut-rpi
Common Collection configuration options
• Restrict access: local users, local groups
• Restrict sharing: paths, local users, local groups
• Allow guest collections à enables sharing
• Enable HTTPS access
• Force data channel encryption
Local account restrictions
• Note: These only apply to mapped collections
• A storage gateway’s allowed identity domains and
identity mapping method determine the universe of local
accounts that may access the mapped collection
• You can further narrow the access universe using…
--user-allow
--user-deny
--posix-group-allow (POSIX storage gateways only)
--posix-group-deny (POSIX storage gateways only)
Path restrictions
• Always use the narrowest base path possible for your storage
gateway(s) and collection(s)
– Storage gateway base specifies where collections may be created
– Collection base specifies the base directory for the collection
• POSIX storage gateway
– Use --restrict_paths to specify narrower read, read/write, or none
access for specific paths
– You provide a JSON doc that lists paths for each permission type
– Note: These are absolute paths on the host system
• Collection: specify narrowest base path that satisfies the need
Restrict collection
access to filesystem
38
Setting path restrictions
• A new storage gateway will limit access to /home
– NB: No change to local permissions, only visibility via Globus
• We specify the path restrictions in paths.json
– This file is in your admin user’s home directory
• Run: storage-gateway create command with the
--restrict-paths option
• Create a new POSIX mapped collection
Cheatsheet
bit.ly/gw-tut-rpi
5. Create a restricted storage gateway, collection
$ globus-connect-server storage-gateway create posix 
> "My Storage Gateway - Restricted" 
> --domain uchicago.edu 
> --authentication-timeout-mins 90 
> --restrict-paths file:/home/adminN/paths.json
$ globus-connect-server collection create 
> 3926bf02-6bc3-11e7-a9c6-22000bf2d287 
> / 
> "My Mapped Collection – Restricted"
Fully qualified filename containing
rule(s) for restricting access to
specific filesystem paths
Revisit your mapped collections
• Your will need to authenticate on your new (restricted
access) collection, and consent
• Note the access behavior differences between the two
mapped collections
Subscriptions and Endpoint Roles
• Subscription(s) configured for your institution
• Multiple Subscription Managers per subscription
• Subscription Manager associates an endpoint with a
subscription
– Results in a “subscribed” endpoint
• Assign additional roles for endpoint management
– Administrator, Manager, Monitor
Converting endpoint(s) to “subscribed”
• As a member of an institution’s subscription, you can
subscribe your own endpoint(s)
• Subscription Managers can subscribe any endpoint at
the institution
• Check role in your institution’s subscription to
determine level of access and control
• But, not yet … J
Associate the
endpoint with a
subscription
44
Convert your endpoint to “subscribed”
• Confirm that your Globus identity is a subscription
manager à we’ll do this for you now
• Run globus-connect-server endpoint set-
subscription-id
• Confirm: globus-connect-server endpoint show
Cheatsheet
bit.ly/gw-tut-rpi
6. Associate endpoint with a subscription
$ globus-connect-server endpoint set-subscription-id DEFAULT
$ globus-connect-server endpoint set-subscription-id 
> 39299902-6bc3-aa56-a9c6-22000bf2d287
Subscription managers can also set this via the web app Console
page: app.globus.org/console (look under the Endpoints tab)
Your identity may already be a
subscription manager on a
subscription
Be identity-, role-, and permission-aware
• Default: Only endpoint owner can configure an endpoint
• Delegate administrator role to other sysadmins
– Best practice: Delegate to a Globus group, not individuals
• Check identity using the session command
• Check resource permissions on storage gateways and
collections with --include-private-policies option
docs.globus.org/globus-connect-server/v5.4/reference/role/
7. Create a guest collection
• Created by user, not endpoint administrator
• Grants access to specific Globus users without a
mapped local account
• “Guest” users have the same (or more limited)
permissions as the guest collection creator
– Access logs show access by the collection creator*
• Guest collection’s root is relative to the mapped
collection’s base path
* High Assurance collections log guest user identities to enable auditing
Sharing restrictions
• Guest collections may be created in any directory accessible by
the collection, by any authorized local account
• You can restrict the authorized accounts…
o --sharing-user-allow
--sharing-user-deny
o --posix-sharing-group-allow
o --posix-sharing-group-deny
• …and sharing paths…
o --sharing-restrict-paths (specify JSON PathRestrictions)
• You can also set policies for specific user/path combinations
o $ globus-connect-server sharing-policy create ...
Create and access a
guest collection
51
Create and access a guest collection
• Enable creation of guest collections
• Run: globus-connect-server collection update
• Access the mapped collection; create /projects
• Create a guest collection on the /projects directory
• Grant read access to the “Tutorial Users” group
• Authenticate and browse guest collection
8. Enable web browser upload/download
• Authorized users can upload,
download files via a browser
• Must have permissions to the
collection
– Collection configuration governs
access
– Web server is a different
application (separate
authentication)
Enable/disable file
download/upload via
browser
54
Enable HTTPS access
• Run: globus-connect-server collection update
• Access your mapped collection
• Upload a file from your laptop (and download it too!)
Cheatsheet
bit.ly/gw-tut-rpi
Using the
management console
56
Things to do with the management console
• Monitor current transfers on your endpoints
– See what’s going on at the transfer request level
– Much better than watching individual file transfers
• Pause (and later resume) a transfer in progress
– Sends a notice to the transfer owner
• Set a pause rule for current and future transfers
– Ideal for maintenance mode
– Notifies transfer owners,
– Tasks resume when endpoint is un-paused
docs.globus.org/management-console-guide/
Resources
• GCSv5 Guides: docs.globus.org/globus-connect-server/
• Migration: docs.globus.org/globus-connect-
server/migrating-to-v5.4/
• Globus support: support@globus.org
58

More Related Content

Similar to Globus for System Administrators

Similar to Globus for System Administrators (20)

Migrating to Globus Connect Server v5
Migrating to Globus Connect Server v5Migrating to Globus Connect Server v5
Migrating to Globus Connect Server v5
 
Introduction to Globus for System Administrators (GlobusWorld Tour - UMich)
Introduction to Globus for System Administrators (GlobusWorld Tour - UMich)Introduction to Globus for System Administrators (GlobusWorld Tour - UMich)
Introduction to Globus for System Administrators (GlobusWorld Tour - UMich)
 
Globus for System Administrators
Globus for System AdministratorsGlobus for System Administrators
Globus for System Administrators
 
Tutorial: Introduction to Globus for System Administrators
Tutorial: Introduction to Globus for System AdministratorsTutorial: Introduction to Globus for System Administrators
Tutorial: Introduction to Globus for System Administrators
 
Globus Connect Server Deep Dive - Advanced Configuration Options and Use Cases
Globus Connect Server Deep Dive - Advanced Configuration Options and Use CasesGlobus Connect Server Deep Dive - Advanced Configuration Options and Use Cases
Globus Connect Server Deep Dive - Advanced Configuration Options and Use Cases
 
Advanced Globus System Administration Topics
Advanced Globus System Administration TopicsAdvanced Globus System Administration Topics
Advanced Globus System Administration Topics
 
Globus for System Administrators (GlobusWorld Tour - UCSD)
Globus for System Administrators (GlobusWorld Tour - UCSD)Globus for System Administrators (GlobusWorld Tour - UCSD)
Globus for System Administrators (GlobusWorld Tour - UCSD)
 
Globus Endpoint Setup and Configuration - XSEDE14 Tutorial
Globus Endpoint Setup and Configuration - XSEDE14 TutorialGlobus Endpoint Setup and Configuration - XSEDE14 Tutorial
Globus Endpoint Setup and Configuration - XSEDE14 Tutorial
 
Globus Endpoint Administration (GlobusWorld Tour - STFC)
Globus Endpoint Administration (GlobusWorld Tour - STFC)Globus Endpoint Administration (GlobusWorld Tour - STFC)
Globus Endpoint Administration (GlobusWorld Tour - STFC)
 
Globus for System Administrators (CHPC 2019 - South Africa)
Globus for System Administrators (CHPC 2019 - South Africa)Globus for System Administrators (CHPC 2019 - South Africa)
Globus for System Administrators (CHPC 2019 - South Africa)
 
Globus for System Administrators (GlobusWorld Tour - Columbia University)
Globus for System Administrators (GlobusWorld Tour - Columbia University)Globus for System Administrators (GlobusWorld Tour - Columbia University)
Globus for System Administrators (GlobusWorld Tour - Columbia University)
 
Globus Command Line Interface (APS Workshop)
Globus Command Line Interface (APS Workshop)Globus Command Line Interface (APS Workshop)
Globus Command Line Interface (APS Workshop)
 
GlobusWorld 2021 Tutorial: The Globus CLI, Platform and SDK
GlobusWorld 2021 Tutorial: The Globus CLI, Platform and SDKGlobusWorld 2021 Tutorial: The Globus CLI, Platform and SDK
GlobusWorld 2021 Tutorial: The Globus CLI, Platform and SDK
 
Globus Connect Server v5 Q&A Briefing
Globus Connect Server v5 Q&A BriefingGlobus Connect Server v5 Q&A Briefing
Globus Connect Server v5 Q&A Briefing
 
Automating Research Data Flows and Introduction to the Globus Platform
Automating Research Data Flows and Introduction to the Globus PlatformAutomating Research Data Flows and Introduction to the Globus Platform
Automating Research Data Flows and Introduction to the Globus Platform
 
Data Publication and Discovery with Globus
Data Publication and Discovery with GlobusData Publication and Discovery with Globus
Data Publication and Discovery with Globus
 
Globus Endpoint Migration and Advanced Administration Topics
Globus Endpoint Migration and Advanced Administration TopicsGlobus Endpoint Migration and Advanced Administration Topics
Globus Endpoint Migration and Advanced Administration Topics
 
Working with Globus Platform Services and Portals
Working with Globus Platform Services and PortalsWorking with Globus Platform Services and Portals
Working with Globus Platform Services and Portals
 
Automating Research Data Flows and an Introduction to the Globus Platform
Automating Research Data Flows and an Introduction to the Globus PlatformAutomating Research Data Flows and an Introduction to the Globus Platform
Automating Research Data Flows and an Introduction to the Globus Platform
 
Globus Connect Server 5.1 Webinar
Globus Connect Server 5.1 WebinarGlobus Connect Server 5.1 Webinar
Globus Connect Server 5.1 Webinar
 

More from Globus

Providing Globus Services to Users Of JASMIN for Environmental Data Analysis
Providing Globus Services to Users Of JASMIN for Environmental Data AnalysisProviding Globus Services to Users Of JASMIN for Environmental Data Analysis
Providing Globus Services to Users Of JASMIN for Environmental Data Analysis
Globus
 
How to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good PracticesHow to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good Practices
Globus
 
Extending Globus into a Site-wide Automated Data Infrastructure
Extending Globus into a Site-wide Automated Data InfrastructureExtending Globus into a Site-wide Automated Data Infrastructure
Extending Globus into a Site-wide Automated Data Infrastructure
Globus
 

More from Globus (20)

The Department of Energy's Integrated Research Infrastructure (IRI).pdf
The Department of Energy's Integrated Research Infrastructure (IRI).pdfThe Department of Energy's Integrated Research Infrastructure (IRI).pdf
The Department of Energy's Integrated Research Infrastructure (IRI).pdf
 
Research Automation with Globus Flows.pptx
Research Automation with Globus Flows.pptxResearch Automation with Globus Flows.pptx
Research Automation with Globus Flows.pptx
 
Reactive Documents and Computational Pipelines
Reactive Documents and Computational PipelinesReactive Documents and Computational Pipelines
Reactive Documents and Computational Pipelines
 
Providing Globus Services to Users Of JASMIN for Environmental Data Analysis
Providing Globus Services to Users Of JASMIN for Environmental Data AnalysisProviding Globus Services to Users Of JASMIN for Environmental Data Analysis
Providing Globus Services to Users Of JASMIN for Environmental Data Analysis
 
Understanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSageUnderstanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSage
 
Innovating Inference: Remote Triggering of Large Language Models on HPC Clust...
Innovating Inference: Remote Triggering of Large Language Models on HPC Clust...Innovating Inference: Remote Triggering of Large Language Models on HPC Clust...
Innovating Inference: Remote Triggering of Large Language Models on HPC Clust...
 
How to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good PracticesHow to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good Practices
 
GlobusWorld 2024: Opening Keynote Address
GlobusWorld 2024: Opening Keynote AddressGlobusWorld 2024: Opening Keynote Address
GlobusWorld 2024: Opening Keynote Address
 
Globus Compute with Integrated Research Infrastructure (IRI) Workflows
Globus Compute with Integrated Research Infrastructure (IRI) WorkflowsGlobus Compute with Integrated Research Infrastructure (IRI) Workflows
Globus Compute with Integrated Research Infrastructure (IRI) Workflows
 
Exploring Innovations in Data Repository Solutions Insights from the U.S. Geo...
Exploring Innovations in Data Repository Solutions Insights from the U.S. Geo...Exploring Innovations in Data Repository Solutions Insights from the U.S. Geo...
Exploring Innovations in Data Repository Solutions Insights from the U.S. Geo...
 
Globus at the U.S. Geological Survey (USGS)
Globus at the U.S. Geological Survey (USGS)Globus at the U.S. Geological Survey (USGS)
Globus at the U.S. Geological Survey (USGS)
 
Globus and the Integrated Research Infrastructure (IRI)
Globus and the Integrated Research Infrastructure (IRI)Globus and the Integrated Research Infrastructure (IRI)
Globus and the Integrated Research Infrastructure (IRI)
 
First Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User EndpointsFirst Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User Endpoints
 
Extending Globus into a Site-wide Automated Data Infrastructure
Extending Globus into a Site-wide Automated Data InfrastructureExtending Globus into a Site-wide Automated Data Infrastructure
Extending Globus into a Site-wide Automated Data Infrastructure
 
Enhancing Research Orchestration Capabilities at ORNL.pptx
Enhancing Research Orchestration Capabilities at ORNL.pptxEnhancing Research Orchestration Capabilities at ORNL.pptx
Enhancing Research Orchestration Capabilities at ORNL.pptx
 
Enhancing Performance with Globus and the Science DMZ.pdf
Enhancing Performance with Globus and the Science DMZ.pdfEnhancing Performance with Globus and the Science DMZ.pdf
Enhancing Performance with Globus and the Science DMZ.pdf
 
Developing Distributed High-performance Computing Capabilities of an Open Sci...
Developing Distributed High-performance Computing Capabilities of an Open Sci...Developing Distributed High-performance Computing Capabilities of an Open Sci...
Developing Distributed High-performance Computing Capabilities of an Open Sci...
 
Climate Science Flows Enabling Petabyte-Scale Climate Analysis with the Earth...
Climate Science Flows Enabling Petabyte-Scale Climate Analysis with the Earth...Climate Science Flows Enabling Petabyte-Scale Climate Analysis with the Earth...
Climate Science Flows Enabling Petabyte-Scale Climate Analysis with the Earth...
 
Introduction to Globus Compute - GlobusWorld 2024
Introduction to Globus Compute - GlobusWorld 2024Introduction to Globus Compute - GlobusWorld 2024
Introduction to Globus Compute - GlobusWorld 2024
 
Instrument Data Automation: The Life of a Flow
Instrument Data Automation: The Life of a FlowInstrument Data Automation: The Life of a Flow
Instrument Data Automation: The Life of a Flow
 

Recently uploaded

Recently uploaded (20)

Optimizing Operations by Aligning Resources with Strategic Objectives Using O...
Optimizing Operations by Aligning Resources with Strategic Objectives Using O...Optimizing Operations by Aligning Resources with Strategic Objectives Using O...
Optimizing Operations by Aligning Resources with Strategic Objectives Using O...
 
Abortion Clinic Pretoria ](+27832195400*)[ Abortion Clinic Near Me ● Abortion...
Abortion Clinic Pretoria ](+27832195400*)[ Abortion Clinic Near Me ● Abortion...Abortion Clinic Pretoria ](+27832195400*)[ Abortion Clinic Near Me ● Abortion...
Abortion Clinic Pretoria ](+27832195400*)[ Abortion Clinic Near Me ● Abortion...
 
Abortion Clinic In Springs ](+27832195400*)[ 🏥 Safe Abortion Pills in Springs...
Abortion Clinic In Springs ](+27832195400*)[ 🏥 Safe Abortion Pills in Springs...Abortion Clinic In Springs ](+27832195400*)[ 🏥 Safe Abortion Pills in Springs...
Abortion Clinic In Springs ](+27832195400*)[ 🏥 Safe Abortion Pills in Springs...
 
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
 
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdf
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdfStrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdf
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdf
 
Auto Affiliate AI Earns First Commission in 3 Hours..pdf
Auto Affiliate  AI Earns First Commission in 3 Hours..pdfAuto Affiliate  AI Earns First Commission in 3 Hours..pdf
Auto Affiliate AI Earns First Commission in 3 Hours..pdf
 
Spring into AI presented by Dan Vega 5/14
Spring into AI presented by Dan Vega 5/14Spring into AI presented by Dan Vega 5/14
Spring into AI presented by Dan Vega 5/14
 
The mythical technical debt. (Brooke, please, forgive me)
The mythical technical debt. (Brooke, please, forgive me)The mythical technical debt. (Brooke, please, forgive me)
The mythical technical debt. (Brooke, please, forgive me)
 
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Era
 
Community is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea GouletCommunity is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea Goulet
 
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit MilanWorkshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
 
What is a Recruitment Management Software?
What is a Recruitment Management Software?What is a Recruitment Management Software?
What is a Recruitment Management Software?
 
Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...
Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...
Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...
 
OpenChain @ LF Japan Executive Briefing - May 2024
OpenChain @ LF Japan Executive Briefing - May 2024OpenChain @ LF Japan Executive Briefing - May 2024
OpenChain @ LF Japan Executive Briefing - May 2024
 
Salesforce Introduced Zero Copy Partner Network to Simplify the Process of In...
Salesforce Introduced Zero Copy Partner Network to Simplify the Process of In...Salesforce Introduced Zero Copy Partner Network to Simplify the Process of In...
Salesforce Introduced Zero Copy Partner Network to Simplify the Process of In...
 
Effective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeConEffective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeCon
 
Lessons Learned from Building a Serverless Notifications System.pdf
Lessons Learned from Building a Serverless Notifications System.pdfLessons Learned from Building a Serverless Notifications System.pdf
Lessons Learned from Building a Serverless Notifications System.pdf
 
Food Delivery Business App Development Guide 2024
Food Delivery Business App Development Guide 2024Food Delivery Business App Development Guide 2024
Food Delivery Business App Development Guide 2024
 
From Theory to Practice: Utilizing SpiraPlan's REST API
From Theory to Practice: Utilizing SpiraPlan's REST APIFrom Theory to Practice: Utilizing SpiraPlan's REST API
From Theory to Practice: Utilizing SpiraPlan's REST API
 
Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024
Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024
Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024
 

Globus for System Administrators

  • 1. Vas Vasiliadis vas@uchicago.edu February 27, 2024 Globus for System Administrators
  • 2. To participate see: bit.ly/gw-tut-rpi Follow instructions under Globus for System Administrators 2
  • 3. Our focus in this session • Makes your storage accessible via Globus • Software/tools installed and managed by sysadmin • Native packaging Linux: DEB, RPM docs.globus.org/globus-connect-server
  • 4. Agenda • Globus Connect Server Overview • Deploying a Globus endpoint and creating collections • Common GCS configuration options • Monitoring and managing user activity 5
  • 7. GCS management conceptual architecture 8 Data Transfer Node GCS Command Line Interface GridFTP Server Globus Transfer Service GCS management requests Globus Auth Service GCS Manager authorize request using client ID/secret GCS Manager endpoint: abc.abc.data.globus.org Installation register GCS client with Globus Auth; first-timers will need to create a Project to contain the registration Define Globus resources (gateways, collections, …)
  • 9. Key prerequisite: network accessibility • Yes, you must have a chat with OpSec, NetOps, … • Control channel: must be on publicly routable IP – Default: port 443; configurable – Inbound and outbound traffic from all – Can restrict to CIDR block, but severely limits functionality • Data channel: can be on private network – Default: 50000-51000 – Configurable, but strongly advise against it • But you already have Science DMZ, right? ;-)
  • 10. Requires a Globus subscription GCSv5 installation/configuration summary 1. Install GCS packages on data transfer node (DTN) 2. Set up the endpoint and add node(s) 3. Create a POSIX storage gateway 4. Create a mapped collection 5. Associate endpoint with a subscription 6. Create a guest collection 7. Enable browser down/upload (HTTPS access) 8. Add other storage systems to the endpoint
  • 11. 1. Install Globus Connect Server v5 packages $ curl -LOs http://downloads.globus.org/globus-connect- server/stable/installers/repo/deb/globus-repo_latest_all.deb $ dpkg -i globus-repo_latest_all.deb $ apt-key add /usr/share/globus-repo/RPM-GPG-KEY-Globus $ apt-get update $ apt-get --assume-yes install globus-connect-server54 Already done on your EC2 instances.
  • 13. 2. Set up endpoint and add node $ globus-connect-server endpoint setup > "My Endpoint" > --organization "My Organization" > --contact-email me@uchicago.edu > --owner me@uchicago.edu $ sudo globus-connect-server node setup Note: endpoint setup command generates deployment-key.json Use this file when setting up additional data transfer nodes Identity must be known to Globus Auth; log in and confirm prior to endpoint setup
  • 14. Set up endpoint and add a DTN • Access server: ssh adminN@rpiN.globusdemo.org • Switch to root: sudo su • Run: globus-connect-server endpoint setup ... – Ensure --owner is the identity you used to register the GCS • Run: globus-connect-server node setup ... • Run: systemctl restart apache2 • Display endpoint details: – globus-connect-server login localhost – globus-connect-server endpoint show Cheatsheet bit.ly/gw-tut-rpi
  • 15. GCS clients and projects • Running the endpoint setup command creates a Globus Auth client • If this is your first GCS installation, a project will also be automatically created • Projects are mostly an organizing construct, necessary for access control over clients and their secrets • View projects and Globus Auth clients: Settings à Developers • If you have existing endpoints/client applications registered with Globus, endpoint setup will ask for a project ID
  • 16. Our setup so far Run globus-connect-server node setup to set up additional data transfer nodes Copy deployment-key.json from original DTN
  • 17. Storage Gateways define a set of access policies • Authentication for local account-holders – Which identity domain(s) are acceptable? – How are identities mapped from domain(s) to local accounts? • Policy scope – Which parts of the storage system are accessible via Globus? – Which local accounts does this policy allow (or deny)? • High Assurance settings • MFA requirements
  • 18. Authentication for local account-holders • Primary access (via a mapped collection) requires an account on the host system* • Two-part authentication configuration: 1. Pick one or more identity domains 2. Configure the method to map the authenticated identity to an account on your system * You may allow primary users to share with others who don’t have accounts on your system
  • 19. Picking identity domains • User must present identity from one of the configured domains – On access attempts, linked identities will be scanned for a match – If no identity from the required domain(s), will be asked to link one • Identity domains may include… – …any organization in Globus federated list – …your institution’s identity provider trusted by Globus – …a local OpenID Connect (OIDC) server using your PAM stack
  • 20. Mapping identities to local accounts • Default: Strip identity domain (everything after “@”) – e.g. userX@uchicago.edu maps to local account userX – Best for campus identities w/synchronized local accounts • Use --identity-mapping option on storage gateway – Specify expression in a JSON document – Execute a custom script docs.globus.org/globus-connect-server/v5.4/identity-mapping-guide/
  • 22. Creating a storage gateway • Our storage gateway will access a POSIX system – This is the only type permitted without a subscription • It will allow access to users with credentials from the uchicago.edu (subsctitute your own) domain • Reauthentication will be required every 90 minutes Cheatsheet bit.ly/gw-tut-rpi
  • 23. 3. Create a storage gateway $ globus-connect-server storage-gateway create posix > "My Storage Gateway" > --domain uchicago.edu > --authentication-timeout-mins 90 Allowed authentication domain Duration of user session when accessing collections via this storage gateway
  • 24. Our setup so far… / (user home)
  • 25. Create a mapped collection on the POSIX gateway 26
  • 26. Creating a collection • Our collection will use the default identity mapping • It will be “rooted” at the user’s home directory • Access will require authentication with an identity from the uchicago.edu (substitute your own) domain Cheatsheet bit.ly/gw-tut-rpi
  • 27. 4. Create a mapped collection $ globus-connect-server collection create > f77ff456-1f18-41d3-94a7-f3fd8858ea4d > / > "My Mapped Collection" Collections are rooted at the specified base path Specifying "/" as the base path sets the collection root to the local user’s home directory Storage gateway ID Collection base path
  • 28. Our setup so far…
  • 30.
  • 32. We are using the default identity mapping, so… • Create a local user account with the same name as your globusid.org (or other IdP) identity – e.g., for me@uchicago.edu create local account “me” – e.g., for me2@orcid.org create local account “me2” • adduser --disabled-password --gecos 'me' me • Access your mapped collection via the web app… • …and move some files, if you like Cheatsheet bit.ly/gw-tut-rpi
  • 33. Common Collection configuration options • Restrict access: local users, local groups • Restrict sharing: paths, local users, local groups • Allow guest collections à enables sharing • Enable HTTPS access • Force data channel encryption
  • 34. Local account restrictions • Note: These only apply to mapped collections • A storage gateway’s allowed identity domains and identity mapping method determine the universe of local accounts that may access the mapped collection • You can further narrow the access universe using… --user-allow --user-deny --posix-group-allow (POSIX storage gateways only) --posix-group-deny (POSIX storage gateways only)
  • 35. Path restrictions • Always use the narrowest base path possible for your storage gateway(s) and collection(s) – Storage gateway base specifies where collections may be created – Collection base specifies the base directory for the collection • POSIX storage gateway – Use --restrict_paths to specify narrower read, read/write, or none access for specific paths – You provide a JSON doc that lists paths for each permission type – Note: These are absolute paths on the host system • Collection: specify narrowest base path that satisfies the need
  • 37. Setting path restrictions • A new storage gateway will limit access to /home – NB: No change to local permissions, only visibility via Globus • We specify the path restrictions in paths.json – This file is in your admin user’s home directory • Run: storage-gateway create command with the --restrict-paths option • Create a new POSIX mapped collection Cheatsheet bit.ly/gw-tut-rpi
  • 38. 5. Create a restricted storage gateway, collection $ globus-connect-server storage-gateway create posix > "My Storage Gateway - Restricted" > --domain uchicago.edu > --authentication-timeout-mins 90 > --restrict-paths file:/home/adminN/paths.json $ globus-connect-server collection create > 3926bf02-6bc3-11e7-a9c6-22000bf2d287 > / > "My Mapped Collection – Restricted" Fully qualified filename containing rule(s) for restricting access to specific filesystem paths
  • 39. Revisit your mapped collections • Your will need to authenticate on your new (restricted access) collection, and consent • Note the access behavior differences between the two mapped collections
  • 40. Subscriptions and Endpoint Roles • Subscription(s) configured for your institution • Multiple Subscription Managers per subscription • Subscription Manager associates an endpoint with a subscription – Results in a “subscribed” endpoint • Assign additional roles for endpoint management – Administrator, Manager, Monitor
  • 41. Converting endpoint(s) to “subscribed” • As a member of an institution’s subscription, you can subscribe your own endpoint(s) • Subscription Managers can subscribe any endpoint at the institution • Check role in your institution’s subscription to determine level of access and control • But, not yet … J
  • 42. Associate the endpoint with a subscription 44
  • 43. Convert your endpoint to “subscribed” • Confirm that your Globus identity is a subscription manager à we’ll do this for you now • Run globus-connect-server endpoint set- subscription-id • Confirm: globus-connect-server endpoint show Cheatsheet bit.ly/gw-tut-rpi
  • 44. 6. Associate endpoint with a subscription $ globus-connect-server endpoint set-subscription-id DEFAULT $ globus-connect-server endpoint set-subscription-id > 39299902-6bc3-aa56-a9c6-22000bf2d287 Subscription managers can also set this via the web app Console page: app.globus.org/console (look under the Endpoints tab) Your identity may already be a subscription manager on a subscription
  • 45. Be identity-, role-, and permission-aware • Default: Only endpoint owner can configure an endpoint • Delegate administrator role to other sysadmins – Best practice: Delegate to a Globus group, not individuals • Check identity using the session command • Check resource permissions on storage gateways and collections with --include-private-policies option docs.globus.org/globus-connect-server/v5.4/reference/role/
  • 46. 7. Create a guest collection • Created by user, not endpoint administrator • Grants access to specific Globus users without a mapped local account • “Guest” users have the same (or more limited) permissions as the guest collection creator – Access logs show access by the collection creator* • Guest collection’s root is relative to the mapped collection’s base path * High Assurance collections log guest user identities to enable auditing
  • 47. Sharing restrictions • Guest collections may be created in any directory accessible by the collection, by any authorized local account • You can restrict the authorized accounts… o --sharing-user-allow --sharing-user-deny o --posix-sharing-group-allow o --posix-sharing-group-deny • …and sharing paths… o --sharing-restrict-paths (specify JSON PathRestrictions) • You can also set policies for specific user/path combinations o $ globus-connect-server sharing-policy create ...
  • 48. Create and access a guest collection 51
  • 49. Create and access a guest collection • Enable creation of guest collections • Run: globus-connect-server collection update • Access the mapped collection; create /projects • Create a guest collection on the /projects directory • Grant read access to the “Tutorial Users” group • Authenticate and browse guest collection
  • 50. 8. Enable web browser upload/download • Authorized users can upload, download files via a browser • Must have permissions to the collection – Collection configuration governs access – Web server is a different application (separate authentication)
  • 52. Enable HTTPS access • Run: globus-connect-server collection update • Access your mapped collection • Upload a file from your laptop (and download it too!) Cheatsheet bit.ly/gw-tut-rpi
  • 54. Things to do with the management console • Monitor current transfers on your endpoints – See what’s going on at the transfer request level – Much better than watching individual file transfers • Pause (and later resume) a transfer in progress – Sends a notice to the transfer owner • Set a pause rule for current and future transfers – Ideal for maintenance mode – Notifies transfer owners, – Tasks resume when endpoint is un-paused docs.globus.org/management-console-guide/
  • 55. Resources • GCSv5 Guides: docs.globus.org/globus-connect-server/ • Migration: docs.globus.org/globus-connect- server/migrating-to-v5.4/ • Globus support: support@globus.org 58