SlideShare a Scribd company logo
Automating Research Data Flows and an
Introduction to the Globus Platform
Greg Nawrocki
greg@globus.org
June 23, 2022
Globus Automation Capabilities
Timer Service
Scheduled and recurring transfers
(a.k.a. Globus cron)
Command Line Interface
Ad hoc scripting and integration
Globus Flows service
Comprehensive task (data and
compute) orchestration with human in
the loop interactions
Globus Auth: Foundational IAM service
• Brokers authentication and authorization among…
– End-users
– Identity providers: enterprise, external (federated identities)
– Services: resource servers with REST APIs
– Apps: web, mobile, desktop, command line clients
– Services acting as clients to other services
• Support high assurance service for use with
protected data (e.g. HIPAA protected data)
5
Securing Apps with Globus Auth
• Native App (with refresh tokens – extend expiration)
– Authentication as user identity
– Authentication URL / come back with a auth code – exchanged for tokens
– Clients can’t keep a secret - tokens in plain text
– Jupyter Notebook examples / Timer Service
• Auth Code Grant – Templated App
– Authentication as user identity
– Browser redirect to Globus Auth, auth code returned (no manual copy)
– Tokens stored securely
– CLI / Jupyter Hub secured with Globus Auth
• Confidential Client:
– Authentication as application
– ClientID and Secret stored securely
– Custom apps
– developers.globus.org - Client ID / Secret / Client Identity Username
6
Globus Timer Service
Use case: Data replication
• For backup: initiated by user or system back up
• Automated transfer of data from science instrument
12
Recurring transfers
with sync option
Copy /ingest
Daily @ 3:30am
The Globus Timer service
• Scheduled/recurring file transfers
• Supports all Globus transfer and sync options
• Service accessible via web app and CLI
• Example: NIH – hpc.nih.gov/storage/globus_cron.html
13
Using the Globus Timer service
14
$ globus–timer session {login, logout, whoami}
$ globus–timer job transfer 
--name example–job 
--label "Timer Transfer Job" 
--interval 28800 
--start '2020–01–01T12:34:56' 
--source–endpoint ddb59aef–6d04–11e5–ba46–22000b92c6ec 
--dest–endpoint ddb59af0–6d04–11e5–ba46–22000b92c6ec 
--item ~/file1.txt ~/new_file1.txt false 
--item ~/file2.txt ~/new_file2.txt false
Timer options
in the Globus
web app
Globus Command Line
Interface (CLI)
Globus Command Line Interface
Open source, uses
the Python SDK
Because of this
correspondence the CLI is
an excellent tool for getting
the gist of how he SDK
functions.
Very scriptable!
Globus CLI
• It’s a stand alone application distributed by Globus
– https://docs.globus.org/cli/
– https://github.com/globus/globus-cli
• Easy install and updates
• Command “globus login” gets access tokens
• All interactions with the service use the tokens
– The CLI is acting as you (your identity)
• Command “globus logout” deletes those
CLI Basics – “globus” is the executable
$ globus endpoint search 'Globus Tutorial'
$ globus task list
$ globus get-identities greg@globus.org --verbose
• Getting help / list of commands
– globus list-commands
– globus –help
– https://docs.globus.org/cli/examples/
• UUIDs for endpoint, task, user identity, groups…
• Can query to discover the UUIDs
– Use search / list / get options
Use case: Sharing out data
Researcher initiates
transfer request; or
requested automatically
by script, science
gateway
Instrument
Globus
controls
access to
shared files on
existing
storage; no
need to move
files to cloud
storage!
Researcher
selects files to
share, selects
user or group,
and sets access
permissions
Collaborator logs in to
Globus and accesses
shared files; no local
account required;
download via Globus
Personal Computer
Transfer
Share
Compute Facility
Globus transfers files
reliably, securely
Step 1: Transfer
• Using a Batch Transfer
– Transfer tasks have one source/destination, but can have any
number of files
– Provide input source-dest pairs via local file
– File may have embedded comments
$ export ep1=ddb59aef-6d04-11e5-ba46-22000b92c6ec
$ export ep2=af7bda53-6d04-11e5-ba46-22000b92c6ec
$ globus transfer $ep1:/share/godata/ $ep2:/~/automation-example-
data-share/outbound/Vas --label "Files to share" --batch
/Users/gregnawrocki/Greg/Globus/Demos/ornl20220623/files.txt
# this is the contents of files.txt:
# a list of source paths followed by destination paths
file1.txt file1.txt
# file2.txt file2.txt # inline-comments are also allowed
file3.txt file3.txt
Step 2: Share - Set permissions
• Set and manage permissions on guest collection
• Requires access manager role
$ globus endpoint search 'automation-example-data-share’
$ export share=bf4445c4-ecfd-11ec-aed7-6f7c2b57b05c
$ globus endpoint permission create --permissions r --identity
vas@uchicago.edu $share:/Vas/
Useful submission commands
• Safe resubmissions
– Applies to all tasks (transfer and delete)
– Get a task UUID, use that in submission
– $ globus task generate-submission-id
– --submission-id option in transfer
– Useful for lazy branching or when dealing with unreliable
networks
• Task wait
– useful for scripting conditionals on transfer task status
Parsing CLI output
$ globus endpoint search --filter-scope my-endpoints
$ globus endpoint search --filter-scope my-endpoints --format json
$ globus endpoint search --filter-scope my-endpoints --jmespath
'DATA[].[id, display_name]'
• Default output is text; for JSON output use --format
json
• Extract specific attributes using --jmespath
<expression>
Managing notifications
• Turn off emails sent for tasks
• Useful when an application manages tasks for a user
• Disable notifications with the --notify option
--notify off (all notifications)
--notify succeeded|failed|inactive (select notifications)
Globus Flows Service
Managed automation of tasks
• Flows: A platform service for defining, applying, and
sharing distributed research automation flows
• Flows comprise Actions
• Action Providers: Called by Flows to perform tasks
• Triggers*: Start flows based on events
* In development
Automation services ecosystem
GET /provider_url/
POST /provider_url/run
GET /provider_url/action_id/status
GET /provider_url/action_id/cancel
GET /provider_url/action_id/status
Create Action
Providers
Define and
deploy flows
{ “StartAt”: ”ToProject”,
”States” : {
”ToProject” : { … },
”SetPermission” : { …},
“ProcessData” : { … } … }}
Run flows
Automation with Globus Flows
• Built on AWS Step Functions
– Simple JSON-based state machine
language
– Conditions, loops, fault tolerance, etc.
– Propagates state through the flow
• Standardized API for integrating
custom event and action services
– Actions: synchronous or asynchronous
– Custom Web forms prompt for user input
• Actions secured with Globus Auth
Globus-provided flows
31
32
Run flows: Guided input
Label
Notify user
Timeout
Dynamic forms generated
from input schema
33
Managing runs at scale
Developing
Globus Flows
jupyter.demo.globus.org
34
Extending the ecosystem: Action providers
35
• Action Provider is a
service endpoint
– Run
– Status
– Cancel
– Release
– Resume
• Action Provider Toolkit
action-provider-
tools.readthedocs.io/en/latest
Search
Transfer
Notification
ACLs Identifier
Delete
Ingest
User
Form
Describe Xtract
funcX Web
Form
Custom built
Globus Provided
The Globus
Platform
APIs and the SDK
36
37
Custom portals? Science Gateways? Unique workflows? Our open
REST APIs and Python SDK empower you to create an integrated
ecosystem of research data services and applications.
Data centric applications leveraging Globus
38
App Access to Collections
• Globus Transfer – Authentication with access tokens
– Individual: Globus login to get tokens
– Application: Apps are people too!
o developers.globus.org - Client ID / Secret / Client Identity Username
• Collection access
– GCSv4 Mapped Collections – user consent
o Activating endpoint means binding a credential to an endpoint for login
– GCSv5 Mapped Collections (no user certificates, OAUTH tokens and consents)
o https://docs.globus.org/globus-connect-server/v5.4/use-client-credentials/
o Request the data_access scope (per collection) to be able to access the collection.
o The storage gateway must permit identities from the 'clients.auth.globus.org' identity domain
o Identity Mapping Policy that maps the ‘UUID@clients.auth.globus.org' identity to a valid local user
– Guest Collections
o Guest Collections auto-activate - need to do this before API calls to endpoints
o Use Guest Collections whenever possible
– Remember to set your ACLs (WebApp)
Automation
https://developers.globus.org
Globus APIs
• Auth
• Groups
• Transfer
• Search
• Timer
• Flows
• GCS Manager
• Globus Web App consumes public
Transfer API
• Resource named by URL (standard
REST approach)
• Globus APIs use JSON for documents
docs.globus.org/api/transfer
Globus Python SDK
• Python client library for the Globus REST APIs
• Largely direct mapping to REST API
• globus_sdk.TransferClient class handles
connection management, security, framing,
marshaling
globus-sdk-python.readthedocs.io/en/stable/
globus.github.io/globus-sdk-python
42
TransferClient higher-level calls
• One method for each API resource and HTTP verb
• Largely direct mapping to REST API
endpoint_search(filter_fulltext=None,
filter_scope=None,
num_results=25,
**params)
44
Synchronous Tasks
• Endpoint search (with scopes)
• List directory contents (ls)
• Make directory (mkdir)
• Rename
• Note:
– Path encoding & UTF gotchas
– Don’t forget to auto-activate first
46
Asynchronous Tasks
• Transfer
– Sync level option
• Delete
• Get submission_id, followed by submit
– Once and only once submission
• Use task id to “follow up”
47
The Globus API / SDK with a Jupyter Notebook in a
Jupyter Hub – Auth Code Grant
login
REST APIs
{ “tokens”:…
{“tokens”:…
REST APIs
REST APIs
Bearer a45cd…
Walkthrough API with our Jupyter Hub
• https://jupyter.demo.globus.org
– Sign in with Globus
– Verify the consents
– Start My Server (this will take about a minute)
– Open folder: globus-jupyter-notebooks
– Run Platform_Introduction_JupyterHub_Auth.ipynb
• If you mess it up and want to “go back to the beginning”
– Just stop and restart the server
• If you want to use the notebook outside of our hub
– https://github.com/globus/globus-jupyter-notebooks
– Authentication is a manual cut and paste of exchanging the
authorization code for an access token – Native App
49
Support Resources
• Globus Documentation: docs.globus.org
• Globus Timer: globus.org/blog/scheduled-and-
recurring-transfers-now-available-globus-web-app
• Globus CLI: docs.globus.org/cli/
• Globus Automation Services and Flows:
docs.globus.org/globus-automation-services/
• YouTube Channel: youtube.com/user/GlobusOnline
Developer References
• Globus API / SDK Documentation
– Transfer API : docs.globus.org/api/transfer/
– SDK: globus-sdk-python.readthedocs.io/en/stable/
• Globus GitHub: github.com/globus/
– Jupyter Notebooks
o Stand alone notebooks and hub integrations that walk through much of the
functionality of our SDK
o https://github.com/globus/globus-jupyter-notebooks
– Automation Examples
o Shell scripted CLI and Python module examples of common research data
management use cases
o https://github.com/globus/automation-examples

More Related Content

Similar to Automating Research Data Flows and an Introduction to the Globus Platform

Introduction to the Globus Platform (GlobusWorld Tour - UMich)
Introduction to the Globus Platform (GlobusWorld Tour - UMich)Introduction to the Globus Platform (GlobusWorld Tour - UMich)
Introduction to the Globus Platform (GlobusWorld Tour - UMich)
Globus
 
Introduction to the Globus Platform (APS Workshop)
Introduction to the Globus Platform (APS Workshop)Introduction to the Globus Platform (APS Workshop)
Introduction to the Globus Platform (APS Workshop)
Globus
 
Automating Research Data Flows with Globus (CHPC 2019 - South Africa)
Automating Research Data Flows with Globus (CHPC 2019 - South Africa)Automating Research Data Flows with Globus (CHPC 2019 - South Africa)
Automating Research Data Flows with Globus (CHPC 2019 - South Africa)
Globus
 
Tutorial: Leveraging Globus in your Research Applications
Tutorial: Leveraging Globus in your Research ApplicationsTutorial: Leveraging Globus in your Research Applications
Tutorial: Leveraging Globus in your Research Applications
Globus
 
Automating Research Data Workflows (GlobusWorld Tour - Columbia University)
Automating Research Data Workflows (GlobusWorld Tour - Columbia University)Automating Research Data Workflows (GlobusWorld Tour - Columbia University)
Automating Research Data Workflows (GlobusWorld Tour - Columbia University)
Globus
 
Automating Research Data Workflows (GlobusWorld Tour - STFC)
Automating Research Data Workflows (GlobusWorld Tour - STFC)Automating Research Data Workflows (GlobusWorld Tour - STFC)
Automating Research Data Workflows (GlobusWorld Tour - STFC)
Globus
 
Automating Data Flows with the Globus CLI (GlobusWorld Tour - UMich)
Automating Data Flows with the Globus CLI (GlobusWorld Tour - UMich)Automating Data Flows with the Globus CLI (GlobusWorld Tour - UMich)
Automating Data Flows with the Globus CLI (GlobusWorld Tour - UMich)
Globus
 
Globus Automation
Globus AutomationGlobus Automation
Globus Automation
Globus
 
Automating Research Data Flows with the Globus Command Line Interface (CLI)
Automating Research Data Flows with the Globus Command Line Interface (CLI)Automating Research Data Flows with the Globus Command Line Interface (CLI)
Automating Research Data Flows with the Globus Command Line Interface (CLI)
Globus
 
Tutorial: Automating Research Data Workflows
Tutorial: Automating Research Data WorkflowsTutorial: Automating Research Data Workflows
Tutorial: Automating Research Data Workflows
Globus
 
Automating Research Data Workflows (GlobusWorld Tour - UCSD)
Automating Research Data Workflows (GlobusWorld Tour - UCSD)Automating Research Data Workflows (GlobusWorld Tour - UCSD)
Automating Research Data Workflows (GlobusWorld Tour - UCSD)
Globus
 
Working with Globus Platform Services and Portals
Working with Globus Platform Services and PortalsWorking with Globus Platform Services and Portals
Working with Globus Platform Services and Portals
Globus
 
Leveraging the Globus Platform (GlobusWorld Tour - Columbia University)
Leveraging the Globus Platform (GlobusWorld Tour - Columbia University)Leveraging the Globus Platform (GlobusWorld Tour - Columbia University)
Leveraging the Globus Platform (GlobusWorld Tour - Columbia University)
Globus
 
Jupyter + Globus: The Foundation for Interactive Data Science
Jupyter + Globus: The Foundation for Interactive Data ScienceJupyter + Globus: The Foundation for Interactive Data Science
Jupyter + Globus: The Foundation for Interactive Data Science
Globus
 
Globus Platform Overview
Globus Platform OverviewGlobus Platform Overview
Globus Platform Overview
Globus
 
Introduction to Research Automation with Globus
Introduction to Research Automation with GlobusIntroduction to Research Automation with Globus
Introduction to Research Automation with Globus
Globus
 
Data Publication and Discovery with Globus
Data Publication and Discovery with GlobusData Publication and Discovery with Globus
Data Publication and Discovery with Globus
Globus
 
Instrument Data Orchestration with Globus Search and Flows
Instrument Data Orchestration with Globus Search and FlowsInstrument Data Orchestration with Globus Search and Flows
Instrument Data Orchestration with Globus Search and Flows
Globus
 
Tutorial: Introduction to Globus for System Administrators
Tutorial: Introduction to Globus for System AdministratorsTutorial: Introduction to Globus for System Administrators
Tutorial: Introduction to Globus for System Administrators
Globus
 
Introduction to Globus for System Administrators
Introduction to Globus for System AdministratorsIntroduction to Globus for System Administrators
Introduction to Globus for System Administrators
Globus
 

Similar to Automating Research Data Flows and an Introduction to the Globus Platform (20)

Introduction to the Globus Platform (GlobusWorld Tour - UMich)
Introduction to the Globus Platform (GlobusWorld Tour - UMich)Introduction to the Globus Platform (GlobusWorld Tour - UMich)
Introduction to the Globus Platform (GlobusWorld Tour - UMich)
 
Introduction to the Globus Platform (APS Workshop)
Introduction to the Globus Platform (APS Workshop)Introduction to the Globus Platform (APS Workshop)
Introduction to the Globus Platform (APS Workshop)
 
Automating Research Data Flows with Globus (CHPC 2019 - South Africa)
Automating Research Data Flows with Globus (CHPC 2019 - South Africa)Automating Research Data Flows with Globus (CHPC 2019 - South Africa)
Automating Research Data Flows with Globus (CHPC 2019 - South Africa)
 
Tutorial: Leveraging Globus in your Research Applications
Tutorial: Leveraging Globus in your Research ApplicationsTutorial: Leveraging Globus in your Research Applications
Tutorial: Leveraging Globus in your Research Applications
 
Automating Research Data Workflows (GlobusWorld Tour - Columbia University)
Automating Research Data Workflows (GlobusWorld Tour - Columbia University)Automating Research Data Workflows (GlobusWorld Tour - Columbia University)
Automating Research Data Workflows (GlobusWorld Tour - Columbia University)
 
Automating Research Data Workflows (GlobusWorld Tour - STFC)
Automating Research Data Workflows (GlobusWorld Tour - STFC)Automating Research Data Workflows (GlobusWorld Tour - STFC)
Automating Research Data Workflows (GlobusWorld Tour - STFC)
 
Automating Data Flows with the Globus CLI (GlobusWorld Tour - UMich)
Automating Data Flows with the Globus CLI (GlobusWorld Tour - UMich)Automating Data Flows with the Globus CLI (GlobusWorld Tour - UMich)
Automating Data Flows with the Globus CLI (GlobusWorld Tour - UMich)
 
Globus Automation
Globus AutomationGlobus Automation
Globus Automation
 
Automating Research Data Flows with the Globus Command Line Interface (CLI)
Automating Research Data Flows with the Globus Command Line Interface (CLI)Automating Research Data Flows with the Globus Command Line Interface (CLI)
Automating Research Data Flows with the Globus Command Line Interface (CLI)
 
Tutorial: Automating Research Data Workflows
Tutorial: Automating Research Data WorkflowsTutorial: Automating Research Data Workflows
Tutorial: Automating Research Data Workflows
 
Automating Research Data Workflows (GlobusWorld Tour - UCSD)
Automating Research Data Workflows (GlobusWorld Tour - UCSD)Automating Research Data Workflows (GlobusWorld Tour - UCSD)
Automating Research Data Workflows (GlobusWorld Tour - UCSD)
 
Working with Globus Platform Services and Portals
Working with Globus Platform Services and PortalsWorking with Globus Platform Services and Portals
Working with Globus Platform Services and Portals
 
Leveraging the Globus Platform (GlobusWorld Tour - Columbia University)
Leveraging the Globus Platform (GlobusWorld Tour - Columbia University)Leveraging the Globus Platform (GlobusWorld Tour - Columbia University)
Leveraging the Globus Platform (GlobusWorld Tour - Columbia University)
 
Jupyter + Globus: The Foundation for Interactive Data Science
Jupyter + Globus: The Foundation for Interactive Data ScienceJupyter + Globus: The Foundation for Interactive Data Science
Jupyter + Globus: The Foundation for Interactive Data Science
 
Globus Platform Overview
Globus Platform OverviewGlobus Platform Overview
Globus Platform Overview
 
Introduction to Research Automation with Globus
Introduction to Research Automation with GlobusIntroduction to Research Automation with Globus
Introduction to Research Automation with Globus
 
Data Publication and Discovery with Globus
Data Publication and Discovery with GlobusData Publication and Discovery with Globus
Data Publication and Discovery with Globus
 
Instrument Data Orchestration with Globus Search and Flows
Instrument Data Orchestration with Globus Search and FlowsInstrument Data Orchestration with Globus Search and Flows
Instrument Data Orchestration with Globus Search and Flows
 
Tutorial: Introduction to Globus for System Administrators
Tutorial: Introduction to Globus for System AdministratorsTutorial: Introduction to Globus for System Administrators
Tutorial: Introduction to Globus for System Administrators
 
Introduction to Globus for System Administrators
Introduction to Globus for System AdministratorsIntroduction to Globus for System Administrators
Introduction to Globus for System Administrators
 

More from Globus

Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus
 
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
Globus
 
Globus Compute Introduction - GlobusWorld 2024
Globus Compute Introduction - GlobusWorld 2024Globus Compute Introduction - GlobusWorld 2024
Globus Compute Introduction - GlobusWorld 2024
Globus
 
Globus Connect Server Deep Dive - GlobusWorld 2024
Globus Connect Server Deep Dive - GlobusWorld 2024Globus Connect Server Deep Dive - GlobusWorld 2024
Globus Connect Server Deep Dive - GlobusWorld 2024
Globus
 
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...
Globus
 
Providing Globus Services to Users of JASMIN for Environmental Data Analysis
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisProviding Globus Services to Users of JASMIN for Environmental Data Analysis
Providing Globus Services to Users of JASMIN for Environmental Data Analysis
Globus
 
First Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User EndpointsFirst Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User Endpoints
Globus
 
Enhancing Research Orchestration Capabilities at ORNL.pdf
Enhancing Research Orchestration Capabilities at ORNL.pdfEnhancing Research Orchestration Capabilities at ORNL.pdf
Enhancing Research Orchestration Capabilities at ORNL.pdf
Globus
 
Understanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSageUnderstanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSage
Globus
 
How to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good PracticesHow to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good Practices
Globus
 
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
Globus
 
Developing Distributed High-performance Computing Capabilities of an Open Sci...
Developing Distributed High-performance Computing Capabilities of an Open Sci...Developing Distributed High-performance Computing Capabilities of an Open Sci...
Developing Distributed High-performance Computing Capabilities of an Open Sci...
Globus
 
The Department of Energy's Integrated Research Infrastructure (IRI)
The Department of Energy's Integrated Research Infrastructure (IRI)The Department of Energy's Integrated Research Infrastructure (IRI)
The Department of Energy's Integrated Research Infrastructure (IRI)
Globus
 
GlobusWorld 2024 Opening Keynote session
GlobusWorld 2024 Opening Keynote sessionGlobusWorld 2024 Opening Keynote session
GlobusWorld 2024 Opening Keynote session
Globus
 
Enhancing Performance with Globus and the Science DMZ
Enhancing Performance with Globus and the Science DMZEnhancing Performance with Globus and the Science DMZ
Enhancing Performance with Globus and the Science DMZ
Globus
 
Extending Globus into a Site-wide Automated Data Infrastructure.pdf
Extending Globus into a Site-wide Automated Data Infrastructure.pdfExtending Globus into a Site-wide Automated Data Infrastructure.pdf
Extending Globus into a Site-wide Automated Data Infrastructure.pdf
Globus
 
Globus at the United States Geological Survey
Globus at the United States Geological SurveyGlobus at the United States Geological Survey
Globus at the United States Geological Survey
Globus
 
Providing Globus Services to Users of JASMIN for Environmental Data Analysis
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisProviding Globus Services to Users of JASMIN for Environmental Data Analysis
Providing Globus Services to Users of JASMIN for Environmental Data Analysis
Globus
 
Globus Compute with Integrated Research Infrastructure (IRI) workflows
Globus Compute with Integrated Research Infrastructure (IRI) workflowsGlobus Compute with Integrated Research Infrastructure (IRI) workflows
Globus Compute with Integrated Research Infrastructure (IRI) workflows
Globus
 
Reactive Documents and Computational Pipelines - Bridging the Gap
Reactive Documents and Computational Pipelines - Bridging the GapReactive Documents and Computational Pipelines - Bridging the Gap
Reactive Documents and Computational Pipelines - Bridging the Gap
Globus
 

More from Globus (20)

Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus Compute wth IRI Workflows - GlobusWorld 2024
 
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
 
Globus Compute Introduction - GlobusWorld 2024
Globus Compute Introduction - GlobusWorld 2024Globus Compute Introduction - GlobusWorld 2024
Globus Compute Introduction - GlobusWorld 2024
 
Globus Connect Server Deep Dive - GlobusWorld 2024
Globus Connect Server Deep Dive - GlobusWorld 2024Globus Connect Server Deep Dive - GlobusWorld 2024
Globus Connect Server Deep Dive - GlobusWorld 2024
 
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...
 
Providing Globus Services to Users of JASMIN for Environmental Data Analysis
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisProviding Globus Services to Users of JASMIN for Environmental Data Analysis
Providing Globus Services to Users of JASMIN for Environmental Data Analysis
 
First Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User EndpointsFirst Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User Endpoints
 
Enhancing Research Orchestration Capabilities at ORNL.pdf
Enhancing Research Orchestration Capabilities at ORNL.pdfEnhancing Research Orchestration Capabilities at ORNL.pdf
Enhancing Research Orchestration Capabilities at ORNL.pdf
 
Understanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSageUnderstanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSage
 
How to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good PracticesHow to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good Practices
 
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
 
Developing Distributed High-performance Computing Capabilities of an Open Sci...
Developing Distributed High-performance Computing Capabilities of an Open Sci...Developing Distributed High-performance Computing Capabilities of an Open Sci...
Developing Distributed High-performance Computing Capabilities of an Open Sci...
 
The Department of Energy's Integrated Research Infrastructure (IRI)
The Department of Energy's Integrated Research Infrastructure (IRI)The Department of Energy's Integrated Research Infrastructure (IRI)
The Department of Energy's Integrated Research Infrastructure (IRI)
 
GlobusWorld 2024 Opening Keynote session
GlobusWorld 2024 Opening Keynote sessionGlobusWorld 2024 Opening Keynote session
GlobusWorld 2024 Opening Keynote session
 
Enhancing Performance with Globus and the Science DMZ
Enhancing Performance with Globus and the Science DMZEnhancing Performance with Globus and the Science DMZ
Enhancing Performance with Globus and the Science DMZ
 
Extending Globus into a Site-wide Automated Data Infrastructure.pdf
Extending Globus into a Site-wide Automated Data Infrastructure.pdfExtending Globus into a Site-wide Automated Data Infrastructure.pdf
Extending Globus into a Site-wide Automated Data Infrastructure.pdf
 
Globus at the United States Geological Survey
Globus at the United States Geological SurveyGlobus at the United States Geological Survey
Globus at the United States Geological Survey
 
Providing Globus Services to Users of JASMIN for Environmental Data Analysis
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisProviding Globus Services to Users of JASMIN for Environmental Data Analysis
Providing Globus Services to Users of JASMIN for Environmental Data Analysis
 
Globus Compute with Integrated Research Infrastructure (IRI) workflows
Globus Compute with Integrated Research Infrastructure (IRI) workflowsGlobus Compute with Integrated Research Infrastructure (IRI) workflows
Globus Compute with Integrated Research Infrastructure (IRI) workflows
 
Reactive Documents and Computational Pipelines - Bridging the Gap
Reactive Documents and Computational Pipelines - Bridging the GapReactive Documents and Computational Pipelines - Bridging the Gap
Reactive Documents and Computational Pipelines - Bridging the Gap
 

Recently uploaded

Automated software refactoring with OpenRewrite and Generative AI.pptx.pdf
Automated software refactoring with OpenRewrite and Generative AI.pptx.pdfAutomated software refactoring with OpenRewrite and Generative AI.pptx.pdf
Automated software refactoring with OpenRewrite and Generative AI.pptx.pdf
timtebeek1
 
Orion Context Broker introduction 20240604
Orion Context Broker introduction 20240604Orion Context Broker introduction 20240604
Orion Context Broker introduction 20240604
Fermin Galan
 
GOING AOT WITH GRAALVM FOR SPRING BOOT (SPRING IO)
GOING AOT WITH GRAALVM FOR  SPRING BOOT (SPRING IO)GOING AOT WITH GRAALVM FOR  SPRING BOOT (SPRING IO)
GOING AOT WITH GRAALVM FOR SPRING BOOT (SPRING IO)
Alina Yurenko
 
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
AI Pilot Review: The World’s First Virtual Assistant Marketing SuiteAI Pilot Review: The World’s First Virtual Assistant Marketing Suite
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
Google
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
Safe Software
 
Fundamentals of Programming and Language Processors
Fundamentals of Programming and Language ProcessorsFundamentals of Programming and Language Processors
Fundamentals of Programming and Language Processors
Rakesh Kumar R
 
socradar-q1-2024-aviation-industry-report.pdf
socradar-q1-2024-aviation-industry-report.pdfsocradar-q1-2024-aviation-industry-report.pdf
socradar-q1-2024-aviation-industry-report.pdf
SOCRadar
 
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
mz5nrf0n
 
Utilocate provides Smarter, Better, Faster, Safer Locate Ticket Management
Utilocate provides Smarter, Better, Faster, Safer Locate Ticket ManagementUtilocate provides Smarter, Better, Faster, Safer Locate Ticket Management
Utilocate provides Smarter, Better, Faster, Safer Locate Ticket Management
Utilocate
 
A Sighting of filterA in Typelevel Rite of Passage
A Sighting of filterA in Typelevel Rite of PassageA Sighting of filterA in Typelevel Rite of Passage
A Sighting of filterA in Typelevel Rite of Passage
Philip Schwarz
 
Transform Your Communication with Cloud-Based IVR Solutions
Transform Your Communication with Cloud-Based IVR SolutionsTransform Your Communication with Cloud-Based IVR Solutions
Transform Your Communication with Cloud-Based IVR Solutions
TheSMSPoint
 
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona InfotechMobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
Drona Infotech
 
Empowering Growth with Best Software Development Company in Noida - Deuglo
Empowering Growth with Best Software  Development Company in Noida - DeugloEmpowering Growth with Best Software  Development Company in Noida - Deuglo
Empowering Growth with Best Software Development Company in Noida - Deuglo
Deuglo Infosystem Pvt Ltd
 
Mobile app Development Services | Drona Infotech
Mobile app Development Services  | Drona InfotechMobile app Development Services  | Drona Infotech
Mobile app Development Services | Drona Infotech
Drona Infotech
 
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOMLORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
lorraineandreiamcidl
 
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...
kalichargn70th171
 
SWEBOK and Education at FUSE Okinawa 2024
SWEBOK and Education at FUSE Okinawa 2024SWEBOK and Education at FUSE Okinawa 2024
SWEBOK and Education at FUSE Okinawa 2024
Hironori Washizaki
 
Quarkus Hidden and Forbidden Extensions
Quarkus Hidden and Forbidden ExtensionsQuarkus Hidden and Forbidden Extensions
Quarkus Hidden and Forbidden Extensions
Max Andersen
 
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Atelier - Innover avec l’IA Générative et les graphes de connaissancesAtelier - Innover avec l’IA Générative et les graphes de connaissances
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Neo4j
 
What is Augmented Reality Image Tracking
What is Augmented Reality Image TrackingWhat is Augmented Reality Image Tracking
What is Augmented Reality Image Tracking
pavan998932
 

Recently uploaded (20)

Automated software refactoring with OpenRewrite and Generative AI.pptx.pdf
Automated software refactoring with OpenRewrite and Generative AI.pptx.pdfAutomated software refactoring with OpenRewrite and Generative AI.pptx.pdf
Automated software refactoring with OpenRewrite and Generative AI.pptx.pdf
 
Orion Context Broker introduction 20240604
Orion Context Broker introduction 20240604Orion Context Broker introduction 20240604
Orion Context Broker introduction 20240604
 
GOING AOT WITH GRAALVM FOR SPRING BOOT (SPRING IO)
GOING AOT WITH GRAALVM FOR  SPRING BOOT (SPRING IO)GOING AOT WITH GRAALVM FOR  SPRING BOOT (SPRING IO)
GOING AOT WITH GRAALVM FOR SPRING BOOT (SPRING IO)
 
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
AI Pilot Review: The World’s First Virtual Assistant Marketing SuiteAI Pilot Review: The World’s First Virtual Assistant Marketing Suite
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
 
Fundamentals of Programming and Language Processors
Fundamentals of Programming and Language ProcessorsFundamentals of Programming and Language Processors
Fundamentals of Programming and Language Processors
 
socradar-q1-2024-aviation-industry-report.pdf
socradar-q1-2024-aviation-industry-report.pdfsocradar-q1-2024-aviation-industry-report.pdf
socradar-q1-2024-aviation-industry-report.pdf
 
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
 
Utilocate provides Smarter, Better, Faster, Safer Locate Ticket Management
Utilocate provides Smarter, Better, Faster, Safer Locate Ticket ManagementUtilocate provides Smarter, Better, Faster, Safer Locate Ticket Management
Utilocate provides Smarter, Better, Faster, Safer Locate Ticket Management
 
A Sighting of filterA in Typelevel Rite of Passage
A Sighting of filterA in Typelevel Rite of PassageA Sighting of filterA in Typelevel Rite of Passage
A Sighting of filterA in Typelevel Rite of Passage
 
Transform Your Communication with Cloud-Based IVR Solutions
Transform Your Communication with Cloud-Based IVR SolutionsTransform Your Communication with Cloud-Based IVR Solutions
Transform Your Communication with Cloud-Based IVR Solutions
 
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona InfotechMobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
 
Empowering Growth with Best Software Development Company in Noida - Deuglo
Empowering Growth with Best Software  Development Company in Noida - DeugloEmpowering Growth with Best Software  Development Company in Noida - Deuglo
Empowering Growth with Best Software Development Company in Noida - Deuglo
 
Mobile app Development Services | Drona Infotech
Mobile app Development Services  | Drona InfotechMobile app Development Services  | Drona Infotech
Mobile app Development Services | Drona Infotech
 
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOMLORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
 
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...
 
SWEBOK and Education at FUSE Okinawa 2024
SWEBOK and Education at FUSE Okinawa 2024SWEBOK and Education at FUSE Okinawa 2024
SWEBOK and Education at FUSE Okinawa 2024
 
Quarkus Hidden and Forbidden Extensions
Quarkus Hidden and Forbidden ExtensionsQuarkus Hidden and Forbidden Extensions
Quarkus Hidden and Forbidden Extensions
 
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Atelier - Innover avec l’IA Générative et les graphes de connaissancesAtelier - Innover avec l’IA Générative et les graphes de connaissances
Atelier - Innover avec l’IA Générative et les graphes de connaissances
 
What is Augmented Reality Image Tracking
What is Augmented Reality Image TrackingWhat is Augmented Reality Image Tracking
What is Augmented Reality Image Tracking
 

Automating Research Data Flows and an Introduction to the Globus Platform

  • 1. Automating Research Data Flows and an Introduction to the Globus Platform Greg Nawrocki greg@globus.org June 23, 2022
  • 2. Globus Automation Capabilities Timer Service Scheduled and recurring transfers (a.k.a. Globus cron) Command Line Interface Ad hoc scripting and integration Globus Flows service Comprehensive task (data and compute) orchestration with human in the loop interactions
  • 3. Globus Auth: Foundational IAM service • Brokers authentication and authorization among… – End-users – Identity providers: enterprise, external (federated identities) – Services: resource servers with REST APIs – Apps: web, mobile, desktop, command line clients – Services acting as clients to other services • Support high assurance service for use with protected data (e.g. HIPAA protected data) 5
  • 4. Securing Apps with Globus Auth • Native App (with refresh tokens – extend expiration) – Authentication as user identity – Authentication URL / come back with a auth code – exchanged for tokens – Clients can’t keep a secret - tokens in plain text – Jupyter Notebook examples / Timer Service • Auth Code Grant – Templated App – Authentication as user identity – Browser redirect to Globus Auth, auth code returned (no manual copy) – Tokens stored securely – CLI / Jupyter Hub secured with Globus Auth • Confidential Client: – Authentication as application – ClientID and Secret stored securely – Custom apps – developers.globus.org - Client ID / Secret / Client Identity Username 6
  • 6. Use case: Data replication • For backup: initiated by user or system back up • Automated transfer of data from science instrument 12 Recurring transfers with sync option Copy /ingest Daily @ 3:30am
  • 7. The Globus Timer service • Scheduled/recurring file transfers • Supports all Globus transfer and sync options • Service accessible via web app and CLI • Example: NIH – hpc.nih.gov/storage/globus_cron.html 13
  • 8. Using the Globus Timer service 14 $ globus–timer session {login, logout, whoami} $ globus–timer job transfer --name example–job --label "Timer Transfer Job" --interval 28800 --start '2020–01–01T12:34:56' --source–endpoint ddb59aef–6d04–11e5–ba46–22000b92c6ec --dest–endpoint ddb59af0–6d04–11e5–ba46–22000b92c6ec --item ~/file1.txt ~/new_file1.txt false --item ~/file2.txt ~/new_file2.txt false
  • 9. Timer options in the Globus web app
  • 11. Globus Command Line Interface Open source, uses the Python SDK Because of this correspondence the CLI is an excellent tool for getting the gist of how he SDK functions. Very scriptable!
  • 12. Globus CLI • It’s a stand alone application distributed by Globus – https://docs.globus.org/cli/ – https://github.com/globus/globus-cli • Easy install and updates • Command “globus login” gets access tokens • All interactions with the service use the tokens – The CLI is acting as you (your identity) • Command “globus logout” deletes those
  • 13. CLI Basics – “globus” is the executable $ globus endpoint search 'Globus Tutorial' $ globus task list $ globus get-identities greg@globus.org --verbose • Getting help / list of commands – globus list-commands – globus –help – https://docs.globus.org/cli/examples/ • UUIDs for endpoint, task, user identity, groups… • Can query to discover the UUIDs – Use search / list / get options
  • 14. Use case: Sharing out data Researcher initiates transfer request; or requested automatically by script, science gateway Instrument Globus controls access to shared files on existing storage; no need to move files to cloud storage! Researcher selects files to share, selects user or group, and sets access permissions Collaborator logs in to Globus and accesses shared files; no local account required; download via Globus Personal Computer Transfer Share Compute Facility Globus transfers files reliably, securely
  • 15. Step 1: Transfer • Using a Batch Transfer – Transfer tasks have one source/destination, but can have any number of files – Provide input source-dest pairs via local file – File may have embedded comments $ export ep1=ddb59aef-6d04-11e5-ba46-22000b92c6ec $ export ep2=af7bda53-6d04-11e5-ba46-22000b92c6ec $ globus transfer $ep1:/share/godata/ $ep2:/~/automation-example- data-share/outbound/Vas --label "Files to share" --batch /Users/gregnawrocki/Greg/Globus/Demos/ornl20220623/files.txt # this is the contents of files.txt: # a list of source paths followed by destination paths file1.txt file1.txt # file2.txt file2.txt # inline-comments are also allowed file3.txt file3.txt
  • 16. Step 2: Share - Set permissions • Set and manage permissions on guest collection • Requires access manager role $ globus endpoint search 'automation-example-data-share’ $ export share=bf4445c4-ecfd-11ec-aed7-6f7c2b57b05c $ globus endpoint permission create --permissions r --identity vas@uchicago.edu $share:/Vas/
  • 17. Useful submission commands • Safe resubmissions – Applies to all tasks (transfer and delete) – Get a task UUID, use that in submission – $ globus task generate-submission-id – --submission-id option in transfer – Useful for lazy branching or when dealing with unreliable networks • Task wait – useful for scripting conditionals on transfer task status
  • 18. Parsing CLI output $ globus endpoint search --filter-scope my-endpoints $ globus endpoint search --filter-scope my-endpoints --format json $ globus endpoint search --filter-scope my-endpoints --jmespath 'DATA[].[id, display_name]' • Default output is text; for JSON output use --format json • Extract specific attributes using --jmespath <expression>
  • 19. Managing notifications • Turn off emails sent for tasks • Useful when an application manages tasks for a user • Disable notifications with the --notify option --notify off (all notifications) --notify succeeded|failed|inactive (select notifications)
  • 21. Managed automation of tasks • Flows: A platform service for defining, applying, and sharing distributed research automation flows • Flows comprise Actions • Action Providers: Called by Flows to perform tasks • Triggers*: Start flows based on events * In development
  • 22. Automation services ecosystem GET /provider_url/ POST /provider_url/run GET /provider_url/action_id/status GET /provider_url/action_id/cancel GET /provider_url/action_id/status Create Action Providers Define and deploy flows { “StartAt”: ”ToProject”, ”States” : { ”ToProject” : { … }, ”SetPermission” : { …}, “ProcessData” : { … } … }} Run flows
  • 23. Automation with Globus Flows • Built on AWS Step Functions – Simple JSON-based state machine language – Conditions, loops, fault tolerance, etc. – Propagates state through the flow • Standardized API for integrating custom event and action services – Actions: synchronous or asynchronous – Custom Web forms prompt for user input • Actions secured with Globus Auth
  • 25. 32 Run flows: Guided input Label Notify user Timeout Dynamic forms generated from input schema
  • 28. Extending the ecosystem: Action providers 35 • Action Provider is a service endpoint – Run – Status – Cancel – Release – Resume • Action Provider Toolkit action-provider- tools.readthedocs.io/en/latest Search Transfer Notification ACLs Identifier Delete Ingest User Form Describe Xtract funcX Web Form Custom built Globus Provided
  • 30. 37 Custom portals? Science Gateways? Unique workflows? Our open REST APIs and Python SDK empower you to create an integrated ecosystem of research data services and applications.
  • 31. Data centric applications leveraging Globus 38
  • 32. App Access to Collections • Globus Transfer – Authentication with access tokens – Individual: Globus login to get tokens – Application: Apps are people too! o developers.globus.org - Client ID / Secret / Client Identity Username • Collection access – GCSv4 Mapped Collections – user consent o Activating endpoint means binding a credential to an endpoint for login – GCSv5 Mapped Collections (no user certificates, OAUTH tokens and consents) o https://docs.globus.org/globus-connect-server/v5.4/use-client-credentials/ o Request the data_access scope (per collection) to be able to access the collection. o The storage gateway must permit identities from the 'clients.auth.globus.org' identity domain o Identity Mapping Policy that maps the ‘UUID@clients.auth.globus.org' identity to a valid local user – Guest Collections o Guest Collections auto-activate - need to do this before API calls to endpoints o Use Guest Collections whenever possible – Remember to set your ACLs (WebApp) Automation
  • 34. Globus APIs • Auth • Groups • Transfer • Search • Timer • Flows • GCS Manager • Globus Web App consumes public Transfer API • Resource named by URL (standard REST approach) • Globus APIs use JSON for documents docs.globus.org/api/transfer
  • 35. Globus Python SDK • Python client library for the Globus REST APIs • Largely direct mapping to REST API • globus_sdk.TransferClient class handles connection management, security, framing, marshaling globus-sdk-python.readthedocs.io/en/stable/ globus.github.io/globus-sdk-python 42
  • 36. TransferClient higher-level calls • One method for each API resource and HTTP verb • Largely direct mapping to REST API endpoint_search(filter_fulltext=None, filter_scope=None, num_results=25, **params) 44
  • 37. Synchronous Tasks • Endpoint search (with scopes) • List directory contents (ls) • Make directory (mkdir) • Rename • Note: – Path encoding & UTF gotchas – Don’t forget to auto-activate first 46
  • 38. Asynchronous Tasks • Transfer – Sync level option • Delete • Get submission_id, followed by submit – Once and only once submission • Use task id to “follow up” 47
  • 39. The Globus API / SDK with a Jupyter Notebook in a Jupyter Hub – Auth Code Grant login REST APIs { “tokens”:… {“tokens”:… REST APIs REST APIs Bearer a45cd…
  • 40. Walkthrough API with our Jupyter Hub • https://jupyter.demo.globus.org – Sign in with Globus – Verify the consents – Start My Server (this will take about a minute) – Open folder: globus-jupyter-notebooks – Run Platform_Introduction_JupyterHub_Auth.ipynb • If you mess it up and want to “go back to the beginning” – Just stop and restart the server • If you want to use the notebook outside of our hub – https://github.com/globus/globus-jupyter-notebooks – Authentication is a manual cut and paste of exchanging the authorization code for an access token – Native App 49
  • 41. Support Resources • Globus Documentation: docs.globus.org • Globus Timer: globus.org/blog/scheduled-and- recurring-transfers-now-available-globus-web-app • Globus CLI: docs.globus.org/cli/ • Globus Automation Services and Flows: docs.globus.org/globus-automation-services/ • YouTube Channel: youtube.com/user/GlobusOnline
  • 42. Developer References • Globus API / SDK Documentation – Transfer API : docs.globus.org/api/transfer/ – SDK: globus-sdk-python.readthedocs.io/en/stable/ • Globus GitHub: github.com/globus/ – Jupyter Notebooks o Stand alone notebooks and hub integrations that walk through much of the functionality of our SDK o https://github.com/globus/globus-jupyter-notebooks – Automation Examples o Shell scripted CLI and Python module examples of common research data management use cases o https://github.com/globus/automation-examples