SlideShare a Scribd company logo
1 of 39
Download to read offline
Working with Globus
Platform Services and Portals
Vas Vasiliadis
vas@uchicago.edu
September 9, 2022
Sponsored by
Globus Platform
Services
…Integrate Globus
capabilities into your
data portal, science
gateway, data commons
or other research app
Globus APIs
• Auth
• Groups
• Transfer
• Search
• Timer
• Flows
• GCS Manager
• Globus web app consumes the
same public APIs
• Resources named by URL
(standard REST approach)
• Request/response body is JSON
docs.globus.org/api
Globus Python SDK
• Python client library for the Globus REST APIs
• Largely direct mapping to REST API
• globus_sdk.TransferClient class handles
connection management, security, framing,
marshaling
globus-sdk-python.readthedocs.io
github.com/globus/globus-sdk-python
5
Globus Auth: Foundational IAM service
• Brokers authentication and authorization among…
– End-users
– Identity providers: enterprise, external (federated identities)
– Services: resource servers with REST APIs
– Apps: web, mobile, desktop, command line clients
– Services acting as clients to other services
• OAuth 2.0 Authorization Framework (a.k.a. OAuth2)
• OpenID Connect Core 1.0 (a.k.a. OIDC)
6
Fundamental Concepts
• Scopes
– APIs that client is requesting access to
– Service and resources within that service
• Consents
– Authorizes a client to access a service, within limited scope, on
the resource owner's behalf
• Multiple methods for user to grant consent depending
on the type of application
7
Several authentication models supported
• Application acting as user with consent
– Authorization code grant: authenticate as user identity
– Browser redirect; auth code returned automatically; tokens stored securely
– Examples: Globus CLI, Jupyter Hub secured with Globus Auth
• Application authenticating as itself
– Client credentials grant: authenticate as application
– Client ID and Secret stored securely
– Examples: Globus Django portal, custom apps
• Application able to manage tokens for offline/long lived tasks
– Request refresh tokens in addition to access tokens
App registration
9
Get app credentials at developers.globus.org
Getting API access
tokens
11
jupyter.demo.globus.org
Getting tokens: jupyter.demo.globus.org
• Sign in with Globus and verify the consents
• Start My Server and open globus-jupyter-notebooks
• Run Platform_Introduction_Native_App_Auth
• If you mess up, just stop and restart the server
• Alternatively, use notebooks on your own machine:
github.com/globus/globus-jupyter-notebooks
12
REST APIs
REST APIs
REST APIs
Request
Bearer a45cd...
Configurable HTTP proxy
Authenticator
User DB
Spawner
Notebook
/api/auth
/hub/
/user/[name]/
login
Browser
{"tokens":...
{"tokens":...
Globus Auth integrated with JupyterHub
The world is your
oyster API…
• Globus Transfer
• Globus Search
• Your app
• Data portal
• Analysis engine
• …
Working with the
Transfer API
19
jupyter.demo.globus.org
TransferClient low-level calls
• Thin wrapper around REST API
– post(), get(), update(), delete()
get(path, params=None, headers=None, auth=None,
response_class=None)
o path – path for the request, with or without leading slash
o params – dict to be encoded as a query string
o headers – dict of HTTP headers to add to the request
o response_class – class response object, overrides the client’s
default_response_class
o Returns: GlobusHTTPResponse object
20
TransferClient higher-level calls
• One method for each API resource and HTTP verb
• Largely direct mapping to REST API
endpoint_search(filter_fulltext=None,
filter_scope=None,
num_results=25,
**params)
21
API walkthrough: jupyter.demo.globus.org
• Run Platform_Introduction_JupyterHub_Auth
• Note automatic token extraction from JupyterHub
• Run the first few cells
• Experiment with endpoint search/filters
22
Making data findable
with Globus Search
23
docs.globus.org/api/search
Data description and discovery
• Metadata store with fine-
grained visibility controls
• Schema agnostic
à dynamic schemas
• Simple search using URL
query parameters
• Complex search using
search request document
24
Search
Index
Distinct access policies
may be applied to
Data and Metadata
…using permissions on
guest collections
…using permissions on
metadata elements
Data ingest with Globus Search
26
Search
Index
POST /index/{index_id}/ingest'
{
"ingest_type": "GMetaList",
"ingest_data": {
"gmeta": [
{
"id": "filetype",
"subject”: "https://search.api.globus.org/abc.txt",
"visible_to": ["public"],
"content": {
"metadata-schema/file#type": "file”
}
},
...
]
}
- Bulk create and update
- Task model for ingest at scale
Data ingest with Globus Search
27
Search
Index
POST /index/{index_id}/ingest'
{
"ingest_type": "GMetaList",
"ingest_data": {
"gmeta": [
{
"id": ”weight",
"subject": "https://search.api.globus.org/abc.txt",
"visible_to": ["urn:globus:auth:identity:46bd0f56-
e24f-11e5-a510-131bef46955c"],
"content": {
"metadata-schema/file#size": ”37.6",
"metadata-schema/file#size_human": ”<50lb”
}
},
...
]
}
Visibility limited to Globus Auth identity
- Single user
- Globus Group
- Registered client application
Data discovery with Globus Search
28
{
"@datatype": "GSearchResult",
"@version": "2017-09-01",
"count": 1,
"gmeta": [
{
"@datatype": "GMetaResult",
"@version": "2019-08-27",
"entries": [
{ ... }
],
"subject": "https://..."
}
],
"offset": 0,
"total": 1
}
GET /index/{index_id}/search?q=type%3Ahdf5
Search
Index
Simple query
Data discovery with Globus Search
29
POST /index/{index_id}/search
Search
Index
Complex query
{
"filters": [
{
"type": "range",
"field_name": ”pubdate",
"values": [
{
"from": "*",
"to": "2020-12-31"
}
]
}
],
"facets": [
{
"name": "Publication Date",
"field_name": "pubdate",
...
}
]
}
Filter
Facets
Boosts
Sort
Limit
Why go beyond the standard tools?
• FAIRness: Enable broad audience to access diverse
research data (type, size, metadata, …)
• …and diverse/complicated data sources (beamlines,
electron microscopes, sequencers, …)
• Add curation and cataloguing to make data findable
• Enforce (sometimes complex) access policies
• Enable researchers to customize their experience
The Modern Research Data
Portal Design Pattern
docs.globus.org/mrdp
MRDP: Key elements
Science DMZ
Fast, clean data path
Data Transfer Nodes
Purpose-built data movers
Globus Platform
Secure, reliable data
orchestration
Globus Connect
Storage system enabler
32
Globus Portal
Framework
Data discovery and access
An exemplar:
The ALCF Data Co-op
33
acdc.alcf.anl.gov
Accessing data via your portal
• Recall: Data independent of portal logic…
• …ideally, on a guest collection
• Step 1: Create a guest collection; requires authN by
human or by app using client credentials
• Step2: Grant the application Access Manager role
– Allows the application to manage permissions on the collection
– Set for application identity: appclientid@clients.auth.globus.org
• Optional: Grant roles for endpoint/task management
10GE
10GE
10GE
10GE
Border Router
WAN
Science DMZ
Switch/Router
Firewall
Enterprise
perfSONAR
perfSONAR
10GE
10GE
10GE
10GE
DTN
DTN
API DTNs
(data access governed
by portal)
DTN
DTN
perfSONAR
Filesystem
(data store)
10GE
Portal
Server
Browsing path
Query path
Portal server applications:
· web server
· search
· database
· authentication
Data Path
Data Transfer Path
Portal Query/Browse Path
Globus Groups simplify permissions management
• Grant group access to
collection(s)
• Restrict search visibility
using group
• Make portal client a group
administrator
• Check authenticated user’s
group membership
• Add/remove user to/from
group
Bootstrap a Simple
(but fully functional and extensible)
Research Data Portal
using the Django Globus
Portal Framework
Django Globus Portal key features
• Federated login
• Data export using Globus
• Browse datasets via Globus Search service
• Template-driven search results and landing pages
• Django-based framework with extensible templates
• Bootstrap your project using Cookiecutter
37
Source: github.com/globus/django-globus-portal-framework
Docs: django-globus-portal-framework.readthedocs.io
Get up and running
with the Globus
portal framework
38
Source: github.com/globus/django-globus-portal-framework
Docs: django-globus-portal-framework.readthedocs.io/en/stable/
Step 0: Application registration
• Set redirect URLs
• Get client ID and secret
• Consents implement
least privileges principle
39
developers.globus.org
Redirect URLs
https://tutN.globusdemo.org:8443/
https://tutN.globusdemo.org:8443/complete/globus/
Portal deployment
• Deploy a portal instance using cookiecutter
• Configure settings
• Run and use!
• Future: containers
• Note: For production, add robust WSGI/ASGI server
Adding a new search
index to your portal
41
Update the search index definition
• Edit ~/$PROJECT_SLUG/$PROJECT_SLUG/settings/search.py
• Index ID and name
• Fields
• Facets
• Multiple search indices may be defined
Update definition of search result fields
• Edit ~/$PROJECT_SLUG/$PROJECT_SLUG/fields.py
• Type definitions
• Formatting functions
• Data transformations
• Globus collection and data path
Modify templates for search results, details
• Fields to be displayed on search item card in list
– Metadata to help help user determine relevance
– Links to actions, e.g., transfer, download, process
– Edit ~/$PROJECT_SLUG/templates/globus-portal-
framework/v2/components/search-results.html
• Fields to be displayed on detail page
– (More) complete metadata
– Preview images (if appropriate)
– Links to actions
– Edit ~/$PROJECT_SLUG/templates/globus-portal-framework/v2/detail-
overview.html
• Uses standard Jinja2 templating language
Developer References
• Documentation
– API: docs.globus.org/api
– SDK: globus-sdk-python.readthedocs.io/en/stable/
• Open source repositories: github.com/globus
• Notebooks: github.com/globus/globus-jupyter-
notebooks

More Related Content

Similar to Working with Globus Platform Services and Portals

Introduction to the Globus Platform (APS Workshop)
Introduction to the Globus Platform (APS Workshop)Introduction to the Globus Platform (APS Workshop)
Introduction to the Globus Platform (APS Workshop)Globus
 
Automating Research Data Flows and an Introduction to the Globus Platform
Automating Research Data Flows and an Introduction to the Globus PlatformAutomating Research Data Flows and an Introduction to the Globus Platform
Automating Research Data Flows and an Introduction to the Globus PlatformGlobus
 
Leveraging the Globus Platform (GlobusWorld Tour - Columbia University)
Leveraging the Globus Platform (GlobusWorld Tour - Columbia University)Leveraging the Globus Platform (GlobusWorld Tour - Columbia University)
Leveraging the Globus Platform (GlobusWorld Tour - Columbia University)Globus
 
Advanced Computing Meets Data FAIRness
Advanced Computing Meets Data FAIRnessAdvanced Computing Meets Data FAIRness
Advanced Computing Meets Data FAIRnessGlobus
 
Enabling Secure Data Discoverability (SC21 Tutorial)
Enabling Secure Data Discoverability (SC21 Tutorial)Enabling Secure Data Discoverability (SC21 Tutorial)
Enabling Secure Data Discoverability (SC21 Tutorial)Globus
 
Globus Command Line Interface (APS Workshop)
Globus Command Line Interface (APS Workshop)Globus Command Line Interface (APS Workshop)
Globus Command Line Interface (APS Workshop)Globus
 
Build an AI/ML-driven image archive processing workflow: Image archive, analy...
Build an AI/ML-driven image archive processing workflow: Image archive, analy...Build an AI/ML-driven image archive processing workflow: Image archive, analy...
Build an AI/ML-driven image archive processing workflow: Image archive, analy...wesley chun
 
Globus Platform Overview
Globus Platform OverviewGlobus Platform Overview
Globus Platform OverviewGlobus
 
GlobusWorld 2021 Tutorial: Building with the Globus Platform
GlobusWorld 2021 Tutorial: Building with the Globus PlatformGlobusWorld 2021 Tutorial: Building with the Globus Platform
GlobusWorld 2021 Tutorial: Building with the Globus PlatformGlobus
 
Gateways 2020 Tutorial - Introduction to Globus
Gateways 2020 Tutorial - Introduction to GlobusGateways 2020 Tutorial - Introduction to Globus
Gateways 2020 Tutorial - Introduction to GlobusGlobus
 
GlobusWorld 2021 Tutorial: The Globus CLI, Platform and SDK
GlobusWorld 2021 Tutorial: The Globus CLI, Platform and SDKGlobusWorld 2021 Tutorial: The Globus CLI, Platform and SDK
GlobusWorld 2021 Tutorial: The Globus CLI, Platform and SDKGlobus
 
Jupyter + Globus: The Foundation for Interactive Data Science
Jupyter + Globus: The Foundation for Interactive Data ScienceJupyter + Globus: The Foundation for Interactive Data Science
Jupyter + Globus: The Foundation for Interactive Data ScienceGlobus
 
Power your apps with Gmail, Google Drive, Calendar, Sheets, Slides & more
Power your apps with Gmail, Google Drive, Calendar, Sheets, Slides & morePower your apps with Gmail, Google Drive, Calendar, Sheets, Slides & more
Power your apps with Gmail, Google Drive, Calendar, Sheets, Slides & morewesley chun
 
Globus Connect Server v5 Q&A Briefing
Globus Connect Server v5 Q&A BriefingGlobus Connect Server v5 Q&A Briefing
Globus Connect Server v5 Q&A BriefingGlobus
 
Data Publication and Discovery with Globus
Data Publication and Discovery with GlobusData Publication and Discovery with Globus
Data Publication and Discovery with GlobusGlobus
 
Simple Data Automation with Globus (GlobusWorld Tour West)
Simple Data Automation with Globus (GlobusWorld Tour West)Simple Data Automation with Globus (GlobusWorld Tour West)
Simple Data Automation with Globus (GlobusWorld Tour West)Globus
 
Using Globus to Streamline Research at Scale
Using Globus to Streamline Research at ScaleUsing Globus to Streamline Research at Scale
Using Globus to Streamline Research at ScaleGlobus
 
Introduction to Globus for System Administrators
Introduction to Globus for System AdministratorsIntroduction to Globus for System Administrators
Introduction to Globus for System AdministratorsGlobus
 
Leveraging the Globus Platform (GlobusWorld Tour - UCSD)
Leveraging the Globus Platform (GlobusWorld Tour - UCSD)Leveraging the Globus Platform (GlobusWorld Tour - UCSD)
Leveraging the Globus Platform (GlobusWorld Tour - UCSD)Globus
 
Introduction to Globus for System Administrators
Introduction to Globus for System AdministratorsIntroduction to Globus for System Administrators
Introduction to Globus for System AdministratorsGlobus
 

Similar to Working with Globus Platform Services and Portals (20)

Introduction to the Globus Platform (APS Workshop)
Introduction to the Globus Platform (APS Workshop)Introduction to the Globus Platform (APS Workshop)
Introduction to the Globus Platform (APS Workshop)
 
Automating Research Data Flows and an Introduction to the Globus Platform
Automating Research Data Flows and an Introduction to the Globus PlatformAutomating Research Data Flows and an Introduction to the Globus Platform
Automating Research Data Flows and an Introduction to the Globus Platform
 
Leveraging the Globus Platform (GlobusWorld Tour - Columbia University)
Leveraging the Globus Platform (GlobusWorld Tour - Columbia University)Leveraging the Globus Platform (GlobusWorld Tour - Columbia University)
Leveraging the Globus Platform (GlobusWorld Tour - Columbia University)
 
Advanced Computing Meets Data FAIRness
Advanced Computing Meets Data FAIRnessAdvanced Computing Meets Data FAIRness
Advanced Computing Meets Data FAIRness
 
Enabling Secure Data Discoverability (SC21 Tutorial)
Enabling Secure Data Discoverability (SC21 Tutorial)Enabling Secure Data Discoverability (SC21 Tutorial)
Enabling Secure Data Discoverability (SC21 Tutorial)
 
Globus Command Line Interface (APS Workshop)
Globus Command Line Interface (APS Workshop)Globus Command Line Interface (APS Workshop)
Globus Command Line Interface (APS Workshop)
 
Build an AI/ML-driven image archive processing workflow: Image archive, analy...
Build an AI/ML-driven image archive processing workflow: Image archive, analy...Build an AI/ML-driven image archive processing workflow: Image archive, analy...
Build an AI/ML-driven image archive processing workflow: Image archive, analy...
 
Globus Platform Overview
Globus Platform OverviewGlobus Platform Overview
Globus Platform Overview
 
GlobusWorld 2021 Tutorial: Building with the Globus Platform
GlobusWorld 2021 Tutorial: Building with the Globus PlatformGlobusWorld 2021 Tutorial: Building with the Globus Platform
GlobusWorld 2021 Tutorial: Building with the Globus Platform
 
Gateways 2020 Tutorial - Introduction to Globus
Gateways 2020 Tutorial - Introduction to GlobusGateways 2020 Tutorial - Introduction to Globus
Gateways 2020 Tutorial - Introduction to Globus
 
GlobusWorld 2021 Tutorial: The Globus CLI, Platform and SDK
GlobusWorld 2021 Tutorial: The Globus CLI, Platform and SDKGlobusWorld 2021 Tutorial: The Globus CLI, Platform and SDK
GlobusWorld 2021 Tutorial: The Globus CLI, Platform and SDK
 
Jupyter + Globus: The Foundation for Interactive Data Science
Jupyter + Globus: The Foundation for Interactive Data ScienceJupyter + Globus: The Foundation for Interactive Data Science
Jupyter + Globus: The Foundation for Interactive Data Science
 
Power your apps with Gmail, Google Drive, Calendar, Sheets, Slides & more
Power your apps with Gmail, Google Drive, Calendar, Sheets, Slides & morePower your apps with Gmail, Google Drive, Calendar, Sheets, Slides & more
Power your apps with Gmail, Google Drive, Calendar, Sheets, Slides & more
 
Globus Connect Server v5 Q&A Briefing
Globus Connect Server v5 Q&A BriefingGlobus Connect Server v5 Q&A Briefing
Globus Connect Server v5 Q&A Briefing
 
Data Publication and Discovery with Globus
Data Publication and Discovery with GlobusData Publication and Discovery with Globus
Data Publication and Discovery with Globus
 
Simple Data Automation with Globus (GlobusWorld Tour West)
Simple Data Automation with Globus (GlobusWorld Tour West)Simple Data Automation with Globus (GlobusWorld Tour West)
Simple Data Automation with Globus (GlobusWorld Tour West)
 
Using Globus to Streamline Research at Scale
Using Globus to Streamline Research at ScaleUsing Globus to Streamline Research at Scale
Using Globus to Streamline Research at Scale
 
Introduction to Globus for System Administrators
Introduction to Globus for System AdministratorsIntroduction to Globus for System Administrators
Introduction to Globus for System Administrators
 
Leveraging the Globus Platform (GlobusWorld Tour - UCSD)
Leveraging the Globus Platform (GlobusWorld Tour - UCSD)Leveraging the Globus Platform (GlobusWorld Tour - UCSD)
Leveraging the Globus Platform (GlobusWorld Tour - UCSD)
 
Introduction to Globus for System Administrators
Introduction to Globus for System AdministratorsIntroduction to Globus for System Administrators
Introduction to Globus for System Administrators
 

More from Globus

Advanced Globus System Administration Topics
Advanced Globus System Administration TopicsAdvanced Globus System Administration Topics
Advanced Globus System Administration TopicsGlobus
 
Instrument Data Automation: The Life of a Flow
Instrument Data Automation: The Life of a FlowInstrument Data Automation: The Life of a Flow
Instrument Data Automation: The Life of a FlowGlobus
 
Reliable, Remote Computation at All Scales
Reliable, Remote Computation at All ScalesReliable, Remote Computation at All Scales
Reliable, Remote Computation at All ScalesGlobus
 
Best Practices for Data Sharing Using Globus
Best Practices for Data Sharing Using GlobusBest Practices for Data Sharing Using Globus
Best Practices for Data Sharing Using GlobusGlobus
 
An Introduction to Globus for Researchers
An Introduction to Globus for ResearchersAn Introduction to Globus for Researchers
An Introduction to Globus for ResearchersGlobus
 
Introduction to Research Automation with Globus
Introduction to Research Automation with GlobusIntroduction to Research Automation with Globus
Introduction to Research Automation with GlobusGlobus
 
Globus for System Administrators
Globus for System AdministratorsGlobus for System Administrators
Globus for System AdministratorsGlobus
 
Introduction to Data Transfer and Sharing for Researchers
Introduction to Data Transfer and Sharing for ResearchersIntroduction to Data Transfer and Sharing for Researchers
Introduction to Data Transfer and Sharing for ResearchersGlobus
 
Introduction to the Globus Platform for Developers
Introduction to the Globus Platform for DevelopersIntroduction to the Globus Platform for Developers
Introduction to the Globus Platform for DevelopersGlobus
 
Introduction to the Command Line Interface (CLI)
Introduction to the Command Line Interface (CLI)Introduction to the Command Line Interface (CLI)
Introduction to the Command Line Interface (CLI)Globus
 
Automating Research Data with Globus Flows and Compute
Automating Research Data with Globus Flows and ComputeAutomating Research Data with Globus Flows and Compute
Automating Research Data with Globus Flows and ComputeGlobus
 
Advanced Globus System Administration
Advanced Globus System AdministrationAdvanced Globus System Administration
Advanced Globus System AdministrationGlobus
 
Introduction to Globus for System Administrators
Introduction to Globus for System AdministratorsIntroduction to Globus for System Administrators
Introduction to Globus for System AdministratorsGlobus
 
Introduction to Globus for New Users
Introduction to Globus for New UsersIntroduction to Globus for New Users
Introduction to Globus for New UsersGlobus
 
Globus Automation
Globus AutomationGlobus Automation
Globus AutomationGlobus
 
Advanced Globus System Administration
Advanced Globus System AdministrationAdvanced Globus System Administration
Advanced Globus System AdministrationGlobus
 
Introduction to Globus
Introduction to GlobusIntroduction to Globus
Introduction to GlobusGlobus
 
Advanced Globus System Administration
Advanced Globus System AdministrationAdvanced Globus System Administration
Advanced Globus System AdministrationGlobus
 
Introduction to Globus for System Administrators
Introduction to Globus for System AdministratorsIntroduction to Globus for System Administrators
Introduction to Globus for System AdministratorsGlobus
 
Introduction to Globus for Researchers
Introduction to Globus for ResearchersIntroduction to Globus for Researchers
Introduction to Globus for ResearchersGlobus
 

More from Globus (20)

Advanced Globus System Administration Topics
Advanced Globus System Administration TopicsAdvanced Globus System Administration Topics
Advanced Globus System Administration Topics
 
Instrument Data Automation: The Life of a Flow
Instrument Data Automation: The Life of a FlowInstrument Data Automation: The Life of a Flow
Instrument Data Automation: The Life of a Flow
 
Reliable, Remote Computation at All Scales
Reliable, Remote Computation at All ScalesReliable, Remote Computation at All Scales
Reliable, Remote Computation at All Scales
 
Best Practices for Data Sharing Using Globus
Best Practices for Data Sharing Using GlobusBest Practices for Data Sharing Using Globus
Best Practices for Data Sharing Using Globus
 
An Introduction to Globus for Researchers
An Introduction to Globus for ResearchersAn Introduction to Globus for Researchers
An Introduction to Globus for Researchers
 
Introduction to Research Automation with Globus
Introduction to Research Automation with GlobusIntroduction to Research Automation with Globus
Introduction to Research Automation with Globus
 
Globus for System Administrators
Globus for System AdministratorsGlobus for System Administrators
Globus for System Administrators
 
Introduction to Data Transfer and Sharing for Researchers
Introduction to Data Transfer and Sharing for ResearchersIntroduction to Data Transfer and Sharing for Researchers
Introduction to Data Transfer and Sharing for Researchers
 
Introduction to the Globus Platform for Developers
Introduction to the Globus Platform for DevelopersIntroduction to the Globus Platform for Developers
Introduction to the Globus Platform for Developers
 
Introduction to the Command Line Interface (CLI)
Introduction to the Command Line Interface (CLI)Introduction to the Command Line Interface (CLI)
Introduction to the Command Line Interface (CLI)
 
Automating Research Data with Globus Flows and Compute
Automating Research Data with Globus Flows and ComputeAutomating Research Data with Globus Flows and Compute
Automating Research Data with Globus Flows and Compute
 
Advanced Globus System Administration
Advanced Globus System AdministrationAdvanced Globus System Administration
Advanced Globus System Administration
 
Introduction to Globus for System Administrators
Introduction to Globus for System AdministratorsIntroduction to Globus for System Administrators
Introduction to Globus for System Administrators
 
Introduction to Globus for New Users
Introduction to Globus for New UsersIntroduction to Globus for New Users
Introduction to Globus for New Users
 
Globus Automation
Globus AutomationGlobus Automation
Globus Automation
 
Advanced Globus System Administration
Advanced Globus System AdministrationAdvanced Globus System Administration
Advanced Globus System Administration
 
Introduction to Globus
Introduction to GlobusIntroduction to Globus
Introduction to Globus
 
Advanced Globus System Administration
Advanced Globus System AdministrationAdvanced Globus System Administration
Advanced Globus System Administration
 
Introduction to Globus for System Administrators
Introduction to Globus for System AdministratorsIntroduction to Globus for System Administrators
Introduction to Globus for System Administrators
 
Introduction to Globus for Researchers
Introduction to Globus for ResearchersIntroduction to Globus for Researchers
Introduction to Globus for Researchers
 

Recently uploaded

Real-time Tracking and Monitoring with Cargo Cloud Solutions.pptx
Real-time Tracking and Monitoring with Cargo Cloud Solutions.pptxReal-time Tracking and Monitoring with Cargo Cloud Solutions.pptx
Real-time Tracking and Monitoring with Cargo Cloud Solutions.pptxRTS corp
 
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full RecordingOpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full RecordingShane Coughlan
 
VictoriaMetrics Q1 Meet Up '24 - Community & News Update
VictoriaMetrics Q1 Meet Up '24 - Community & News UpdateVictoriaMetrics Q1 Meet Up '24 - Community & News Update
VictoriaMetrics Q1 Meet Up '24 - Community & News UpdateVictoriaMetrics
 
Strategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero resultsStrategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero resultsJean Silva
 
Powering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsPowering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsSafe Software
 
Introduction to Firebase Workshop Slides
Introduction to Firebase Workshop SlidesIntroduction to Firebase Workshop Slides
Introduction to Firebase Workshop Slidesvaideheekore1
 
Salesforce Implementation Services PPT By ABSYZ
Salesforce Implementation Services PPT By ABSYZSalesforce Implementation Services PPT By ABSYZ
Salesforce Implementation Services PPT By ABSYZABSYZ Inc
 
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...OnePlan Solutions
 
Patterns for automating API delivery. API conference
Patterns for automating API delivery. API conferencePatterns for automating API delivery. API conference
Patterns for automating API delivery. API conferencessuser9e7c64
 
SensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving CarsSensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving CarsChristian Birchler
 
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...Bert Jan Schrijver
 
Best Angular 17 Classroom & Online training - Naresh IT
Best Angular 17 Classroom & Online training - Naresh ITBest Angular 17 Classroom & Online training - Naresh IT
Best Angular 17 Classroom & Online training - Naresh ITmanoharjgpsolutions
 
Leveraging AI for Mobile App Testing on Real Devices | Applitools + Kobiton
Leveraging AI for Mobile App Testing on Real Devices | Applitools + KobitonLeveraging AI for Mobile App Testing on Real Devices | Applitools + Kobiton
Leveraging AI for Mobile App Testing on Real Devices | Applitools + KobitonApplitools
 
Enhancing Supply Chain Visibility with Cargo Cloud Solutions.pdf
Enhancing Supply Chain Visibility with Cargo Cloud Solutions.pdfEnhancing Supply Chain Visibility with Cargo Cloud Solutions.pdf
Enhancing Supply Chain Visibility with Cargo Cloud Solutions.pdfRTS corp
 
eSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration toolseSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration toolsosttopstonverter
 
SAM Training Session - How to use EXCEL ?
SAM Training Session - How to use EXCEL ?SAM Training Session - How to use EXCEL ?
SAM Training Session - How to use EXCEL ?Alexandre Beguel
 
Keeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository worldKeeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository worldRoberto Pérez Alcolea
 
Post Quantum Cryptography – The Impact on Identity
Post Quantum Cryptography – The Impact on IdentityPost Quantum Cryptography – The Impact on Identity
Post Quantum Cryptography – The Impact on Identityteam-WIBU
 
What’s New in VictoriaMetrics: Q1 2024 Updates
What’s New in VictoriaMetrics: Q1 2024 UpdatesWhat’s New in VictoriaMetrics: Q1 2024 Updates
What’s New in VictoriaMetrics: Q1 2024 UpdatesVictoriaMetrics
 
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...OnePlan Solutions
 

Recently uploaded (20)

Real-time Tracking and Monitoring with Cargo Cloud Solutions.pptx
Real-time Tracking and Monitoring with Cargo Cloud Solutions.pptxReal-time Tracking and Monitoring with Cargo Cloud Solutions.pptx
Real-time Tracking and Monitoring with Cargo Cloud Solutions.pptx
 
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full RecordingOpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
 
VictoriaMetrics Q1 Meet Up '24 - Community & News Update
VictoriaMetrics Q1 Meet Up '24 - Community & News UpdateVictoriaMetrics Q1 Meet Up '24 - Community & News Update
VictoriaMetrics Q1 Meet Up '24 - Community & News Update
 
Strategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero resultsStrategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero results
 
Powering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsPowering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data Streams
 
Introduction to Firebase Workshop Slides
Introduction to Firebase Workshop SlidesIntroduction to Firebase Workshop Slides
Introduction to Firebase Workshop Slides
 
Salesforce Implementation Services PPT By ABSYZ
Salesforce Implementation Services PPT By ABSYZSalesforce Implementation Services PPT By ABSYZ
Salesforce Implementation Services PPT By ABSYZ
 
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
 
Patterns for automating API delivery. API conference
Patterns for automating API delivery. API conferencePatterns for automating API delivery. API conference
Patterns for automating API delivery. API conference
 
SensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving CarsSensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving Cars
 
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...
 
Best Angular 17 Classroom & Online training - Naresh IT
Best Angular 17 Classroom & Online training - Naresh ITBest Angular 17 Classroom & Online training - Naresh IT
Best Angular 17 Classroom & Online training - Naresh IT
 
Leveraging AI for Mobile App Testing on Real Devices | Applitools + Kobiton
Leveraging AI for Mobile App Testing on Real Devices | Applitools + KobitonLeveraging AI for Mobile App Testing on Real Devices | Applitools + Kobiton
Leveraging AI for Mobile App Testing on Real Devices | Applitools + Kobiton
 
Enhancing Supply Chain Visibility with Cargo Cloud Solutions.pdf
Enhancing Supply Chain Visibility with Cargo Cloud Solutions.pdfEnhancing Supply Chain Visibility with Cargo Cloud Solutions.pdf
Enhancing Supply Chain Visibility with Cargo Cloud Solutions.pdf
 
eSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration toolseSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration tools
 
SAM Training Session - How to use EXCEL ?
SAM Training Session - How to use EXCEL ?SAM Training Session - How to use EXCEL ?
SAM Training Session - How to use EXCEL ?
 
Keeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository worldKeeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository world
 
Post Quantum Cryptography – The Impact on Identity
Post Quantum Cryptography – The Impact on IdentityPost Quantum Cryptography – The Impact on Identity
Post Quantum Cryptography – The Impact on Identity
 
What’s New in VictoriaMetrics: Q1 2024 Updates
What’s New in VictoriaMetrics: Q1 2024 UpdatesWhat’s New in VictoriaMetrics: Q1 2024 Updates
What’s New in VictoriaMetrics: Q1 2024 Updates
 
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
 

Working with Globus Platform Services and Portals

  • 1. Working with Globus Platform Services and Portals Vas Vasiliadis vas@uchicago.edu September 9, 2022 Sponsored by
  • 2. Globus Platform Services …Integrate Globus capabilities into your data portal, science gateway, data commons or other research app
  • 3. Globus APIs • Auth • Groups • Transfer • Search • Timer • Flows • GCS Manager • Globus web app consumes the same public APIs • Resources named by URL (standard REST approach) • Request/response body is JSON docs.globus.org/api
  • 4. Globus Python SDK • Python client library for the Globus REST APIs • Largely direct mapping to REST API • globus_sdk.TransferClient class handles connection management, security, framing, marshaling globus-sdk-python.readthedocs.io github.com/globus/globus-sdk-python 5
  • 5. Globus Auth: Foundational IAM service • Brokers authentication and authorization among… – End-users – Identity providers: enterprise, external (federated identities) – Services: resource servers with REST APIs – Apps: web, mobile, desktop, command line clients – Services acting as clients to other services • OAuth 2.0 Authorization Framework (a.k.a. OAuth2) • OpenID Connect Core 1.0 (a.k.a. OIDC) 6
  • 6. Fundamental Concepts • Scopes – APIs that client is requesting access to – Service and resources within that service • Consents – Authorizes a client to access a service, within limited scope, on the resource owner's behalf • Multiple methods for user to grant consent depending on the type of application 7
  • 7. Several authentication models supported • Application acting as user with consent – Authorization code grant: authenticate as user identity – Browser redirect; auth code returned automatically; tokens stored securely – Examples: Globus CLI, Jupyter Hub secured with Globus Auth • Application authenticating as itself – Client credentials grant: authenticate as application – Client ID and Secret stored securely – Examples: Globus Django portal, custom apps • Application able to manage tokens for offline/long lived tasks – Request refresh tokens in addition to access tokens
  • 9. Get app credentials at developers.globus.org
  • 11. Getting tokens: jupyter.demo.globus.org • Sign in with Globus and verify the consents • Start My Server and open globus-jupyter-notebooks • Run Platform_Introduction_Native_App_Auth • If you mess up, just stop and restart the server • Alternatively, use notebooks on your own machine: github.com/globus/globus-jupyter-notebooks 12
  • 12. REST APIs REST APIs REST APIs Request Bearer a45cd... Configurable HTTP proxy Authenticator User DB Spawner Notebook /api/auth /hub/ /user/[name]/ login Browser {"tokens":... {"tokens":... Globus Auth integrated with JupyterHub The world is your oyster API… • Globus Transfer • Globus Search • Your app • Data portal • Analysis engine • …
  • 13. Working with the Transfer API 19 jupyter.demo.globus.org
  • 14. TransferClient low-level calls • Thin wrapper around REST API – post(), get(), update(), delete() get(path, params=None, headers=None, auth=None, response_class=None) o path – path for the request, with or without leading slash o params – dict to be encoded as a query string o headers – dict of HTTP headers to add to the request o response_class – class response object, overrides the client’s default_response_class o Returns: GlobusHTTPResponse object 20
  • 15. TransferClient higher-level calls • One method for each API resource and HTTP verb • Largely direct mapping to REST API endpoint_search(filter_fulltext=None, filter_scope=None, num_results=25, **params) 21
  • 16. API walkthrough: jupyter.demo.globus.org • Run Platform_Introduction_JupyterHub_Auth • Note automatic token extraction from JupyterHub • Run the first few cells • Experiment with endpoint search/filters 22
  • 17. Making data findable with Globus Search 23 docs.globus.org/api/search
  • 18. Data description and discovery • Metadata store with fine- grained visibility controls • Schema agnostic à dynamic schemas • Simple search using URL query parameters • Complex search using search request document 24 Search Index
  • 19. Distinct access policies may be applied to Data and Metadata …using permissions on guest collections …using permissions on metadata elements
  • 20. Data ingest with Globus Search 26 Search Index POST /index/{index_id}/ingest' { "ingest_type": "GMetaList", "ingest_data": { "gmeta": [ { "id": "filetype", "subject”: "https://search.api.globus.org/abc.txt", "visible_to": ["public"], "content": { "metadata-schema/file#type": "file” } }, ... ] } - Bulk create and update - Task model for ingest at scale
  • 21. Data ingest with Globus Search 27 Search Index POST /index/{index_id}/ingest' { "ingest_type": "GMetaList", "ingest_data": { "gmeta": [ { "id": ”weight", "subject": "https://search.api.globus.org/abc.txt", "visible_to": ["urn:globus:auth:identity:46bd0f56- e24f-11e5-a510-131bef46955c"], "content": { "metadata-schema/file#size": ”37.6", "metadata-schema/file#size_human": ”<50lb” } }, ... ] } Visibility limited to Globus Auth identity - Single user - Globus Group - Registered client application
  • 22. Data discovery with Globus Search 28 { "@datatype": "GSearchResult", "@version": "2017-09-01", "count": 1, "gmeta": [ { "@datatype": "GMetaResult", "@version": "2019-08-27", "entries": [ { ... } ], "subject": "https://..." } ], "offset": 0, "total": 1 } GET /index/{index_id}/search?q=type%3Ahdf5 Search Index Simple query
  • 23. Data discovery with Globus Search 29 POST /index/{index_id}/search Search Index Complex query { "filters": [ { "type": "range", "field_name": ”pubdate", "values": [ { "from": "*", "to": "2020-12-31" } ] } ], "facets": [ { "name": "Publication Date", "field_name": "pubdate", ... } ] } Filter Facets Boosts Sort Limit
  • 24. Why go beyond the standard tools? • FAIRness: Enable broad audience to access diverse research data (type, size, metadata, …) • …and diverse/complicated data sources (beamlines, electron microscopes, sequencers, …) • Add curation and cataloguing to make data findable • Enforce (sometimes complex) access policies • Enable researchers to customize their experience
  • 25. The Modern Research Data Portal Design Pattern docs.globus.org/mrdp
  • 26. MRDP: Key elements Science DMZ Fast, clean data path Data Transfer Nodes Purpose-built data movers Globus Platform Secure, reliable data orchestration Globus Connect Storage system enabler 32 Globus Portal Framework Data discovery and access
  • 27. An exemplar: The ALCF Data Co-op 33 acdc.alcf.anl.gov
  • 28. Accessing data via your portal • Recall: Data independent of portal logic… • …ideally, on a guest collection • Step 1: Create a guest collection; requires authN by human or by app using client credentials • Step2: Grant the application Access Manager role – Allows the application to manage permissions on the collection – Set for application identity: appclientid@clients.auth.globus.org • Optional: Grant roles for endpoint/task management 10GE 10GE 10GE 10GE Border Router WAN Science DMZ Switch/Router Firewall Enterprise perfSONAR perfSONAR 10GE 10GE 10GE 10GE DTN DTN API DTNs (data access governed by portal) DTN DTN perfSONAR Filesystem (data store) 10GE Portal Server Browsing path Query path Portal server applications: · web server · search · database · authentication Data Path Data Transfer Path Portal Query/Browse Path
  • 29. Globus Groups simplify permissions management • Grant group access to collection(s) • Restrict search visibility using group • Make portal client a group administrator • Check authenticated user’s group membership • Add/remove user to/from group
  • 30. Bootstrap a Simple (but fully functional and extensible) Research Data Portal using the Django Globus Portal Framework
  • 31. Django Globus Portal key features • Federated login • Data export using Globus • Browse datasets via Globus Search service • Template-driven search results and landing pages • Django-based framework with extensible templates • Bootstrap your project using Cookiecutter 37 Source: github.com/globus/django-globus-portal-framework Docs: django-globus-portal-framework.readthedocs.io
  • 32. Get up and running with the Globus portal framework 38 Source: github.com/globus/django-globus-portal-framework Docs: django-globus-portal-framework.readthedocs.io/en/stable/
  • 33. Step 0: Application registration • Set redirect URLs • Get client ID and secret • Consents implement least privileges principle 39 developers.globus.org Redirect URLs https://tutN.globusdemo.org:8443/ https://tutN.globusdemo.org:8443/complete/globus/
  • 34. Portal deployment • Deploy a portal instance using cookiecutter • Configure settings • Run and use! • Future: containers • Note: For production, add robust WSGI/ASGI server
  • 35. Adding a new search index to your portal 41
  • 36. Update the search index definition • Edit ~/$PROJECT_SLUG/$PROJECT_SLUG/settings/search.py • Index ID and name • Fields • Facets • Multiple search indices may be defined
  • 37. Update definition of search result fields • Edit ~/$PROJECT_SLUG/$PROJECT_SLUG/fields.py • Type definitions • Formatting functions • Data transformations • Globus collection and data path
  • 38. Modify templates for search results, details • Fields to be displayed on search item card in list – Metadata to help help user determine relevance – Links to actions, e.g., transfer, download, process – Edit ~/$PROJECT_SLUG/templates/globus-portal- framework/v2/components/search-results.html • Fields to be displayed on detail page – (More) complete metadata – Preview images (if appropriate) – Links to actions – Edit ~/$PROJECT_SLUG/templates/globus-portal-framework/v2/detail- overview.html • Uses standard Jinja2 templating language
  • 39. Developer References • Documentation – API: docs.globus.org/api – SDK: globus-sdk-python.readthedocs.io/en/stable/ • Open source repositories: github.com/globus • Notebooks: github.com/globus/globus-jupyter- notebooks