SlideShare a Scribd company logo
Jupyter + Globus: The Foundation for
Interactive Data Science
Rick Wagner
rick@globus.org
Gatways 2018 – September 25, 2018
Globus
Beyond File Transfer
3
Research Computing HPC
Desktop Workstations
Mass Storage Instruments
Personal Resources
Public Cloud
National Resources
Unify access to data across tiers
IBM Spectrum Scale
Current Planned
Storage Connectors - globus.org/connectors
Public / private cloud stores
External
campus
storage
EC2
Project
repositories,
replication stores
Public repositories
Share with collaborators/community
Analysis
store
Next-Gen Sequencer
MRI
Advanced Light Source
Personal system
Remote visualization
Light Sheet Microscope
High-durability,
low-cost store
Manage data from instruments
Cryo-EM
Develop apps, services, and workflows
7
8,000
active shared
endpoints
90
subscribers
425 PB
transferred
18,000
active GCP
endpoints
70 billion
files processed
1,700
active GCS
endpoints
3 months
longest running transfer
1 PB
largest single
transfer to date
99.9%
availability
500
identity providers
1,042
most shared
endpoints
at a single
institution 100,000
users
Globus by the numbers
Conceptual architecture: Hybrid SaaS
DATA
Channel
CONTROL
Channel
Source
Endpoint
Destination
Endpoint
Subscriber owned
and administered
storage system
Globus
“connector”
software
No data relay or
staging via Globus
cloud service
Subscriber
Control
Domain
Globus
Control
Domain
Single, globally accessible
multi-tenant service
Conceptual architecture: Sharing
Managed
Endpoint
Subscriber
Control
Domain
Globus
Control
Domain Globus managed
”overlay” permissions
Shared
Endpoint
DATA
Channel
CONTROL
Channel Subscriber managed
filesystem permissions
External User
Control
Domain
Endpoints (Collections)
• Storage abstraction
– All transfers happen between two endpoints
– Globus Connect instantiates endpoints
• Collection ~= Endpoint
• Test / Demo Endpoints
– Globus Tutorial Endpoint 1
– Globus Tutorial Endpoint 2
– ESnet Test Endpoints
o Contain file samples of various sizes
• Globus Connect Personal
– Now your laptop is an endpoint
– https://www.globus.org/globus-connect-personal
11
Globus Connect Personal
• Installers do not require admin access
• Zero configuration; auto updating
• Handles NATs
• Installs in seconds – easy to delete - I’ll prove it!
The Globus Web App - Accounts
• A Globus Account is
– A Primary Identity
– Possible Linked Identities
• Linking Identities
• Managing Identities
• Consents
The Globus Web App - Hidden in Plain Sight
• The Hamburger Menu
– Not always the same – based on the type of endpoint and the storage behind it
– A great place to get the link to a share
• Transfer Settings
– label – when using the activity monitor it’s nice to see a recognizable name
– sync - only transfer new or changed files
– delete files on destination that do not exist on source
– preserve source file modification times
– verify file integrity after transfer
– encrypt transfer
• Search
– Look for the magnifying glass
– Search for: Endpoints / Users / Groups
Activity Monitoring
• Recent / History / Filter
• Drilling Down
– File transfer statistics
– Overview
– Event Log
– Cancelling an active task
Groups
• What can they be used for?
– Sharing: Access permissions for more than one person
– Roles: Endpoint management and monitoring
• Groups
– Creating groups and setting the visibility
– Members (invitations), Subgroups, Settings
– Settings
o Policies / Membership Fields / Terms & Conditions
– Roles
o Giving others authority over your groups
Endpoint Sharing and Roles
• Sharing
– Select the directory and create the “share”
– A ”share” is another type of endpoint
– Share with: Users / Groups / All Globus Users
• Roles
– Give others (users or groups) rights on your endpoint to:
o Monitor activity on the endpoint
o Manage activity on the endpoint (e.g., cancel, pause)
o Manage access control permissions on the endpoint
Bookmarks
• Just like browser bookmarks – frequently used, or
maybe not used frequently enough!
• Creating a bookmark
• Using a bookmark
• Sorting and Filtering
• Editing and Deleting
Globus Command Line Interface
Open source, uses
Python SDK
docs.globus.org/cli
github.com/globus/
globus-cli
The Globus CLI
• Installation
– docs.globus.org/cli/installation
– Prerequisites
• Logging On (remember the consents?)
– globus login / logout
• Getting help / list of commands
– globus –help
– globus list-commands
• Doing something
– It all about the UUIDs
– Don’t forget the file paths!
The Globus CLI – Let’s do a few things…
• Find endpoints
– globus endpoint search Midway
– globus endpoint search ESNet
– globus endpoint search --filter-scope=recently-used
• Find endpoint contents
– globus ls af7bda53-6d04-11e5-ba46-22000b92c6ec
– globus ls af7bda53-6d04-11e5-ba46-22000b92c6ec:RMACC2018
• Transfer a file
– From ESnet Read-Only Test DTN at CERN to Midway
– Note the specific paths
– globus transfer d8eb36b6-6d04-11e5-ba46-22000b92c6ec:/~/data1/1M.dat af7bda53-6d04-11e5-
ba46-22000b92c6ec:/~/1M.dat
• Transfer a directory
– From Globus Tutorial Endpoint 2 to Midway (create directory and contents)
– globus transfer --recursive ddb59af0-6d04-11e5-ba46-22000b92c6ec:/~/sync-demo af7bda53-
6d04-11e5-ba46-22000b92c6ec:/~/syncDemo
• https://docs.globus.org/cli/examples/
Building the Services
Ecosystem
Globus Auth enables
an integrated ecosystem
of services and applications
for the research community
23
Based on widely used web standards
• OAuth 2.0 Authorization Framework (a.k.a. OAuth2)
• OpenID Connect Core 1.0 (a.k.a. OIDC)
24
docs.globus.org/api/auth
What is a services ecosystem?
• Build services with secure REST APIs
• Services can leverage other services securely
App
Your Service
Globus Transfer
App Your Service
Globus Transfer
Other Service
Why create your own services?
• Make your specialized capabilities available to your
research community as a service
• Extend your web portal with a public REST API, so
that other developers can integrate with and extend it
• Front-end / back-end within your portal / app
– Remote backend for portal
– Backend for pure Javascript browser apps
26
Why Globus Auth for your service?
• Outsource all identity management and authentication
– Federated identity with InCommon, Google, etc.
• Outsource your REST API security
– Consent, token issuance, validation, revocation
– You provide service-specific authorization
• Apps use your service like all others, with standard OAuth2 & OIDC
• Your service can seamlessly leverage other services
• Other services can leverage your service
• Implement your service using any language and framework
Add your service to the science services ecosystem
27
Role of Globus Auth for services
• Issue and check OAuth2 access tokens
• With a token, your service can get attributes about
the user, which it can use to authorize the request
App
Your Service
Globus Transfer
Globus Auth
1. Issue
access token 3. Check
access token
2. Access
token
Fundamental Concepts
• Scopes: APIs that client is requesting access to
– Scope syntax: OpenID Connect: openid, email, profile
– https://auth.globus.org/scopes/<service-name>:<scope-name>
– A service can have multiple scopes
• Consents: authorize client to access a service, within
limited scope, on the resource owner’s (user’s) behalf
29
Globus account
• Globus Account = Primary identity + Linked Identities
– An identity can be primary on only one account
– (Currently) Identities can be linked to only one account
• Account does not have own identifier
– An account is uniquely identified using its primary identity
• Effective identity = linked identity from a particular
identity provider required by a client or service
30
Identity id vs. username
• Identity id
– Unique among all Globus Auth identities; will never be reused
– UUID
– Always use this to refer to an identity
• Identity username
– Unique at any point in time; may change, may be re-used
– Case-insensitive user@domain
– Can map to/from id, for user experience
• Auth API allows mapping back and forth
31
App registration
• Admin registers an App ( “confidential client”) with Globus Auth
– https://developers.globus.org
• Gets client_id and client_secret for service
• Sets app display name
• Declare required scopes (optional)
– Need long-term, offline refresh tokens?
– May require authorization from scope admin
• OAuth2 redirect URIs
• Links for terms of service & privacy policy
• Effective identity policy (optional)
32
developers.globus.org
Native App Auto-registration (new)
• New Native App bootstrap model
• Developer registers a native app template at
developers.globus.org
• Each installed instance of the native app auto-registers
itself on first use via Globus Auth Client API
– Registers with a native app template
– Gets a client id and secret for itself
– Securely stores client id and secret (e.g., into user read-only file)
• Native app is now a “confidential client”, just like any
other
Client
(Web Portal,
Application,
Jupyter)
Globus Transfer
(Resource Server)
Globus Auth
(Authorization
Server)
5. Authenticate using client id
and secret, send authorization
code
Authorization Code Grant
Browser (User)
1. Access
portal
2.
Redirects
user
3. User authenticates and
consents
4. Authorization
code
6. Access token(s)
7. Authenticate with access
token(s) to give the client
the authority invoke the
transfer service
Identity
Provider
OAuth2 grants for apps
• Authorization code grant
– Native app grant variant
• Refresh token grants
– For apps that need long-lived, “offline” access to a service
• Client credential grant
– For app invoking services as itself, instead of as the user
What is a services ecosystem?
• Build services with secure REST APIs
• Services can leverage other services securely
App
Your Service
Globus Transfer
App Your Service
Globus Transfer
Other Service
Service registration
• Client_id and client_secret for service
• Service display name
• Validated DNS name for service
• One or more scopes
– Who is authorized to use each scope: all client (public API) or specific clients
• Declare dependent scopes
– Need long-term, offline refresh tokens?
– May require authorization from scope admin
• Links for terms of service & privacy policy
• Effective identity policy (optional)
37
mailto:support@globus.org
Typical service interactions
• Service receives HTTPS request with header
– Authorization: Bearer <request-access-token>
• Introspects the request access token
– Auth API: POST /v2/oauth2/token/introspect
– Authorized by client_id and client_secret
– Returns: active, client_id, scope, sub (identity), identities_set
• Verifies token info (e.g., active, aud, scope)
• Authorizes request based on token info (e.g., sub)
• Service processes request
• Responds to client HTTPS request
38
Sample Research Data Portal
Service Walk-through
39
https://github.com/globus/globus-sample-data-portal/tree/master/service
Authorization based on identity set
• Use identities_set when authorizing a request based
on the resource owner associated with an access
token
– E.g., ACLs on Globus shared endpoints
• Authorizing based on set of identities is same
complexity as authorizing based on group
membership set
40
Groups
• Globus group service is
identity set aware
– “Tell me all groups for all
identities of the logged in user”
• Services can leverage this
for authorization (soon)
41
Django REST Framework
• Use Globus Auth to protect REST APIs in the Django
REST Framework
• Simple enhancements to do token introspection,
validation, and map to Django account
• Contact support@globus.org if you are interested
What is a services ecosystem?
• Build services with secure REST APIs
• Services can leverage other services securely
App
Your Service
Globus Transfer
App Your Service
Globus Transfer
Other Service
Examples uses of dependent services
• NCAR RDA REST API
– App à RDA à Globus Transfer
• Globus Transfer
– Globus Connect Server v5 expects Globus Auth access tokens
– App à Transfer à Collections
Dependent tokens
• Your service can act as client to other services (scopes)
– Globus Transfer, Search, Identities, Auth
– Other community services
• Entire service call tree consented by user and service
owners
– Rescinding consent revokes all dependent tokens
• Dependent tokens are restricted to a particular client,
calling a particular scope, on behalf of a particular
resource owner (e.g., user)
– Restricted delegation!
45
Typical service interactions
• Service receives HTTPS request with header
• Introspects the request access token
• Verifies token info (e.g., active, aud, scope)
• Authorizes request based on token info (e.g., sub)
• If service needs to act as client to other services:
– Calls Globus Auth Dependent Token Grant
o Returns a token for each dependent service
– Uses correct dependent token for downstream REST call
• Service processes request, including calls to other services
• Responds to client HTTPS request
46
Sessions
• Higher authentication assurance
– For services with PHI, PII,
sensitive but unclassified data
• Introspection response includes session attributes
– Which identities have authenticated in this “session”
– When each authenticated last
– Whether multi-factor authentication was used
• Return authentication assurance error if not sufficient
Refresh tokens
• For “offline service”: Service working on your behalf
even when you are offline
– Example use: Globus Transfer service, for async transfers
• Refresh tokens issued to a particular client for use with a
particular scope
• Client uses refresh token to get access token
– Client_id and client_secret required
• Refresh token good for 6 months after last use
• Consent rescindment revokes resource token
48
Token caching
• Service should cache tokens and related information
– Improves performance of service
– Reduces load on Globus Auth
• Access token -> introspect response
– Cache timeout: 1-30 seconds recommended
– To improve performance and load related to bursty use of REST API
– Validity: Timeout duration determines responsiveness to token revocation and
rescinding consent
• Access token -> dependent access tokens
– Cache timeout: lifetime of access token
– To avoid costly dependent token re-issuance
– Rescinding consent will invalidate everything
• Refresh tokens
– For however long they are needed for specific operations.
– Keep distinct refresh tokens for each access token.
49
Coming soon to Auth
• Incremental auth
– Add consents and tokens for new services dynamically
• Optional scopes
– Allow user to optionally deny access to a scope, but allow the
client to continue functioning with reduced capability
• SSH with Globus Auth
Leveraging the Globus
Platform in your Web
Applications
Globus serves as…
A platform for building science
gateways, web portals and
other applications in support of
research and education
52
Example web apps that leverage Globus
53
Globus Auth
(and Group Management)
…
GlobusAPIs
GlobusConnect
Data Automation, Publication
& Search
File Sharing
File Transfer & Replication
Globus Platform-as-a-Service
Use existing institutional
ID systems in external
web applications
Integrate file transfer and sharing
capabilities into scientific web
apps, portals, gateways, etc...
PaaS Security Challenges – Globus Auth
• How to provide:
– Login to apps
o Web apps (Jupyter Notebook, Portals), Mobile, Desktop, Command line
– Protect all REST API communications
o App à Globus service (Jupyter Notebook, MRDP)
o App à non-Globus service (MRDP)
o Service à service (MRDP)
• While:
– Not introducing even more identities
o Providing a platform to consolidate those identities
– Providing least privileges security model (consents)
– Being agnostic to programming language and framework
– Being web friendly
– Making it easy for users and developers
55
Client
(Web Portal,
Application,
Jupyter)
Globus Transfer
(Resource Server)
Globus Auth
(Authorization
Server)
5. Authenticate using client id
and secret, send authorization
code
Authorization Code Grant
Browser (User)
1. Access
portal
2.
Redirects
user
3. User authenticates and
consents
4. Authorization
code
6. Access token(s)
7. Authenticate with access
token(s) to give the client
the authority invoke the
transfer service
Identity
Provider
Globus Platform
Transfer API
Globus Transfer API
• Globus Web App consumes public Transfer API
• REST API: Resources/actions named by URL
– Query params allow refinement (e.g., filter)
• Globus APIs use JSON for documents and resource
representations
• Requests authorized via Globus Auth issued OAuth2
access token
– Authorization: Bearer asdflkqhafsdafeawk
docs.globus.org/api/transfer
58
Globus Python SDK
• Python client library for the Globus Auth and Transfer
REST APIs
• globus_sdk.TransferClient class handles
connection management, security, framing,
marshaling
from globus_sdk import TransferClient
tc = TransferClient()
globus.github.io/globus-sdk-python
59
TransferClient low-level calls
• Thin wrapper around REST API
– post(), get(), update(), delete()
get(path, params=None, headers=None, auth=None,
response_class=None)
o path – path for the request, with or without leading slash
o params – dict to be encoded as a query string
o headers – dict of HTTP headers to add to the request
o response_class – class response object, overrides the client’s
default_response_class
o Returns: GlobusHTTPResponse object
60
TransferClient higher-level calls
• One method for each API resource and HTTP verb
• Largely direct mapping to REST API
endpoint_search(filter_fulltext=None,
filter_scope=None,
num_results=25,
**params)
61
Walkthrough Jupyter Notebook with our Hub
• https://jupyter.demo.globus.org
– Sign in with Globus
– Verify the consents
– Start My Server (this will take about a minute)
– Open folder: globus-jupyter-notebooks
– Open folder: globusWorld2018
– Run SDK_autoAuth.ipynb
• If you mess it up and want to “go back to the beginning”
– Back down to the root folder
– Run NotebookPuller.ipynb
• If you want to use the notebook outside of our hub
– https://github.com/globus/globus-jupyter-notebooks
– Autentication is a manual cut and paste of exchanging the authorization
code for an access token
62
Endpoint Search
• Plain text search for endpoint
– Searches owner, display name, keywords, description,
organization, department
– Full word and prefix match
• Limit search to pre-defined scopes
– all, my-endpoints, recently-used, in-use, shared-
by-me, shared-with-me
• Returns: List of endpoint documents
63
Endpoint Management
• Get endpoint (by id)
• Update endpoint
• Create & delete (shared) endpoints
• Manage endpoint servers
64
Endpoint Activation
• Activating endpoint means binding a credential to an
endpoint for login
• Globus Connect Server endpoint that have MyProxy
or MyProxy OAuth identity provider require login via
web
• Auto-activate
– Globus Connect Personal and Shared endpoints use Globus-
provided credential
– Must auto-activate before any API calls to endpoints
65
File operations
• List directory contents (ls)
• Make directory (mkdir)
• Rename
• Note:
– Path encoding & UTF gotchas
– Don’t forget to auto-activate first
66
Task submission
• Asynchronous operations
– Transfer
o Sync level option
– Delete
• Get submission_id, followed by submit
– Once and only once submission
67
Task management
• Get task by id
• Get task_list
• Update task by id (label, deadline)
• Cancel task by id
• Get event list for task
• Get task pause info
68
Bookmarks
• Get list of bookmarks
• Create bookmark
• Get bookmark by id
• Update bookmark
• Delete bookmark by id
• Cannot perform other operations directly on bookmarks
– Requires client-side resolution
69
Shared endpoints and access rules (ACL)
• Shared Endpoint – create / delete / get info / get list
• Administrator role required to delegate access managers
• Access manager role required to manage
permissions/ACL
• Operations:
– Get list of access rules
– Get access rule by id
– Create access rule
– Update access rule
– Delete access rule
70
Management API
• Allow endpoint administrators to monitor and manage
all tasks with endpoint
– Task API is essentially the same as for users
– Information limited to what they could see locally
• Cancel tasks
• Pause rules
71
Globus Helper Pages
• Globus pages designed for use by your web apps
– Browse Endpoint
– Activate Endpoint
– Select Group
– Manage Identities
– Manage Consents
– Logout
docs.globus.org/api/helper-pages
72
Example
Modern Research
Data Portal
https://docs.globus.org/modern-research-data-portal/
Modern data apps leverage the Science DMZ
74
fasterdata.es.net/
10GE10GE
10GE
10GE
Border Router
WAN
Science DMZ
Switch/Router
Firewall
Enterprise
perfSONAR
perfSONAR
10GE
10GE
10GE
10GE
DTN
DTN
API DTNs
(data access governed
by portal)
DTN
DTN
perfSONAR
Filesystem
(data store)
10GE
Portal
Server
Browsing path
Query path
Portal server applications:
· web server
· search
· database
· authentication
Data Path
Data Transfer Path
Portal Query/Browse Path
Globus PaaS developer resources
Python SDK
Sample
Application
docs.globus.org/api github.com/globus
Jupyter Notebook
Now, about Jupyter…
Andre Schleife, UIUC
16,000 CPU-hours per simulation
Sample'Experimental'
sca0ering'
Material'
composi4on'
Simulated'
structure'
Simulated'
sca0ering'
Sr'40%'
Evolu4onary'op4miza4on'
786,432 CPUs, 10 PFLOPS supercomputer
Argonne Leadership Computing Facility
Modeling stopping power with time-dependent
density functional theory
@python_app
Logan Ward
Jupyter notebooks enable rapid iteration/results
But the data are big, distributed…
…and our science is collaborative
petrel.alcf.anl.gov
materialsdatafacility.org
2PB, 80Gbps store
3.2M materials data
Cooley: 290 TFLOPS
Query1 Share4
Transfer2
Learn3
Need multi-credential, multi-service authentication and data management
Hub
Configurable HTTP proxy
Authenticator
User DB
Spawner
Notebook
/api/auth
Browser
/hub/
/user/[name]/
• Multi-user hub
• Manages multiple instances
of Jupyter notebook server
• Configurable HTTP proxy
JupyterHub
Goal: Liberate the notebook!
• Tokens for remote services
• APIs for remote actions, e.g. data
management via Globus service
petrel.alcf.anl.gov
Securing JupyterHub with Globus Auth plugin
• Existing OAuth
framework
• Can restrict IdP
• Custom scopes
• Tokens passed into
notebook environment
github.com/jupyterhub/oauthenticator
github.com/jupyterhub/oauthenticator#globus-setup
Securing JupyterHub with Globus Auth
REST APIs
REST APIs
REST APIs
Bearer a45cd...
Hub
Configurable HTTP proxy
Authenticator
User DB
Spawner
Notebook
/api/auth
/hub/
/user/[name]/
login
Browser
{"tokens":...
{"tokens":...
Tokens in Jupyter notebooks
The world is your
oyster API…
• Globus Transfer
• Globus Search
• Your app
• Data portal
• Analysis engine
• …
Automated data analysis/results distribution
Notebook
Data
Repository
Bearer a45cd…
Dataset
Shared
endpoint
POST '/endpoint/a3c345f... /mkdir’
200 OK
...
X-Transfer-API-Version: 0.10
Content-Type: application/json
...
Analyze
Experiment with the demo notebook
• Login into our JupyterHub*: jupyter.demo.globus.org
• Launch (spawn) a notebook server; get tokens
• Access Globus APIs; download some data
• “Analyze” data (generate plot)
• PUT results (graph) on an HTTPS endpoint
*zero-to-jupyterhub.readthedocs.io
Futures…
Juan David Garrido
Browse data on
local storage
Search remote
storage systems
Select files on
remote storage
Transfer data to
local storage
There it is!
Search remote
databases
Select materials
data
Inspect materials
data
Incorporate seamless parallel computing
• (Data) science apps require…
– Interactivity
– Scalability (more than a desktop)
– Reproducibility (publish code, docs)
• Solution: JupyterHub + Parsl
– Interactive computing environment
– Notebooks for publication
– Can run on dedicated hardware
@python_app
def compute_features(chunk):
for f in featurizers:
chunk = f.featurize_dataframe(chunk, 'atoms’)
return chunk
chunks = [compute_features(chunk)
for chunk in np.array_split(data, chunks)]
Python parallel library
• Tasks exposed as functions (Python, bash)
• Python code to glue functions together
• Globus for auth and data movement
parsl-project.org
Containerized data science ecosystem
Auth
Transfer
Search
container metadata
container recipes
containers
ALCF Petrel
Uniform IAM
platform scalable for
distributed objectsUsers select container
to execute custom
Jupyter environment
Registry for
container discovery
and referencing
Container definitions
tracked in version
control systems
Containers used
for both Jupyter
notebook server
and compute
nodes
Containers used other
tasks; analysis, ML, etc.Supercomputer
Compute
Compute
Compute
Compute
Containers staged to
local file systems
Container
Registry
Notebook Server
Support resources
• Globus documentation: docs.globus.org
• Sample code: github.com/globus
• Helpdesk and issue escalation: support@globus.org
• Customer engagement team
• Globus professional services team
– Assist with portal/gateway/app architecture and design
– Develop custom applications that leverage the Globus platform
– Advise on customized deployment and integration scenarios
Join the Globus community
• Access the service: globus.org/login
• Create a personal endpoint: globus.org/app/endpoints/create-gcp
• Documentation: docs.globus.org
• Engage: globus.org/mailing-lists
• Subscribe: globus.org/subscriptions
• Need help? support@globus.org
• Follow us: @globusonline

More Related Content

What's hot

GlobusWorld 2021 Tutorial: The Globus CLI, Platform and SDK
GlobusWorld 2021 Tutorial: The Globus CLI, Platform and SDKGlobusWorld 2021 Tutorial: The Globus CLI, Platform and SDK
GlobusWorld 2021 Tutorial: The Globus CLI, Platform and SDK
Globus
 
Tutorial: Managing Protected Data with Globus Connect Server v5
Tutorial: Managing Protected Data with Globus Connect Server v5Tutorial: Managing Protected Data with Globus Connect Server v5
Tutorial: Managing Protected Data with Globus Connect Server v5
Globus
 
Globus for System Administrators
Globus for System AdministratorsGlobus for System Administrators
Globus for System Administrators
Globus
 
Leveraging the Globus Platform in Web Applications (CHPC 2019 - South Africa)
Leveraging the Globus Platform in Web Applications (CHPC 2019 - South Africa)Leveraging the Globus Platform in Web Applications (CHPC 2019 - South Africa)
Leveraging the Globus Platform in Web Applications (CHPC 2019 - South Africa)
Globus
 
Globus Platform Overview
Globus Platform OverviewGlobus Platform Overview
Globus Platform Overview
Globus
 
Introduction to the Globus Platform (GlobusWorld Tour - UMich)
Introduction to the Globus Platform (GlobusWorld Tour - UMich)Introduction to the Globus Platform (GlobusWorld Tour - UMich)
Introduction to the Globus Platform (GlobusWorld Tour - UMich)
Globus
 
Managing Protected and Controlled Data with Globus
Managing Protected and Controlled Data with Globus Managing Protected and Controlled Data with Globus
Managing Protected and Controlled Data with Globus
Globus
 
Globus for System Administrators (GlobusWorld Tour - UCSD)
Globus for System Administrators (GlobusWorld Tour - UCSD)Globus for System Administrators (GlobusWorld Tour - UCSD)
Globus for System Administrators (GlobusWorld Tour - UCSD)
Globus
 
How AD has been re-engineered to extend to the cloud
How AD has been re-engineered to extend to the cloudHow AD has been re-engineered to extend to the cloud
How AD has been re-engineered to extend to the cloud
LDAPCon
 
GlobusWorld 2021 Tutorial: Introduction to Globus
GlobusWorld 2021 Tutorial: Introduction to GlobusGlobusWorld 2021 Tutorial: Introduction to Globus
GlobusWorld 2021 Tutorial: Introduction to Globus
Globus
 
Introduction to Globus for New Users (GlobusWorld Tour - UCSD)
Introduction to Globus for New Users (GlobusWorld Tour - UCSD)Introduction to Globus for New Users (GlobusWorld Tour - UCSD)
Introduction to Globus for New Users (GlobusWorld Tour - UCSD)
Globus
 
Globus Endpoint Setup and Configuration - XSEDE14 Tutorial
Globus Endpoint Setup and Configuration - XSEDE14 TutorialGlobus Endpoint Setup and Configuration - XSEDE14 Tutorial
Globus Endpoint Setup and Configuration - XSEDE14 Tutorial
Globus
 
Linux AD integration with OpenDJ
Linux AD integration with OpenDJLinux AD integration with OpenDJ
Linux AD integration with OpenDJ
Pieter Baele
 
Synchronize AD and OpenLDAP with LSC
Synchronize AD and OpenLDAP with LSCSynchronize AD and OpenLDAP with LSC
Synchronize AD and OpenLDAP with LSC
LDAPCon
 
Tutorial: Automating Research Data Workflows
Tutorial: Automating Research Data WorkflowsTutorial: Automating Research Data Workflows
Tutorial: Automating Research Data Workflows
Globus
 
Building Open Source Identity Management with FreeIPA
Building Open Source Identity Management with FreeIPABuilding Open Source Identity Management with FreeIPA
Building Open Source Identity Management with FreeIPA
LDAPCon
 
Bridging the gap: Adding missing client (security) features using OpenLDAP pr...
Bridging the gap: Adding missing client (security) features using OpenLDAP pr...Bridging the gap: Adding missing client (security) features using OpenLDAP pr...
Bridging the gap: Adding missing client (security) features using OpenLDAP pr...
LDAPCon
 
Introduction to Globus: Research Data Management Software at the ALCF
Introduction to Globus: Research Data Management Software at the ALCFIntroduction to Globus: Research Data Management Software at the ALCF
Introduction to Globus: Research Data Management Software at the ALCF
Globus
 
Automating Data Flows with the Globus CLI (GlobusWorld Tour - UMich)
Automating Data Flows with the Globus CLI (GlobusWorld Tour - UMich)Automating Data Flows with the Globus CLI (GlobusWorld Tour - UMich)
Automating Data Flows with the Globus CLI (GlobusWorld Tour - UMich)
Globus
 
Securing Your Deployment with MongoDB and Red Hat's Identity Management in Re...
Securing Your Deployment with MongoDB and Red Hat's Identity Management in Re...Securing Your Deployment with MongoDB and Red Hat's Identity Management in Re...
Securing Your Deployment with MongoDB and Red Hat's Identity Management in Re...
MongoDB
 

What's hot (20)

GlobusWorld 2021 Tutorial: The Globus CLI, Platform and SDK
GlobusWorld 2021 Tutorial: The Globus CLI, Platform and SDKGlobusWorld 2021 Tutorial: The Globus CLI, Platform and SDK
GlobusWorld 2021 Tutorial: The Globus CLI, Platform and SDK
 
Tutorial: Managing Protected Data with Globus Connect Server v5
Tutorial: Managing Protected Data with Globus Connect Server v5Tutorial: Managing Protected Data with Globus Connect Server v5
Tutorial: Managing Protected Data with Globus Connect Server v5
 
Globus for System Administrators
Globus for System AdministratorsGlobus for System Administrators
Globus for System Administrators
 
Leveraging the Globus Platform in Web Applications (CHPC 2019 - South Africa)
Leveraging the Globus Platform in Web Applications (CHPC 2019 - South Africa)Leveraging the Globus Platform in Web Applications (CHPC 2019 - South Africa)
Leveraging the Globus Platform in Web Applications (CHPC 2019 - South Africa)
 
Globus Platform Overview
Globus Platform OverviewGlobus Platform Overview
Globus Platform Overview
 
Introduction to the Globus Platform (GlobusWorld Tour - UMich)
Introduction to the Globus Platform (GlobusWorld Tour - UMich)Introduction to the Globus Platform (GlobusWorld Tour - UMich)
Introduction to the Globus Platform (GlobusWorld Tour - UMich)
 
Managing Protected and Controlled Data with Globus
Managing Protected and Controlled Data with Globus Managing Protected and Controlled Data with Globus
Managing Protected and Controlled Data with Globus
 
Globus for System Administrators (GlobusWorld Tour - UCSD)
Globus for System Administrators (GlobusWorld Tour - UCSD)Globus for System Administrators (GlobusWorld Tour - UCSD)
Globus for System Administrators (GlobusWorld Tour - UCSD)
 
How AD has been re-engineered to extend to the cloud
How AD has been re-engineered to extend to the cloudHow AD has been re-engineered to extend to the cloud
How AD has been re-engineered to extend to the cloud
 
GlobusWorld 2021 Tutorial: Introduction to Globus
GlobusWorld 2021 Tutorial: Introduction to GlobusGlobusWorld 2021 Tutorial: Introduction to Globus
GlobusWorld 2021 Tutorial: Introduction to Globus
 
Introduction to Globus for New Users (GlobusWorld Tour - UCSD)
Introduction to Globus for New Users (GlobusWorld Tour - UCSD)Introduction to Globus for New Users (GlobusWorld Tour - UCSD)
Introduction to Globus for New Users (GlobusWorld Tour - UCSD)
 
Globus Endpoint Setup and Configuration - XSEDE14 Tutorial
Globus Endpoint Setup and Configuration - XSEDE14 TutorialGlobus Endpoint Setup and Configuration - XSEDE14 Tutorial
Globus Endpoint Setup and Configuration - XSEDE14 Tutorial
 
Linux AD integration with OpenDJ
Linux AD integration with OpenDJLinux AD integration with OpenDJ
Linux AD integration with OpenDJ
 
Synchronize AD and OpenLDAP with LSC
Synchronize AD and OpenLDAP with LSCSynchronize AD and OpenLDAP with LSC
Synchronize AD and OpenLDAP with LSC
 
Tutorial: Automating Research Data Workflows
Tutorial: Automating Research Data WorkflowsTutorial: Automating Research Data Workflows
Tutorial: Automating Research Data Workflows
 
Building Open Source Identity Management with FreeIPA
Building Open Source Identity Management with FreeIPABuilding Open Source Identity Management with FreeIPA
Building Open Source Identity Management with FreeIPA
 
Bridging the gap: Adding missing client (security) features using OpenLDAP pr...
Bridging the gap: Adding missing client (security) features using OpenLDAP pr...Bridging the gap: Adding missing client (security) features using OpenLDAP pr...
Bridging the gap: Adding missing client (security) features using OpenLDAP pr...
 
Introduction to Globus: Research Data Management Software at the ALCF
Introduction to Globus: Research Data Management Software at the ALCFIntroduction to Globus: Research Data Management Software at the ALCF
Introduction to Globus: Research Data Management Software at the ALCF
 
Automating Data Flows with the Globus CLI (GlobusWorld Tour - UMich)
Automating Data Flows with the Globus CLI (GlobusWorld Tour - UMich)Automating Data Flows with the Globus CLI (GlobusWorld Tour - UMich)
Automating Data Flows with the Globus CLI (GlobusWorld Tour - UMich)
 
Securing Your Deployment with MongoDB and Red Hat's Identity Management in Re...
Securing Your Deployment with MongoDB and Red Hat's Identity Management in Re...Securing Your Deployment with MongoDB and Red Hat's Identity Management in Re...
Securing Your Deployment with MongoDB and Red Hat's Identity Management in Re...
 

Similar to Jupyter + Globus: The Foundation for Interactive Data Science

Tutorial: Leveraging Globus in your Research Applications
Tutorial: Leveraging Globus in your Research ApplicationsTutorial: Leveraging Globus in your Research Applications
Tutorial: Leveraging Globus in your Research Applications
Globus
 
Gateways 2020 Tutorial - Introduction to Globus
Gateways 2020 Tutorial - Introduction to GlobusGateways 2020 Tutorial - Introduction to Globus
Gateways 2020 Tutorial - Introduction to Globus
Globus
 
Facilitating Collaboration with Globus (GlobusWorld Tour - STFC)
Facilitating Collaboration with Globus (GlobusWorld Tour - STFC)Facilitating Collaboration with Globus (GlobusWorld Tour - STFC)
Facilitating Collaboration with Globus (GlobusWorld Tour - STFC)
Globus
 
"What's New With Globus" Webinar: Spring 2018
"What's New With Globus" Webinar: Spring 2018"What's New With Globus" Webinar: Spring 2018
"What's New With Globus" Webinar: Spring 2018
Globus
 
Leveraging the Globus Platform (GlobusWorld Tour - Columbia University)
Leveraging the Globus Platform (GlobusWorld Tour - Columbia University)Leveraging the Globus Platform (GlobusWorld Tour - Columbia University)
Leveraging the Globus Platform (GlobusWorld Tour - Columbia University)
Globus
 
Tutorial: Best Practices for Data Sharing
Tutorial: Best Practices for Data SharingTutorial: Best Practices for Data Sharing
Tutorial: Best Practices for Data Sharing
Globus
 
Introduction to Globus for New Users
Introduction to Globus for New UsersIntroduction to Globus for New Users
Introduction to Globus for New Users
Globus
 
Best Practices for Data Sharing (GlobusWorld Tour - Columbia University)
Best Practices for Data Sharing (GlobusWorld Tour - Columbia University)Best Practices for Data Sharing (GlobusWorld Tour - Columbia University)
Best Practices for Data Sharing (GlobusWorld Tour - Columbia University)
Globus
 
Building Data Portals and Science Gateways with Globus
Building Data Portals and Science Gateways with GlobusBuilding Data Portals and Science Gateways with Globus
Building Data Portals and Science Gateways with Globus
Globus
 
Gateways 2020 Tutorial - Large Scale Data Transfer with Globus
Gateways 2020 Tutorial - Large Scale Data Transfer with GlobusGateways 2020 Tutorial - Large Scale Data Transfer with Globus
Gateways 2020 Tutorial - Large Scale Data Transfer with Globus
Globus
 
Automating Research Data Flows and Introduction to the Globus Platform
Automating Research Data Flows and Introduction to the Globus PlatformAutomating Research Data Flows and Introduction to the Globus Platform
Automating Research Data Flows and Introduction to the Globus Platform
Globus
 
Globus: Research Data Management as Service and Platform - pearc17
Globus: Research Data Management as Service and Platform - pearc17Globus: Research Data Management as Service and Platform - pearc17
Globus: Research Data Management as Service and Platform - pearc17
Mary Bass
 
Automating Research Data Flows and an Introduction to the Globus Platform
Automating Research Data Flows and an Introduction to the Globus PlatformAutomating Research Data Flows and an Introduction to the Globus Platform
Automating Research Data Flows and an Introduction to the Globus Platform
Globus
 
Leveraging the Globus Platform (GlobusWorld Tour - UCSD)
Leveraging the Globus Platform (GlobusWorld Tour - UCSD)Leveraging the Globus Platform (GlobusWorld Tour - UCSD)
Leveraging the Globus Platform (GlobusWorld Tour - UCSD)
Globus
 
Introduction to the Globus Platform (APS Workshop)
Introduction to the Globus Platform (APS Workshop)Introduction to the Globus Platform (APS Workshop)
Introduction to the Globus Platform (APS Workshop)
Globus
 
Introduction to Globus for New Users
Introduction to Globus for New UsersIntroduction to Globus for New Users
Introduction to Globus for New Users
Globus
 
Automating Research Data Workflows (GlobusWorld Tour - Columbia University)
Automating Research Data Workflows (GlobusWorld Tour - Columbia University)Automating Research Data Workflows (GlobusWorld Tour - Columbia University)
Automating Research Data Workflows (GlobusWorld Tour - Columbia University)
Globus
 
Scalable Data Management: Automation and the Modern Research Data Portal
Scalable Data Management: Automation and the Modern Research Data PortalScalable Data Management: Automation and the Modern Research Data Portal
Scalable Data Management: Automation and the Modern Research Data Portal
Globus
 
Introduction to the Globus PaaS (GlobusWorld Tour - STFC)
Introduction to the Globus PaaS (GlobusWorld Tour - STFC)Introduction to the Globus PaaS (GlobusWorld Tour - STFC)
Introduction to the Globus PaaS (GlobusWorld Tour - STFC)
Globus
 
Globus: Beyond File Transfer
Globus: Beyond File TransferGlobus: Beyond File Transfer
Globus: Beyond File Transfer
Globus
 

Similar to Jupyter + Globus: The Foundation for Interactive Data Science (20)

Tutorial: Leveraging Globus in your Research Applications
Tutorial: Leveraging Globus in your Research ApplicationsTutorial: Leveraging Globus in your Research Applications
Tutorial: Leveraging Globus in your Research Applications
 
Gateways 2020 Tutorial - Introduction to Globus
Gateways 2020 Tutorial - Introduction to GlobusGateways 2020 Tutorial - Introduction to Globus
Gateways 2020 Tutorial - Introduction to Globus
 
Facilitating Collaboration with Globus (GlobusWorld Tour - STFC)
Facilitating Collaboration with Globus (GlobusWorld Tour - STFC)Facilitating Collaboration with Globus (GlobusWorld Tour - STFC)
Facilitating Collaboration with Globus (GlobusWorld Tour - STFC)
 
"What's New With Globus" Webinar: Spring 2018
"What's New With Globus" Webinar: Spring 2018"What's New With Globus" Webinar: Spring 2018
"What's New With Globus" Webinar: Spring 2018
 
Leveraging the Globus Platform (GlobusWorld Tour - Columbia University)
Leveraging the Globus Platform (GlobusWorld Tour - Columbia University)Leveraging the Globus Platform (GlobusWorld Tour - Columbia University)
Leveraging the Globus Platform (GlobusWorld Tour - Columbia University)
 
Tutorial: Best Practices for Data Sharing
Tutorial: Best Practices for Data SharingTutorial: Best Practices for Data Sharing
Tutorial: Best Practices for Data Sharing
 
Introduction to Globus for New Users
Introduction to Globus for New UsersIntroduction to Globus for New Users
Introduction to Globus for New Users
 
Best Practices for Data Sharing (GlobusWorld Tour - Columbia University)
Best Practices for Data Sharing (GlobusWorld Tour - Columbia University)Best Practices for Data Sharing (GlobusWorld Tour - Columbia University)
Best Practices for Data Sharing (GlobusWorld Tour - Columbia University)
 
Building Data Portals and Science Gateways with Globus
Building Data Portals and Science Gateways with GlobusBuilding Data Portals and Science Gateways with Globus
Building Data Portals and Science Gateways with Globus
 
Gateways 2020 Tutorial - Large Scale Data Transfer with Globus
Gateways 2020 Tutorial - Large Scale Data Transfer with GlobusGateways 2020 Tutorial - Large Scale Data Transfer with Globus
Gateways 2020 Tutorial - Large Scale Data Transfer with Globus
 
Automating Research Data Flows and Introduction to the Globus Platform
Automating Research Data Flows and Introduction to the Globus PlatformAutomating Research Data Flows and Introduction to the Globus Platform
Automating Research Data Flows and Introduction to the Globus Platform
 
Globus: Research Data Management as Service and Platform - pearc17
Globus: Research Data Management as Service and Platform - pearc17Globus: Research Data Management as Service and Platform - pearc17
Globus: Research Data Management as Service and Platform - pearc17
 
Automating Research Data Flows and an Introduction to the Globus Platform
Automating Research Data Flows and an Introduction to the Globus PlatformAutomating Research Data Flows and an Introduction to the Globus Platform
Automating Research Data Flows and an Introduction to the Globus Platform
 
Leveraging the Globus Platform (GlobusWorld Tour - UCSD)
Leveraging the Globus Platform (GlobusWorld Tour - UCSD)Leveraging the Globus Platform (GlobusWorld Tour - UCSD)
Leveraging the Globus Platform (GlobusWorld Tour - UCSD)
 
Introduction to the Globus Platform (APS Workshop)
Introduction to the Globus Platform (APS Workshop)Introduction to the Globus Platform (APS Workshop)
Introduction to the Globus Platform (APS Workshop)
 
Introduction to Globus for New Users
Introduction to Globus for New UsersIntroduction to Globus for New Users
Introduction to Globus for New Users
 
Automating Research Data Workflows (GlobusWorld Tour - Columbia University)
Automating Research Data Workflows (GlobusWorld Tour - Columbia University)Automating Research Data Workflows (GlobusWorld Tour - Columbia University)
Automating Research Data Workflows (GlobusWorld Tour - Columbia University)
 
Scalable Data Management: Automation and the Modern Research Data Portal
Scalable Data Management: Automation and the Modern Research Data PortalScalable Data Management: Automation and the Modern Research Data Portal
Scalable Data Management: Automation and the Modern Research Data Portal
 
Introduction to the Globus PaaS (GlobusWorld Tour - STFC)
Introduction to the Globus PaaS (GlobusWorld Tour - STFC)Introduction to the Globus PaaS (GlobusWorld Tour - STFC)
Introduction to the Globus PaaS (GlobusWorld Tour - STFC)
 
Globus: Beyond File Transfer
Globus: Beyond File TransferGlobus: Beyond File Transfer
Globus: Beyond File Transfer
 

More from Globus

Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus
 
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
Globus
 
Globus Compute Introduction - GlobusWorld 2024
Globus Compute Introduction - GlobusWorld 2024Globus Compute Introduction - GlobusWorld 2024
Globus Compute Introduction - GlobusWorld 2024
Globus
 
Globus Connect Server Deep Dive - GlobusWorld 2024
Globus Connect Server Deep Dive - GlobusWorld 2024Globus Connect Server Deep Dive - GlobusWorld 2024
Globus Connect Server Deep Dive - GlobusWorld 2024
Globus
 
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...
Globus
 
Providing Globus Services to Users of JASMIN for Environmental Data Analysis
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisProviding Globus Services to Users of JASMIN for Environmental Data Analysis
Providing Globus Services to Users of JASMIN for Environmental Data Analysis
Globus
 
First Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User EndpointsFirst Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User Endpoints
Globus
 
Enhancing Research Orchestration Capabilities at ORNL.pdf
Enhancing Research Orchestration Capabilities at ORNL.pdfEnhancing Research Orchestration Capabilities at ORNL.pdf
Enhancing Research Orchestration Capabilities at ORNL.pdf
Globus
 
Understanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSageUnderstanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSage
Globus
 
How to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good PracticesHow to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good Practices
Globus
 
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
Globus
 
Developing Distributed High-performance Computing Capabilities of an Open Sci...
Developing Distributed High-performance Computing Capabilities of an Open Sci...Developing Distributed High-performance Computing Capabilities of an Open Sci...
Developing Distributed High-performance Computing Capabilities of an Open Sci...
Globus
 
The Department of Energy's Integrated Research Infrastructure (IRI)
The Department of Energy's Integrated Research Infrastructure (IRI)The Department of Energy's Integrated Research Infrastructure (IRI)
The Department of Energy's Integrated Research Infrastructure (IRI)
Globus
 
GlobusWorld 2024 Opening Keynote session
GlobusWorld 2024 Opening Keynote sessionGlobusWorld 2024 Opening Keynote session
GlobusWorld 2024 Opening Keynote session
Globus
 
Enhancing Performance with Globus and the Science DMZ
Enhancing Performance with Globus and the Science DMZEnhancing Performance with Globus and the Science DMZ
Enhancing Performance with Globus and the Science DMZ
Globus
 
Extending Globus into a Site-wide Automated Data Infrastructure.pdf
Extending Globus into a Site-wide Automated Data Infrastructure.pdfExtending Globus into a Site-wide Automated Data Infrastructure.pdf
Extending Globus into a Site-wide Automated Data Infrastructure.pdf
Globus
 
Globus at the United States Geological Survey
Globus at the United States Geological SurveyGlobus at the United States Geological Survey
Globus at the United States Geological Survey
Globus
 
Providing Globus Services to Users of JASMIN for Environmental Data Analysis
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisProviding Globus Services to Users of JASMIN for Environmental Data Analysis
Providing Globus Services to Users of JASMIN for Environmental Data Analysis
Globus
 
Globus Compute with Integrated Research Infrastructure (IRI) workflows
Globus Compute with Integrated Research Infrastructure (IRI) workflowsGlobus Compute with Integrated Research Infrastructure (IRI) workflows
Globus Compute with Integrated Research Infrastructure (IRI) workflows
Globus
 
Reactive Documents and Computational Pipelines - Bridging the Gap
Reactive Documents and Computational Pipelines - Bridging the GapReactive Documents and Computational Pipelines - Bridging the Gap
Reactive Documents and Computational Pipelines - Bridging the Gap
Globus
 

More from Globus (20)

Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus Compute wth IRI Workflows - GlobusWorld 2024
 
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
 
Globus Compute Introduction - GlobusWorld 2024
Globus Compute Introduction - GlobusWorld 2024Globus Compute Introduction - GlobusWorld 2024
Globus Compute Introduction - GlobusWorld 2024
 
Globus Connect Server Deep Dive - GlobusWorld 2024
Globus Connect Server Deep Dive - GlobusWorld 2024Globus Connect Server Deep Dive - GlobusWorld 2024
Globus Connect Server Deep Dive - GlobusWorld 2024
 
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...
 
Providing Globus Services to Users of JASMIN for Environmental Data Analysis
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisProviding Globus Services to Users of JASMIN for Environmental Data Analysis
Providing Globus Services to Users of JASMIN for Environmental Data Analysis
 
First Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User EndpointsFirst Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User Endpoints
 
Enhancing Research Orchestration Capabilities at ORNL.pdf
Enhancing Research Orchestration Capabilities at ORNL.pdfEnhancing Research Orchestration Capabilities at ORNL.pdf
Enhancing Research Orchestration Capabilities at ORNL.pdf
 
Understanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSageUnderstanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSage
 
How to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good PracticesHow to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good Practices
 
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
 
Developing Distributed High-performance Computing Capabilities of an Open Sci...
Developing Distributed High-performance Computing Capabilities of an Open Sci...Developing Distributed High-performance Computing Capabilities of an Open Sci...
Developing Distributed High-performance Computing Capabilities of an Open Sci...
 
The Department of Energy's Integrated Research Infrastructure (IRI)
The Department of Energy's Integrated Research Infrastructure (IRI)The Department of Energy's Integrated Research Infrastructure (IRI)
The Department of Energy's Integrated Research Infrastructure (IRI)
 
GlobusWorld 2024 Opening Keynote session
GlobusWorld 2024 Opening Keynote sessionGlobusWorld 2024 Opening Keynote session
GlobusWorld 2024 Opening Keynote session
 
Enhancing Performance with Globus and the Science DMZ
Enhancing Performance with Globus and the Science DMZEnhancing Performance with Globus and the Science DMZ
Enhancing Performance with Globus and the Science DMZ
 
Extending Globus into a Site-wide Automated Data Infrastructure.pdf
Extending Globus into a Site-wide Automated Data Infrastructure.pdfExtending Globus into a Site-wide Automated Data Infrastructure.pdf
Extending Globus into a Site-wide Automated Data Infrastructure.pdf
 
Globus at the United States Geological Survey
Globus at the United States Geological SurveyGlobus at the United States Geological Survey
Globus at the United States Geological Survey
 
Providing Globus Services to Users of JASMIN for Environmental Data Analysis
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisProviding Globus Services to Users of JASMIN for Environmental Data Analysis
Providing Globus Services to Users of JASMIN for Environmental Data Analysis
 
Globus Compute with Integrated Research Infrastructure (IRI) workflows
Globus Compute with Integrated Research Infrastructure (IRI) workflowsGlobus Compute with Integrated Research Infrastructure (IRI) workflows
Globus Compute with Integrated Research Infrastructure (IRI) workflows
 
Reactive Documents and Computational Pipelines - Bridging the Gap
Reactive Documents and Computational Pipelines - Bridging the GapReactive Documents and Computational Pipelines - Bridging the Gap
Reactive Documents and Computational Pipelines - Bridging the Gap
 

Recently uploaded

Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Tatiana Kojar
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
Zilliz
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Brandon Minnick, MBA
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Safe Software
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Chart Kalyan
 
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024
Jason Packer
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
panagenda
 
Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)
Jakub Marek
 
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
alexjohnson7307
 
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their MainframeDigital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Precisely
 
GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)
Javier Junquera
 
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Jeffrey Haguewood
 
Serial Arm Control in Real Time Presentation
Serial Arm Control in Real Time PresentationSerial Arm Control in Real Time Presentation
Serial Arm Control in Real Time Presentation
tolgahangng
 
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERPBest 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
Pixlogix Infotech
 
Skybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoptionSkybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoption
Tatiana Kojar
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
Miro Wengner
 
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development ProvidersYour One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
akankshawande
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
AstuteBusiness
 
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing InstancesEnergy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Alpen-Adria-Universität
 

Recently uploaded (20)

Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
 
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
 
Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)
 
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
 
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their MainframeDigital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
 
GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)
 
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
 
Serial Arm Control in Real Time Presentation
Serial Arm Control in Real Time PresentationSerial Arm Control in Real Time Presentation
Serial Arm Control in Real Time Presentation
 
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERPBest 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
 
Skybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoptionSkybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoption
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
 
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development ProvidersYour One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
 
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing InstancesEnergy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
 

Jupyter + Globus: The Foundation for Interactive Data Science

  • 1. Jupyter + Globus: The Foundation for Interactive Data Science Rick Wagner rick@globus.org Gatways 2018 – September 25, 2018
  • 3. 3 Research Computing HPC Desktop Workstations Mass Storage Instruments Personal Resources Public Cloud National Resources Unify access to data across tiers
  • 4. IBM Spectrum Scale Current Planned Storage Connectors - globus.org/connectors
  • 5. Public / private cloud stores External campus storage EC2 Project repositories, replication stores Public repositories Share with collaborators/community
  • 6. Analysis store Next-Gen Sequencer MRI Advanced Light Source Personal system Remote visualization Light Sheet Microscope High-durability, low-cost store Manage data from instruments Cryo-EM
  • 7. Develop apps, services, and workflows 7
  • 8. 8,000 active shared endpoints 90 subscribers 425 PB transferred 18,000 active GCP endpoints 70 billion files processed 1,700 active GCS endpoints 3 months longest running transfer 1 PB largest single transfer to date 99.9% availability 500 identity providers 1,042 most shared endpoints at a single institution 100,000 users Globus by the numbers
  • 9. Conceptual architecture: Hybrid SaaS DATA Channel CONTROL Channel Source Endpoint Destination Endpoint Subscriber owned and administered storage system Globus “connector” software No data relay or staging via Globus cloud service Subscriber Control Domain Globus Control Domain Single, globally accessible multi-tenant service
  • 10. Conceptual architecture: Sharing Managed Endpoint Subscriber Control Domain Globus Control Domain Globus managed ”overlay” permissions Shared Endpoint DATA Channel CONTROL Channel Subscriber managed filesystem permissions External User Control Domain
  • 11. Endpoints (Collections) • Storage abstraction – All transfers happen between two endpoints – Globus Connect instantiates endpoints • Collection ~= Endpoint • Test / Demo Endpoints – Globus Tutorial Endpoint 1 – Globus Tutorial Endpoint 2 – ESnet Test Endpoints o Contain file samples of various sizes • Globus Connect Personal – Now your laptop is an endpoint – https://www.globus.org/globus-connect-personal 11
  • 12. Globus Connect Personal • Installers do not require admin access • Zero configuration; auto updating • Handles NATs • Installs in seconds – easy to delete - I’ll prove it!
  • 13. The Globus Web App - Accounts • A Globus Account is – A Primary Identity – Possible Linked Identities • Linking Identities • Managing Identities • Consents
  • 14. The Globus Web App - Hidden in Plain Sight • The Hamburger Menu – Not always the same – based on the type of endpoint and the storage behind it – A great place to get the link to a share • Transfer Settings – label – when using the activity monitor it’s nice to see a recognizable name – sync - only transfer new or changed files – delete files on destination that do not exist on source – preserve source file modification times – verify file integrity after transfer – encrypt transfer • Search – Look for the magnifying glass – Search for: Endpoints / Users / Groups
  • 15. Activity Monitoring • Recent / History / Filter • Drilling Down – File transfer statistics – Overview – Event Log – Cancelling an active task
  • 16. Groups • What can they be used for? – Sharing: Access permissions for more than one person – Roles: Endpoint management and monitoring • Groups – Creating groups and setting the visibility – Members (invitations), Subgroups, Settings – Settings o Policies / Membership Fields / Terms & Conditions – Roles o Giving others authority over your groups
  • 17. Endpoint Sharing and Roles • Sharing – Select the directory and create the “share” – A ”share” is another type of endpoint – Share with: Users / Groups / All Globus Users • Roles – Give others (users or groups) rights on your endpoint to: o Monitor activity on the endpoint o Manage activity on the endpoint (e.g., cancel, pause) o Manage access control permissions on the endpoint
  • 18. Bookmarks • Just like browser bookmarks – frequently used, or maybe not used frequently enough! • Creating a bookmark • Using a bookmark • Sorting and Filtering • Editing and Deleting
  • 19. Globus Command Line Interface Open source, uses Python SDK docs.globus.org/cli github.com/globus/ globus-cli
  • 20. The Globus CLI • Installation – docs.globus.org/cli/installation – Prerequisites • Logging On (remember the consents?) – globus login / logout • Getting help / list of commands – globus –help – globus list-commands • Doing something – It all about the UUIDs – Don’t forget the file paths!
  • 21. The Globus CLI – Let’s do a few things… • Find endpoints – globus endpoint search Midway – globus endpoint search ESNet – globus endpoint search --filter-scope=recently-used • Find endpoint contents – globus ls af7bda53-6d04-11e5-ba46-22000b92c6ec – globus ls af7bda53-6d04-11e5-ba46-22000b92c6ec:RMACC2018 • Transfer a file – From ESnet Read-Only Test DTN at CERN to Midway – Note the specific paths – globus transfer d8eb36b6-6d04-11e5-ba46-22000b92c6ec:/~/data1/1M.dat af7bda53-6d04-11e5- ba46-22000b92c6ec:/~/1M.dat • Transfer a directory – From Globus Tutorial Endpoint 2 to Midway (create directory and contents) – globus transfer --recursive ddb59af0-6d04-11e5-ba46-22000b92c6ec:/~/sync-demo af7bda53- 6d04-11e5-ba46-22000b92c6ec:/~/syncDemo • https://docs.globus.org/cli/examples/
  • 23. Globus Auth enables an integrated ecosystem of services and applications for the research community 23
  • 24. Based on widely used web standards • OAuth 2.0 Authorization Framework (a.k.a. OAuth2) • OpenID Connect Core 1.0 (a.k.a. OIDC) 24 docs.globus.org/api/auth
  • 25. What is a services ecosystem? • Build services with secure REST APIs • Services can leverage other services securely App Your Service Globus Transfer App Your Service Globus Transfer Other Service
  • 26. Why create your own services? • Make your specialized capabilities available to your research community as a service • Extend your web portal with a public REST API, so that other developers can integrate with and extend it • Front-end / back-end within your portal / app – Remote backend for portal – Backend for pure Javascript browser apps 26
  • 27. Why Globus Auth for your service? • Outsource all identity management and authentication – Federated identity with InCommon, Google, etc. • Outsource your REST API security – Consent, token issuance, validation, revocation – You provide service-specific authorization • Apps use your service like all others, with standard OAuth2 & OIDC • Your service can seamlessly leverage other services • Other services can leverage your service • Implement your service using any language and framework Add your service to the science services ecosystem 27
  • 28. Role of Globus Auth for services • Issue and check OAuth2 access tokens • With a token, your service can get attributes about the user, which it can use to authorize the request App Your Service Globus Transfer Globus Auth 1. Issue access token 3. Check access token 2. Access token
  • 29. Fundamental Concepts • Scopes: APIs that client is requesting access to – Scope syntax: OpenID Connect: openid, email, profile – https://auth.globus.org/scopes/<service-name>:<scope-name> – A service can have multiple scopes • Consents: authorize client to access a service, within limited scope, on the resource owner’s (user’s) behalf 29
  • 30. Globus account • Globus Account = Primary identity + Linked Identities – An identity can be primary on only one account – (Currently) Identities can be linked to only one account • Account does not have own identifier – An account is uniquely identified using its primary identity • Effective identity = linked identity from a particular identity provider required by a client or service 30
  • 31. Identity id vs. username • Identity id – Unique among all Globus Auth identities; will never be reused – UUID – Always use this to refer to an identity • Identity username – Unique at any point in time; may change, may be re-used – Case-insensitive user@domain – Can map to/from id, for user experience • Auth API allows mapping back and forth 31
  • 32. App registration • Admin registers an App ( “confidential client”) with Globus Auth – https://developers.globus.org • Gets client_id and client_secret for service • Sets app display name • Declare required scopes (optional) – Need long-term, offline refresh tokens? – May require authorization from scope admin • OAuth2 redirect URIs • Links for terms of service & privacy policy • Effective identity policy (optional) 32 developers.globus.org
  • 33. Native App Auto-registration (new) • New Native App bootstrap model • Developer registers a native app template at developers.globus.org • Each installed instance of the native app auto-registers itself on first use via Globus Auth Client API – Registers with a native app template – Gets a client id and secret for itself – Securely stores client id and secret (e.g., into user read-only file) • Native app is now a “confidential client”, just like any other
  • 34. Client (Web Portal, Application, Jupyter) Globus Transfer (Resource Server) Globus Auth (Authorization Server) 5. Authenticate using client id and secret, send authorization code Authorization Code Grant Browser (User) 1. Access portal 2. Redirects user 3. User authenticates and consents 4. Authorization code 6. Access token(s) 7. Authenticate with access token(s) to give the client the authority invoke the transfer service Identity Provider
  • 35. OAuth2 grants for apps • Authorization code grant – Native app grant variant • Refresh token grants – For apps that need long-lived, “offline” access to a service • Client credential grant – For app invoking services as itself, instead of as the user
  • 36. What is a services ecosystem? • Build services with secure REST APIs • Services can leverage other services securely App Your Service Globus Transfer App Your Service Globus Transfer Other Service
  • 37. Service registration • Client_id and client_secret for service • Service display name • Validated DNS name for service • One or more scopes – Who is authorized to use each scope: all client (public API) or specific clients • Declare dependent scopes – Need long-term, offline refresh tokens? – May require authorization from scope admin • Links for terms of service & privacy policy • Effective identity policy (optional) 37 mailto:support@globus.org
  • 38. Typical service interactions • Service receives HTTPS request with header – Authorization: Bearer <request-access-token> • Introspects the request access token – Auth API: POST /v2/oauth2/token/introspect – Authorized by client_id and client_secret – Returns: active, client_id, scope, sub (identity), identities_set • Verifies token info (e.g., active, aud, scope) • Authorizes request based on token info (e.g., sub) • Service processes request • Responds to client HTTPS request 38
  • 39. Sample Research Data Portal Service Walk-through 39 https://github.com/globus/globus-sample-data-portal/tree/master/service
  • 40. Authorization based on identity set • Use identities_set when authorizing a request based on the resource owner associated with an access token – E.g., ACLs on Globus shared endpoints • Authorizing based on set of identities is same complexity as authorizing based on group membership set 40
  • 41. Groups • Globus group service is identity set aware – “Tell me all groups for all identities of the logged in user” • Services can leverage this for authorization (soon) 41
  • 42. Django REST Framework • Use Globus Auth to protect REST APIs in the Django REST Framework • Simple enhancements to do token introspection, validation, and map to Django account • Contact support@globus.org if you are interested
  • 43. What is a services ecosystem? • Build services with secure REST APIs • Services can leverage other services securely App Your Service Globus Transfer App Your Service Globus Transfer Other Service
  • 44. Examples uses of dependent services • NCAR RDA REST API – App à RDA à Globus Transfer • Globus Transfer – Globus Connect Server v5 expects Globus Auth access tokens – App à Transfer à Collections
  • 45. Dependent tokens • Your service can act as client to other services (scopes) – Globus Transfer, Search, Identities, Auth – Other community services • Entire service call tree consented by user and service owners – Rescinding consent revokes all dependent tokens • Dependent tokens are restricted to a particular client, calling a particular scope, on behalf of a particular resource owner (e.g., user) – Restricted delegation! 45
  • 46. Typical service interactions • Service receives HTTPS request with header • Introspects the request access token • Verifies token info (e.g., active, aud, scope) • Authorizes request based on token info (e.g., sub) • If service needs to act as client to other services: – Calls Globus Auth Dependent Token Grant o Returns a token for each dependent service – Uses correct dependent token for downstream REST call • Service processes request, including calls to other services • Responds to client HTTPS request 46
  • 47. Sessions • Higher authentication assurance – For services with PHI, PII, sensitive but unclassified data • Introspection response includes session attributes – Which identities have authenticated in this “session” – When each authenticated last – Whether multi-factor authentication was used • Return authentication assurance error if not sufficient
  • 48. Refresh tokens • For “offline service”: Service working on your behalf even when you are offline – Example use: Globus Transfer service, for async transfers • Refresh tokens issued to a particular client for use with a particular scope • Client uses refresh token to get access token – Client_id and client_secret required • Refresh token good for 6 months after last use • Consent rescindment revokes resource token 48
  • 49. Token caching • Service should cache tokens and related information – Improves performance of service – Reduces load on Globus Auth • Access token -> introspect response – Cache timeout: 1-30 seconds recommended – To improve performance and load related to bursty use of REST API – Validity: Timeout duration determines responsiveness to token revocation and rescinding consent • Access token -> dependent access tokens – Cache timeout: lifetime of access token – To avoid costly dependent token re-issuance – Rescinding consent will invalidate everything • Refresh tokens – For however long they are needed for specific operations. – Keep distinct refresh tokens for each access token. 49
  • 50. Coming soon to Auth • Incremental auth – Add consents and tokens for new services dynamically • Optional scopes – Allow user to optionally deny access to a scope, but allow the client to continue functioning with reduced capability • SSH with Globus Auth
  • 51. Leveraging the Globus Platform in your Web Applications
  • 52. Globus serves as… A platform for building science gateways, web portals and other applications in support of research and education 52
  • 53. Example web apps that leverage Globus 53
  • 54. Globus Auth (and Group Management) … GlobusAPIs GlobusConnect Data Automation, Publication & Search File Sharing File Transfer & Replication Globus Platform-as-a-Service Use existing institutional ID systems in external web applications Integrate file transfer and sharing capabilities into scientific web apps, portals, gateways, etc...
  • 55. PaaS Security Challenges – Globus Auth • How to provide: – Login to apps o Web apps (Jupyter Notebook, Portals), Mobile, Desktop, Command line – Protect all REST API communications o App à Globus service (Jupyter Notebook, MRDP) o App à non-Globus service (MRDP) o Service à service (MRDP) • While: – Not introducing even more identities o Providing a platform to consolidate those identities – Providing least privileges security model (consents) – Being agnostic to programming language and framework – Being web friendly – Making it easy for users and developers 55
  • 56. Client (Web Portal, Application, Jupyter) Globus Transfer (Resource Server) Globus Auth (Authorization Server) 5. Authenticate using client id and secret, send authorization code Authorization Code Grant Browser (User) 1. Access portal 2. Redirects user 3. User authenticates and consents 4. Authorization code 6. Access token(s) 7. Authenticate with access token(s) to give the client the authority invoke the transfer service Identity Provider
  • 58. Globus Transfer API • Globus Web App consumes public Transfer API • REST API: Resources/actions named by URL – Query params allow refinement (e.g., filter) • Globus APIs use JSON for documents and resource representations • Requests authorized via Globus Auth issued OAuth2 access token – Authorization: Bearer asdflkqhafsdafeawk docs.globus.org/api/transfer 58
  • 59. Globus Python SDK • Python client library for the Globus Auth and Transfer REST APIs • globus_sdk.TransferClient class handles connection management, security, framing, marshaling from globus_sdk import TransferClient tc = TransferClient() globus.github.io/globus-sdk-python 59
  • 60. TransferClient low-level calls • Thin wrapper around REST API – post(), get(), update(), delete() get(path, params=None, headers=None, auth=None, response_class=None) o path – path for the request, with or without leading slash o params – dict to be encoded as a query string o headers – dict of HTTP headers to add to the request o response_class – class response object, overrides the client’s default_response_class o Returns: GlobusHTTPResponse object 60
  • 61. TransferClient higher-level calls • One method for each API resource and HTTP verb • Largely direct mapping to REST API endpoint_search(filter_fulltext=None, filter_scope=None, num_results=25, **params) 61
  • 62. Walkthrough Jupyter Notebook with our Hub • https://jupyter.demo.globus.org – Sign in with Globus – Verify the consents – Start My Server (this will take about a minute) – Open folder: globus-jupyter-notebooks – Open folder: globusWorld2018 – Run SDK_autoAuth.ipynb • If you mess it up and want to “go back to the beginning” – Back down to the root folder – Run NotebookPuller.ipynb • If you want to use the notebook outside of our hub – https://github.com/globus/globus-jupyter-notebooks – Autentication is a manual cut and paste of exchanging the authorization code for an access token 62
  • 63. Endpoint Search • Plain text search for endpoint – Searches owner, display name, keywords, description, organization, department – Full word and prefix match • Limit search to pre-defined scopes – all, my-endpoints, recently-used, in-use, shared- by-me, shared-with-me • Returns: List of endpoint documents 63
  • 64. Endpoint Management • Get endpoint (by id) • Update endpoint • Create & delete (shared) endpoints • Manage endpoint servers 64
  • 65. Endpoint Activation • Activating endpoint means binding a credential to an endpoint for login • Globus Connect Server endpoint that have MyProxy or MyProxy OAuth identity provider require login via web • Auto-activate – Globus Connect Personal and Shared endpoints use Globus- provided credential – Must auto-activate before any API calls to endpoints 65
  • 66. File operations • List directory contents (ls) • Make directory (mkdir) • Rename • Note: – Path encoding & UTF gotchas – Don’t forget to auto-activate first 66
  • 67. Task submission • Asynchronous operations – Transfer o Sync level option – Delete • Get submission_id, followed by submit – Once and only once submission 67
  • 68. Task management • Get task by id • Get task_list • Update task by id (label, deadline) • Cancel task by id • Get event list for task • Get task pause info 68
  • 69. Bookmarks • Get list of bookmarks • Create bookmark • Get bookmark by id • Update bookmark • Delete bookmark by id • Cannot perform other operations directly on bookmarks – Requires client-side resolution 69
  • 70. Shared endpoints and access rules (ACL) • Shared Endpoint – create / delete / get info / get list • Administrator role required to delegate access managers • Access manager role required to manage permissions/ACL • Operations: – Get list of access rules – Get access rule by id – Create access rule – Update access rule – Delete access rule 70
  • 71. Management API • Allow endpoint administrators to monitor and manage all tasks with endpoint – Task API is essentially the same as for users – Information limited to what they could see locally • Cancel tasks • Pause rules 71
  • 72. Globus Helper Pages • Globus pages designed for use by your web apps – Browse Endpoint – Activate Endpoint – Select Group – Manage Identities – Manage Consents – Logout docs.globus.org/api/helper-pages 72
  • 74. Modern data apps leverage the Science DMZ 74 fasterdata.es.net/ 10GE10GE 10GE 10GE Border Router WAN Science DMZ Switch/Router Firewall Enterprise perfSONAR perfSONAR 10GE 10GE 10GE 10GE DTN DTN API DTNs (data access governed by portal) DTN DTN perfSONAR Filesystem (data store) 10GE Portal Server Browsing path Query path Portal server applications: · web server · search · database · authentication Data Path Data Transfer Path Portal Query/Browse Path
  • 75. Globus PaaS developer resources Python SDK Sample Application docs.globus.org/api github.com/globus Jupyter Notebook
  • 77. Andre Schleife, UIUC 16,000 CPU-hours per simulation Sample'Experimental' sca0ering' Material' composi4on' Simulated' structure' Simulated' sca0ering' Sr'40%' Evolu4onary'op4miza4on' 786,432 CPUs, 10 PFLOPS supercomputer Argonne Leadership Computing Facility Modeling stopping power with time-dependent density functional theory
  • 78. @python_app Logan Ward Jupyter notebooks enable rapid iteration/results
  • 79. But the data are big, distributed… …and our science is collaborative petrel.alcf.anl.gov materialsdatafacility.org 2PB, 80Gbps store 3.2M materials data Cooley: 290 TFLOPS Query1 Share4 Transfer2 Learn3 Need multi-credential, multi-service authentication and data management
  • 80. Hub Configurable HTTP proxy Authenticator User DB Spawner Notebook /api/auth Browser /hub/ /user/[name]/ • Multi-user hub • Manages multiple instances of Jupyter notebook server • Configurable HTTP proxy JupyterHub Goal: Liberate the notebook! • Tokens for remote services • APIs for remote actions, e.g. data management via Globus service petrel.alcf.anl.gov
  • 81. Securing JupyterHub with Globus Auth plugin • Existing OAuth framework • Can restrict IdP • Custom scopes • Tokens passed into notebook environment github.com/jupyterhub/oauthenticator
  • 83. REST APIs REST APIs REST APIs Bearer a45cd... Hub Configurable HTTP proxy Authenticator User DB Spawner Notebook /api/auth /hub/ /user/[name]/ login Browser {"tokens":... {"tokens":... Tokens in Jupyter notebooks The world is your oyster API… • Globus Transfer • Globus Search • Your app • Data portal • Analysis engine • …
  • 84. Automated data analysis/results distribution Notebook Data Repository Bearer a45cd… Dataset Shared endpoint POST '/endpoint/a3c345f... /mkdir’ 200 OK ... X-Transfer-API-Version: 0.10 Content-Type: application/json ... Analyze
  • 85. Experiment with the demo notebook • Login into our JupyterHub*: jupyter.demo.globus.org • Launch (spawn) a notebook server; get tokens • Access Globus APIs; download some data • “Analyze” data (generate plot) • PUT results (graph) on an HTTPS endpoint *zero-to-jupyterhub.readthedocs.io
  • 96. Incorporate seamless parallel computing • (Data) science apps require… – Interactivity – Scalability (more than a desktop) – Reproducibility (publish code, docs) • Solution: JupyterHub + Parsl – Interactive computing environment – Notebooks for publication – Can run on dedicated hardware @python_app def compute_features(chunk): for f in featurizers: chunk = f.featurize_dataframe(chunk, 'atoms’) return chunk chunks = [compute_features(chunk) for chunk in np.array_split(data, chunks)] Python parallel library • Tasks exposed as functions (Python, bash) • Python code to glue functions together • Globus for auth and data movement parsl-project.org
  • 97. Containerized data science ecosystem Auth Transfer Search container metadata container recipes containers ALCF Petrel Uniform IAM platform scalable for distributed objectsUsers select container to execute custom Jupyter environment Registry for container discovery and referencing Container definitions tracked in version control systems Containers used for both Jupyter notebook server and compute nodes Containers used other tasks; analysis, ML, etc.Supercomputer Compute Compute Compute Compute Containers staged to local file systems Container Registry Notebook Server
  • 98. Support resources • Globus documentation: docs.globus.org • Sample code: github.com/globus • Helpdesk and issue escalation: support@globus.org • Customer engagement team • Globus professional services team – Assist with portal/gateway/app architecture and design – Develop custom applications that leverage the Globus platform – Advise on customized deployment and integration scenarios
  • 99. Join the Globus community • Access the service: globus.org/login • Create a personal endpoint: globus.org/app/endpoints/create-gcp • Documentation: docs.globus.org • Engage: globus.org/mailing-lists • Subscribe: globus.org/subscriptions • Need help? support@globus.org • Follow us: @globusonline