Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

GlobusWorld 2021 Tutorial: The Globus CLI, Platform and SDK

An introduction to the Globus command line interface and the SDK for accessing Globus platform services. This tutorial was presented at the GlobusWorld 2021 conference in Chicago, IL by Greg Nawrocki.

  • Be the first to comment

GlobusWorld 2021 Tutorial: The Globus CLI, Platform and SDK

  1. 1. The Globus CLI, Platform and SDK Greg Nawrocki GlobusWorld - May 13, 2021
  2. 2. 2 Custom portals? Science Gateways? Unique workflows? Our Command Line Interface, open REST APIs and Python SDK empower you to create an integrated ecosystem of research data services and applications.
  3. 3. Globus Command Line Interface Open source, uses the Python SDK Because of this correspondence the CLI is an excellent tool for getting the gist of how he SDK functions.
  4. 4. PaaS Security Challenges – Globus Auth • How to provide: – Login to apps o Web apps (Jupyter Notebook, Portals), Mobile, Desktop, Command line – Protect all REST API communications o App à Globus service (Jupyter Notebook, Portals) o App à non-Globus service (Portals) o Service à service (Portals) • While: – Not introducing even more identities o Providing a platform to consolidate those identities – Providing least privileges security model (consents) – Being agnostic to programming language and framework – Being web friendly – Making it easy for users and developers 4
  5. 5. Globus Auth: Native apps • Client that cannot keep a secret, e.g… – Command line, desktop apps – The Globus CLI – Mobile apps – Jupyter notebooks • Native app is registered with Globus Auth – Not a confidential client, the task on behalf of an authenticated user – Redirect to give consent • Globus SDK: – To get tokens: NativeAppAuthClient – To use tokens: AccessTokenAuthorizer 5
  6. 6. Browser Native App grant 6 Native App (Client) 1. Run application 2. URL to authenticate 3. Authenticate and consent 4. Auth code 5. Register auth code 6. Exchange code 7. Access tokens 8. Authenticate with access tokens to invoke transfer service as user App/Service (Resource Server) Globus Auth (Authorization Server)
  7. 7. Refresh tokens • Common use cases – Portal checking transfer status when user is not logged in – Running command line app from script o The CLI gets access and refresh tokens upon ”globus login” • Refresh tokens issued to client, in particular scope • Client uses refresh token to get access token – Confidential client: client_id and client_secret required – Native app: client_secret not required • Refresh token good for 6 months after last use • Consent rescindment revokes resource token 7
  8. 8. Refresh tokens 8 Native App (Client) App/Service (Resource Server) Globus Auth (Authorization Server) 1. Run application 2. URL to authenticate Browser 3. Authenticate and consent 4. Auth code 5. Register auth code 6. Exchange code, request refresh tokens 7. Access tokens and refresh tokens 9. Exchange refresh token for new access tokens 8. Store refresh tokens 10. Access tokens 11. Authenticate with access tokens to invoke service as user
  9. 9. Globus CLI • It’s a native application distributed by Globus – – • Easy install and updates • Command “globus login” gets access tokens and refresh tokens – Stores the token locally (~/.globus.cfg ) • All interactions with the service use the tokens – Tokens for Globus Auth and Transfer services • Command “globus logout” deletes those
  10. 10. CLI Basics • Getting help / list of commands – globus –help – globus list-commands • UUIDs for endpoint, task, user identity, groups… – Use search/list options • get-identities for identity username to UUID $ globus endpoint search 'Globus Tutorial' $ globus task list $ globus get-identities 2867d9fb- d5b5-4f21-95e7-312b288b8d11 --verbose
  11. 11. The Globus CLI – Simple tasks • Find endpoints – globus endpoint search 'Globus Tutorial' – globus endpoint search Midway – globus endpoint search --filter-scope=recently-used • Find endpoint contents – globus ls af7bda53-6d04-11e5-ba46-22000b92c6ec – globus ls ddb59aef-6d04-11e5-ba46-22000b92c6ec:/share/godata/ • Transfer a file – From Globus Tutorial Endpoint 1 to Globus Tutorial Endpoint 2 – Note the specific paths – globus transfer ddb59aef-6d04-11e5-ba46-22000b92c6ec:/share/godata/file3.txt ddb59af0-6d04- 11e5-ba46-22000b92c6ec:/~/file3.txt • Transfer a directory – From Globus Tutorial Endpoint 1 to Globus Tutorial Endpoint 2 – globus transfer --recursive ddb59aef-6d04-11e5-ba46-22000b92c6ec:/share/godata/ ddb59af0- 6d04-11e5-ba46-22000b92c6ec:/~/syncDemo •
  12. 12. Batch Transfers • Transfer tasks have one source/destination, but can have any number of files • Provide input source-dest pairs via local file • e.g. move files listed in files.txt from $ep1 to $ep2 $ ep1=ddb59aef-6d04-11e5-ba46-22000b92c6ec $ ep2=ddb59af0-6d04-11e5-ba46-22000b92c6ec $ globus transfer $ep1:/share/godata/ $ep2:/~/ -- batch --label 'CLI Batch' < files.txt
  13. 13. Parsing CLI output • Default output is text; for JSON output use --format json $ globus endpoint search --filter-scope my-endpoints $ globus endpoint search --filter-scope my-endpoints -- format json • Extract specific attributes using --jmespath <expression> $ globus endpoint search --filter-scope my-endpoints -- jmespath 'DATA[].[id, display_name]'
  14. 14. Managing notifications • Turn off emails sent for tasks • Useful when an application manages tasks for a user • Disable notifications with the --notify option --notify off (all notifications) --notify succeeded|failed|inactive (select notifications)
  15. 15. Permission management • Set and manage permissions on shared endpoint • Requires access manager role $ share=<shared_endpoint_UUID> $ globus endpoint permission create --permissions r -- identity $share:/nawrockipersonal/ $ globus endpoint permission list $share $ globus endpoint permission delete $share <perm_UUID>
  16. 16. Automation with the CLI • Interactions are as user: both for data access and to Globus services – Globus login to get tokens • Collection access – Mapped Collections o Use the –skip-activation-check to submit the task even if endpoint is not activated at submit time – Guest Collections o Guest Collection / Shared Endpoints auto-activate o Use Guest Collections whenever possible • Reference – Basic Data Automation with the Globus Command Line Interface (CLI) o
  17. 17. Data centric applications leveraging Globus 17
  18. 18. JupyterHub • Multi-user hub • Manages multiple instances of Jupyter notebook server – Python, R, etc. • Use it to serve notebooks to research team, class, etc. • Configurable HTTP proxy
  19. 19. Securing JupyterHub with Globus Auth plugin • Existing OAuth framework • Can restrict IdP • Custom scopes • Tokens passed into notebook environment • Documentation covers app registration and config
  20. 20. Securing JupyterHub with Globus Auth
  21. 21. Client (Web Portal, Application) Globus Transfer (Resource Server) Globus Auth (Authorization Server) 5. Authenticate using client id and secret, send authorization code Authorization Code Grant Browser (User) 1. Access portal 2. Redirects user 3. User authenticates and consents 4. Authorization code 6. Access token(s) 7. Authenticate with access token(s) to give the client the authority invoke the transfer service Identity Provider
  22. 22. Tokens and Jupyter Hub login REST APIs { “tokens”:… {“tokens”:… REST APIs REST APIs Bearer a45cd…
  23. 23. Globus Transfer API • Globus Web App consumes public Transfer API • Resource named by URL (standard REST approach) – Query params allow refinement (e.g., subset of fields) • Globus APIs use JSON for documents and resource representations • Requests authorized via OAuth2 access token – Authorization: Bearer asdflkqhafsdafeawk 23
  24. 24. Globus Python SDK • Python client library for the Globus Auth and Transfer REST APIs • globus_sdk.TransferClient class handles connection management, security, framing, marshaling from globus_sdk import TransferClient tc = TransferClient() 24
  25. 25. TransferClient higher-level calls • One method for each API resource and HTTP verb • Largely direct mapping to REST API endpoint_search(filter_fulltext=None, filter_scope=None, num_results=25, **params) 25
  26. 26. Endpoint Search • Plain text search for endpoint – Searches owner, display name, keywords, description, organization, department – Full word and prefix match • Limit search to pre-defined scopes – all, my-endpoints, recently-used, in-use, shared- by-me, shared-with-me • Returns: List of endpoint documents 26
  27. 27. Endpoint Activation • Activating endpoint means binding a credential to an endpoint for login • Mapped Collections require login via web app • Auto-activate – Globus Connect Personal and Guest Collections use Globus-provided credential – Must auto-activate before any API calls to endpoints 27
  28. 28. File operations • List directory contents (ls) • Make directory (mkdir) • Rename • Note: – Path encoding & UTF gotchas – Don’t forget to auto-activate first 28
  29. 29. Task submission • Asynchronous operations – Transfer o Sync level option – Delete • Get submission_id, followed by submit – Once and only once submission 29
  30. 30. Walkthrough API with our Jupyter Hub • – Sign in with Globus – Verify the consents – Start My Server (this will take about a minute) – Open folder: globus-jupyter-notebooks – Run Platform_Introduction_JupyterHub_Auth.ipynb • If you mess it up and want to “go back to the beginning” – Just stop and restart the notebook • If you want to use the notebook outside of our hub – – Authentication is a manual cut and paste of exchanging the authorization code for an access token – Native App 30
  31. 31. Automation Examples • Simple code examples for various use cases using Globus – – Syncing a directory o Bash script that calls the Globus CLI and a Python module that can be run as a script or imported as a module. – Staging data in a shared directory o Bash / Python – Removing directories after files are transferred o Python script 31
  32. 32. Resources • Documentation: • YouTube channel: • Helpdesk and issue escalation: • Mailing list: • Globus customer team: – We’d love to be part of your events – GlobusWorld Tours – Office Hours