Successfully reported this slideshow.
Your SlideShare is downloading. ×

Working with Globus Platform Services

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 26 Ad

Working with Globus Platform Services

Download to read offline

We describe the various Globus APIs and demonstrate how developers can use them to integrate robust data management capabilties into their research applications. We also provide an overview of advanced services such as Globus Search and tools such as the data portal framework that you can use to simplify data search and discovery.

Presented at a workshop at KU Leuven on July 8, 2022.

We describe the various Globus APIs and demonstrate how developers can use them to integrate robust data management capabilties into their research applications. We also provide an overview of advanced services such as Globus Search and tools such as the data portal framework that you can use to simplify data search and discovery.

Presented at a workshop at KU Leuven on July 8, 2022.

Advertisement
Advertisement

More Related Content

More from Globus (20)

Recently uploaded (20)

Advertisement

Working with Globus Platform Services

  1. 1. Working with Globus Platform Services Vas Vasiliadis vas@uchicago.edu 8 July 2022
  2. 2. Globus Platform Services
  3. 3. Globus APIs • Auth • Groups • Transfer • Search • Timer • Flows • GCS Manager • Globus web app consumes the same public APIs • Resources named by URL (standard REST approach) • Request/response body is JSON docs.globus.org/api
  4. 4. Globus Python SDK • Python client library for the Globus REST APIs • Largely direct mapping to REST API • globus_sdk.TransferClient class handles connection management, security, framing, marshaling globus-sdk-python.readthedocs.io/en/stable/ globus.github.io/globus-sdk-python 4
  5. 5. Globus Auth: Foundational IAM service • Brokers authentication and authorization among… – End-users – Identity providers: enterprise, external (federated identities) – Services: resource servers with REST APIs – Apps: web, mobile, desktop, command line clients – Services acting as clients to other services • OAuth 2.0 Authorization Framework (a.k.a. OAuth2) • OpenID Connect Core 1.0 (a.k.a. OIDC) 5
  6. 6. Fundamental Concepts • Scopes – APIs that client is requesting access to – Service and resources within that service • Consents – Authorizes a client to access a service, within limited scope, on the resource owner's behalf • Multiple methods for user to grant consent depending on the type of application 6
  7. 7. Several authentication models supported • Application acting as user with consent – Authorization code grant • Application authenticating as itself – Client credentials grant • Application able to manage tokens for offline or long running tasks – Refresh tokens
  8. 8. App registration 8
  9. 9. Get app credentials at developers.globus.org
  10. 10. Getting API access tokens 10 jupyter.demo.globus.org
  11. 11. Accessing collections via an app • Use guest collections whenever possible – No need to worry about identity mapping – Grant your app the Access Manager role on the collection – Grant your app roles to manage endpoints and tasks • Mapped collections use tokens and consents – Configure storage gateway to accept identities from 'clients.auth.globus.org' domain – Request data_access scope (per collection) to access collection – Use identity mapping policy to maps ‘UUID@clients.auth.globus.org’ to local user docs.globus.org/globus-connect-server/v5.4/use-client-credentials
  12. 12. Working with the Transfer API 18 jupyter.demo.globus.org
  13. 13. TransferClient low-level calls • Thin wrapper around REST API – post(), get(), update(), delete() get(path, params=None, headers=None, auth=None, response_class=None) o path – path for the request, with or without leading slash o params – dict to be encoded as a query string o headers – dict of HTTP headers to add to the request o response_class – class response object, overrides the client’s default_response_class o Returns: GlobusHTTPResponse object 19
  14. 14. TransferClient higher-level calls • One method for each API resource and HTTP verb • Largely direct mapping to REST API endpoint_search(filter_fulltext=None, filter_scope=None, num_results=25, **params) 20
  15. 15. API walkthrough: jupyter.demo.globus.org • Sign in with Globus and verify the consents • Start My Server and open globus-jupyter-notebooks • Run Platform_Introduction_JupyterHub_Auth.ipynb • If you mess up just stop and restart the server • Use the notebooks on your own machine github.com/globus/globus-jupyter-notebooks 21
  16. 16. Making Data Findable with Globus Search
  17. 17. Data description and discovery • Metadata store with fine- grained visibility controls • Schema agnostic à dynamic schemas • Simple search using URL query parameters • Complex search using search request document 23 docs.globus.org/api/search Search Index
  18. 18. Distinct access policies may be applied to Data and Metadata …(ideally) using permissions on guest collections …using permissions on metadata elements
  19. 19. Data ingest with Globus Search 25 Search Index POST /index/{index_id}/ingest' { "ingest_type": "GMetaList", "ingest_data": { "gmeta": [ { "id": "filetype", "subject”: "https://search.api.globus.org/abc.txt", "visible_to": ["public"], "content": { "metadata-schema/file#type": "file” } }, ... ] } - Bulk create and update - Task model for ingest at scale
  20. 20. Data ingest with Globus Search 26 Search Index POST /index/{index_id}/ingest' { "ingest_type": "GMetaList", "ingest_data": { "gmeta": [ { "id": ”weight", "subject": "https://search.api.globus.org/abc.txt", "visible_to": ["urn:globus:auth:identity:46bd0f56- e24f-11e5-a510-131bef46955c"], "content": { "metadata-schema/file#size": ”37.6", "metadata-schema/file#size_human": ”<50lb” } }, ... ] } Visibility limited to Globus Auth identity - Single user - Globus Group - Registered client application
  21. 21. Data discovery with Globus Search 27 { "@datatype": "GSearchResult", "@version": "2017-09-01", "count": 1, "gmeta": [ { "@datatype": "GMetaResult", "@version": "2019-08-27", "entries": [ { ... } ], "subject": "https://..." } ], "offset": 0, "total": 1 } GET /index/{index_id}/search?q=type%3Ahdf5 Search Index Simple query
  22. 22. Data discovery with Globus Search 28 POST /index/{index_id}/search Search Index Complex query { "filters": [ { "type": "range", "field_name": ”pubdate", "values": [ { "from": "*", "to": "2020-12-31" } ] } ], "facets": [ { "name": "Publication Date", "field_name": "pubdate", ... } ] } Filter Facets Boosts Sort Limit
  23. 23. Exemplar portal The ALCF Data Co-op 29 acdc.alcf.anl.gov/
  24. 24. Creating your own using the Django Globus Portal Framework 30
  25. 25. Django Globus Portal key features • Federated login • Data export using Globus • Browse datasets via Globus Search service • Template-driven search results and landing pages • Django-based framework with extensible templates • Bootstrap your project using Cookiecutter 31 Source: github.com/globus/django-globus-portal-framework Docs: django-globus-portal-framework.readthedocs.io/en/stable
  26. 26. Developer References • Documentation – API: docs.globus.org/api – SDK: globus-sdk-python.readthedocs.io/en/stable/ • Open source repositories: github.com/globus • Notebooks: github.com/globus/globus-jupyter- notebooks

×