Successfully reported this slideshow.
Your SlideShare is downloading. ×

Working with Globus Platform Services and Portals

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 39 Ad

Working with Globus Platform Services and Portals

Download to read offline

We describe how developers can use Globus APIs to integrate robust data management capabilities into their research applications. We also demonstrate the new Globus portal framework that can be used in conjunction with the Globus Search service to simplify data search and discovery.

We describe how developers can use Globus APIs to integrate robust data management capabilities into their research applications. We also demonstrate the new Globus portal framework that can be used in conjunction with the Globus Search service to simplify data search and discovery.

Advertisement
Advertisement

More Related Content

More from Globus (20)

Recently uploaded (20)

Advertisement

Working with Globus Platform Services and Portals

  1. 1. Working with Globus Platform Services and Portals Vas Vasiliadis vas@uchicago.edu September 9, 2022 Sponsored by
  2. 2. Globus Platform Services …Integrate Globus capabilities into your data portal, science gateway, data commons or other research app
  3. 3. Globus APIs • Auth • Groups • Transfer • Search • Timer • Flows • GCS Manager • Globus web app consumes the same public APIs • Resources named by URL (standard REST approach) • Request/response body is JSON docs.globus.org/api
  4. 4. Globus Python SDK • Python client library for the Globus REST APIs • Largely direct mapping to REST API • globus_sdk.TransferClient class handles connection management, security, framing, marshaling globus-sdk-python.readthedocs.io github.com/globus/globus-sdk-python 5
  5. 5. Globus Auth: Foundational IAM service • Brokers authentication and authorization among… – End-users – Identity providers: enterprise, external (federated identities) – Services: resource servers with REST APIs – Apps: web, mobile, desktop, command line clients – Services acting as clients to other services • OAuth 2.0 Authorization Framework (a.k.a. OAuth2) • OpenID Connect Core 1.0 (a.k.a. OIDC) 6
  6. 6. Fundamental Concepts • Scopes – APIs that client is requesting access to – Service and resources within that service • Consents – Authorizes a client to access a service, within limited scope, on the resource owner's behalf • Multiple methods for user to grant consent depending on the type of application 7
  7. 7. Several authentication models supported • Application acting as user with consent – Authorization code grant: authenticate as user identity – Browser redirect; auth code returned automatically; tokens stored securely – Examples: Globus CLI, Jupyter Hub secured with Globus Auth • Application authenticating as itself – Client credentials grant: authenticate as application – Client ID and Secret stored securely – Examples: Globus Django portal, custom apps • Application able to manage tokens for offline/long lived tasks – Request refresh tokens in addition to access tokens
  8. 8. App registration 9
  9. 9. Get app credentials at developers.globus.org
  10. 10. Getting API access tokens 11 jupyter.demo.globus.org
  11. 11. Getting tokens: jupyter.demo.globus.org • Sign in with Globus and verify the consents • Start My Server and open globus-jupyter-notebooks • Run Platform_Introduction_Native_App_Auth • If you mess up, just stop and restart the server • Alternatively, use notebooks on your own machine: github.com/globus/globus-jupyter-notebooks 12
  12. 12. REST APIs REST APIs REST APIs Request Bearer a45cd... Configurable HTTP proxy Authenticator User DB Spawner Notebook /api/auth /hub/ /user/[name]/ login Browser {"tokens":... {"tokens":... Globus Auth integrated with JupyterHub The world is your oyster API… • Globus Transfer • Globus Search • Your app • Data portal • Analysis engine • …
  13. 13. Working with the Transfer API 19 jupyter.demo.globus.org
  14. 14. TransferClient low-level calls • Thin wrapper around REST API – post(), get(), update(), delete() get(path, params=None, headers=None, auth=None, response_class=None) o path – path for the request, with or without leading slash o params – dict to be encoded as a query string o headers – dict of HTTP headers to add to the request o response_class – class response object, overrides the client’s default_response_class o Returns: GlobusHTTPResponse object 20
  15. 15. TransferClient higher-level calls • One method for each API resource and HTTP verb • Largely direct mapping to REST API endpoint_search(filter_fulltext=None, filter_scope=None, num_results=25, **params) 21
  16. 16. API walkthrough: jupyter.demo.globus.org • Run Platform_Introduction_JupyterHub_Auth • Note automatic token extraction from JupyterHub • Run the first few cells • Experiment with endpoint search/filters 22
  17. 17. Making data findable with Globus Search 23 docs.globus.org/api/search
  18. 18. Data description and discovery • Metadata store with fine- grained visibility controls • Schema agnostic à dynamic schemas • Simple search using URL query parameters • Complex search using search request document 24 Search Index
  19. 19. Distinct access policies may be applied to Data and Metadata …using permissions on guest collections …using permissions on metadata elements
  20. 20. Data ingest with Globus Search 26 Search Index POST /index/{index_id}/ingest' { "ingest_type": "GMetaList", "ingest_data": { "gmeta": [ { "id": "filetype", "subject”: "https://search.api.globus.org/abc.txt", "visible_to": ["public"], "content": { "metadata-schema/file#type": "file” } }, ... ] } - Bulk create and update - Task model for ingest at scale
  21. 21. Data ingest with Globus Search 27 Search Index POST /index/{index_id}/ingest' { "ingest_type": "GMetaList", "ingest_data": { "gmeta": [ { "id": ”weight", "subject": "https://search.api.globus.org/abc.txt", "visible_to": ["urn:globus:auth:identity:46bd0f56- e24f-11e5-a510-131bef46955c"], "content": { "metadata-schema/file#size": ”37.6", "metadata-schema/file#size_human": ”<50lb” } }, ... ] } Visibility limited to Globus Auth identity - Single user - Globus Group - Registered client application
  22. 22. Data discovery with Globus Search 28 { "@datatype": "GSearchResult", "@version": "2017-09-01", "count": 1, "gmeta": [ { "@datatype": "GMetaResult", "@version": "2019-08-27", "entries": [ { ... } ], "subject": "https://..." } ], "offset": 0, "total": 1 } GET /index/{index_id}/search?q=type%3Ahdf5 Search Index Simple query
  23. 23. Data discovery with Globus Search 29 POST /index/{index_id}/search Search Index Complex query { "filters": [ { "type": "range", "field_name": ”pubdate", "values": [ { "from": "*", "to": "2020-12-31" } ] } ], "facets": [ { "name": "Publication Date", "field_name": "pubdate", ... } ] } Filter Facets Boosts Sort Limit
  24. 24. Why go beyond the standard tools? • FAIRness: Enable broad audience to access diverse research data (type, size, metadata, …) • …and diverse/complicated data sources (beamlines, electron microscopes, sequencers, …) • Add curation and cataloguing to make data findable • Enforce (sometimes complex) access policies • Enable researchers to customize their experience
  25. 25. The Modern Research Data Portal Design Pattern docs.globus.org/mrdp
  26. 26. MRDP: Key elements Science DMZ Fast, clean data path Data Transfer Nodes Purpose-built data movers Globus Platform Secure, reliable data orchestration Globus Connect Storage system enabler 32 Globus Portal Framework Data discovery and access
  27. 27. An exemplar: The ALCF Data Co-op 33 acdc.alcf.anl.gov
  28. 28. Accessing data via your portal • Recall: Data independent of portal logic… • …ideally, on a guest collection • Step 1: Create a guest collection; requires authN by human or by app using client credentials • Step2: Grant the application Access Manager role – Allows the application to manage permissions on the collection – Set for application identity: appclientid@clients.auth.globus.org • Optional: Grant roles for endpoint/task management 10GE 10GE 10GE 10GE Border Router WAN Science DMZ Switch/Router Firewall Enterprise perfSONAR perfSONAR 10GE 10GE 10GE 10GE DTN DTN API DTNs (data access governed by portal) DTN DTN perfSONAR Filesystem (data store) 10GE Portal Server Browsing path Query path Portal server applications: · web server · search · database · authentication Data Path Data Transfer Path Portal Query/Browse Path
  29. 29. Globus Groups simplify permissions management • Grant group access to collection(s) • Restrict search visibility using group • Make portal client a group administrator • Check authenticated user’s group membership • Add/remove user to/from group
  30. 30. Bootstrap a Simple (but fully functional and extensible) Research Data Portal using the Django Globus Portal Framework
  31. 31. Django Globus Portal key features • Federated login • Data export using Globus • Browse datasets via Globus Search service • Template-driven search results and landing pages • Django-based framework with extensible templates • Bootstrap your project using Cookiecutter 37 Source: github.com/globus/django-globus-portal-framework Docs: django-globus-portal-framework.readthedocs.io
  32. 32. Get up and running with the Globus portal framework 38 Source: github.com/globus/django-globus-portal-framework Docs: django-globus-portal-framework.readthedocs.io/en/stable/
  33. 33. Step 0: Application registration • Set redirect URLs • Get client ID and secret • Consents implement least privileges principle 39 developers.globus.org Redirect URLs https://tutN.globusdemo.org:8443/ https://tutN.globusdemo.org:8443/complete/globus/
  34. 34. Portal deployment • Deploy a portal instance using cookiecutter • Configure settings • Run and use! • Future: containers • Note: For production, add robust WSGI/ASGI server
  35. 35. Adding a new search index to your portal 41
  36. 36. Update the search index definition • Edit ~/$PROJECT_SLUG/$PROJECT_SLUG/settings/search.py • Index ID and name • Fields • Facets • Multiple search indices may be defined
  37. 37. Update definition of search result fields • Edit ~/$PROJECT_SLUG/$PROJECT_SLUG/fields.py • Type definitions • Formatting functions • Data transformations • Globus collection and data path
  38. 38. Modify templates for search results, details • Fields to be displayed on search item card in list – Metadata to help help user determine relevance – Links to actions, e.g., transfer, download, process – Edit ~/$PROJECT_SLUG/templates/globus-portal- framework/v2/components/search-results.html • Fields to be displayed on detail page – (More) complete metadata – Preview images (if appropriate) – Links to actions – Edit ~/$PROJECT_SLUG/templates/globus-portal-framework/v2/detail- overview.html • Uses standard Jinja2 templating language
  39. 39. Developer References • Documentation – API: docs.globus.org/api – SDK: globus-sdk-python.readthedocs.io/en/stable/ • Open source repositories: github.com/globus • Notebooks: github.com/globus/globus-jupyter- notebooks

×