Successfully reported this slideshow.

Globus Portal Framework (APS Workshop)

0

Share

Loading in …3
×
1 of 27
1 of 27

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Related Audiobooks

Free with a 14 day trial from Scribd

See all

Globus Portal Framework (APS Workshop)

  1. 1. Globus Portal Framework An introduction to Globus Data Portals October 15, 2021
  2. 2. Agenda • Globus Data Portals – FAIR data – Data discovery – Custom use-cases • Django Globus Portal Framework – Globus Auth – Globus Services – Integration with custom resource servers • Portal Architecture – Lifecycle of a request • Live Demo: Build your own portal
  3. 3. What is a Globus Data Portal A Web interface for a Globus Search index with integrated Globus capabilities to find, access, and act on data regardless of location
  4. 4. Why we use portals • Different experiments (beamlines, electron microscopes, biology, etc) generate data with different types, size and experimental information • Processing, curation, and cataloging need to happen as soon as possible so data are not lost • Standardize secure access between users • Work toward FAIR datasets to enable more science
  5. 5. Advantages of a portal • Make data FAIRer • Track lots of data – Tens of thousands of datasets • Discovery – Free text search in Globus Search – Filtering on specific values – User Friendly GUI • Authorized Access – Public/private, group-based, subject-level ACLs • Integration with Globus Services • Customizability
  6. 6. Portal Features • Watch curated data appear in your portal! – A Web interface to data collection – Data acquisition in real time o Synchrotron serial crystallography • HTTPS Access to data – Show images in a portal from a Globus Endpoint – Add links to specific files for user download • Easy customizability – Add your own templates – Rewrite existing views – Create and call out to custom services
  7. 7. ALCF Community Data Co-Op: acdc.alcf.anl.gov • A collection of data portals in collaboration with ALCF • Leverage ALCF resources – Eagle: 100PB store, 650GB/s transfer, Globus interfaces
  8. 8. CosmoFlow Search
  9. 9. Serial Crystallography
  10. 10. Leveraging the Globus data platform…
  11. 11. Globus Search Globus Search • Scalable, secure search for research data • Metadata store with fine- grained access controls • Schema agnostic • Free text, faceted search • Integrated with core platform services, e.g. Auth, Groups Input form Extract Metadata Ingest metadata, set visibility policies Discovery POST /index/123 {"filters": [ {"field_name": "record_year", "values": ["2020"], "type": "match_all" }, {"field_name": "temp_farenheit", "values": [{"from": 90, "to": "*"}], "type": "range" ]} ]} Query Bulk ingest
  12. 12. Globus Portal Framework • Django framework for rapidly creating portals • Python Package for bootstrapping a portal – Install from PyPi using pip – github.com/globus/django-globus-portal-framework • Globus Portal Framework Deployment – ALCF Community Data Co-Op (acdc.alcf.anl.gov) • Example Repo – github.com/globus/django-globus-portal-framework-example
  13. 13. Globus Portal Framework Features • Authorized Access with Globus Auth • Plugins for Globus Services – Globus Search – Globus Transfer – Globus Flows – Third-Party Globus Services • Customizability – Redesign how your portal talks to these services – Extend Globus Portal Framework with new features
  14. 14. APS XPCS: secure data discovery 14 Globus Auth Globus Groups Globus Search
  15. 15. APS XPCS: data access & preview 15 Globus Transfer HTTPS access
  16. 16. APS XPCS: automated processing & indexing 16 Globus Flows: Transfer, analysis, and ingest to search index
  17. 17. Portal Architecture
  18. 18. Portal Architecture • User Authentication • Globus Portal Framework – Portal URL Mappings – Token Loading • Service Calls to Globus Search • Lifecycle of a Request • Post Processing search requests
  19. 19. User authentication • Scopes are configured in the portal • Users authenticate with Globus using 3-legged flow – Python Social Auth used for Authentication backend • User tokens are saved in the database • Future requests authorized with user access tokens – New searches use Search bearer token.
  20. 20. Service Calls • Most Portal Service calls use the Globus SDK – Globus Portal Framework loads tokens from DB – Globus SDK object instantiated with token – Call to Globus Service – Portal renders result in templates
  21. 21. Globus Portal Framework URLs • URLs are laid out in three major categories – Index Selection – Index Search page – Search Subject detail page • Many Globus Search Indices are supported • Index Search page links to many result subjects • Each subject has a unique URL
  22. 22. Example: Cosmology 22
  23. 23. Format of a URL
  24. 24. An index is Configuration Driven • A Search Index can be setup through configuration – Add Globus Search index uuid – Add a name – Add facets – Add fields • Start Searching!
  25. 25. Lifecycle of a request • User makes a query • Portal sends request to Globus Search – Request contains user bearer token • Portal receives response • Portal does processing on response – Parse Dates, build URL for Globus webapp, etc. • Portal renders data into templates • User receives a search page
  26. 26. Pilot • Tooling to simplify ingesting metadata • Create valid GMeta entries from JSON • Upload files to an associated endpoints and track HTTPS URLs to add to GMeta
  27. 27. Demo: Creating your own portal • Django Globus Portal Framework – github.com/globus/django-globus-portal-framework – django-globus-portal-framework.readthedocs.io/en/stable/ • Example Repo – github.com/globus/django-globus-portal-framework-example

×