Download to read offline
![Hub
Configurable HTTP
proxy
Authenticator
User DB
Spawner
Notebook
/api/auth
Browser
/hub/
/user/[name]/
• ALCF runs a production
instance tied to ALCF IdP
• Version with Globus Auth
on PetrelKube
JupyterHub
Goal: Liberate the notebook!
• Tokens for remote services
• APIs for remote actions, e.g. data
management via Globus service
petrel.alcf.anl.gov](https://image.slidesharecdn.com/190731pearcpetrel-190807170515/75/Petrel-A-Programmatically-Accessible-Research-Data-Service-25-2048.jpg)
Uploaded byGlobus
Petrel: A Programmatically Accessible Research Data Service
Petrel is a programmatically accessible research data service hosted at ALCF that provides 3.2 petabytes of storage accessible at 100 Gbps via Globus. It allows large datasets to be kept close to computational resources, with PI-managed access control and configurable sharing of data internally and with collaborators.
More Related Content
Similar to Petrel: A Programmatically Accessible Research Data Service
Petrel: A Programmatically Accessible Research Data Service
- 1. Petrel: A Programmatically AccessibleResearch Data Service William E. Allcock Benjamin S. Allen Rachana Ananthakrishnan Ben Blaiszik Kyle Chard Ryan Chard Ian Foster Lukasz Lacinski Michael E. Papka Rick Wagner rick@globus.org PEARC rwagner@anl.gov July 31, 2019
- 2. 2 Storage O( 50PB ) Globus SimulationAnalysis User Community via web and Exp/Obs-specific clients Web/Data Interfaces - Galaxy - Jupyter - PDACS - DESCQA - Globus - Smaash - Visualization simulation job description analysis job description Experimental/Observational Facility (Advanced Photon Source, LSST-DESC, ATLAS, DIII-D) CooleyMira (10PF) Adaptation of Leadership Facilities to accommodate large exp/obs collaborations Machine Learning job submission/adaptation layer Exp/Obs datasets ALCF-hosted Exp/Obs-controlled Resources
- 3. But the dataare big, distributed… …and the science is collaborative petrel.alcf.anl.gov materialsdatafacility.org 2PB, 80Gbps store 3.2M materials data Cooley: 290 TFLOPS Query1 Share4 Transfer2 Learn3 Need multi-credential, multi-service authentication and data management
- 4. What is Globus? (thevery fast version) 4
- 5. 5 Research Computing HPC DesktopWorkstations Mass Storage Instruments Personal Resources Public Cloud National Resources Unify access to data across tiers
- 6. IBM Spectrum Scale Current Planned StorageConnectors - globus.org/connectors
- 7. Conceptual architecture: HybridSaaS DATA Channel CONTROL Channel Source Endpoint Destination Endpoint Subscriber owned and administered storage system Globus “connector” software No data relay or staging via Globus cloud service Subscriber Control Domain Globus Control Domain Single, globally accessible multi-tenant service
- 8. Conceptual architecture: Sharing Managed Endpoint Subscriber Control Domain Globus Control DomainGlobus managed ”overlay” permissions Shared Endpoint DATA Channel CONTROL Channel Subscriber managed filesystem permissions External User Control Domain
- 9. Develop apps, services,and workflows 9
- 10. Use(r)-appropriate interfaces 10 GET /endpoint/go%23ep1 PUT/endpoint/vas#my_endpt 200 OK X-Transfer-API-Version: 0.10 Content-Type: application/json … Globus service Web CLI Rest API
- 11. What is Petrel? (theexecutive version) 11
- 12. 12 Petrel online store petrel.alcf.anl.gov 94Gbit/s Petrel—Blue Waters Petrel 3.2 petabytes 100 Gbps X X Ceph 3.2
- 13. Petrel • Accessible viaGlobus (GCS & HTTPS) • WAN & LAN connections at 100+ Gbps • 100TB default allocation • Keep large datasets close to computational resource • PI-managed access control – PIs can delegate ACL management • Configurable solution to share data – Internal to project – With collaborators
- 14. What does Petrelenable? (the cool stuff) 14
- 15. Petrel: cornerstone ofa data foundation 15 Capability Application Key
- 16.
- 17. 17 PetrelKube Kubernetes cluster foron- demand computing GCS+HTTPS Domain-specific data portals PetrelKube On-demand compute on Petrel data Materials Data Facility RAMSES Neuro- carto- graphy Cappello compress Capability Application Key Petrel: cornerstone of a data foundation
- 18. Expanded Petrel 18 PetrelKube Kubernetes clusterfor on- demand computing GCS+HTTPS Domain-specific data portals PetrelKube On-demand compute on Petrel data Petrel-3 Expand to 3PB with Ceph file system Materials Data Facility RAMSES Neuro- carto- graphy Cappello compress Capability Application Key Petrel: cornerstone of a data foundation
- 19. Expanded Petrel 19 PetrelKube Kubernetes clusterfor on- demand computing GCS+HTTPS Domain-specific data portals PetrelKube On-demand compute on Petrel data Petrel-3 Expand to 3PB with Ceph file system JupyterHub Notebook serving with access to Petrel data JupyterHub Notebook serving on Cooley Stopping power More, better results Materials Data Facility RAMSES Neuro- carto- graphy Cappello compress Capability Application Key
- 20. Expanded Petrel 20 PetrelKube Kubernetes clusterfor on- demand computing GCS+HTTPS Domain-specific data portals PetrelKube On-demand compute on Petrel data Petrel-3 Expand to 3PB with Ceph file system JupyterHub Notebook serving with access to Petrel data JupyterHub Notebook serving on Cooley Automate Declarative Flows Neurocartography APS/ALCF Kasthuri wflow Stopping power More, better results Materials Data Facility RAMSES Neuro- carto- graphy Cappello compress Capability Application Key
- 21. Data Source Collector Storage and Compute •Capture dataset creation • Review center position APS beamline 32-ID ALCF Cooley Cluster • Generate preview and center images • Reconstruct image • Extract metadata Ingest in Globus Search Set sharing ACLs Data publication 1 2 1 Rules 2 • IF new HDF5 files THEN transfer to Cooley • IF new center_pos THEN initiate reconstruction • IF transfer complete THEN execute preview and center finding • IF results THEN return data to APS • IF reconstruction THEN transfer data to Petrel AND publish dataset ALCF Petrel Archive Visualize with Neuroglancer Mosaic tomography for neurocartography (N. Kasthuri, R. Chard, R. Vescovi, et al.) http://tomofish.kasthurilab.com Human-in-the-loop validation
- 22.
- 23. ALCFPortal Project2 Project1 Globus Search 23Petrel DTNs: GCS+HTTPS Index 2 Custompage templates (display) Custom views (logic) Custom page templates (display) Custom views (logic) Index 1 Data 1 Data 2 users ALCF Project Portals Custom CLIs
- 24.
- 25. Hub Configurable HTTP proxy Authenticator User DB Spawner Notebook /api/auth Browser /hub/ /user/[name]/ •ALCF runs a production instance tied to ALCF IdP • Version with Globus Auth on PetrelKube JupyterHub Goal: Liberate the notebook! • Tokens for remote services • APIs for remote actions, e.g. data management via Globus service petrel.alcf.anl.gov
- 26. Example Ad hocdata analysis/results Notebook Petrel Bearer a45cd… Dataset Shared endpoint POST '/endpoint/a3c345f... /mkdir’ 200 OK ... X-Transfer-API-Version: 0.10 Content-Type: application/json ... Analyze
- 27. PetrelKube • JLSE KubernetesTestbed • Modest: 14 nodes • Several workloads are machine learning • JupyterHub with Globus Auth using Zero-to-JupyterHub (z2jh) • FuncX (Function-as-a-Service) 27
- 28. Petrel: Summary • Accessiblevia Globus (GCS & HTTPS) • WAN & LAN connections at 100+ Gbps • 100TB default allocation • Keep large datasets close to computational resource • PI-managed access control – PIs can delegate ACL management • Configurable solution to share data – Internal to project – With collaborators