What's New With Globus
•Download as PPTX, PDF•
1 like•366 views
These slides were prepared for a briefing at the 2019 PEARC conference, by Vas Vasiliadis, Globus chief customer officer.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to What's New With Globus
Similar to What's New With Globus (20)
More from Globus
More from Globus (20)
Recently uploaded
Recently uploaded (20)
What's New With Globus
- 1. What’s New with Globus? Vas Vasiliadis vas@uchicago.edu
- 2. 6,902 active shared endpoints 70+ petabyte movers 675 PB moved 23,450 active personal endpoints 93 billion files processed 1,868 active server endpoints 110+ subscribers 2.9 PB largest transfer to date 99.9% availability 710 identity providers 1,923 most shared endpoints at a single institution 111,000 registered users Globus by the numbers
- 3. Manage Protected Data 3 Higher assurance levels for HIPAA and other regulated data • Transfer and share… – PHI (Protected Health Information) – PII (Personally identifiable information) – Controlled Unclassified Information • Security controls comply with… – NIST 800-53 Low – Superset of NIST 800-171 Low • Optional BAA with UChicago
- 4. Product enhancements for high assurance • Additional authentication assurance – Authenticate with specific identity… – …within specific time – …within specific session • Application instance isolation – Per application – Per session (~browser session) • Encryption of user data in transit and Globus data at rest • Detailed audit logs: Globus service + your DTNs
- 5. Product enhancements for high assurance • Additional security requirements enforced on management of all high assurance resources – Data access, and any interaction that can lead to data access – Examples: Groups, Management Console • Enhanced user interfaces (web app and CLI) for seamless management of protected data
- 6. Services enabled • Globus Services: Auth, Transfer & Sharing, Groups • Globus Connect Server v5.2 and above • Globus Connect Personal v3.x • Web app (app.globus.org) • Globus Command Line Interface (CLI) • Connectors: POSIX, Google Drive, AWS S3, CEPH
- 7. Operational enhancements for high assurance • Intrusion detection and prevention • Encryption • Enhanced logging • Secure remote access, access control, and secure practices for laptops • Uniform configuration management and change control • AWS best practices for secure environment: VPCs, security groups, IAM best practices
- 8. New subscription levels • High Assurance – 33% uplift on Standard subscription and on premium connectors • BAA – 50% uplift on Standard subscription and on premium connectors
- 10. Web app enhancements • Accessibility – Target WCAG 2.0 AA compliance • Responsiveness and touch • Works with new connectors collections.globus.org 10
- 11. Web app enhancements • Customizable interface – Single vs. dual panel – Compact file listing display – Columns displayed • Continue incorporating user feedback
- 12. CLI enhancements • Support for use with high assurance collections • '--format UNIX': output suitable for line-oriented processing with typical Unix tools • Added 'globus rm' command • 'globus whoami --linked-identities': shows all linked identities • '--timeout-exit-code': overrides the default exit code for commands which wait on tasks • Enhancements to SDK as needed 12
- 13. Globus for Box • Extends the value of your Box deployment • Unifies access to cloud and on-premise storage • Transitions protected data (HIPAA-regulated, CUI) seamlessly between Box and other storage systems 13
- 16. Make Box part of your research storage ecosystem globus.org/connectors/box docs.globus.org/premium-storage-connectors/box
- 17. Connector updates • Enhanced user experience for credential handling for several connectors (GCSv5) • AWS S3 – Automated multi-region support • Google Drive – Enhancement to retry handling for large transfers • HPSS – Support added for HPSS 7.5 (7.3 to 7.5 supported) – Improved asynchronous staging from tape 17
- 18. S3 compatible systems • Initial customer deployments • Validation, testing and vendor engagement planned • Additional systems driven by customer demand PLEASE CONTACT US BEFORE DEPLOYING ANY OF THESE! 18
- 20. GCSv5 architecture and new terminology
- 21. Globus Connect Server v5.3 • Subsumes GCS version 5.0, 5.1, 5.2 (upgrade now) • Standard and high assurance guest collections (sharing) • High assurance mapped collections • Connectors: POSIX, AWS S3, CEPH, Google Drive, Box • High assurance, standard gateways on same endpoint • Data access protocols: GridFTP and HTTPS
- 22. HTTPS access to Globus endpoints • Browser based up/download • Put your (research) storage “on the web” • Enforce same security policies 22
- 23. Globus Connect Server v5 Milestones v5.0: Google Drive v5.1: POSIX guest collections, HTTPS v5.x: v4 feature parity+ v5.3 • Multi DTN support • Additional storage systems • Endpoint specific identity providers • … Other features v5.2: High assurance v5.4: …
- 24. Recent Transfer enhancements • Verify transfer using client provided checksums – User provided checksum used rather than source checksum for verification • Improvements for scaling transfer service – Multiple nodes for transfer service for higher availability and reliability – Allows for code updates with no downtime 24
- 25. SSH with OAuth • Securely access resource using SSH with federated identity – Facilitates automation, eliminates SSH key management – Replacement for deprecated GSI OpenSSH • First version released – Server side PAM module with Globus Auth support – Command line client • Open source, community support – Not part of the standard subscription – OAuth SSH Client: https://pypi.org/project/oauth-ssh/ – OAuth SSH Server PAM module: https://github.com/xsede/oauth-ssh
- 26. Where are we headed?
- 27. Enhancing the core: Transfer Building the future: Platform
- 28. Globus Transfer: A complete solution ☑ Bulk transfer and sync ☑ Good end-to-end performance in myriad of real world settings ☑ End-to-end reliability ☑ Robust security, with federated identities ☑ Layers onto diverse storage systems ☑ Web-compatible client/server remote access ☑ Easy to use interfaces ☑ Easy installation and administration ☑ Sharing data with guest users ☑ Dedicated professional support 28
- 29. Rethinking data publication • Limited adoption – Not easily customizable • Maintenance Challenges – Costly to maintain – JRE licensing concerns • Going forward – Code will be open source – Leverage platform • Invest in higher priorities
- 30. JLSEUChicago ALCFAPS Publication7 Kasthuri Lab: Building the connectome Imaging1 Lab Server 1 Acquisition2 Lab Server 2 Pre-processing3 Preview/Center4 Reconstruction6Visualization8 User validation5 Science!9 Neuroanatomy reconstruction pipeline
- 31. Automation: Neuroanatomy Web form User input Search Ingest Share Set policy Identifier Mint DOI funcX Auth Get credentials Automate Run job Describe Get metadata Transfer Transfer data funcX Run job Transfer Transfer data
- 32. Our (ambitious) goals for the Globus platform • Transform how research applications, services, and workflows are created, delivered, used, and sustained – Scientific instrument data processing – Repositories: Make data more FAIR – Science gateways • Facilitate creation of interoperable app ecosystem 32
- 33. Globus platform services • Identity and Access Management (IAM) – Federated identity login, Groups, Attributes, Access Control – Globus Auth: Oauth authorization provider • Connect • Transfer – Building a family of services • Execution • Search, Identifiers • Automation – Queues, Events – Triggers, Actions, Flows 33
- 34. Platform status • Generally Available in a few years • Separate product with separate sustainability model • Early engagements help shape product direction – Argonne Leadership Computing Facility, Materials Data Facility, – NCAR Research Data Archive, NSO, … – Use in Globus products • Multiple integrations facilitate more complete solution – e.g. Django, JupyterHub – Follow progress: globus-integration-examples.readthedocs.io • Currently accessible via professional services team
- 35. Thank you to our sponsors... U . S . D E P A R T M E N T O F ENERGY
Editor's Notes
- Use cases – HIPAA/protected data enclave, multi-institutional trials,
- Access Control Identities provided and managed by institution Globus acts as identity broker only, does not access or store any institutional user credentials Institution controls all access policies (at multiple levels) who can access what data and with what permissions who can share what data and with what permissions all access policies can be changed or revoked at any time
- Protected Environment using either AWS KMS encryption or AWS service-specific encryption options.
- Data access CLI access Private window Groups page
- we are working towards being compliant with W3 established accessibility standards –increased visual contrast, ability to resize all of the GUI text using standard browser controls, code to support screen readers, the GUI adapts to a users’ screen from mobile devices all the way up to large high resolution desktop displays
- Greg’s talk
- Backup, data management plans, archive…use Globus for all those use cases Layers on Box
- Our current set of premium connectors; care and feed
- Box collection creation Data transfer Sharing
- Pricing – same as Google Drive connector
- Care and feed
- Customer demand + sustainble model for maintaining the connectors. Ask for any input.
- And what we once called endpoints are now called Collections Mapped = host endpoint Guest = shared endpoint Don’t forget we now also offer HTTPS as well!
- Hand to Steve who takes it from here…
- BE POSITIVE! IT’S ALL GOOD! WE’RE DOING IT FOR YOUR OWN GOOD! WE CAN INVEST ON COOL NEW STUFF
- A good example of the science that Globus facilitates is the work being done by Bobby Kasthuri Lab at Argonne Lab HIs group have set out to map the brain, or build the connectome, as it’s known It’s an ambitious undertaking that involves massive mounts of data They start with samples that are imaged using a beamline at the Advanced Photon Source They get time from APS every 2 to 3 days, and efficiency and automation allows them to make the most of the time at APS. The initial set of raw images undergo some prep-processing and are sent to the Argonne Leadership Computing Facility for analysis A scientist then previews the images and makes any needed adjustments to the experiment Once everything is properly configured the sample is fully imaged and reconstructed at the ALCF The datasets are then moved to a petascale storage system called Petrel where they are annotated with metadata and published with a persistent identifier Researchers at Uchicago and elsewhere can then search and extract relevant subsets of the data to analyze further …and then Science happens!
- No timeframe for general availability Last year we talked about services such as Automate and Search. These are fundamentally platform services, for developers to leverage in their own applications, services, and solutions. We continue to make good progress on these However, realistically, due to funding constraints, these platforms services will not be generally available for several more year In the mean time we are partnering with select groups to use prototype and limited production platform services To learn what is really needed before going GA If you are interested in experimenting with them, please talk with us. We will be selective.