Download to read offline
Uploaded byGlobus
What's New With Globus
Globus has reported significant usage metrics including over 6,900 active shared endpoints and the transfer of 675 petabytes of data. It offers enhanced security features tailored for high assurance data management, including compliance with HIPAA and NIST standards, and has rolled out new subscription levels with various pricing increases. Upcoming developments focus on enhancing the platform's core transfer capabilities, building interoperable applications, and transitioning to an open-source model for more customizable solutions.
More Related Content
![[DevDay 2016] OpenStack and approaches for new users - Speaker: Chi Le – Head...](https://cdn.slidesharecdn.com/ss_thumbnails/slideopenstack-devday2016-160414042927-thumbnail.jpg?width=640&height=640&fit=bounds)
What's New With Globus
- 1. What’s New withGlobus? Vas Vasiliadis vas@uchicago.edu
- 2. 6,902 active shared endpoints 70+ petabyte movers 675PB moved 23,450 active personal endpoints 93 billion files processed 1,868 active server endpoints 110+ subscribers 2.9 PB largest transfer to date 99.9% availability 710 identity providers 1,923 most shared endpoints at a single institution 111,000 registered users Globus by the numbers
- 3. Manage Protected Data 3 Higherassurance levels for HIPAA and other regulated data • Transfer and share… – PHI (Protected Health Information) – PII (Personally identifiable information) – Controlled Unclassified Information • Security controls comply with… – NIST 800-53 Low – Superset of NIST 800-171 Low • Optional BAA with UChicago
- 4. Product enhancements forhigh assurance • Additional authentication assurance – Authenticate with specific identity… – …within specific time – …within specific session • Application instance isolation – Per application – Per session (~browser session) • Encryption of user data in transit and Globus data at rest • Detailed audit logs: Globus service + your DTNs
- 5. Product enhancements forhigh assurance • Additional security requirements enforced on management of all high assurance resources – Data access, and any interaction that can lead to data access – Examples: Groups, Management Console • Enhanced user interfaces (web app and CLI) for seamless management of protected data
- 6. Services enabled • GlobusServices: Auth, Transfer & Sharing, Groups • Globus Connect Server v5.2 and above • Globus Connect Personal v3.x • Web app (app.globus.org) • Globus Command Line Interface (CLI) • Connectors: POSIX, Google Drive, AWS S3, CEPH
- 7. Operational enhancements forhigh assurance • Intrusion detection and prevention • Encryption • Enhanced logging • Secure remote access, access control, and secure practices for laptops • Uniform configuration management and change control • AWS best practices for secure environment: VPCs, security groups, IAM best practices
- 8. New subscription levels •High Assurance – 33% uplift on Standard subscription and on premium connectors • BAA – 50% uplift on Standard subscription and on premium connectors
- 9.
- 10. Web app enhancements •Accessibility – Target WCAG 2.0 AA compliance • Responsiveness and touch • Works with new connectors collections.globus.org 10
- 11. Web app enhancements •Customizable interface – Single vs. dual panel – Compact file listing display – Columns displayed • Continue incorporating user feedback
- 12. CLI enhancements • Supportfor use with high assurance collections • '--format UNIX': output suitable for line-oriented processing with typical Unix tools • Added 'globus rm' command • 'globus whoami --linked-identities': shows all linked identities • '--timeout-exit-code': overrides the default exit code for commands which wait on tasks • Enhancements to SDK as needed 12
- 13. Globus for Box •Extends the value of your Box deployment • Unifies access to cloud and on-premise storage • Transitions protected data (HIPAA-regulated, CUI) seamlessly between Box and other storage systems 13
- 15.
- 16. Make Box partof your research storage ecosystem globus.org/connectors/box docs.globus.org/premium-storage-connectors/box
- 17. Connector updates • Enhanceduser experience for credential handling for several connectors (GCSv5) • AWS S3 – Automated multi-region support • Google Drive – Enhancement to retry handling for large transfers • HPSS – Support added for HPSS 7.5 (7.3 to 7.5 supported) – Improved asynchronous staging from tape 17
- 18. S3 compatible systems •Initial customer deployments • Validation, testing and vendor engagement planned • Additional systems driven by customer demand PLEASE CONTACT US BEFORE DEPLOYING ANY OF THESE! 18
- 20. GCSv5 architecture andnew terminology
- 21. Globus Connect Serverv5.3 • Subsumes GCS version 5.0, 5.1, 5.2 (upgrade now) • Standard and high assurance guest collections (sharing) • High assurance mapped collections • Connectors: POSIX, AWS S3, CEPH, Google Drive, Box • High assurance, standard gateways on same endpoint • Data access protocols: GridFTP and HTTPS
- 22. HTTPS access toGlobus endpoints • Browser based up/download • Put your (research) storage “on the web” • Enforce same security policies 22
- 23. Globus Connect Serverv5 Milestones v5.0: Google Drive v5.1: POSIX guest collections, HTTPS v5.x: v4 feature parity+ v5.3 • Multi DTN support • Additional storage systems • Endpoint specific identity providers • … Other features v5.2: High assurance v5.4: …
- 24. Recent Transfer enhancements •Verify transfer using client provided checksums – User provided checksum used rather than source checksum for verification • Improvements for scaling transfer service – Multiple nodes for transfer service for higher availability and reliability – Allows for code updates with no downtime 24
- 25. SSH with OAuth •Securely access resource using SSH with federated identity – Facilitates automation, eliminates SSH key management – Replacement for deprecated GSI OpenSSH • First version released – Server side PAM module with Globus Auth support – Command line client • Open source, community support – Not part of the standard subscription – OAuth SSH Client: https://pypi.org/project/oauth-ssh/ – OAuth SSH Server PAM module: https://github.com/xsede/oauth-ssh
- 26. Where are weheaded?
- 27. Enhancing the core: Transfer Buildingthe future: Platform
- 28. Globus Transfer: Acomplete solution ☑ Bulk transfer and sync ☑ Good end-to-end performance in myriad of real world settings ☑ End-to-end reliability ☑ Robust security, with federated identities ☑ Layers onto diverse storage systems ☑ Web-compatible client/server remote access ☑ Easy to use interfaces ☑ Easy installation and administration ☑ Sharing data with guest users ☑ Dedicated professional support 28
- 29. Rethinking data publication •Limited adoption – Not easily customizable • Maintenance Challenges – Costly to maintain – JRE licensing concerns • Going forward – Code will be open source – Leverage platform • Invest in higher priorities
- 30. JLSEUChicago ALCFAPS Publication7 Kasthuri Lab: Buildingthe connectome Imaging1 Lab Server 1 Acquisition2 Lab Server 2 Pre-processing3 Preview/Center4 Reconstruction6Visualization8 User validation5 Science!9 Neuroanatomy reconstruction pipeline
- 31. Automation: Neuroanatomy Web form User input Search Ingest Share Setpolicy Identifier Mint DOI funcX Auth Get credentials Automate Run job Describe Get metadata Transfer Transfer data funcX Run job Transfer Transfer data
- 32. Our (ambitious) goalsfor the Globus platform • Transform how research applications, services, and workflows are created, delivered, used, and sustained – Scientific instrument data processing – Repositories: Make data more FAIR – Science gateways • Facilitate creation of interoperable app ecosystem 32
- 33. Globus platform services •Identity and Access Management (IAM) – Federated identity login, Groups, Attributes, Access Control – Globus Auth: Oauth authorization provider • Connect • Transfer – Building a family of services • Execution • Search, Identifiers • Automation – Queues, Events – Triggers, Actions, Flows 33
- 34. Platform status • GenerallyAvailable in a few years • Separate product with separate sustainability model • Early engagements help shape product direction – Argonne Leadership Computing Facility, Materials Data Facility, – NCAR Research Data Archive, NSO, … – Use in Globus products • Multiple integrations facilitate more complete solution – e.g. Django, JupyterHub – Follow progress: globus-integration-examples.readthedocs.io • Currently accessible via professional services team
- 35. Thank you toour sponsors... U . S . D E P A R T M E N T O F ENERGY
- 36.
Editor's Notes
- #4 Use cases – HIPAA/protected data enclave, multi-institutional trials,
- #5 Access Control Identities provided and managed by institution Globus acts as identity broker only, does not access or store any institutional user credentials Institution controls all access policies (at multiple levels) who can access what data and with what permissions who can share what data and with what permissions all access policies can be changed or revoked at any time
- #8 Protected Environment using either AWS KMS encryption or AWS service-specific encryption options.
- #10 Data access CLI access Private window Groups page
- #11 we are working towards being compliant with W3 established accessibility standards –increased visual contrast, ability to resize all of the GUI text using standard browser controls, code to support screen readers, the GUI adapts to a users’ screen from mobile devices all the way up to large high resolution desktop displays
- #12 Greg’s talk
- #14 Backup, data management plans, archive…use Globus for all those use cases Layers on Box
- #15 Our current set of premium connectors; care and feed
- #16 Box collection creation Data transfer Sharing
- #17 Pricing – same as Google Drive connector
- #18 Care and feed
- #20 Customer demand + sustainble model for maintaining the connectors. Ask for any input.
- #21 And what we once called endpoints are now called Collections Mapped = host endpoint Guest = shared endpoint Don’t forget we now also offer HTTPS as well!
- #27 Hand to Steve who takes it from here…
- #30 BE POSITIVE! IT’S ALL GOOD! WE’RE DOING IT FOR YOUR OWN GOOD! WE CAN INVEST ON COOL NEW STUFF
- #31 A good example of the science that Globus facilitates is the work being done by Bobby Kasthuri Lab at Argonne Lab HIs group have set out to map the brain, or build the connectome, as it’s known It’s an ambitious undertaking that involves massive mounts of data They start with samples that are imaged using a beamline at the Advanced Photon Source They get time from APS every 2 to 3 days, and efficiency and automation allows them to make the most of the time at APS. The initial set of raw images undergo some prep-processing and are sent to the Argonne Leadership Computing Facility for analysis A scientist then previews the images and makes any needed adjustments to the experiment Once everything is properly configured the sample is fully imaged and reconstructed at the ALCF The datasets are then moved to a petascale storage system called Petrel where they are annotated with metadata and published with a persistent identifier Researchers at Uchicago and elsewhere can then search and extract relevant subsets of the data to analyze further …and then Science happens!
- #35 No timeframe for general availability Last year we talked about services such as Automate and Search. These are fundamentally platform services, for developers to leverage in their own applications, services, and solutions. We continue to make good progress on these However, realistically, due to funding constraints, these platforms services will not be generally available for several more year In the mean time we are partnering with select groups to use prototype and limited production platform services To learn what is really needed before going GA If you are interested in experimenting with them, please talk with us. We will be selective.