1.
What’s New with Globus
Q&A Webinar Series
June 26, 2018
Steve Tuecke, Globus Co-Founder

2.
2
Research Computing HPC
Desktop Workstations
Mass Storage Instruments
Personal Resources
Public Cloud
National Resources
Unify access to data across tiers

3.
Public / private cloud stores
External
campus
storage
EC2
Project
repositories,
replication stores
Public repositories
Share with collaborators/community

4.
Analysis
store
Next-Gen Sequencer
MRI
Advanced Light Source
Personal system
Remote visualization
Light Sheet Microscope
High-durability,
low-cost store
Manage data from instruments
Cryo-EM

5.
Develop apps, services, and workflows
5

6.
8,000
active shared
endpoints
90
subscribers
425 PB
transferred
18,000
active GCP
endpoints
70 billion
files processed
1,700
active GCS
endpoints
3 months
longest running transfer
1 PB
largest single
transfer to date
99.9%
availability
500
identity providers
1,042
most shared
endpoints
at a single
institution 100,000
users
Globus by the numbers

7.
What’s New

8.
IBM Spectrum
Scale
Current Planned
Storage Connectors - globus.org/connectors

9.
Connectors for S3 ”compatible” systems
• S3 API is de-facto standard API for object storage
• Make it easier to validate and support connectors for
S3 “compatible” object storage systems
– Functionality and performance test suite
– Improving connector robustness and performance
– E.g., Ceph, ActiveScale, SwiftStack, Wasabi,
IBM Cloud Object Storage System (CleverSafe)
• Also requires vendor engagement and market interest
9

10.
HPSS Connector
• Community has agreed on sustainability model
• NERSC & ORNL investing in enhancements
• Premium storage connector subscription
10

11.
Globus Connect Server version 5.x
• HTTPS access to storage
• Globus Auth (OAuth2) for authentication and authorization
• Scale out deployment without shared file system
• Multiple storage systems simultaneously
• Single port for data access
• Improved endpoint administration
• And more
GCSv5.1 Webinar: https://www.youtube.com/watch?v=Ubu0KhIbIA0

12.
GCS v5 Milestones
v5.0: Google
Drive
v5.1: POSIX guest
collections, HTTPS
v5.2: High assurance
(e.g. HIPAA)
v5.N: includes all
version 4.0 features
v5.3: …
Multi DTN support,
other storage types,
custom identity
providers
…
Other features
v5.1: POSIX guest
collections, HTTPS
Upgrade from GCS v4

13.
Globus Connect Server v5
CONCEPTUAL ARCHITECTURE

14.
GCS v5 differences from GCS v4
• Globus ID is not needed
• Endpoint created using endpoint client identity
– <client_id>@clients.auth.globus.org
– Managed through https://developers.globus.org
• Port 443 rather than 2811 used for GridFTP control channel
– Also HTTPS access, and (eventually) GridFTP data channel
• Each collection is assigned a DNS name under dn.glob.us
– E.g. 988c.8540.dn.glob.us
• Certificates are obtained from Let’s Encrypt
• Need to create storage gateway(s) for data access

15.
Globus Connect Server v5.1 features
• HTTPS (and GridFTP) access to data
• Multiple storage connectors
– POSIX
– Google Drive
• Guest collections (shared endpoints) only
• Single DTN install only
• Authentication for data access using only identity
providers used to login to Globus
https://docs.globus.org/globus-connect-server-v5-installation-guide

16.
Use GCS v5.1 only if you need…
• Google Drive support
– Migrate from 5.0 to 5.1
– Contact us for migration documentation
• HTTPS access to data
– with guest collections (shared endpoint)
• Else wait for feature complete GCS 5.N
GCSv5.1 Webinar: https://www.youtube.com/watch?v=Ubu0KhIbIA0

17.
Coming soon…

18.
Protected data
• High assurance endpoints
– User must authenticate with specific identity within a specified time period,
with browser session and native app device instance isolation
– Audit logging
– Multi-factor authentication
• For data that requires additional security
– HIPAA Personal Health Information (PHI) w/ BAA
– Personally Identifiable Information (PII)
– Sensitive but unclassified
• NIST 800-171 Low
• Requires Globus Connect Server v5.2
• Two additional subscription tiers
– High assurance tier: for all added security features
– BAA tier: high assurance features plus BAA with Uchicago
• Available this Summer
– Transfer, sharing, web app, CLI only.
– Excludes publish, search, identifiers, hosted CLI, GlobusID 24

19.
Command Line Interface
• New Globus CLI is
generally available
– Fully functional
– Many enhancements
– Simple updater
• Deprecating old
hosted SSH CLI
– Will be turned off August 1
• pip install --upgrade --user
globus-cli
https://docs.globus.org/cli

20.
Globus CLI Demo

21.
Publication v1
• Publication v1 app
– Publish datasets
to Globus Search
– Internationalization
• Canadian Federated
Research Data Repository
– https://frdr.ca/
– Uses v1 open source
and Globus Search
27

22.
• Decompose Publication v1 into platform components
• Allow flexible re-composition & adaptation by customers
Describe
Get
metadata
Auth
Get
credentials
Identifiers
Mint DOI
Search
Catalog
Transfer
Create
folder
Transfer
data
Set ACLAutomate… … ……
Publication v2 platform

23.
Globus Search platform service
• Search service:
– Schema agnostic: can use standard (e.g., DataCite) or custom metadata
– Fine grain access control: only returns results that are visible to user
– Plain text search: ranked results
– Faceted search: for data discovery
– Rich query language: ranges, expressions, regex, fuzzy, stemming, etc.
– Scalable: to billions of entries
• Limited production
– Contact us at support@globus.org if you are interested in using it
29

24.
Django Globus Portal App
30

25.
Django Globus Portal Demo

26.
Globus Identifiers platform service
• Issue persistent identifiers
– DOI, ARK, Handle, Globus
– E.g., https://identifiers.globus.org/doi:10.1145/2076450.2076468
• Within a namespace
– E.g., Your University’s DataCite namespace
– Control which identities and groups can create identifiers in your namespace
• Each identifier has:
– Link to data: one or more https URLs, to file, folder or manifest
– Landing page: provided by service, or by user
– Visibility: which identities and groups can see identifier
– Checksum: of the file or manifest
– Metadata: as required by identifier (e.g., DataCite), extensible
– Replaces / Replaced-by: for versioning
• Limited beta available now
– Contact us at support@globus.org if you are interested in using it
32

27.
Jupyter integration
• Authenticate to JupyterHub
with Globus Auth
– Passes tokens into notebooks
as environment variable
• Use Globus data management platform from notebooks
– With Globus Python SDK
33
https://github.com/globus/globus-jupyter-notebooks

28.
Jupyter Integration Demo

29.
SSH with Globus Auth
• Securely access resource using SSH with federated identity
– Leverage same security model as rest of data infrastructure
– Facilitates automation
– Eliminate need to manage SSH key lifecycle and provisioning
• Replaces GSI SSH
• Client side wrapper around local SSH client (globus-ssh …)
• No changes to the SSH server (PAM module)
• Status:
– Beta is imminent, for early customer feedback
– Generally available by end of year
35

30.
SSH with Globus Auth Demo

31.
Groups
• Generally available in web app
• REST API has been in limited production
• Plan on opening some portion to general availability
– Please tell us your use cases
37

32.
New web app
• Complete file manager for any research storage
• Improved browser experience
– Accessibility: WCAG 2.0 AA
– Responsiveness: from large desktop to small phone
– Touch support: for phones and pads
• Leverage Globus Connect HTTPS
– E.g., Preview, download
• Beta available now:
https://app.globus.org
38

33.
New Web App Demo

34.
Join the Globus community
• Access the service: globus.org/login
• Create a personal endpoint: globus.org/app/endpoints/create-gcp
• Documentation: docs.globus.org
• Engage: globus.org/mailing-lists
• Subscribe: globus.org/subscriptions
• Need help? support@globus.org
• Follow us: @globus

35.
Thank you to our sponsors
U . S . D E P A R T M E N T O F
ENERGY

36.
Questions?