Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Easy Object Storage Import/Export Using the S3 Connector on Jetstream

11 views

Published on

This presentation was given at the 2019 GlobusWorld Conference in Chicago, IL by Lee Liming from University of Chicago and Globus.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Easy Object Storage Import/Export Using the S3 Connector on Jetstream

  1. 1. Easy Object Storage Import/Export Using the S3 Storage Connector Lee Liming (LLiming@uchicago.edu) University of Chicago funded by the National Science Foundation Award #ACI-1445604
  2. 2. funded by the National Science Foundation Award #ACI-1445604 http://jetstream-cloud.org/ Cloud, specialized for the research community Researchers browse VM images saved by other researchers – Web interface provided by CyVerse – Share fully-installed systems with applications ready to go – OpenStack API/CLI access for advanced users Images can be launched on VMs scaled for research – 1-44 vCPUs – 2-120 GB RAM – 8-480GB disk Use is allocated by NSF/XSEDE
  3. 3. funded by the National Science Foundation Award #ACI-1445604 http://jetstream-cloud.org/ Jetstream’s object storage (Swift/Ceph) • 1.25 PB usable storage (If all of JS was object store, no VMs, volumes, etc. Of course, it’s not…) • Access via Swift APIs (and any tools that use them) – Horizon web app – OpenStack CLI • AWS S3-compatible • A place to learn & evaluate before the commercial cloud
  4. 4. funded by the National Science Foundation Award #ACI-1445604 http://jetstream-cloud.org/ Research data starts & lives at home • Campus systems or research facilities/labs – Data often comes from facilities/labs/observatories – Results are often needed elsewhere – Publishing requirements are becoming important • Very often, available via Globus • So…how do we get it from there into Jetstream’s object store for use via Ceph, Swift, S3, etc? – Must be: easy, fast, reliable • (Jetstream to “next step location” is also valuable…)
  5. 5. funded by the National Science Foundation Award #ACI-1445604 http://jetstream-cloud.org/ Fire-and-forget transfers into Jetstream buckets
  6. 6. funded by the National Science Foundation Award #ACI-1445604 http://jetstream-cloud.org/ So how does this work? • Globus Connect Server (the server version of Globus) offers storage connectors to subscribers, including an AWS S3 connector. • The AWS S3 connector works equally well with OpenStack Swift’s S3 dialect.
  7. 7. Setting it up 1. Create a VM w/floating IP. 2. Install GCSv5.2 w/S3 connector. 3. Create an S3 storage gateway requiring an XSEDE ID. 4. Use Globus web app to create a collection and supply S3-style credentials. 5. Use the collection to transfer data in or out. 6. Shelve VM until needed again.
  8. 8. Credential management When the GCSv5 server admin does this… …it allows users to do this.
  9. 9. funded by the National Science Foundation Award #ACI-1445604 http://jetstream-cloud.org/ Load dataset from EBI into a Jetstream bucket

×