1. Easy Object Storage Import/Export
Using the S3 Storage Connector
Lee Liming (LLiming@uchicago.edu)
University of Chicago
funded by the National Science Foundation
Award #ACI-1445604

2. funded by the National Science Foundation
Award #ACI-1445604
http://jetstream-cloud.org/
Cloud, specialized for the research community
Researchers browse VM images
saved by other researchers
– Web interface provided by
CyVerse
– Share fully-installed systems with
applications ready to go
– OpenStack API/CLI access for
advanced users
Images can be launched on VMs
scaled for research
– 1-44 vCPUs
– 2-120 GB RAM
– 8-480GB disk
Use is allocated by NSF/XSEDE

3. funded by the National Science Foundation
Award #ACI-1445604
http://jetstream-cloud.org/
Jetstream’s object storage (Swift/Ceph)
• 1.25 PB usable storage
(If all of JS was object store, no VMs,
volumes, etc. Of course, it’s not…)
• Access via Swift APIs
(and any tools that use them)
– Horizon web app
– OpenStack CLI
• AWS S3-compatible
• A place to learn &
evaluate before the
commercial cloud

4. funded by the National Science Foundation
Award #ACI-1445604
http://jetstream-cloud.org/
Research data starts & lives at home
• Campus systems or research facilities/labs
– Data often comes from
facilities/labs/observatories
– Results are often needed elsewhere
– Publishing requirements are becoming important
• Very often, available via Globus
• So…how do we get it from there into Jetstream’s
object store for use via Ceph, Swift, S3, etc?
– Must be: easy, fast, reliable
• (Jetstream to “next step location” is also
valuable…)

5. funded by the National Science Foundation
Award #ACI-1445604
http://jetstream-cloud.org/
Fire-and-forget transfers into Jetstream buckets

6. funded by the National Science Foundation
Award #ACI-1445604
http://jetstream-cloud.org/
So how does this work?
• Globus Connect Server (the
server version of Globus)
offers storage connectors to
subscribers, including an
AWS S3 connector.
• The AWS S3 connector works
equally well with OpenStack
Swift’s S3 dialect.

7. Setting it up
1. Create a VM w/floating IP.
2. Install GCSv5.2 w/S3 connector.
3. Create an S3 storage gateway
requiring an XSEDE ID.
4. Use Globus web app to create a
collection and supply S3-style
credentials.
5. Use the collection to transfer data
in or out.
6. Shelve VM until needed again.

8. Credential management
When the GCSv5 server admin
does this…
…it allows users to
do this.

9. funded by the National Science Foundation
Award #ACI-1445604
http://jetstream-cloud.org/
Load dataset from EBI into a Jetstream bucket