SwiftHLM - OpenStack Swift Extension for High Latency Media Such as Tape and Optical Disc, Internals and Interfaces as of Feb 21, 2017

1. SwiftHLM
OpenStack Swift Extension for
High Latency Media
(such as tape, optical)
Slavisa Sarafijanovic, IBM Research Zurich
sla@zurich.ibm.com
OpenStack Project Team Gathering (PTG), Atlanta
Feb 20-24, 2017

2. End User Application
Load Balancer
Swift Proxy Server Swift Proxy Server Swift Proxy Server
Swift Storage
Server #1
Swift Storage
Server #2
Swift Storage
Server #3
Swift Storage
Server#4
SwiftHLM
Midleware
SwiftHLM
Handler
Request
queue
Tape
Backend
Optical
Backend
Backend API
Connector
Backend API
Connector
SwiftHLM
Dispatcher
(M/R Process)
create_per-node_lists_of_objects()
create_per-storage-node_lists_of_objects()
map-objects-to-files()
(or to storage backend objects…)
Dispatcher – asynchronous Migration/Recall background process that
can run on a proxy or a storage server, or on any node with Swift
installed and Swift configuration and ring definition files locally available
Backends (examples of suitable backends*):
•Tape-equipped file system
• IBM Spectrum Archive (PoC integrated w/ SwiftHLM)
• BDT Tape Library Connector (TLC)
•Optical storage device:
• Fujitsu DS700 Blu-Ray (exposes REST API)
* Some of the listed names are company product trademarks
SwiftHLM Architecture
SwiftHLM
Handler
SwiftHLM
Handler
SwiftHLM
Handler
Legend:
Migration/Recall request
Request status query
Object status query
Async w.r.t. user request
2

3. SwiftHLM user/application API
* Migrate/Recall
POST http://<host>:<port>/hlm/v1/<action>/<account>/<cont>/<obj>
POST http://<host>:<port>/hlm/v1/<action>/<account>/<cont>
<action> is MIGRATE or RECALL (case insensitive)
return code: 202 (ok), or an error code
* RequestsStatus (query pending/non-completed requests)
GET http://<host>:<port>/hlm/v1/REQUESTS/<account>/<cont>/<obj>
GET http://<host>:<port>/hlm/v1/REQUESTS/<account>/<cont>
return code: 200 (ok), or a standard error
return value: JSON-encoded list of pending requests for object or container
-> faster and more efficient than “Status”
* Status (query status of object or container)
GET http://<host>:<port>/hlm/v1/STATUS/<account>/<cont>/<obj>
GET http://<host>:<port>/hlm/v1/STATUS/<account>/<cont>
return code: 200 (ok), or a standard error
return value: JSON-encoded list of objects and their states
-> likely needs limiting number of objects per request (ranged requests)
3

4. SwiftHLM backend API
* Input/output in JSON format
* File backend (example):
{“command” : “status”, “objects” : [
{ “object”:”/a/c/obj1”, ”file”:“/ibm/gpfs/filepath1” },
{ “object”:”/a/c/obj2”, ”file”:“/ibm/gpfs/filepath2” }]}
{“objects” : [
{“object”:”/a/c/obj1”, “file” : “/ibm/gpfs/filepath1”, “status” : “migrated”,},
{“object”:”/a/c/obj2”, “file” : “/srv/node/filepath2”, “status” : “resident”},
{“object”:”/a/c/obj3”, “file” : “/ibm/gpfs/filepath3”, “status” : “premigrated”},
{“object”:”/a/c/obj4”, “file” : “/ibm/gpfs/filepath4”, “status” : “unknown”}]}
* Object backend (to discuss)
{“command” : “status”, “objects” : [
{ “object”:”/a/c/obj1”, ”hlm-object”:“hlm-object-path1” },
{ “object”:”/a/c/obj2”, ”hlm-object”:“/ hlm-object-path2” }]}
{“objects” : [
{“object”:”/a/c/obj1”, “hlm-object” : “/hlm-object-path1”, “status” : “migrated”},
{“object”:”/a/c/obj2”, “hlm-object” : “/hlm-object-path2”, “status” : “resident”},
{“object”:”/a/c/obj3”, “hlm-object” : “/hlm-object-path3”, “status” : “premigrated”},
{“object”:”/a/c/obj4”, “hlm-object” : “/hlm-object-path2”, “status” : “unknown”}]}
4