OpenStack Swift currently allows one storage policy for a container. In a typical setup, a storage policy represents backend devices with similar quality of service (QoS) characteristics like IOPS, time-to-first-byte, etc. This conflation of logical namespace with physical resources puts the burden of placing objects based on QoS through namespace partitioning on the application i.e., store objects with QoS1 in Container1, and objects with QoS2 in Container2, etc. This reduces the flexibility of applications in utilizing containers for organizing objects into logically related buckets. This slide deck describes Hybrid Container for OpenStack Swift. A Hybrid Container allows objects within a container to be stored on backends with different QoS characteristics e.g., different storage media like SSD/HDD/tape, or different storage policies such as replication and erasure coding.

1. Hybrid Container Proposal
© 2015 NetApp, Inc. All rights reserved.1
Saeed Ghanbari
Shiva Chaitanya
Deepak Kenchammana
Barry Benight

2. Problem Statement
§ Swift currently allows one storage policy for a container .
§ A swift container is assigned to a single ring (single storage policy) at
creation time.
§ As a result, all objects of that container are stored in one set of backend
devices – which are typically uniform in performance but might vary in
capacity.
§ This conflation of logical namespace with physical resources puts the
burden of placing objects based on service level objectives on the client
- by namespace partitioning
© 2015 NetApp, Inc. All rights reserved.2

3. Proposal
§ In a hybrid container, object placement will be based on QoS (Quality of
Service) requirement.
§ Client specifies QoS, Swift enforces it.
§ QoS requirement supplied through a special header: x-qos.
§ This header could either directly specify the physical back-end or
§ Alternatively, the operator may declare some rules based on which Swift will infer the
appropriate backend, e.g., objects larger than a threshold to be stored on SMR drives.
§ Enforcement mechanism seamless to the swift client:
§ No changes to the externally-visible namespace
§ Minimal changes to client software (just needs to add x-qos header)
© 2015 NetApp, Inc. All rights reserved.3

4. Why not storage policy?
§ Applications can choose the right back-end for a particular QoS through
Storage Policy. Why Hybrid Container?
§ This proposal addresses use cases where Swift operators manage
multiple storage devices transparent to the application.
§ Operator’s key objective to achieve best TCO across archival devices
spanning multiple hardware generations and media types (SMR drives,
tape, optical).
§ This proposal:
§ Caters to applications that will find it harder to adapt to changes in workload &
underlying infrastructure over time. Selection of the right backend for placing the data
and movement mechanisms are best deferred to Swift in such instances.
© 2015 NetApp, Inc. All rights reserved.4

5. © 2015 NetApp, Inc. All rights reserved.5
Proxy Server
Stub ObjA
E
ObjA
meta-data
Swift DAS backend
Overview of Hybrid Container Prototype
HB-proxy
BackendAPI
E Swift Tape …
WSGI middleware
Object Server
HB-obj
BackendAPI
E Swift Tape …
WSGI middleware
Obj
App
Add-on/pluggable
middleware
Patches to
existing code
‘E’ stands for a generic external device that
can be accessed through BackendAPI

6. Code changes
§ Code changes primarily in the form of WSGI middleware at proxy &
object servers
§ Middleware at proxy server:
§ Constructs stub objects for PUTs destined to external backend
§ Talks to external backend directly via a standard multi-backend compatible
API
§ Middleware at object server:
§ Handles requests from background processes. E.g GETs from container-sync,
DELETEs from account-reaper
§ Latency optimization: Offloads internal HEADs from proxy server to determine
presence of stubs
§ Change to object server function:
§ Pass on (size, etag) of full object to container update operation – instead of
the stub object’s.
© 2015 NetApp, Inc. All rights reserved.6

7. Stub object
§ Stub is a zero-sized sparse reference object that points to an object
stored in a different backend
§ Object metadata are attached to the stub
§ All the ‘pointer’ information is attached to the stub as system
metadata
§ Example:
§ X-Object-Sysmeta-Eb-Dest-Path
§ X-Object-Sysmeta-Eb-Dest-Size
§ X-Object-Sysmeta-Eb-Dest-Etag
© 2015 NetApp, Inc. All rights reserved.7

8. Stub object (cont’d)
§ X-Object-Sysmeta-Eb-Dest-Path header helps locate the main object
on a GET request
§ X-Object-Sysmeta-Eb-Dest-Size/Etag headers help service HEAD
operations without accessing main object
§ A stub object is treated as just another regular swift object by
background processes, i.e.,
§ Subject to 3-way replication by object-replicator to ensure reliability
§ Processed by object-auditor to ensure integrity
© 2015 NetApp, Inc. All rights reserved.8

9. Backend API
§ Currently using a simple four-function abstract class that can be
extended for custom implementations for Swift, Tape, Glacier, etc.
§ __init__(args)
§ ‘args’ is an opaque list of params that can encapsulate arbitrary backend-
specific configuration info to enable setup, authentication, etc.
§ PUT(content_iter, path)
§ GET(path)
§ DELETE(path)
§ We have API implementation for a Swift container backend. This can
be used to combine two types of Swift backends (e.g SSD + HDD) to
present one (hybrid) container to user .
© 2015 NetApp, Inc. All rights reserved.9

10. © 2015 NetApp, Inc. All rights reserved.10
Proxy Server
Stub ObjA
E
ObjA
meta-data
Swift DAS backend
Hybrid Container Prototype – PUT flow
HB-proxy
BackendAPI
E Swift Tape …
…
WSGI middleware
Proxy
App
Object Server
HB-obj
BackendAPI
E Swift Tape …
WSGI middleware
Obj
App
Add-on/pluggable
middleware
Patches to
existing code
31
6
4
5
2
Incoming Obj with user
metadata
Write Obj to external
backend
Construct & Write Stub
with system metadata to
point to external object.
Check via an internal
HEAD if PUT is
overwriting an existing
stub
If yes, delete older object
in external backend
Write stub + user
metadata to swift
backend
Update container DB with
(size, etag) of real object
1
2
3
4
5
6
7
7
3
‘E’ stands for a generic external device that
can be accessed through BackendAPI

11. © 2015 NetApp, Inc. All rights reserved.11
Proxy Server
Stub ObjA
E
ObjA
meta-data
Swift DAS backend
Hybrid Container Prototype – GET flow
HB-proxy
BackendAPI
E Swift Tape …
…
WSGI middleware
Proxy
App
Object Server
HB-obj
BackendAPI
E Swift Tape …
WSGI middleware
Obj
App
Add-on/pluggable
middleware
Patches to
existing code
21
34
Incoming GET request
Retrieve object from
object server
Retrieve object from DAS
backend
If object received is a
stub, read the object
contents from backend
1
2
3
4
2
‘E’ stands for a generic external device that
can be accessed through BackendAPI

12. © 2015 NetApp, Inc. All rights reserved.12
Proxy Server
Stub ObjA
E
ObjA
meta-data
Swift DAS backend
Hybrid Container Prototype – DELETE flow
HB-proxy
BackendAPI
E Swift Tape …
…
WSGI middleware
Proxy
App
Object Server
HB-obj
BackendAPI
E Swift Tape …
WSGI middleware
Obj
App
Add-on/pluggable
middleware
Patches to
existing code
21
5
3
4
Incoming DELETE
request
Send DELETE request to
object server
Check via an internal
HEAD if DELETE is
removing an existing stub
If yes, delete object in
external backend
Delete Stub object in
DAS backend
Remove object entry from
container DB
1
2
3
4
5
6
6
2
‘E’ stands for a generic external device that
can be accessed through BackendAPI

13. © 2015 NetApp, Inc. All rights reserved.13
Container server
Stub ObjA
E
ObjA
meta-data
Swift DAS backend
Internal requests – e.g from container-sync
Container-sync
Object Server
HB-obj
BackendAPI
E Swift Tape …
WSGI middleware
Obj
App
Add-on/pluggable
middleware
Patches to
existing code
2
3
GET request from
container-sync to object
server
Fetch object from DAS
backend
If stub, fetch the object
contents from external
backend
1
2
3
1
‘E’ stands for a generic external device that
can be accessed through BackendAPI