Implementing a Cloud Storage     Standard for SwiftMark Carlson – SNIA Cloud Storage Co-Chair – mark.carlson@oracle.comDou...
Abstract•   There is growing momentum behind a cloud storage standard that can    level the playing field and expand the m...
Agenda•   Overview of Blueprint•   CDMI Overview•   Design Decisions•   Modules and what they do•   Installing and testing...
Votes• How many people are here?• How many people have heard of CDMI?
Overview of Blueprint• Objects, Containers and Metadata• Map CDMI Standard to Swift functions• CDMI would live side by sid...
CDMI Overview• Cloud Data Management Interface status as a  standard• CDMI conceptual model• CDMI use of Metadata• CDMI Ca...
CDMI Status• CDMI (1.x) was released in April 2010• CDMI 1.0.2 (errata release) in September 2011• CDMI now in Publicly Av...
CDMI Model Related to Swift                                    Root                        System          Container      ...
CDMI APIs• REST/HTTP based APIs• Normal HTTP CRUD operations• Metadata is not stored as HTTP Headers  – Allows for more ex...
CDMI Metadata• System Metadata  – Storage System Metadata – object timestamps,    ACL permissions, counts, etc. (i.e. file...
CDMI Capabilities• Allow a client to discover what standardized  features are available from THIS cloud, and from  THIS ob...
Sample Capability• ...
CDMI Implementations so far• Mezeo - http://www.mezeo.com/cdmi• Scality - http://www.scality.com/ring-organic-storage/• Da...
Design Decisions• Important to maintain fidelity (completeness) between  CDMI and Swift APIs   – Can alternate between the...
Design Decisions – Containers• Both Swift and CDMI support hierarchical folders but  the implement and semantics are diffe...
Design Decisions - Metadata• Swift metadata storage facility is leveraged to  store CDMI metadata• Swift has a limit on me...
Modules and What they do• cdmi.py – entry point• cdmicommoncontroller.py – common ops for CDMI  controllers• cdmicontrolle...
Module Structure• The implementation includes total of 8 modules.  – Six modules are the implementation and two modules   ...
Module Structure• cdmicommoncontroller.py is created to handle  common operations among containers and data  objects such ...
Module Structure• cdmibase.py defines a base controller class so that the  other controllers can inherit from it   • Initi...
Installation• swift/common/middleware   – cdmi.py• swift/common/middleware/cdmiapps   –   cdmicommoncontroller.py   –   cd...
Setup1. In /etc/swift/proxy-server.conf, add cdmi filter before proxy-server[pipeline:main]pipeline = healthcheck cache te...
Optional ConfigurationOptionally, this implementation can be configured to use different accessroot.By default, the access...
TestingCDMI test cases were developed as functional tests, it will access a runningSwift system with CDMI filter enabled. ...
SAMPLE MESSAGE FLOWS
LoginUse the same hostname and port number Swift usesGET /cdmi HTTP/1.1                         (swift: /v1/auth)User-Agen...
Non-CDMI GET : Top-Level ContainerGET /cdmi/AUTH_test/ HTTP/1.1User-Agent: FiddlerX-Auth-Token: AUTH_tk1ad04b87999e4643b50...
CDMI GET : Top-Level ContainerGET /cdmi/AUTH_test/ HTTP/1.1User-Agent: FiddlerX-Auth-Token: AUTH_tk1ad04b87999e4643b5033f1...
Non-CDMI GET : Data ObjectGET /cdmi/AUTH_test/c1/a/b/c HTTP/1.1User-Agent: FiddlerX-Auth-Token: AUTH_tk1ad04b87999e4643b50...
CDMI GET : Data ObjectGET /cdmi/AUTH_test/c1/a/b/c HTTP/1.1User-Agent: FiddlerX-Auth-Token: AUTH_tk1ad04b87999e4643b5033f1...
SAMPLE CAPABILITIES
Capability : Top-Level ContainerGET /cdmi/cdmi_capabilities/AUTH_test/HTTP/1.1 200 OKContent-Type: application/cdmi-capabi...
Capability: Nested ContainerGET /cdmi/cdmi_capabilities/AUTH_test/c1/a/b/HTTP/1.1 200 OKContent-Type: application/cdmi-cap...
Capability : Data ObjectGET /cdmi/cdmi_capabilities/AUTH_test/c1/a/b/c/HTTP/1.1 200 OKContent-Type: application/cdmi-capab...
Demo• Time permitting...
Futures• Swift Proposed Blueprints  – Look for some• Using CDMI for other types of storage & data  paths
Votes• How many people would actually deploy  CDMI if available?• How many people would be interested in  using CMDI as pa...
Q&A                        Thank You!More Info:CDMI Spec: http://cdmi.sniacloud.comCDMI Blueprint: https://blueprints.laun...
Upcoming SlideShare
Loading in …5
×

CDMI For Swift

3,500 views

Published on

Presentation at the OpenStack Folsom Design Summit on the CDMI interface for Swift

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,500
On SlideShare
0
From Embeds
0
Number of Embeds
71
Actions
Shares
0
Downloads
103
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide
  • CDMI - max items, max size, max total size
  • CDMI For Swift

    1. 1. Implementing a Cloud Storage Standard for SwiftMark Carlson – SNIA Cloud Storage Co-Chair – mark.carlson@oracle.comDoug Davis – IBM, STMS Software Standards – dug@us.ibm.comTong Li – IBM, Software Engineer – Software Standards – litong01@us.ibm.comEtherpad: http://etherpad.openstack.org/ImplementingCDMI4SWIFT
    2. 2. Abstract• There is growing momentum behind a cloud storage standard that can level the playing field and expand the market for cloud storage overall by removing the friction of moving from cloud to cloud.• As Swift gains increasing functionality from the broad community of support, it just makes sense to use a standard API to guide the interface to those features.• The Cloud Data Management Interface (CDMI) is a cloud storage standard for both the data path to cloud storage as well as a standard for managing the data once it is in the cloud.• CDMI is being implemented by commercial and academic organizations and there is a blueprint for incorporating an implementation in OpenStack.• This talk will look at the existing features of Essex and planned features for Folsom, comparing them to the features that are already standardized in CDMI. The talk is intended for a technical audience that is already familiar with the workings of cloud storage in general and Swift in particular.
    3. 3. Agenda• Overview of Blueprint• CDMI Overview• Design Decisions• Modules and what they do• Installing and testing• API Examples, demo• Possible Folsom Swift Functions and how to map• Q&A
    4. 4. Votes• How many people are here?• How many people have heard of CDMI?
    5. 5. Overview of Blueprint• Objects, Containers and Metadata• Map CDMI Standard to Swift functions• CDMI would live side by side with the Native Swift and S3 APIs• Done as a Filter• Initial code contribution under review: – https://review.openstack.org/#change,5539
    6. 6. CDMI Overview• Cloud Data Management Interface status as a standard• CDMI conceptual model• CDMI use of Metadata• CDMI Capabilities• CDMI implementations so far
    7. 7. CDMI Status• CDMI (1.x) was released in April 2010• CDMI 1.0.2 (errata release) in September 2011• CDMI now in Publicly Available Specification (PAS) process in JTC 1 (ISO/IEC) to become an international standard• The Cloud Storage TWG in SNIA is creating and publishing Extensions and Profiles of CDMI to evolve the standard as use cases arise
    8. 8. CDMI Model Related to Swift Root System Container Container Capabilities Container Container Capabilities Directory Data Data Object Object Capabilities ObjectLegend: CDMI Model Each CDMI entity has System & User metadata associated with it Swift Model
    9. 9. CDMI APIs• REST/HTTP based APIs• Normal HTTP CRUD operations• Metadata is not stored as HTTP Headers – Allows for more expressive metadata – Presence of X-CDMI-Specification-Version Header indicates were operating on CMDI structures instead of directly against the Data Object contents – Although, Data Object contents is a field in the Object
    10. 10. CDMI Metadata• System Metadata – Storage System Metadata – object timestamps, ACL permissions, counts, etc. (i.e. filesystems) – Data System Metadata – requirements for the object or container of objects • Which Data Services (backup, replication, retention) should apply? • What Data Services are currently applied?• User Metadata – Application specified, and searchable
    11. 11. CDMI Capabilities• Allow a client to discover what standardized features are available from THIS cloud, and from THIS object/container – Capabilities are not permissions – Rather describe what is implemented• Allows CDMI to shrink to fit any cloud offering’s feature set, expand over time without breaking clients• Exposed as a reference from each container and data object
    12. 12. Sample Capability• ...
    13. 13. CDMI Implementations so far• Mezeo - http://www.mezeo.com/cdmi• Scality - http://www.scality.com/ring-organic-storage/• Data Direct Networks http://www.storagebytesnow.com/2011/10/13/datadirect- /• StorageSwitch - http://www.storageswitch.com/index.php/products• CompatibleOne - http://www.slideshare.net/enovance/compatible-one-open• More than a dozen vendors participating in quarterly Cloud Plugfests with pre-release products
    14. 14. Design Decisions• Important to maintain fidelity (completeness) between CDMI and Swift APIs – Can alternate between the two APIs to access the same data• CDMI is a “shrink to fit” standard – Implemented subset of CDMI corresponds to Swifts current functionality• Implemented as a Filter to live side by side with other APIs• Make use of Swift authentication filter• If there is a discrepancy between “world views”, CDMI implementation will “fault”
    15. 15. Design Decisions – Containers• Both Swift and CDMI support hierarchical folders but the implement and semantics are different • Implicit Swift folders are implicit containers through CDMI • Swift pseudo-folders are viewed as containers through CDMI• Creating Containers with CDMI – Explicit creation of each level of sub Container – which then map to pseudo-hierarchical folders/directories• Fault Example: – Via Swift we create two "files": /a/b/c & /a/b – Retrieval of /a/b/cs CDMI metadata will fault since its parent is inconsistent
    16. 16. Design Decisions - Metadata• Swift metadata storage facility is leveraged to store CDMI metadata• Swift has a limit on metadata size and this thus limits CDMI metadata as well • Investigating how to expose these limits via CDMI Capabilities• Metadata stored through Swift API does not (yet) show up as user metadata in CDMI• Metadata stored through CDMI does show up as metadata through Swift API
    17. 17. Modules and What they do• cdmi.py – entry point• cdmicommoncontroller.py – common ops for CDMI controllers• cdmicontrollers.py – if a CDMI mime-type request• noncdmicontrollers.py – non-CDMI mime-type request• cdmibase.py – base class to be extended• cdmiutils.py – utility classes and methods• test_cdmi_container.py – testing of CDMI container functions• test_cdmi_object.py – testing of CDMI object functions
    18. 18. Module Structure• The implementation includes total of 8 modules. – Six modules are the implementation and two modules are the test cases.• The first module is named cdmi.py which serves as a bootstrap or SWGI server. – This module also defines the filter_factory method which is to make the entire module an OpenStack filter. – Inspect each request and dispatch the request to container or data object controllers in other modules
    19. 19. Module Structure• cdmicommoncontroller.py is created to handle common operations among containers and data objects such as read, delete, and capabilities – Responds to the capability retrieval request for both containers and data objects• cdmicontrollers.py and noncdmicontrollers.py are created to handle cdmi content type and non-cdmi content type request operations such as container creation and update
    20. 20. Module Structure• cdmibase.py defines a base controller class so that the other controllers can inherit from it • Initialization and config management function • Login and account controller• cdmiutils.py a few utility methods so the entire implementation can be a little cleaner• test_cdmi_container.py tests containers and capabilities• test_cdmi_object.py tests objects• 70 test cases created to verify various use cases
    21. 21. Installation• swift/common/middleware – cdmi.py• swift/common/middleware/cdmiapps – cdmicommoncontroller.py – cdmicontrollers.py – noncdmicontrollers.py – cdmibase.py – cdmiutils.py• swift/test/functional/cdmi – test_cdmi_container.py – test_cdmi_object.py
    22. 22. Setup1. In /etc/swift/proxy-server.conf, add cdmi filter before proxy-server[pipeline:main]pipeline = healthcheck cache tempauth cdmi proxy-server[filter:cdmi]use = egg:swift#cdmi2. In <swiftroot>/swift.egg-info/entry-points.txt, add the following line at the end of the "paste.filter_factory" section[paste.filter_factory]...cdmi = swift.common.middleware.cdmi:filter_factory3. Restart swift.
    23. 23. Optional ConfigurationOptionally, this implementation can be configured to use different accessroot.By default, the access root is "http://hostname:port/cdmi", you can configurethe implementation to use different root – e.g.:http://hostname:port/another/rootSimply change /etc/swift/proxy-server.conf file by adding the following line inthe [default] section:[default]...cdmi_root = another/root
    24. 24. TestingCDMI test cases were developed as functional tests, it will access a runningSwift system with CDMI filter enabled. Before you run the test cases, makesure CDMI filter configuration is correct by checking the proxy-server.conf fileof your installation.Once OpenStack Swift is up and running, switch to the following directory <SwiftInstallRoot>/test/functional/cdmiRun the following two commands to test container and objects operations. To test container behaviors: > python test_cdmi_container.py To test object behaviors: > python test_cdmi_object.py
    25. 25. SAMPLE MESSAGE FLOWS
    26. 26. LoginUse the same hostname and port number Swift usesGET /cdmi HTTP/1.1 (swift: /v1/auth)User-Agent: FiddlerX-Storage-User: test:testerX-Storage-Pass: testingHost: example.comHTTP/1.1 200 OKContent-Type: text/html; charset=UTF-8Content-Length: 0X-Auth-Token: AUTH_tk1ad04b87999e4643b5033f1d04883343X-Storage-Token: AUTH_tk1ad04b87999e4643b5033f1d04883343X-Storage-Url: http://example.com/cdmi/AUTH_testDate: Thu, 12 Apr 2012 19:30:02 GMT
    27. 27. Non-CDMI GET : Top-Level ContainerGET /cdmi/AUTH_test/ HTTP/1.1User-Agent: FiddlerX-Auth-Token: AUTH_tk1ad04b87999e4643b5033f1d04883343Host: example.comHTTP/1.1 200 OKX-Account-Object-Count: 7X-Account-Bytes-Used: 130X-Account-Container-Count: 3Accept-Ranges: bytesContent-Length: 69Content-Type: text/plain; charset=utf-8Date: Tue, 24 Jan 2012 14:21:46 GMTcdmi_test_container_11327335466cdmi_test_container_11327335467picsNote: this is not defined by the CDMI spec
    28. 28. CDMI GET : Top-Level ContainerGET /cdmi/AUTH_test/ HTTP/1.1User-Agent: FiddlerX-Auth-Token: AUTH_tk1ad04b87999e4643b5033f1d04883343X-CDMI-Specification-Version: 1.0.1Host: example.comHTTP/1.1 200 OKX-Account-Object-Count: 0X-Account-Bytes-Used: 0X-Account-Container-Count: 1Accept-Ranges: bytesContent-Type: application/json; charset=utf-8Content-Length: 248Date: Thu, 12 Apr 2012 19:41:51 GMT{ "mimetype": "application/cdmi-container", "objectName": "AUTH_test/", "capabilitiesURI": "cdmi_capabilities/AUTH_test/", "parentURI": "", "objectType": "application/cdmi-container", "children": [ "cdmi_test_container_11327335466/", "cdmi_test_container_11327335467/", "pics/" ], "metadata": {}}
    29. 29. Non-CDMI GET : Data ObjectGET /cdmi/AUTH_test/c1/a/b/c HTTP/1.1User-Agent: FiddlerX-Auth-Token: AUTH_tk1ad04b87999e4643b5033f1d04883343Host: example.comContent-Length: 0HTTP/1.1 200 OKLast-Modified: Thu, 12 Apr 2012 18:47:38 GMTEtag: 8a492a080b777657ad5e9941a3f903d9Accept-Ranges: bytesContent-Length: 11Content-Type: application/octet-streamDate: Thu, 12 Apr 2012 18:47:57 GMTsome values
    30. 30. CDMI GET : Data ObjectGET /cdmi/AUTH_test/c1/a/b/c HTTP/1.1User-Agent: FiddlerX-Auth-Token: AUTH_tk1ad04b87999e4643b5033f1d04883343X-CDMI-Specification-Version: 1.0.1Host: example.comHTTP/1.1 200 OKX-CDMI-Specification-Version: 1.0.1Content-Type: application/cdmi-objectContent-Length: 346Date: Thu, 12 Apr 2012 18:49:57 GMT{ "completionStatus": "Complete", "mimetype": "application/octet-stream", "valuetransferencoding": "utf-8", "objectName": "c", "capabilitiesURI": "cdmi_capabilities/AUTH_test/c1/a/b/c", "parentURI": "AUTH_test/c1/a/b/", "value": "some values", "valuerange": "0-11", "objectType": "application/cdmi-object", "metadata": {}}
    31. 31. SAMPLE CAPABILITIES
    32. 32. Capability : Top-Level ContainerGET /cdmi/cdmi_capabilities/AUTH_test/HTTP/1.1 200 OKContent-Type: application/cdmi-capabilityX-Cdmi-Specification-Version: 1.0.1Content-Length: 172Date: Thu, 12 Apr 2012 18:20:37 GMT{ "capabilities": { "cdmi_list_children": true, "cdmi_create_container": true }, "objectName": "AUTH_test/", "objectType": "application/cdmi-capability"}
    33. 33. Capability: Nested ContainerGET /cdmi/cdmi_capabilities/AUTH_test/c1/a/b/HTTP/1.1 200 OKContent-Type: application/cdmi-capabilityX-Cdmi-Specification-Version: 1.0.1Content-Length: 355Date: Thu, 12 Apr 2012 18:21:41 GMT{ "parentURI": "cdmi_capabilities/AUTH_test/c1/a/", "objectName": "b/", "capabilities": { "cdmi_create_dataobject": true, "cdmi_read_metadata": true, "cdmi_modify_metadata": true, "cdmi_list_children": true, "cdmi_create_container": true, "cdmi_delete_container": true }, "objectType": "application/cdmi-capability"}
    34. 34. Capability : Data ObjectGET /cdmi/cdmi_capabilities/AUTH_test/c1/a/b/c/HTTP/1.1 200 OKContent-Type: application/cdmi-capabilityX-Cdmi-Specification-Version: 1.0.1Content-Length: 314Date: Thu, 12 Apr 2012 18:23:35 GMT{ "parentURI": "cdmi_capabilities/AUTH_test/c1/a/b/", "objectName": "c", "capabilities": { "cdmi_read_metadata": true, "cdmi_modify_metadata": true, "cdmi_modify_value": true, "cdmi_delete_dataobject": true, "cdmi_read_value": true }, "objectType": "application/cdmi-capability"}
    35. 35. Demo• Time permitting...
    36. 36. Futures• Swift Proposed Blueprints – Look for some• Using CDMI for other types of storage & data paths
    37. 37. Votes• How many people would actually deploy CDMI if available?• How many people would be interested in using CMDI as part of Nova?• Interest in contributing to code or spec?
    38. 38. Q&A Thank You!More Info:CDMI Spec: http://cdmi.sniacloud.comCDMI Blueprint: https://blueprints.launchpad.net/swift/+spec/cdmiCode: https://review.openstack.org/#change,5539

    ×