![Copyright©2015 NTT corp. All Rights Reserved.
Swift Automated Tiering
Kazuhiro MIYAHARA [IRC: m_kazuhiro]
miyahara.kazuhiro@lab.ntt.co.jp
NTT Software Innovation Center
Copyright(c)2009-2016 NTT CORPORATION. All Rights Reserved.](https://image.slidesharecdn.com/swiftautomatedtiering20160303-160302235504/85/Swift-Automated-Tiering-Swift-Hackathon-Bristol-1-320.jpg)
Recommended
Recommended
More Related Content
Similar to Swift Automated Tiering (Swift Hackathon Bristol)
Similar to Swift Automated Tiering (Swift Hackathon Bristol) (20)
Recently uploaded
Recently uploaded (20)
Swift Automated Tiering (Swift Hackathon Bristol)
- 1. Copyright©2015 NTT corp. All Rights Reserved. Swift Automated Tiering Kazuhiro MIYAHARA [IRC: m_kazuhiro] miyahara.kazuhiro@lab.ntt.co.jp NTT Software Innovation Center Copyright(c)2009-2016 NTT CORPORATION. All Rights Reserved.
- 2. 2Copyright©2015 NTT corp. All Rights Reserved. Goal • Discuss and get agreement for a particular design for our “Automated-Tiering” implementation. (Especially focus on moving and linking objects among tiers) Note : This slide will show you overview of our implementation. Note : For more detail, please see the gerrit code <https://review.openstack.org/#/c /287057/> Agenda • Background • Changes of Swift • Components of Our Proposal • Overview of Deliverer • Deliverer’s Process (STEP 1) • States in STEP 2 ~ 4 • Deliverer’s Process (STEP 2 ~ 4) • Demo Goal & Agenda
- 3. 3Copyright©2015 NTT corp. All Rights Reserved. Our Use Case Contents (e.g. movies, pictures, music) providing service. In this service, new contents are requested frequently and gradually becoming less frequently requested. Our Requirement Store frequently requested objects into replication policy containers for performance and infrequently requested objects into erasure coding policy containers for capacity efficiency. Background
- 4. 4Copyright©2015 NTT corp. All Rights Reserved. Seemingly, containers have variable storage policy objects. • Objects move among some different storage policy containers automatically. • Swift ensures transparent access to the automatically moved objects. Changes of Swift “a/c” policy index : 0 “a/c/o2” Policy index : 1 “a/c/o4” Policy index : 0 “a/c/o6” Policy index : 1 “a/c/o1” Policy index : 0 “a/c/o3” Policy index : 2 “a/c/o5” Policy index : 0 Container Object
- 5. 5Copyright©2015 NTT corp. All Rights Reserved. Deliverer (daemon) : what I focus on today • Move objects to the suitable policy container. Addresser (daemon) : future work • Search objects whose policy is not suitable and decide their suitable policy. Components of Our Proposal addresser deliverer “.delivery_task” PUT GET delivery task object : Includes an object path “a/c/o” and a storage policy index “N” In the name. The name means “N” is the suitable policy for “a/c/o”. Account Object Container … … … …
- 6. 6Copyright©2015 NTT corp. All Rights Reserved. • Deliverers run on storage nodes (as well as object-replicator). • 4 steps in Deliverer’s process. • STEP 1 : GET a list of delivery task objects. • STEP 2 : COPY each entity object to a suitable policy container. • STEP 3 : PUT transparent link objects. • STEP 4 : DELETE extra objects. Note : STEP 3 and 4 may be skipped. Overview of Deliverer deliverer object-server node deliverer object-server node deliverer object-server node
- 7. 7Copyright©2015 NTT corp. All Rights Reserved. GET lists of objects from a special account “.delivery_task” Deliverer’s process (STEP 1) deliverer “.delivery_task” “12345600” “2:a1/c1/o1” “0:a3/c3/o3” “1:a2/c2/o2” … GET .delivery_task/12345600 List of delivery task objects object name format : “<storage policy index>:<object path>” storage policy index : The suitable policy index. object path : The object path whose policy is not suitable. Users can transparently access the moved object with this path. Account Object Container Container names are the hours in Unix time format when the delivery task objects are created.
- 8. 8Copyright©2015 NTT corp. All Rights Reserved. 2 states of tiered objects • State A : stored policy = original policy • State B : stored policy ≠ original policy States in STEP 2 ~ 4 “a/c_silver” policy index : 1 policy name : silver “a/c” policy index : 0 policy name : gold “a/c_silver/o” Policy index : 1 Entity “a/c” policy index : 0 policy name : gold “a/c/o” Policy index : 0 Entity State B of a/c/oState A of a/c/o A SLO manifest object with only one segment “a/c_silver/o” original container derivative container The object which an user PUT first to “a/c/o” original policy derivative policy Object Container Link object “a/c/o” Policy index : 0 Link to “a/c_silver/o”
- 9. 9Copyright©2015 NTT corp. All Rights Reserved. 3 cases in STEP 2 ~ 4. Case 1 : From the original policy to a derivative policy (STEP 2 [COPY], STEP 3 [PUT Link]) Case 2 : From a derivative policy to the original policy (STEP 2 [COPY], STEP 4 [DELETE]) Case 3 : From a derivative policy to another derivative policy (STEP 2 [COPY], STEP 3 [PUT Link], STEP 4 [DELETE]) Deliverer’s Process (STEP 2 ~ 4)
- 10. 10Copyright©2015 NTT corp. All Rights Reserved. Case 1 : From the original policy to a derivative policy • STEP 2 : COPY each entity object to a suitable policy container. • STEP 3 : PUT transparent link object. Deliverer’s Process (STEP 2 ~ 4) “a/c_silver” policy index : 1 policy name : silver after “a/c” policy index : 0 policy name : gold “a/c/o” Policy index : 0 Entity “a/c_silver/o” Policy index : 1 Entity “a/c_silver” policy index : 1 policy name : silver before “a/c” policy index : 0 policy name : gold “a/c/o” Policy index : 0 Entity Object Container Link object
- 11. 11Copyright©2015 NTT corp. All Rights Reserved. Case 1 : From the original policy to a derivative policy • STEP 2 : COPY each entity object to a suitable policy container. • STEP 3 : PUT transparent link objects. Deliverer’s Process (STEP 2 ~ 4) “a/c_silver” policy index : 1 policy name : silver before “a/c” policy index : 0 policy name : gold “a/c/o” Policy index : 0 Entity “a/c_silver/o” Policy index : 1 Entity “a/c_silver” policy index : 1 policy name : silver after “a/c” policy index : 0 policy name : gold “a/c_silver/o” Policy index : 1 Entity “a/c/o” Policy index : 0 Link to “a/c_silver/o” Object Container Link object
- 12. 12Copyright©2015 NTT corp. All Rights Reserved. Case 2 : From a derivative policy to the original policy • STEP 2 : COPY each entity object to a suitable policy container. • STEP 4 : DELETE extra objects. Deliverer’s Process (STEP 2 ~ 4) “a/c_silver” policy index : 1 policy name : silver before “a/c” policy index : 0 policy name : gold “a/c_silver/o” Policy index : 1 Entity “a/c_silver” policy index : 1 policy name : silver after “a/c” policy index : 0 policy name : gold “a/c/o” Policy index : 0 Entity “a/c_silver/o” Policy index : 1 Entity Object Container Link object “a/c/o” Policy index : 0 Link to “a/c_silver/o”
- 13. 13Copyright©2015 NTT corp. All Rights Reserved. Case 2 : From a derivative policy to the original policy • STEP 2 : COPY each entity object to a suitable policy container. • STEP 4 : DELETE extra objects. Deliverer’s Process (STEP 2 ~ 4) “a/c_silver” policy index : 1 policy name : silver before “a/c” policy index : 0 policy name : gold “a/c/o” Policy index : 0 Entity “a/c_silver/o” Policy index : 1 Entity “a/c_silver” policy index : 1 policy name : silver after “a/c” policy index : 0 policy name : gold “a/c/o” Policy index : 0 Entity Object Container Link object
- 14. 14Copyright©2015 NTT corp. All Rights Reserved. Case 3 : From a derivative policy to another derivative policy • STEP 2 : COPY each entity object to a suitable policy container. • STEP 3 : PUT transparent link objects. • STEP 4 : DELETE extra objects. Deliverer’s Process (STEP 2 ~ 4) before “a/c” policy index : 0 policy name : gold “a/c_bronze” policy index : 2 policy name : bronze “a/c_silver” policy index : 1 policy name : silver “a/c_silver/o” Policy index : 1 Entity after “a/c” policy index : 0 policy name : gold “a/c_bronze” policy index : 2 policy name : bronze “a/c_silver” policy index : 1 policy name : silver “a/c_bronze/o” Policy index : 2 Entity “a/c_silver/o” Policy index : 1 Entity Object Container Link object “a/c/o” Policy index : 0 Link to “a/c_silver/o” “a/c/o” Policy index : 0 Link to “a/c_silver/o”
- 15. 15Copyright©2015 NTT corp. All Rights Reserved. Case 3 : From a derivative policy to another derivative policy • STEP 2 : COPY each entity object to a suitable policy container. • STEP 3 : PUT transparent link objects. • STEP 4 : DELETE extra objects. Deliverer’s Process (STEP 2 ~ 4) before “a/c” policy index : 0 policy name : gold “a/c_bronze” policy index : 2 policy name : bronze “a/c_silver” policy index : 1 policy name : silver “a/c_bronze/o” Policy index : 2 Entity “a/c_silver/o” Policy index : 1 Entity after “a/c” policy index : 0 policy name : gold “a/c_bronze” policy index : 2 policy name : bronze “a/c_silver” policy index : 1 policy name : silver “a/c_bronze/o” Policy index : 2 Entity “a/c_silver/o” Policy index : 1 Entity Object Container Link object “a/c/o” Policy index : 0 Link to “a/c_silver/o” “a/c/o” Policy index : 0 Link to “a/c_bronze/o”
- 16. 16Copyright©2015 NTT corp. All Rights Reserved. Case 3 : From a derivative policy to another derivative policy • STEP 2 : COPY each entity object to a suitable policy container. • STEP 3 : PUT transparent link objects. • STEP 4 : DELETE extra objects. Deliverer’s Process (STEP 2 ~ 4) before “a/c” policy index : 0 policy name : gold “a/c_bronze” policy index : 2 policy name : bronze “a/c_silver” policy index : 1 policy name : silver “a/c_bronze/o” Policy index : 2 Entity “a/c_silver/o” Policy index : 1 Entity after “a/c” policy index : 0 policy name : gold “a/c_bronze” policy index : 2 policy name : bronze “a/c_silver” policy index : 1 policy name : silver “a/c_bronze/o” Policy index : 2 Entity Object Container Link object “a/c/o” Policy index : 0 Link to “a/c_bronze/o” “a/c/o” Policy index : 0 Link to “a/c_bronze/o”
- 17. 17Copyright©2015 NTT corp. All Rights Reserved. This Demo show you states of Swift with a Deliverer. • State 1 • A object is in original container • State 2 • The object’s policy is changed from the original to a derivative • State 3 • The object’s policy is changed from a derivative to another derivative • State 4 • The object’s policy is changed from a derivative to the original Note : In this demo, our demo middleware “delivery-address” is in Swift. The delivery-address middleware convert “POST a/c /o ‘X-Delivery-Address:silver’” to “PUT .delivery_task/XXX/<index of silver>:a/c /o” The delivery-address” middleware is NOT a component of automated- tiering. DEMO
- 18. 18Copyright©2015 NTT corp. All Rights Reserved. • Thank you for your listening.