Building an backup-backend using OpenStack Swift

1. Designing and Building
Multi-Region Swift Deployment
KINX(www.kinx.net)
CLOUD TECH GROUP
Shawn Kim
1

2. Table of Contents
Swift
Architecture Review
Node Roles & deployment
Ring
Enhancements
Demonstration
Q&A
2

3. Swift
Object Store Project of OpenStack
Characteristics
High availability and high scalability
By keeping replications, Swift avoids data loss.
Recommended copies >= 3
3 Layered data: Account / Container / Object
Account: Namespace which divides Containers.
Container: Namespace which divides Objects. Min ACL configuration unit.
3

4. Swift Cont
Swift Architecture
Servier Processes + Consistency Processes 로 구성됨
Server Processes provide Swift services.
Proxy server
Account / Container / Object servers
Consistency Processes maintains consistency of A/C/O objects or metadata.
A/C/O Auditor
A/C/O Replicator
4

5. Architecture
Review
Problem:
Region down induces all data
loss.
Alternative:
Multi region Swift.
>=3 copy replications.
Dedicate Keystones with DB
synchronization.
Multi Proxies with GSLB.
5
Figure 1. Multi-Region Architecture

6. Architectural Review - Design Decisions
Replication >= 3
Proxy and Keystone are deployment on the same node
Proxy and Keystone share the same DNS name and SSL configuration.
Easy Keystone HA configuration.
Multi-Region architecture:
Region level high availability
Equally distributed replications
#zones = #replictions, Each zone has one replication. 6

7. Architectural Review - Design Decisions Cont.
Connection btw regions
Storages across different regions cannot be connected with internal IPs.
Under the multi-region environment, storages are connected by public IPs with iptables configuration for
security consideration.
Swift Master
Master Role: Swift installation base and gateway for all nodes.
MGMT Role: Ring building, management and distribution.
Auth Module:
Keystone versus. SWAuth: Keystone Win! 7

8. Architectural Review - Design Result
2 Regions, 3 Zones, 3 storage
nodes with 9 volumes.
Node naming convention:
r<digit>-[z<digit>]-<role><digit>
ex) r1-p1
ex) r2-z1-s1
It can be referred when Ring is
built.
8Figure 2. Deployed Multi-Region Swift : Minimum Configuration

9. Node Roles &
Package Deployment
Keystone Node
Python-openstackclient
Mariadb-server-5.5
Keystone
Memcached
9
Proxy Node
Python-openstackclient
Swift
Swift-proxy
Python-swiftclient
Python-keystonemiddleware
Storage Node
Python-openstackclient
Xfsprogs, rsync
Swift
Swift-account
Swift-container
Swift-object
Keystone node is
integrated with
Proxy nodes

10. 10
Node Roles &
Package Deployment
Swift Master + MGMT
Python-openstackclient
Swift
Python-swiftclient
LMA
EX) SwiftStack: Zabbix + ELK

11. Ring
Ring is abbreviation of Modified Consistent Hashing Ring.
Swift’s Object is located by using Hash function:
[Drive_id] = md5([account_x]/[container_y]/[object_z]) MOD [total_drives]
11
Each drive’s weight determine its length of range.
All drives are placed fare random order.
Ex) [hash of object] is placed in range of drive 4.
Figure 3. Drives and the hashing ring
(OpenStack Swift 2014)*
*Joe Arnold & member of the SwiftStack team. OpenStack Swift: USING,
ADMINISTERING, AND DEVELOPING FOR SWIFT OBJECT STORAGE.
O’REILLY, 2014.

12. Consistent
Hashing Ring Cont.
Minimize object moving even if disk is added or removed.
12
Ex)
Object is originally located at drive 4.
After adding drive 5, the hash value of object is belong to drive 5.
Figure 5. New drive added to a
ring (OpenStack Swift 2014)**
Figure 4. Drives and the hashing
ring (OpenStack Swift 2014)*
*, **Joe Annold & member of the SwiftStack team. OpenStack Swift: USING,
ADMINISTERING, AND DEVELOPING FOR SWIFT OBJECT STORAGE.
O’REILLY, 2014.

13. (Real) Consistent Hashing Ring Cont.
In the Swift rings, each drive is distributed with
many narrow ranges in fare random order.
When a new disk is added, new ranges are
added tail of arbitrary ranges.
See the ranges of drive 5.
13
Figure 6. Many ranges for each drive
(OpenStack Swift 2014)*
*Joe Annold & member of the SwiftStack team. OpenStack Swift: USING,
ADMINISTERING, AND DEVELOPING FOR SWIFT OBJECT STORAGE.
O’REILLY, 2014.

14. Modified Consistent Hashing Ring
Some ‘Modifications’ applied Consistent Hashing Ring:
Partitions
Swift ring is comprised with fixed ranged partitions.
Partition stores and indicates Object’s location(s).
Partitions power
Total partitions in cluster = 2^partitions_power
Arbitrary integer which was defined when cluster creation.
Replica count
Number of replications is defined when ring is built. 14

15. Modified Consistent Hashing Ring Cont.
Hash(path) = md5(path + per_cluster_suffix)
Partitions_power
Partitions_power = UPPER_BOUND[log2(number_of_disks * 100)]
After selecting partitions_power, it is Immutable.
Bigger partitions_power makes ring bigger with more indices.
Bigger rings use more memory.
Partition = hash >> part_shift
Part_shift = 32 - partitions_power
15

16. Enhancements
Swift3: Apply S3 API compatible Middleware.
LMA Node: ELK and Zabbix
Enable swift middlewares such as Tempurl, domap_remap, and staticweb
Test automation
Swift Tox: test coverage is 50 ~ 80%.
After S3 API (Swift3) integration, use Boto unit Test.
Swift Deployment Automation using PuppetSwift module.
Chef Cookbook-openstack-object-storage was deprecated after Mitaka release. 16

17. Demonstration
Uploading sample data
LOGs @ each storage nodes
Trying download after killing 2 object server.
Swift-ring-builder and builder file.
Swift-dispersion-populate and report
17

18. QnA
18
Q: When Swift’s PUT request returns success after making how many replications?
A: Just one object.
Q: With 3 object servers and 3 replication configured Swift, and PUT request is
occasionally failed when 2 arbitrary object servers are stopped. Is it a default
behavior? Also, can I configure this behavior?
A: Unfortunately, object’s hash values (maybe 3 counts) can indicate disabled object servers. In
this case, the object cannot be written because all object servers are not be reachable.
Q: With HA configured proxy servers, and trying to read right after writing an
object, it is possible for end-user to be unreachable to the object?
A: No. By searching its ring, swift can locate the written object among the object’s replication(s)