7 STEPSto Roll-out a PrivateOpenStack Swift Cluster          Joe Arnold          OpenStack Conference                     ...
OpenStack Swift Powers Rackspace Cloud Files
Data is                                           Stickyhttp://www.flickr.com/photos/josefgrunig/
App       App       AppStorage   Storage   StorageCompute   Compute   Compute
Rackspace     OpenstackCloud Files    Launch                          59 developers                          contributing ...
When to use Swift (and when not to)Achieving High Durability and Availability (cluster design)         Convincing Your Bos...
When to use Swift (and when not to)Achieving High Durability and Availability (cluster design)         Convincing Your Bos...
Swi ft is good for          - Application data          - Filesharing          - Backups          - Unstructured data
Swift is laser focused   Swift Attributes          Scalable
Swift Attributes      Durable
Swift AttributesConcurrency
Not good at . . . Databases - transactional workloads No File Locking
Not good at . . .            Single            Stream              I/O
When to use Swift (and when not to)Achieving High Durability and Availability (cluster design)         Convincing Your Bos...
Data Durability - Zones
Data Durability - Replication & Consistency
High Availability Proxy   +Zones
Example: 2 Nodes  - Whole Files  - 3 Replicas  - Withstand multiple    disk failures
Example: 5 Nodes- Withstand multiple  Node Failures- Consider Separate  Proxy Tier
Example: 50 Nodes- Withstand Rack Failures- Requires Separate  Proxy Tier
When to use Swift (and when not to)Achieving High Durability and Availability (cluster design)         Convincing Your Bos...
Swi ft is economical                     $0.40/GB CapexExample: $42,520 for 105TB usable1 x Dell r815 ($4,500)2x Supermicr...
Swi ft is economical     Size        Configuration            Cost    $ / GB             2 ToR Switch             2 Proxy/A...
When to use Swift (and when not to)Achieving High Durability and Availability (cluster design)         Convincing Your Bos...
High volume web /              mobile application Various     Enterprise FilesharingWork loads                  Backup /  ...
Measure and Test                                  Frequency      File Sizes   Concurrency   of upload vs                  ...
RAM - 12GB PROXY NODES     Networking - 2 x 10GbE            CPU - quad-core           RAM - 1GB of RAM / 2TB           Dr...
When to use Swift (and when not to)Achieving High Durability and Availability (cluster design)         Convincing Your Bos...
People areExpensive
Added StatsD to Watch EVERYTHING       Swift Process        Swift Process         Swift Process          Swift Process    ...
Integrating Components around Swift                   Load Balancing         SSL Termination     Authentication           ...
Managing a distributed system                                Controller
Capacity adjustmentsCapacity                       time
When to use Swift (and when not to)Achieving High Durability and Availability (cluster design)         Convincing Your Bos...
S3           Cloud Files                         ’10’06        ’08
When to use Swift (and when not to)Achieving High Durability and Availability (cluster design)         Convincing Your Bos...
HTTP APIData              Storage
Media DeliveryDirect to DeviceIntegrates to CDNStreamingStatic DocumentsDirect from Device
TRY              http://swiftstack.com                    joe@swiftstack.com                    @joearnold - @swiftstack  ...
7 steps to roll out a private open stack swift cluster - joe arnold - swiftstack - OpenStack Conference -2012-04-17
7 steps to roll out a private open stack swift cluster - joe arnold - swiftstack - OpenStack Conference -2012-04-17
7 steps to roll out a private open stack swift cluster - joe arnold - swiftstack - OpenStack Conference -2012-04-17
Upcoming SlideShare
Loading in …5
×

7 steps to roll out a private open stack swift cluster - joe arnold - swiftstack - OpenStack Conference -2012-04-17

7,216 views

Published on

The OpenStack Object Storage system, aka "Swift", was built to power Rackspace's public cloud storage service, Cloud Files. Subsequently, Swift was released into open source as part of the OpenStack project. While Swift is increasingly being utilized to run many large scale public storage clouds outside of Rackspace, Swift is also showing great promise for use in private storage clouds - which look and feel like the public object storage systems from Amazon/Rackspace, yet meeting the needs of the individual company.
In this talk, I will present a step-by-step plan on getting OpenStack Swift up and running in your organization. I will cover:
- when to use Swift (and when not to)
- convincing your boss (the economics)
- selecting appropriate hardware (benchmarking)
- achieving high durability and availability (cluster design)
- knowing how it's working (management & monitoring)
- what interfaces exist (compatible clients)
- how to build killer apps (features for developers)

About Joe Arnold:
Joe Arnold is the CEO of SwiftStack. Joe managed the first public OpenStack launch of Swift independent of Rackspace, and has subsequently deployed multiple large-scale cloud storage systems. He is currently building Swift deployment and management tools with his firm, SwiftStack. Prior to SwiftStack, Joe held leadership positions at Cloudscaling, Engine Yard and Yahoo!.

Published in: Technology
0 Comments
9 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
7,216
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
201
Comments
0
Likes
9
Embeds 0
No embeds

No notes for slide

7 steps to roll out a private open stack swift cluster - joe arnold - swiftstack - OpenStack Conference -2012-04-17

  1. 1. 7 STEPSto Roll-out a PrivateOpenStack Swift Cluster Joe Arnold OpenStack Conference April 19, 2012
  2. 2. OpenStack Swift Powers Rackspace Cloud Files
  3. 3. Data is Stickyhttp://www.flickr.com/photos/josefgrunig/
  4. 4. App App AppStorage Storage StorageCompute Compute Compute
  5. 5. Rackspace OpenstackCloud Files Launch 59 developers contributing to Swift ’09 ’10
  6. 6. When to use Swift (and when not to)Achieving High Durability and Availability (cluster design) Convincing Your Boss (the economics) Selecting Appropriate Hardware (benchmarking) Knowing How It’s Working (management & monitoring) What Interfaces Exist (compatible clients) How to Build Killer Apps (features for developers)
  7. 7. When to use Swift (and when not to)Achieving High Durability and Availability (cluster design) Convincing Your Boss (the economics) Selecting Appropriate Hardware (benchmarking) Knowing How It’s Working (management & monitoring) What Interfaces Exist (compatible clients) How to Build Killer Apps (features for developers)
  8. 8. Swi ft is good for - Application data - Filesharing - Backups - Unstructured data
  9. 9. Swift is laser focused Swift Attributes Scalable
  10. 10. Swift Attributes Durable
  11. 11. Swift AttributesConcurrency
  12. 12. Not good at . . . Databases - transactional workloads No File Locking
  13. 13. Not good at . . . Single Stream I/O
  14. 14. When to use Swift (and when not to)Achieving High Durability and Availability (cluster design) Convincing Your Boss (the economics) Selecting Appropriate Hardware (benchmarking) Knowing How It’s Working (management & monitoring) What Interfaces Exist (compatible clients) How to Build Killer Apps (features for developers)
  15. 15. Data Durability - Zones
  16. 16. Data Durability - Replication & Consistency
  17. 17. High Availability Proxy +Zones
  18. 18. Example: 2 Nodes - Whole Files - 3 Replicas - Withstand multiple disk failures
  19. 19. Example: 5 Nodes- Withstand multiple Node Failures- Consider Separate Proxy Tier
  20. 20. Example: 50 Nodes- Withstand Rack Failures- Requires Separate Proxy Tier
  21. 21. When to use Swift (and when not to)Achieving High Durability and Availability (cluster design) Convincing Your Boss (the economics) Selecting Appropriate Hardware (benchmarking) Knowing How It’s Working (management & monitoring) What Interfaces Exist (compatible clients) How to Build Killer Apps (features for developers)
  22. 22. Swi ft is economical $0.40/GB CapexExample: $42,520 for 105TB usable1 x Dell r815 ($4,500)2x Supermicro JBODs ($5,400)144 x 3TB hard drives ($33,120)Thailand flooding drive prices
  23. 23. Swi ft is economical Size Configuration Cost $ / GB 2 ToR Switch 2 Proxy/Auth Servers 145 TB 5 Object Stores $95k $0.65 ...and cables, rack, etc 2 Agg Switches 6 Proxy/Auth Servers 1.3 PB 5 ToR Switches $750k $0.52 50 Object Stores ...and cables, racks, etc
  24. 24. When to use Swift (and when not to)Achieving High Durability and Availability (cluster design) Convincing Your Boss (the economics) Selecting Appropriate Hardware (benchmarking) Knowing How It’s Working (management & monitoring) What Interfaces Exist (compatible clients) How to Build Killer Apps (features for developers)
  25. 25. High volume web / mobile application Various Enterprise FilesharingWork loads Backup / Active Archive
  26. 26. Measure and Test Frequency File Sizes Concurrency of upload vs download
  27. 27. RAM - 12GB PROXY NODES Networking - 2 x 10GbE CPU - quad-core RAM - 1GB of RAM / 2TB Drives - 2-3 TB Drives Controller Cards - No RAID Using SSDs - User Accounts & Container ListingsSTORAGE NODES
  28. 28. When to use Swift (and when not to)Achieving High Durability and Availability (cluster design) Convincing Your Boss (the economics) Selecting Appropriate Hardware (benchmarking) Knowing How It’s Working (management & monitoring) What Interfaces Exist (compatible clients) How to Build Killer Apps (features for developers)
  29. 29. People areExpensive
  30. 30. Added StatsD to Watch EVERYTHING Swift Process Swift Process Swift Process Swift Process Lightweight Swift Process Swift Process UDP StatsD StatsD Time-Series Backend Ganglia / Graphite
  31. 31. Integrating Components around Swift Load Balancing SSL Termination Authentication OpenStack Object Storage OS, Node & Drive Swift Runtime Monitoring Monitoring Operating System + Kernel
  32. 32. Managing a distributed system Controller
  33. 33. Capacity adjustmentsCapacity time
  34. 34. When to use Swift (and when not to)Achieving High Durability and Availability (cluster design) Convincing Your Boss (the economics) Selecting Appropriate Hardware (benchmarking) Knowing How It’s Working (management & monitoring) What Interfaces Exist (compatible clients) How to Build Killer Apps (features for developers)
  35. 35. S3 Cloud Files ’10’06 ’08
  36. 36. When to use Swift (and when not to)Achieving High Durability and Availability (cluster design) Convincing Your Boss (the economics) Selecting Appropriate Hardware (benchmarking) Knowing How It’s Working (management & monitoring) What Interfaces Exist (compatible clients) How to Build Killer Apps (features for developers)
  37. 37. HTTP APIData Storage
  38. 38. Media DeliveryDirect to DeviceIntegrates to CDNStreamingStatic DocumentsDirect from Device
  39. 39. TRY http://swiftstack.com joe@swiftstack.com @joearnold - @swiftstack rkshop ๏InstallSwi ft wo ๏Build an Application tom orrow at 1:3 0 - 4 : 00

×