Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Open stack swift_essex_meetup_2012_06_21_judd_maltin


Published on

OpenStack Object Storage (code-named Swift) is open source software for creating redundant, scalable object storage using clusters of standardized servers to store petabytes of accessible data. It is not a file system or real-time data storage system, but rather a long-term storage system for a more permanent type of static data that can be retrieved, leveraged, and then updated if necessary. Primary examples of data that best fit this type of storage model are virtual machine images, photo storage, email storage and backup archiving. Having no central "brain" or master point of control provides greater scalability, redundancy and permanence.

Swift Version 1.5 is set for release the week of June 11th. Get a clear understanding of the power of Swift, the changes in the current release and the features slated for the next major OpenStack release: Folsom. As a bonus, watch how easy it is to setup OpenStack Swift with Dell's "Crowbar - the Cloud Unboxer." Discussion that follows will include scalability considerations, monitoring, and management of the system.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Open stack swift_essex_meetup_2012_06_21_judd_maltin

  1. 1. Insiders Perspective: Exploring OpenStack Swift – Essex ReleaseOpenStack Boston MeetupJune 21, 2012Judd Maltin
  2. 2. OpenStack Timeline Fulsom Essex “Platform for Innovation” Diablo “Production Ready” Core Platform for Workable Innovation Cactus Foundation Stable Foundation Network as a Community Bexar Development Exposes Gaps Included in Service Forming Solidify Ubuntu 12.04 Block Storage Austin First Community Public Working Incubated: Public Adoptoin Code Prototypes Loses VMware Network & Block Multiple Scale Formation & HyperV Storage Deployments 2011 2012Nov 2010 Dec Feb Apr Jun Aug Oct Dec Feb Apr Jun Aug Nov 2010: Feb 2011: Apr 2011: Sep 2011: Mar 2012: Oct 2012: Austin Bexar Cactus Diablo Essex Fulsom Release Release Release Release Release Release Oct 2010: Apr 2011: Oct 2011: Apr 2012: Design Design Design Design Summit Summit Summit Summit
  3. 3. A highly scalable, redundant,unstructured data store designed tostore large amounts of data cheaply.
  4. 4. Why Swift? Good use cases:●Storing media libraries (photos, music, videos, etc.)●Archiving video surveillance files●Archiving phone call audio recordings●Archiving un/compressed log files●Archiving backups●Storing and loading of OS Images, etc.●Storing file populations that grow continuously on a practically infinitebasis.●Storing small files (<50 KB). OpenStack Object Storage is great at this.●Storing billions of files.●Storing Petabytes (millions of Gigabytes) of data.
  5. 5. What Swift is (the 411)● Written in Python ● SQLite, Rsync, Memcache, XFS● HTTP/ReST API● Logical Parts ● Accounts ● Containers ● Objects● CDN Integration● No single point of failure● Last write wins
  6. 6. What Swift AintNot a filesystem or block storage:Does not use the typical POSIX filesystems semantics likeopen(),read(),write(), seek() … rather HTTP actions likePUT,GET,DELETE,POST.Not a database:Stores unstructured, file based data.Only download, upload, retrieve data but does not process data.Only basic tagging.Barely even hierarchical:Only one level of container depth. Vague tagging.
  7. 7. System Components
  8. 8. Example OpenStack Object To Load Balancers Storage Hardware Proxies5 Zones2 Proxies per 25Storage Nodes10 GigE to Proxies1 GigE toStorage Nodes24 x 2TB Drives per Storage Node Example Large Scale Deployment -- Many Configs Possible
  9. 9. Object Storage Key FeaturesReST-based API Data distributed evenly throughout system Scalable to multiple petabytes, billions of objects Account/Container/Object structure (not file system, no nesting) plus Replication (N copies of accounts, containers, objects) No central database Hardware agnostic: standard hardware, RAID not required
  10. 10. System ComponentsPYTHON PROCS:Proxy Server: Authentication, ACLs, quotas, rate-limitingAccount Server: Handles listing of containers, stored asSQLite DBContainer Server: Handles listing of objects, stored asSQLite DBObject Server: Blob storage server, metadata kept in xattrs,data in binary formatStore your objects on XFSObject location based on hash of name & timestamp
  11. 11. System Components (cont.)● The Ring: Mapping of names to entities (accounts, containers, objects) on disk.● Stores data based on zones, devices, partitions, and replicas● Weights can be used to balance the distribution of partitions● Used by the Proxy Server for many background processes● Proxy Server: Request routing, exposes the public API● Replication: Keep the system consistent, handle failures● Updaters: Process failed or queued updates● Auditors: Verify integrity of objects, containers, and accounts● Deleters: Account Reaper, Replication
  12. 12. System Components (cont., cont.)● Auth System: Completely pluggable. Temp Auth, Keystone, Roll-your- own● Replication: (again?) replication works by crawling the procs local filesystems and querying the cluster for the file existing elsewhere. Of course, it just compares MD5 sums. ● DB Replication vs. Object Replication● Container to Container Synchronization: Spread your clusters out, sync out-of-band
  13. 13. Swift Containers Based Auth: ● X-Container-Read: accountname ● X-Container-Write: accountname:username ● X-Container-Read: referer:The Container Rules: ● Make lots of containers! Cheap! ● No nested containers ● No / in container names ● <256 bytes (incl URL encoding)
  14. 14. Coming in Folsom (1.5)* Versioned Objects!* Large Object Support (with no need for clientsupport)* Expiring Object support* StatsD Support* Logging is now middleware* Amazon S3 compatibility is now external project* DB Preallocation: great for spinning media, horriblefor SSDs
  15. 15. Coming Later!Ring Builder Web ServiceUpgraded WebOb (1.2) support(unicode, etc.)Drive Failure Detector and RemediationToken Service
  16. 16. OpenStack Community Resources ● IRC (freenode) ● #openstack ● #openstack-dev ● #openstack-meeting ● Docs ● ● ● http://swift.openstack.orga ● ● Twitter ● @openstack