Your SlideShare is downloading. ×
0
Scalable Object Storage withApache CloudStack and ApacheHadoopApril 30 2013Chiradeep Vittal@chiradeep
Agenda• What is CloudStack• Object Storage for IAAS• Current Architecture and Limitations• Requirements for Object Storage...
• History• Incubating in the ApacheSoftware Foundation sinceApril 2012• Open Source since May2010• In production since 200...
How did Amazon build its cloud?CommodityServersCommodityStorageNetworkingOpen Source Xen HypervisorAmazon Orchestration So...
How can YOU build a cloud?Servers StorageNetworkingOpen Source Xen HypervisorAmazon Orchestration SoftwareAWS API (EC2, S3...
Secondary StorageImageL3/L2 coreDC EdgeEnd usersPod Pod Pod PodZone ArchitecturePodAccess SwMySQLCloudStackAdmin/User APIP...
Cloud-Style Workloads• Low cost– Standardized, cookie cutter infrastructure– Highly automated and efficient• Application o...
Secondary StorageImageL3/L2 coreDC EdgePod Pod Pod PodAt scale…everything breaksPodAccess SwPrimary StorageNFS/ISCSI/FCHyp...
Region “West”Zone “West-Alpha”Zone “West-Beta”Zone “West-Gamma”Zone “West-Delta”Low Latency Backbone(e.g., SONET ring)Regi...
Region “East”Region “South”InternetGeographicseparationRegion “West”Low Latency
Secondary Storage in CloudStack 4.0• NFS server default– can be mounted by hypervisor– Easy to obtain, set up and operate•...
Object Storage TechnologyRegion “West”Zone “West-Alpha”Zone “West-Beta”Zone “West-Gamma”Zone “West-Delta”Object Storage in...
Region “West”Object Storage enables reliability
Object Storage TechnologyRegion “West”Object Storage also enables otherapplicationsObject StoreAPI Servers• DropBox• Stati...
Object Storage characteristics• Highly reliable and durable– 99.9 % availability for AWS S3– 99.999999999 % durability• Ma...
CloudStack S3 API ServerObject Storage TechnologyS3API ServersMySQL
CloudStack S3 API Server• Understands AWS S3 REST-style and SOAP API• Pluggable backend– Backend storage needs to map simp...
Object Store Integration intoCloudStack• For images and snapshots• Replacement for NFS secondary storageOrAugmentation for...
What do we want to build ?• Open source, ASL licensed object storage• Scales to at least 1 billion objects• Reliability an...
The following slides are a designdiscussion
Architecture of Scalable ObjectStorageAPI ServersAuth ServersObject Servers Replicators/AuditorsObjectLookupServers
Why HDFS• ASF Project (Apache Hadoop)• Immutable objects, replication• Reliability, scale and performance– 200 million obj...
HDFS-based Object StorageS3 API ServersS3 Auth ServersData nodesNamenodepairHDFS API
BUT• Name Node Scalability– 150 bytes RAM / block– GC issues• Name Node SPOF– Being addressed in the community✔• Cross-zon...
Name Node scalability• 1 billion objects = 3 billion blocks (chunks)– Average of 5 MB/object = 5 PB (actual), 15PB (raw)– ...
Name Node FederationExtension: Federated NameNodes are HA pairs
Federation issues• HA for name nodes• Namespace shards– Map object -> name node• Requires another scalable key-value store...
Replication over lossy/slower linksA. Asynchronous replication– Use distcp to replicate between clusters– 6 copies vs. 3– ...
CAP TheoremConsistency or Availability during partitionMany nuances
Storage for object metadataA. Store it in HDFS along with the object– Reads are expensive (e.g., to check ACL)– Mutable da...
Object store on HDFS Future• Viable for small-sized deployments– Up to 100-200 million objects– Datacenters close together...
Conclusion• CloudStack needs object storage for“cloud-style” workloads• Object Storage is not easy• HDFS comes close but n...
Upcoming SlideShare
Loading in...5
×

Scalable Object Storage with Apache CloudStack and Apache Hadoop

1,168

Published on

Object Storage (like AWS S3) in the cloud is a key enabler of scalability and reliability in Cloud Computing. Apache CloudStack. We will discuss how CloudStack integrates Object Storage solutions and discuss specifically how HDFS can provide the storage engine for the Object Storage component.

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,168
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
25
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • Need a better slide than this
  • Frequently require CCNA , Vmwareceritification, EMC training, etc etc. But they chose commondity systems. And simple networking.Can also sell cheaply since they use their own commerce platform.
  • The key here is the API on top of the infrastructure. This is the disruptive piece for the industry. Forget about CCNA, Vmware cert, now people can programmatically control their infrastructure as well as the VMs on top of it.
  • Transcript of "Scalable Object Storage with Apache CloudStack and Apache Hadoop"

    1. 1. Scalable Object Storage withApache CloudStack and ApacheHadoopApril 30 2013Chiradeep Vittal@chiradeep
    2. 2. Agenda• What is CloudStack• Object Storage for IAAS• Current Architecture and Limitations• Requirements for Object Storage• Object Storage integrations in CloudStack• HDFS for Object Storage• Future directions
    3. 3. • History• Incubating in the ApacheSoftware Foundation sinceApril 2012• Open Source since May2010• In production since 2009– Turnkey platform for deliveringIaaS clouds– Full featured GUI, end-user APIand admin APIApache CloudStackBuild your cloud the way theworld’s most successfulclouds are built
    4. 4. How did Amazon build its cloud?CommodityServersCommodityStorageNetworkingOpen Source Xen HypervisorAmazon Orchestration SoftwareAWS API (EC2, S3, …)Amazon eCommerce Platform
    5. 5. How can YOU build a cloud?Servers StorageNetworkingOpen Source Xen HypervisorAmazon Orchestration SoftwareAWS API (EC2, S3, …)Amazon eCommerce PlatformHypervisor (Xen/KVM/VMW/)CloudStack Orchestration SoftwareOptional PortalCloudStack or AWS API
    6. 6. Secondary StorageImageL3/L2 coreDC EdgeEnd usersPod Pod Pod PodZone ArchitecturePodAccess SwMySQLCloudStackAdmin/User APIPrimary StorageNFS/ISCSI/FCHypervisor (Xen/VMWare/KVM)VMVMSnapshotSnapshotImageDisk DiskVM
    7. 7. Cloud-Style Workloads• Low cost– Standardized, cookie cutter infrastructure– Highly automated and efficient• Application owns availability– At scale everything breaks– Focus on MTTR instead of MTBF
    8. 8. Secondary StorageImageL3/L2 coreDC EdgePod Pod Pod PodAt scale…everything breaksPodAccess SwPrimary StorageNFS/ISCSI/FCHypervisor (Xen/VMWare/KVM)VMVMSnapshotSnapshotImageDisk DiskVM
    9. 9. Region “West”Zone “West-Alpha”Zone “West-Beta”Zone “West-Gamma”Zone “West-Delta”Low Latency Backbone(e.g., SONET ring)Regions and zones
    10. 10. Region “East”Region “South”InternetGeographicseparationRegion “West”Low Latency
    11. 11. Secondary Storage in CloudStack 4.0• NFS server default– can be mounted by hypervisor– Easy to obtain, set up and operate• Problems with NFS:– Scale: max limits of file systems• Solution: CloudStack can manage multiple NFS stores (+complexity)– Performance• N hypervisors : 1 storage CPU / 1 network link– Wide area suitability for cross-region storage• Chatty protocol– Lack of replication
    12. 12. Object Storage TechnologyRegion “West”Zone “West-Alpha”Zone “West-Beta”Zone “West-Gamma”Zone “West-Delta”Object Storage in a region• Replication• Audit• Repair• Maintenance
    13. 13. Region “West”Object Storage enables reliability
    14. 14. Object Storage TechnologyRegion “West”Object Storage also enables otherapplicationsObject StoreAPI Servers• DropBox• Static Content• Archival
    15. 15. Object Storage characteristics• Highly reliable and durable– 99.9 % availability for AWS S3– 99.999999999 % durability• Massive scale– 1.3 trillion objects stored across 7 AWS regions [Nov 2012 figures]– Throughput: 830,000 requests per second• Immutable objects– Objects cannot be modified, only deleted• Simple API– PUT/POST objects, GET objects, DELETE objects– No seek / no mutation / no POSIX API• Flat namespace– Everything stored in buckets.– Bucket names are unique– Buckets can only contain objects, not other buckets• Cheap and getting cheaper
    16. 16. CloudStack S3 API ServerObject Storage TechnologyS3API ServersMySQL
    17. 17. CloudStack S3 API Server• Understands AWS S3 REST-style and SOAP API• Pluggable backend– Backend storage needs to map simple calls to theirAPI• E.g., createContainer, saveObject, loadObject– Default backend is a POSIX filesystem– Backend with Caringo Object Store (commercialvendor) available– HDFS backend also available• MySQL storage– Bucket -> object mapping– ACLs, bucket policies
    18. 18. Object Store Integration intoCloudStack• For images and snapshots• Replacement for NFS secondary storageOrAugmentation for NFS secondary storage• Integrations available with– Riak CS– Openstack Swift• New in 4.2 (upcoming):– Framework for integrating storage providers
    19. 19. What do we want to build ?• Open source, ASL licensed object storage• Scales to at least 1 billion objects• Reliability and durability on par with S3• S3 API (or similar, e.g., Google Storage)• Tooling around maintenance andoperation, specific to object storage
    20. 20. The following slides are a designdiscussion
    21. 21. Architecture of Scalable ObjectStorageAPI ServersAuth ServersObject Servers Replicators/AuditorsObjectLookupServers
    22. 22. Why HDFS• ASF Project (Apache Hadoop)• Immutable objects, replication• Reliability, scale and performance– 200 million objects in 1 cluster [Facebook]– 100 PB in 1 cluster [Facebook]• Simple operation– Just add data nodes
    23. 23. HDFS-based Object StorageS3 API ServersS3 Auth ServersData nodesNamenodepairHDFS API
    24. 24. BUT• Name Node Scalability– 150 bytes RAM / block– GC issues• Name Node SPOF– Being addressed in the community✔• Cross-zone replication– Rack-awareness placement ✔– What if the zones are spread a little further apart?• Storage for object metadata– ACLs, policies, timers
    25. 25. Name Node scalability• 1 billion objects = 3 billion blocks (chunks)– Average of 5 MB/object = 5 PB (actual), 15PB (raw)– 450 GB of RAM per Name Node• 150b x 3 x 10^9– 16 TB / node => 1000 Data nodes• Requires Name Node federation ?• Or an approach like HAR files
    26. 26. Name Node FederationExtension: Federated NameNodes are HA pairs
    27. 27. Federation issues• HA for name nodes• Namespace shards– Map object -> name node• Requires another scalable key-value store– HBase?• Rebalancing between name nodes
    28. 28. Replication over lossy/slower linksA. Asynchronous replication– Use distcp to replicate between clusters– 6 copies vs. 3– Master/Slave relationship• Possibility of loss of data during failover• Need coordination logic outside of HDFSB. Synchronous replication– API server writes to 2 clusters and acks onlywhen both writes are successful– Availability compromised when one zone isdown
    29. 29. CAP TheoremConsistency or Availability during partitionMany nuances
    30. 30. Storage for object metadataA. Store it in HDFS along with the object– Reads are expensive (e.g., to check ACL)– Mutable data, needs layer over HDFSB. Use another storage system (e.g. HBase)– Name node federation also requires this.C. Modify Name Node to store metadata– High performance– Not extensible
    31. 31. Object store on HDFS Future• Viable for small-sized deployments– Up to 100-200 million objects– Datacenters close together• Larger deployments needs development– No effort ongoing at this time
    32. 32. Conclusion• CloudStack needs object storage for“cloud-style” workloads• Object Storage is not easy• HDFS comes close but not close enough• Join the community!
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×