John Burwell presents on using Apache CloudStack and Riak CS to build better, faster, and cheaper private cloud infrastructure. CloudStack is an open-source cloud computing platform that can deploy and manage large networks of virtual machines. Riak CS provides simple and reliable object storage accessed via S3 or Swift APIs. Using these technologies together allows organizations to build a complete private cloud with composable infrastructure, self-service provisioning, and horizontally scalable storage.

1. BETTER, FASTER, CHEAPER
INFRASTRUCTURE: APACHE
CLOUDSTACK + RIAK CS
John Burwell
(jburwell@basho.com | @john_burwell)
Friday, November 8, 13

2. Who Am I
•
Consulting Engineer @ Basho Technologies
•
Member, Apache CloudStack PMC
•
Ran operations and designed automated provisioning for hybrid
analytic/virtualization clouds
•
Led architectural design and server-side development of a SaaS
physical security platform
Friday, November 8, 13

3. ... but I submitted a ticket.
Friday, November 8, 13

4. Shifting Role Of IT
From Cost Center
To Business Opportunity
Friday, November 8, 13

5. Shifting Role Of IT
From Cost Center
To Business Opportunity
Infrastructure Be Boring
Friday, November 8, 13

6. Better
•
Resilience
•
Scalability
•
Service
Friday, November 8, 13

7. Faster
•
Resource Provisioning
•
Systems
•
Failure recovery
Friday, November 8, 13

8. Cheaper
•
Hardware
•
Labor
•
Hosting
Friday, November 8, 13

9. Composable Infrastructure
•
Resource Pooling
•
Rapid Elasticity
•
Self Service
•
Metering/Charge Back
Friday, November 8, 13

10. Composable Infrastructure
•
Resource Pooling
•
Rapid Elasticity
•
Self Service
•
Metering/Charge Back
Late-binding of hardware roles
Friday, November 8, 13

11. Got It ... Why Go Private?
•
Leverage existing infrastructure investment
•
Customization and Control
•
Regulatory/Security Requirements
•
Cost
Friday, November 8, 13

12. Doesn’t vSphere/
XCP/XenServer
already do that?
Friday, November 8, 13

13. Friday, November 8, 13

14. Hopefully not ...
Friday, November 8, 13

15. Friday, November 8, 13

16. Traditional Workload
•
•
Fault tolerant
•
Management
Reliable
Specialized hardware
•
IT managed provisioning
•
Static scale out model
Friday, November 8, 13
Network
Hypervisor
Cluster
Hypervisor
Cluster
Shared

17. Storage

18. Traditional Workload
•
•
Fault tolerant
•
Management
Reliable
Specialized hardware
•
IT managed provisioning
•
Static scale out model
Network
Hypervisor
Cluster
Hypervisor
Cluster
Shared

19. Storage
Focused on resource consolidation not agility
Friday, November 8, 13

20. Software Is Eating Hardware
•
Moore’s Law has pulled fault tolerance up the stack
•
Commodity Hardware
•
•
•
Reduced cost
Services more workload types
Dynamic resource allocation
Friday, November 8, 13

21. Software Is Eating Hardware
•
Moore’s Law has pulled fault tolerance up the stack
•
Commodity Hardware
•
•
•
Reduced cost
Services more workload types
Dynamic resource allocation
Specialized Hardware Reduces Agility
Friday, November 8, 13

22. Cloud Workload
•
Reliable
•
Fault Tolerant
•
Self-Service provisioning
•
Control Plane - Dynamic
scale out model
Rack
Rack
App
App
Commodity Hardware
•
SDN
Friday, November 8, 13
Object

23. Storage

24. Cloud Workload
•
Reliable
•
Fault Tolerant
•
Self-Service provisioning
•
Control Plane - Dynamic
scale out model
Rack
Rack
App
App
Commodity Hardware
•
SDN
Object

25. Storage
Resource optimization increases agility
Friday, November 8, 13

26. Reality
Traditional
Workload
Friday, November 8, 13
Cloud
Workload

27. Complete Private Cloud
+
Compute Services
Friday, November 8, 13
Object Storage
Services

28. What Is Apache CloudStack?
§Open source software that deploys
and manages large networks of
virtual machines, as a highly
available, highly scalable
Infrastructure as a Service (IaaS)
cloud computing platform.
§CloudStack is a Top-Level Project at
the Apache Software Foundation.
Friday, November 8, 13

29. What Is Apache CloudStack?
§Open source software that deploys
and manages large networks of
virtual machines, as a highly
available, highly scalable
Infrastructure as a Service (IaaS)
cloud computing platform.
§CloudStack is a Top-Level Project at
the Apache Software Foundation.
Friday, November 8, 13

30. ~300: Acknowledged production implementations
~43,000: Hosts in a single CloudStack instance
~85: Project committers
Friday, November 8, 13

31. Apache CloudStack Users
Friday, November 8, 13

32. Anatomy Of CloudStack
•
Management Server Farm
•
•
VM
Network
Host
Collection of clusters in the same failure boundary
Primary
Storage
A grouping of hosts and their associated storage
Hosts
•
•
Collection of pods, network offerings and secondary storage
Cluster
•
•
Host
Pod
•
•
VM
Zone
•
•
Management and provisioning tasks
Cluster
Secondary
Storage
Cluster
Servers onto which services will be provisioned
Primary Storage
CloudStack Pod
•
•
VM disk storage
Network
•
CloudStack Pod
Logical network associated with service offerings
Zone
•
Secondary Storage
•
Template, snapshot and ISO storage
Friday, November 8, 13

33. Management Server
•
Orchestrates infrastructure management
•
A single management server instance can manage 10,000 hosts
•
For production, run n +1 management servers with at least one
instance in each zone
•
Cluster MySQL
Friday, November 8, 13

34. Control And Data Planes
•
Automation control of
resources (e.g. hypervisors,
storage, switches, etc)
•
Small, linear bandwidth
consumption
•
Low latency
Friday, November 8, 13
•
Transfer of data between
infrastructure components
(e.g. templates, snapshots,
etc)
•
Large, variable bandwidth
consumption
•
High throughput

35. Control And Data Plane Scalability
•
Separate the control and data plane compute and network
traffic
•
Data transfer operations offloaded to system virtual machines
•
Separate management and storage networks
Friday, November 8, 13

36. Primary Vs Secondary Storage
•
Random access read/write
•
Write once/read many
•
Changes often
•
Append only
•
Zone Scoped
•
Zone or Region Scoped
•
Focused on throughput
•
Focused on reliability
Friday, November 8, 13

37. Primary Vs Secondary Storage
•
Random access read/write
•
Write once/read many
•
Changes often
•
Append only
•
Zone Scoped
•
Zone or Region Scoped
•
Focused on throughput
•
Focused on reliability
Secondary Storage is the lifeboat.
Friday, November 8, 13

38. What Is Riak CS?
•
Simple, reliable storage of large objects accessed using the S3 or Swift
API
•
Objects are stored in buckets with a unique key
•
Multi-tenant security model with per object ACLs
•
Usage Accounting and Reporting
•
Open Source (Apache2 Licensed)
Friday, November 8, 13

39. What Is Riak CS?
•
Simple, reliable storage of large objects accessed using the S3 or Swift
API
•
Objects are stored in buckets with a unique key
•
Multi-tenant security model with per object ACLs
•
Usage Accounting and Reporting
•
Open Source (Apache2 Licensed)
Your Own Private S3
Friday, November 8, 13

40. Anatomy Of Riak CS
•
S3/Swift API Proxy + Riak key-value store
•
Peer-to-Peer Design: Any node can service any read or accept
any write request
•
Maintains multiple replicas (default 3) spread across multiple
nodes
•
Ops Friendly
Friday, November 8, 13

41. Anatomy Of Riak CS
•
S3/Swift API Proxy + Riak key-value store
•
Peer-to-Peer Design: Any node can service any read or accept
any write request
•
Maintains multiple replicas (default 3) spread across multiple
nodes
•
Ops Friendly
Horizontally scaleable storage on
commodity hardware
Friday, November 8, 13

42. Riak CS Use Cases
•
Secondary Storage for VM snapshot and template data
•
Database Backup Storage
•
Serving Static Content
•
Application Log Storage
Friday, November 8, 13

43. Achieving Commodity Reliability
•
Create a zone per power source/backbone network connection
•
Establish independent primary storage in each zone
•
Utilize intra-datacenter block-level replication between zones
•
Distribute Riak CS nodes evenly across zones
Friday, November 8, 13

44. The Complete Private Cloud
Traditional