Presentation I gave at LinuxCon/CloudOpen Japan 2013 on building a dev/test cloud with Apache CloudStack

1. Building a Test/Dev Cloud with
Apache CloudStack
David Nalley
ke4qqq@apache.org
@ke4qqq

2. #whoami
● Sysadmin of a decade
● Apache Software Foundation Member
● Apache CloudStack Committer & PMC Member
● Fedora Project Contributor
● Employed by Citrix in the Open Source Business Office

3. Why use cloud?
From a dev POV the process normally looks like:
● Start new project
● File ticket for resources....wait....wait....wait
● Get resources, that aren't configured....wait...
● Get network access.....get permission....wait
● Get things done.

4. Why use cloud?
● What IT Operations provides is not what a developer wants.

5. Get rid of the waiting
● Remove the constraints - developers empowered to get things
done.
● Agility
● Enforce automated process instead of manual ones

6. What does a dev/test cloud look like?
● Self-service - developers can provision their own environments
● Usage measurement - we worry about VM sprawl
● Isolated networks - must not let dev/test interfere with the real
world.
● Commodity - as cheap as practical
● May also house production workloads

7. Self service
● Provisioning manually doesn't add value
● Can be completely automated
● Do they need full control or just pushing pre-configured
environments?

8. Self-service
● UI?
● API?
● Some external tool?

10. API or Commandline?
� cloudmonkey> deploy virtualmachine
serviceofferingid=d8611d07-acf5-4cd4-a630-5c4d937ef043
templateid=081358ff-2427-44f8-adcc-1bb002fab361
zoneid=d06193b2-7980-4ad1-bd8-7b2f2eda63c3
curl 'http://localhost:8096/client/api?
command=listUsers'

11. Config-mgmt based deployment

12. {
"name": "hadoop_cluster_a",
"description": "A small hadoop cluster with hbase",
"version": "1.0",
"environment": "production",
"servers": [
{
"name": "zookeeper-a, zookeeper-b, zookeeper-c",
"description": "Zookeeper nodes",
"template": "rhel-5.6-base",
"service": "small",
"port_rules": "2181",
"run_list": "role[cluster_a], role[zookeeper_server]",
"actions": [
{ "knife_ssh": ["role:zookeeper_server", "sudo chef-client"] }
]
},
{
"name": "hadoop-master",
"description": "Hadoop master node",
"template": "rhel-5.6-base",
"service": "large",
"networks": "app-net, storage-net",
"port_rules": "50070, 50030, 60010",
"run_list": "role[cluster_a], role[hadoop_master], role[hbase_master]"
},
{
"name": "hadoop-worker-a hadoop-worker-b hadoop-worker-c",
"description": "Hadoop worker nodes",
"template": "rhel-5.6-base",
"service": "medium",
"port_rules": "50075, 50060, 60030",
"run_list": "role[cluster_a], role[hadoop_worker], role[hbase_regionserver]",
"actions": [
{ "knife_ssh": ["role:hadoop_master", "sudo chef-client"] },
{ "http_request": "http://${hadoop-master}:50070/index.jsp" }
]
}
}

13. Use a tool

14. Usage
● Jevons Paradox
● Plenty of waste possible as well - will developers always
destroy a machine when they are done with it?
● Important to show what projects and groups are consuming
resources as well as how they are using those resources

15. Commodity - Storage
● Commodity storage - this is dev/test environment - high
performance, resilient storage isn't needed.
● Local storage tends to be the best mix of cheap and performant
● No failover, but it's dev/test - do you need it?

16. Commodity - Networking
● Layer 3 isolation - (aka Security Groups)
● VLANs - (not as commodity, but still relatively cheap on a small
scale, but not at scale)
● Virtual routers (provide DHCP, DNS, LB, Firewall, PF, NAT, etc)

17. Commodity Hypervisor
● KVM is my personal choice in this space.
● Easiest to consume - completely open source

18. Limiting resources
● Limit the number of VMs, snapshots, IP addresses, etc.
● Use 'projects' to share resources
● This means most folks will never have problems, but heaviest
users will not be able to interrupt service for others.

19. Questions?

20. Resources
● http://cloudstack.apache.org
● http://cloudstack.jp
● #cloudstack on irc.freenode.net
● Mailing lists/google group