Hands onVirtualization with      Ganeti          (part 1)      Lance Albertson        @ramereth     Associate Director    ...
About us● OSU Open Source Lab● Server hosting for Open Source  Projects  ○ Linux Foundation, Apache Software Foundation,  ...
Session Overview (part 1)● Ganeti Introduction  ● Terminology  ● Major Components● Latest Features● Using Ganeti in Practi...
Session Overview (part 2)● Hands on Demo● Installation and Initialization● Cluster Management  ● Adding instances (VMs)  ●...
What can Ganeti do?●   Virtual machine management software tool●   Manages clusters of physical machines●   Xen/KVM/LXC VM...
Ganeti Cluster
Comparing Ganeti●   Private IaaS●   Primarily utilizes local storage●   Designed for hardware failures●   Mature project● ...
Project Background●   Google funded project●   Used in internal corporate env●   Open Sourced in 2007 GPLv2●   Team based ...
Goals of Ganeti
Goals: Low Entry Level● Keeping the entry level as low as  possible● Easy to install, manage and upgrade● No specialized h...
Goals: Enterprise Scale● Manage simultaneously from 1 to ~200  host machines● Access to advanced features  ● drbd, live mi...
Goals: Open Source Citizen● Design and code discussions are open● External contributions are welcome● Cooperate with other...
Terminology
TerminologyNode         virtualization hostNode Group   homogeneous set of nodes   (i.e. rack of nodes)Instance     virtua...
Architecture
Components● Linux & standard utils  ○ (iproute2, bridge-utils, ssh)● KVM, Xen or LXC● DRBD, LVM, RDB, or SAN● Python  ○ (p...
Nodes Roles              (management level)                    Runs ganeti-masterd, rapi, nodedMaster Node                ...
Nodes Roles          (instance hosting level)VM Capable Node   Can run virtual machinesDrained Nodes     Are being evacuat...
Instances● Virtual machine that runs on the cluster● fault tolerant/HA entity within cluster
Instance Parameters● Hypervisor: hvparams● General: beparams● Networking: nicparams● Modifiable at the instance or  cluste...
hvparams● Boot order, CDROM Image● NIC Type, Disk Type● VNC Parameters, Serial console● Kernel Path, initrd, args● Other H...
beparams / nicparams● Memory / Virtual CPUs● Adding or removing disks● MAC● NIC mode (routed or bridged)● Link
Disk Templatedrbd              LVM + DRBD between 2 nodesrbd               RBD volumes residing inside a RADOS cluster *pl...
Primary & Secondary Concepts● Instances always runs on primary● Uses secondary node for disk replication● Depends on disk ...
Instance creation scripts           also known as OS Definitions●   Requires Operating System installation    script●   Pr...
OS Variants● Variants of the OS Definition● Used for defining guest operating  system● Types of deployment settings:  ● Ex...
Latest Features            2.4                             2.5          March 2011                      April 2012●   Out ...
Latest Features             2.6                          Upcoming           July 2012                   Just ideas, not pr...
Initializing your clusterThe node needs to be set up following the ganeti installation guide.gnt-cluster init [-s ip] ... ...
gnt-clusterCluster wide operations:gnt-cluster      infognt-cluster      modify [-B/H/N ...]gnt-cluster      verifygnt-clu...
Adding nodesgnt-node add [-s ip] node2gnt-node add [-s ip] node3gnt-node add [-s ip] node4
Adding instances# install instance-{debootstrap, image}gnt-os listgnt-instance add -t drbd   {-n node3:node2 | -I hail }  ...
gnt-nodePer node operations:gnt-node remove node4gnt-node modify   [ --master-candidate yes|no ]   [ --drained yes|no ]   ...
-t drbdDRBD provides redundancy to instance data, andmakes it possible to perform live migration withouthaving shared stor...
Recovering from failure# set the node offlinegnt-node modify -O yes node3
Recovering from failure# failover instances to their secondariesgnt-node failover --ignore-consistency node3# or, for each...
Recovering from failure# restore redundancygnt-node evacuate -I hail node3# or, for each instance:gnt-instance replace-dis...
gnt-backupManage instance exports/backups:gnt-backup export -n node1 webgnt-backup import -t plain   {-n node3 | -I hail }...
htools: cluster resource management● Written in Haskell● Where do I put a new instance?● Where do I move an existing one? ...
Controlling Ganeti● Command line *● Ganeti Web Manager  ● Developed by OSUOSL● RAPI (Rest-full HTTP Interface) *● On-clust...
Job Queue● Ganeti operations generate jobs in the master  ○ with the exception of queries● Jobs execute concurrently● You ...
gnt-groupManaging node groups:gnt-group addgnt-group assign-nodesgnt-group evacuategnt-group listgnt-group modifygnt-group...
Running Ganeti in Production             What should you add?●   Monitoring/Automation    ● Check host disks, memory, load...
Ganeti in practice● Medium to small virtualization  environments● High performance  ● Dedicated hardware, faster disks, mo...
Ganeti as a "cloud"● Not a traditional cloud environment  ● No AWS APIs (yet at least), no object store  ● Ganeti specific...
How the OSL uses Ganeti● Powers all managed virtualization● Project hosting● KVM based● Hundreds of VMs● Web hosts, code h...
Ganeti at OSL● Node OS: Gentoo  ● Migrating towards CentOS● CFEngine for node configuration setup● Utilize instance-image ...
Ganeti at OSL● Production cluster  ● busybox, darcs, inkscape, musicbrainz, openmrs,       php.net, qemu, freenode, yum   ...
People running Ganeti● Google  ● Corporate Computing Infra● osuosl.org  ● Oregon State University Open Source Lab● grnet.g...
Questions?                     (Part 1 Conclusion)               Lance Albertson              lance@osuosl.org            ...
Upcoming SlideShare
Loading in …5
×

Hands on Virtualization with Ganeti (part 1) - LinuxCon 2012

5,138 views

Published on

Ganeti is a robust cluster virtualization management software tool. It’s built on top of existing virtualization technologies such as Xen and KVM and other Open Source software. Its integration with various technologies such as DRBD and LVM results in a cheaper High Availability infrastructure and linear scaling.

This hands-on tutorial will cover a basic overview of Ganeti, the step-by-step install & setup of a single-node and multi-node Ganeti cluster, operating the cluster, and some best practices of Ganeti.

Published in: Technology
2 Comments
2 Likes
Statistics
Notes
No Downloads
Views
Total views
5,138
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
69
Comments
2
Likes
2
Embeds 0
No embeds

No notes for slide

Hands on Virtualization with Ganeti (part 1) - LinuxCon 2012

  1. Hands onVirtualization with Ganeti (part 1) Lance Albertson @ramereth Associate Director OSU Open Source Lab
  2. About us● OSU Open Source Lab● Server hosting for Open Source Projects ○ Linux Foundation, Apache Software Foundation, Drupal, Python Software Foundation, Freenode, Gentoo, Debian, CentOS, Fedora, etc etc ...● Open Source development projects ○ Ganeti Web Manager
  3. Session Overview (part 1)● Ganeti Introduction ● Terminology ● Major Components● Latest Features● Using Ganeti in Practice● How Ganeti is deployed at OSUOSL
  4. Session Overview (part 2)● Hands on Demo● Installation and Initialization● Cluster Management ● Adding instances (VMs) ● Controlling instances ● Auto Allocation● Dealing with node failures
  5. What can Ganeti do?● Virtual machine management software tool● Manages clusters of physical machines● Xen/KVM/LXC VM deployment● Live Migration● Resiliency to failure ● data redundancy via DRBD● Cluster Balancing● Ease of repairs and hardware swaps
  6. Ganeti Cluster
  7. Comparing Ganeti● Private IaaS● Primarily utilizes local storage● Designed for hardware failures● Mature project● Low package requirements● Simple administration● Easily pluggable via hooks & RAPI
  8. Project Background● Google funded project● Used in internal corporate env● Open Sourced in 2007 GPLv2● Team based in Google Switzerland● Active mailing list & IRC channel● Started internally before libvirt, openstack, etc
  9. Goals of Ganeti
  10. Goals: Low Entry Level● Keeping the entry level as low as possible● Easy to install, manage and upgrade● No specialized hardware needed ● i.e. SANs● Lightweight ● no "expensive" package dependencies
  11. Goals: Enterprise Scale● Manage simultaneously from 1 to ~200 host machines● Access to advanced features ● drbd, live migration, API, OOB control● Batch VM deployments● Ease of lateral expansion and rebalancing
  12. Goals: Open Source Citizen● Design and code discussions are open● External contributions are welcome● Cooperate with other "big scale" Ganeti users● Welcome third-party projects ● Ganeti Web Manager (OSL), Synnefo (GRNET)
  13. Terminology
  14. TerminologyNode virtualization hostNode Group homogeneous set of nodes (i.e. rack of nodes)Instance virtualization guestCluster set of nodes, managed as a collectiveJob ganeti operation
  15. Architecture
  16. Components● Linux & standard utils ○ (iproute2, bridge-utils, ssh)● KVM, Xen or LXC● DRBD, LVM, RDB, or SAN● Python ○ (plus a few modules)● socat● Haskell (optional, for auto-allocation)
  17. Nodes Roles (management level) Runs ganeti-masterd, rapi, nodedMaster Node and confd Have a full copy of the config, can become masterMaster Candidates Run ganeti-confd and noded Cannot become masterRegular Nodes Get only part of the configOffline nodes In repair or decommissioned
  18. Nodes Roles (instance hosting level)VM Capable Node Can run virtual machinesDrained Nodes Are being evacuatedOffline Nodes Are in repair
  19. Instances● Virtual machine that runs on the cluster● fault tolerant/HA entity within cluster
  20. Instance Parameters● Hypervisor: hvparams● General: beparams● Networking: nicparams● Modifiable at the instance or cluster level
  21. hvparams● Boot order, CDROM Image● NIC Type, Disk Type● VNC Parameters, Serial console● Kernel Path, initrd, args● Other Hypervisor specific parameters
  22. beparams / nicparams● Memory / Virtual CPUs● Adding or removing disks● MAC● NIC mode (routed or bridged)● Link
  23. Disk Templatedrbd LVM + DRBD between 2 nodesrbd RBD volumes residing inside a RADOS cluster *plain LVM with no redundancydiskless No disks. Useful for testing only* experimental support added in 2.6
  24. Primary & Secondary Concepts● Instances always runs on primary● Uses secondary node for disk replication● Depends on disk template (i.e. drbd, plain)
  25. Instance creation scripts also known as OS Definitions● Requires Operating System installation script● Provide scripts to deploy various operating systems● Ganeti Instance Debootstrap ● upstream supported● Ganeti Instance Image ● written by me
  26. OS Variants● Variants of the OS Definition● Used for defining guest operating system● Types of deployment settings: ● Extra packages ● Filesystem ● Image directory ● Image Name
  27. Latest Features 2.4 2.5 March 2011 April 2012● Out of Band management ● shared storage (SAN)● vhost net support (KVM) support● hugepages support (KVM) ● improved node groups● initial node groups (scalability, evacuate, commands) ● master IP turnup customization ● full SPICE support (KVM)
  28. Latest Features 2.6 Upcoming July 2012 Just ideas, not promises● RBD support (ceph) ● Full dynamic memory support● initial memory balloning ● Better instance networking (KVM, Xen) customization● cpu pinning ● Rolling Reboot● OVF export/import support ● Better automation, self-● customized drbd parameters healing, availability● policies for better resource ● Higher Scalability modeling ● KVM block device migration● Optional haskell ganeti-confd ● Better OS Installation
  29. Initializing your clusterThe node needs to be set up following the ganeti installation guide.gnt-cluster init [-s ip] ... --enabled-hypervisors=kvm cluster
  30. gnt-clusterCluster wide operations:gnt-cluster infognt-cluster modify [-B/H/N ...]gnt-cluster verifygnt-cluster master-failovergnt-cluster command/copyfile ...
  31. Adding nodesgnt-node add [-s ip] node2gnt-node add [-s ip] node3gnt-node add [-s ip] node4
  32. Adding instances# install instance-{debootstrap, image}gnt-os listgnt-instance add -t drbd {-n node3:node2 | -I hail } -o debootstrap+default webping webssh web # easy with OS hooks
  33. gnt-nodePer node operations:gnt-node remove node4gnt-node modify [ --master-candidate yes|no ] [ --drained yes|no ] [ --offline yes|no ] node2gnt-node evacuate/failover/migrategnt-node powercycle
  34. -t drbdDRBD provides redundancy to instance data, andmakes it possible to perform live migration withouthaving shared storage between the nodes. "RAID1" over the network
  35. Recovering from failure# set the node offlinegnt-node modify -O yes node3
  36. Recovering from failure# failover instances to their secondariesgnt-node failover --ignore-consistency node3# or, for each instance:gnt-instance failover --ignore-consistency web
  37. Recovering from failure# restore redundancygnt-node evacuate -I hail node3# or, for each instance:gnt-instance replace-disks {-n node1 | -I hail } web
  38. gnt-backupManage instance exports/backups:gnt-backup export -n node1 webgnt-backup import -t plain {-n node3 | -I hail } --src-node node1 --src-dir /tmp/myexport webgnt-backup listgnt-backup remove
  39. htools: cluster resource management● Written in Haskell● Where do I put a new instance?● Where do I move an existing one? ● hail: the H iallocator● How much space do I have? ● hspace: the H space calculator● How do I fix an N+1 error? ● hbal: the cluster balancer
  40. Controlling Ganeti● Command line *● Ganeti Web Manager ● Developed by OSUOSL● RAPI (Rest-full HTTP Interface) *● On-cluster "luxi" interface * ● luxi is currently json over unix socket ● there is code for python and haskell* programmable interfaces
  41. Job Queue● Ganeti operations generate jobs in the master ○ with the exception of queries● Jobs execute concurrently● You can cancel non-started jobs, inspect the queue status, and inspect jobsgnt-job listgnt-job infognt-job watchgnt-job cancel
  42. gnt-groupManaging node groups:gnt-group addgnt-group assign-nodesgnt-group evacuategnt-group listgnt-group modifygnt-group removegnt-group renamegnt-instance change-group
  43. Running Ganeti in Production What should you add?● Monitoring/Automation ● Check host disks, memory, load ● Trigger events (evacuate, send to repairs, readd node, rebalance) ● Automated host installation/setup (config management)● Self service use ● Instance creation and resize ● Instance console access
  44. Ganeti in practice● Medium to small virtualization environments● High performance ● Dedicated hardware, faster disks, more spindles on local storage● Cheap hardware to high-end hardware● Higher reliability
  45. Ganeti as a "cloud"● Not a traditional cloud environment ● No AWS APIs (yet at least), no object store ● Ganeti specific API● Tools to extend it ● Ganeti Web Manager, Syssnefo, GlusterFS, Ceph● Storage layer differences ● block devices instead of disk images (typically)
  46. How the OSL uses Ganeti● Powers all managed virtualization● Project hosting● KVM based● Hundreds of VMs● Web hosts, code hosting, etc● Per-project clusters: PSF, OSGeo, phpBB, Gentoo● Powers Supercell
  47. Ganeti at OSL● Node OS: Gentoo ● Migrating towards CentOS● CFEngine for node configuration setup● Utilize instance-image for guest installs ● Flexibility on guest operating systems we can deploy● 10 clusters, 27 nodes, 230 instances● Ganeti Web Manager
  48. Ganeti at OSL● Production cluster ● busybox, darcs, inkscape, musicbrainz, openmrs, php.net, qemu, freenode, yum ● 5 nodes, 20 instances per machine ● 64G Ram / 3-7TB / 24 cores (2) ● 24G Ram / 670G / 4 cores (3)● Reduced cooling footprint● Per-project clusters enabled flexibility
  49. People running Ganeti● Google ● Corporate Computing Infra● osuosl.org ● Oregon State University Open Source Lab● grnet.gr ● Greek Research & Technology Network● nero.net ● Network for Education & Research in Oregon
  50. Questions? (Part 1 Conclusion) Lance Albertson lance@osuosl.org @ramereth http://lancealbertson.com Check it out at: http://code.google.com/p/ganeti/ Or just search for "Ganeti" Try it. Love it. Improve it. Contribute back (CLA required). © 2009-2012 Oregon State UniversityUse under CC-by-SA / Some content borrowed/modified from Iustin Pop (with permission)

×