Ganeti Web Manager: Cluster Management Made Simple

•

1 like•2,410 views

Looking for an easy, scalable way to manage your Ganeti-based clusters? Ganeti Web Manager provides admins an easy to deploy, Django based GUI that effectively manages private clusters & works equally well for providing customers access. With a caching system designed to scale to thousands of virtual machines without decreasing performance, Ganeti Web Manager makes cluster management truly simple.

Technology

Ganeti Web Manager
Cluster Management Made Simple
Lance Albertson
Peter Krenesky
http://is.gd/oscongwm | http://is.gd/oscongwmpdf

About us
OSU Open Source Lab
Server hosting for Open Source Projects
Open Source development projects
Lance / Lead Systems Administrator
Peter / Lead Software Engineer

Session Overview
Ganeti Basics
Ganeti Web Manager
GWM Internals
Using GWM

What is Ganeti?
Cluster virtual server management
software tool
Built on top of existing OSS
hypervisors
Fast & simple recovery after physical
failures
Using cheap commodity hardware
Private IaaS

Comparing Ganeti
Utilizes local storage
Built to deal with hardware failures
Mature project
Low package requirements
Easily pluggable via hooks & RAPI

Project Background
Google funded project
Used in internal corporate env
Open Sourced in 2007 GPLv2
Team based in Google Switzerland
Active mailing list & IRC channel
Started internally before libvirt

Components
Python
Haskell
DRBD
LVM
Hypervisor

Nodes
Physical machine
Fault tolerance not required
Added/removed at will from cluster
No data loss with loss of node

Instances

Virtual machine that runs on the
cluster
fault tolerant/HA entity within cluster

Instance Parameters
Hypervisor (called hvparams)
General (called beparams)
Networking (called nicparams)
Modified via instance or cluster
defaults

Disk template
drbd : LVM + DRBD between 2 nodes
plain : LVM w/ no redundancy
file : Plain files, no redundancy
diskless : Special purposes

IAllocator
Automatic placement of instances
Eliminates manual node specification
htools
External scripts used to compute

Components
Automatic allocation
hbal : Cluster rebalancer
hail : IAllocator script
hspace : Cluster capacity estimator

Primary & Secondary concepts

Instances always runs on primary
Uses secondary node for disk
replication
Depends on disk template (i.e. drbd)

Remote API
External tools
Retrieve cluster state
Execute commands
JSON over HTTP via REST

RAPI Security
Users & Passwords
RFC 2617 HTTP Authentication
Read-only or Read-write

Easy management of Ganeti
Client facing service

Releases
Project Founded - 9-10-2011
Version 0.4 - 12-20-2010
Version 0.5 - 02-03-2011
Version 0.6 - 03-04-2011
Version 0.7 - 06-17-2011

Open Sourced Libraries
Django Object Permissions
Django Object Log
Twisted VNC Auth Proxy

Dependencies
Python >=2.5, 2.7 recommended
Pip - Python package installer
Fabric - Install scripts
VirtualEnv - Python virtual
environments
Git - Distributed Source Control

Fabric Installer
$ fab dev deploy

$ fab prod deploy

Import Tools
Find Orphaned Instances
Import New Nodes & Instances
Remove Deleted Nodes & Instances

Cache Updater
Imports Nodes and Virtual Machines too
$ twistd --pidfile=cache.pid gwm_cache

VNC Auth Proxy

allows proxying through firewall / VPN
no need for passwords

Personas
Users can act on behalf of groups

Ownership
vs.
Permissions
Ownership is for book keeping
Permissions let you do things

Future Features
Instance Template
Instance Defaults
API
And much more!

Questions?
Lance Albertson Peter Krenesky

lance@osuosl.org peter@osuosl.org

@ramereth @kreneskyp

http://www.lancealbertson.com http://blogs.osuosl.org/kreneskyp/

http://code.google.com/p/ganeti/
http://code.osuosl.org/projects/ganeti-webmgr

Presentation made with showoff
http://github.com/ramereth/presentation-ganeti-tutorial
http://is.gd/oscongwm | http://is.gd/oscongwmpdf

Ganeti is a robust cluster virtualization management software tool. It’s built on top of existing virtualization technologies such as Xen and KVM and other Open Source software. Its integration with various technologies such as DRBD and LVM results in a cheaper High Availability infrastructure and linear scaling. This hands-on tutorial will cover a basic overview of Ganeti, the step-by-step install & setup of a single-node and multi-node Ganeti cluster, operating the cluster, and some best practices of Ganeti. If attendees want to participate in the optional hands-on portions of the tutorial, there will be virtual machine images available online and at the tutorial itself. We’ll be using VirtualBox and Vagrant to deploy Ganeti on two to three Ubuntu nodes.

Bare Metal to OpenStack with Razor and Chef

Matt Ray

The Kitchen Cloud How To: Automating Joyent SmartMachines with Chef

Chef Software, Inc.

Learning a new OS can be intimidating, especially one with less support in terms of open source Chef cookbooks. At Wanelo we’ve found the rewards of using Chef with Joyent’s SmartOS to be well worth the effort. SmartOS is an open source fork of Illumos (think Solaris) that runs in the Joyent Public Cloud. Over the last year we’ve grown to love SmartOS as a deployment environment, and with the help of Chef have grown Wanelo’s infrastructure more than ten times in six months to meet the demands our exponential user growth. In the next year, we expect to grow our infrastructure by another factor of ten. On another public cloud, our business growth would have required a significantly larger infrastructure at every step. In this session I’ll explain why we appreciate SmartOS so much and how you can get started. What’s the terminology? What plugins do you need, and how do you use them? What providers should you learn and where can you find them? I’ll provide bootstrap scripts, basic roles and cookbooks on Github to get people provisioning and using SmartMachines immediately. For larger infrastructures, I’ll walk through some of the dependencies that have made our lives easier, and explain why. By the end, you should have the code at your fingertips to deploy a Ruby or Rails application to the Joyent Public Cloud, with all of the dependent services up and running.

Containerization is more than the new Virtualization: enabling separation of ...

Jérôme Petazzoni

Docker offers a new, lightweight approach to application portability. Applications are shipped using a common container format, and managed with a high-level API. Their processes run within isolated namespaces which abstract the operating environment, independently of the distribution, versions, network setup, and other details of this environment. This "containerization" has often been nicknamed "the new virtualization". But containers are more than lightweight virtual machines. Beyond their smaller footprint, shorter boot times, and higher consolidation factors, they also bring a lot of new features and use cases which were not possible with classical virtual machines. We will focus on one of those features: separation of operational concerns. Specifically, we will demonstrate how some fundamental tasks like logging, remote access, backups, and troubleshooting can be entirely decoupled from the deployment of applications and services. This decoupling results in independent, smaller, simpler moving parts; just like microservice architectures break down large monolithic apps in more manageable components.

Cassandra via-docker

Chris Ballance

Introduction openstack-meetup-nov-28

Sadique Puthen

Running Cassandra in a docker environment to give you a flexible development environment that uses only a very small set of resources, both locally and with your favorite cloud provider. Lessons learned running Cassandra with a very small set of resources are applicable to both your local development environment and larger, less constrained production deployments. From http://www.meetup.com/Docker-Santa-Clara/events/232789407/

Kubernetes networking

Sim Janghoon

Scaling and Managing Cassandra with docker, CoreOS and Presto

Vali-Marius Malinoiu

Cassandra on Docker

Instaclustr

The state of the swarm

Mathieu Buffenoir

Docker - From Walking To Running

Giacomo Vacca

Docker Container Orchestration

Fernand Galiana

LXC, Docker, and the future of software delivery | LinuxCon 2013

dotCloud

Optimizing VM images for OpenStack with KVM/QEMUOpenStack Foundation

Docker and Containers for Development and Deployment — SCALE12X

Jérôme Petazzoni

Docker is an Open Source engine to build, run, and manage containers. We'll explain what are Linux Containers, what powers them (under the hood), and what extra value Docker brings to the table. Then we'll see what the typical Docker workflow looks like from a developer point of view. We'll also give an Ops perspective, including deployment options. If you already saw a "Docker 101", consider this presentation as the February 2014 update! :-)

Consuming Cinder from Docker

Tesora

Kvm performance optimization for ubuntuSim Janghoon

moscmy2016: Extending Docker

Mohammad Fairus Khalid

A Hands-on Introduction to Docker

CodeOps Technologies LLP

Docker is popular open-source software containerization platform. It provides an ability to package software into standardised units on Docker for software development. In this hands-on introductory session, I introduce the concept of containers and provide an overview of Docker. Participants can learn important concepts in Docker step-by-step and learn by example by running commands with me. The main session involves using Docker CLI (Command Line Interface) covering all the key concepts such as creating images and managing containers. What is more, this workshop ends with a complete example of getting some amazing work done with ease using Docker. Presented in OSI Days '16: http://opensourceindia.in/osidays/workshops-osi-2016/

Docker, Docker Swarm mangement tool - Gorae

Rhio kim

OpenStack Cinder Best Practices - Meet Up

Aaron Delp

Best Deep Learning Post from LinkedIn Group

Farshid Pirahansiah

Best Deep Learning Post from LinkedIn Group Datasets for Deep Learning (Slide share) http://www.slideshare.net/pirahansiah/computer-vision-deep-learning-opencv Deep Learning for Video Analysis – part 1 (DeepStream SDK NVIDIA TensorRT || NVIDIA GPU Inference Engine (GIE)) https://www.linkedin.com/lite/external-redirect?url=http%3A%2F%2Fwww%2Eslideshare%2Enet%2Fpirahansiah%2Fdeep-learning-for-video-analysis-part-1-deepstream-sdk-nvidia-tensorrt-nvidia-gpu-inference-engine-gie&urlHash=F2e8 How to install Digits 5.1 on Ubuntu 14 https://www.linkedin.com/redir/redirect?url=http%3A%2F%2Fwww%2Eslideshare%2Enet%2Fpirahansiah%2Fhow-to-install-digits-51-on-ubuntu-14&urlhash=8V-j&_t=tracking_anet example of using deep learning in opencv 3.1 http://docs.opencv.org/3.1.0/d5/de7/tutorial_dnn_googlenet.html Layers in Deep Learning & Caffe layers (model architecture ) https://www.linkedin.com/lite/external-redirect?url=http%3A%2F%2Fwww%2Eslideshare%2Enet%2Fpirahansiah%2Flayers-in-deep-learningcaffe-layers-model-architecture&urlHash=BsHf Install, Compile, Setup, Setting OpenCV 3.2, Visual C++ 2015, Win 64bit, https://www.linkedin.com/lite/external-redirect?url=http%3A%2F%2Fwww%2Eslideshare%2Enet%2Fpirahansiah%2Finstall-compile-setup-setting-opencv-32-visual-c-2015-win-64bit&urlHash=dC7R Autonomous driving vehicles (Computer Vision, Deep Learning) https://www.linkedin.com/pulse/autonomous-driving-vehicles-computer-vision-deep-farshid-pirahansiah-1?trk=mp-author-card Compile OpenCV 3.2 with Visual Studio 2017 (C++) on Windows 10 x64 bit and test with TensorFlow https://www.youtube.com/watch?v=mdeP8SdvSJw TensorFlow in OpenCV 3.2 Visual Studio 2017 (C++ deep learning) application https://www.youtube.com/watch?v=QK1kLTfS97c

Demystifying kubernetes

Works Applications

Libvirt/KVM Driver Update (Kilo)

Stephen Gordon

KVM (Kernel-based Virtual Machine) is a full virtualization solution built into the Linux kernel. OpenStack Foundation user surveys consistently indicate that KVM is the most commonly used Hypervisor for OpenStack deployments, managed using the Libvirt driver for OpenStack Compute (Nova). Despite this sustained popularity development of the driver, and indeed the underlying Hypervisor itself, continues at a frantic pace. This presentation will help you make sense of it all starting with an overview of the way Nova, Libvirt, and KVM interact before analysing progress made in Kilo on utilizing key Libvirt/KVM features in Nova including: Instance vCPU pinning Huge page backed instances Enhanced NUMA topology awareness ...and more! The session will close with a discussion of how in addition to exposing existing Libvirt/KVM features emerging OpenStack use cases - such as Network Function Virtualization (NFV) and High Performance Computing (HPC) - are driving open innovation in the Libvirt, QEMU, and KVM projects themselves.

Docker from A to Z, including Swarm and OCCS

Frank Munz

ZendCon 2015 - DevOps for Small Teams

Joe Ferguson

DevOps is a large part of a company of any size. In the 9+ years that I have been a professional developer I have always taken an interest in DevOps and have been the "server person" for most of the teams I have been a part of. I would like to teach others how easy it is to implement modern tools to make their everyday development and development processes better. I will cover a range of topics from "Stop using WAMP/MAMP and start using Vagrant", "version control isn't renaming files", "Automate common tasks with shell scripts / command line PHP apps" and "From Vagrant to Production".

Madison PHP 2015 - DevOps For Small Teams

Joe Ferguson

What's hot

Cassandra and docker

Ben Bromhead

Kubernetes networking

Sim Janghoon

Scaling and Managing Cassandra with docker, CoreOS and Presto

Vali-Marius Malinoiu

Cassandra on Docker

Instaclustr

The state of the swarm

Mathieu Buffenoir

Docker - From Walking To Running

Giacomo Vacca

Docker Container Orchestration

Fernand Galiana

LXC, Docker, and the future of software delivery | LinuxCon 2013

dotCloud

Optimizing VM images for OpenStack with KVM/QEMUOpenStack Foundation

Docker and Containers for Development and Deployment — SCALE12X

Jérôme Petazzoni

Consuming Cinder from Docker

Tesora

Kvm performance optimization for ubuntuSim Janghoon

moscmy2016: Extending Docker

Mohammad Fairus Khalid

A Hands-on Introduction to Docker

CodeOps Technologies LLP

Docker, Docker Swarm mangement tool - Gorae

Rhio kim

OpenStack Cinder Best Practices - Meet Up

Aaron Delp

Best Deep Learning Post from LinkedIn Group

Farshid Pirahansiah

Demystifying kubernetes

Works Applications

Libvirt/KVM Driver Update (Kilo)

Stephen Gordon

Docker from A to Z, including Swarm and OCCS

Frank Munz

What's hot (20)

Cassandra and docker

Kubernetes networking

Scaling and Managing Cassandra with docker, CoreOS and Presto

Cassandra on Docker

The state of the swarm

Docker - From Walking To Running

Docker Container Orchestration

LXC, Docker, and the future of software delivery | LinuxCon 2013

Optimizing VM images for OpenStack with KVM/QEMU

Docker and Containers for Development and Deployment — SCALE12X

Consuming Cinder from Docker

Kvm performance optimization for ubuntu

moscmy2016: Extending Docker

A Hands-on Introduction to Docker

Docker, Docker Swarm mangement tool - Gorae

OpenStack Cinder Best Practices - Meet Up

Best Deep Learning Post from LinkedIn Group

Demystifying kubernetes

Libvirt/KVM Driver Update (Kilo)

Docker from A to Z, including Swarm and OCCS

Similar to Ganeti Web Manager: Cluster Management Made Simple

ZendCon 2015 - DevOps for Small Teams

Joe Ferguson

Madison PHP 2015 - DevOps For Small Teams

Joe Ferguson

Midwest PHP 2017 DevOps For Small team

Joe Ferguson

See Hudson Run, Run Hudson, Run [SELF 2010]

Vincent Batts

Slides from LAX & DEN usergroup meetings10n Software, LLC

Planning For High Performance Web Application

Yue Tian

DevOps For Small Teams

Joe Ferguson

Planning for-high-performance-web-applicationNguyễn Duy Nhân

OSDC 2016 - Continous Integration in Data Centers - Further 3 Years later by ...

NETWAYS

I gave a talk titled "Continuous Integration in data centers“ at OSDC in 2013, presenting ways how to realize continuous integration/delivery with Jenkins and related tools.Three years later we gained new tools in our continuous delivery pipeline, including Docker, Gerrit and Goss. Over the years we also had to deal with different problems caused by faster release cycles, a growing team and gaining new projects. We therefore established code review in our pipeline, improved our test infrastructure and invested in our infrastructure automation.In this talk I will discuss the lessons we learned over the last years, demonstrate how a proper continuous delivery pipeline can improve your life and how open source tools like Jenkins, Docker and Gerrit can be leveraged for setting up such an environment.

One-Man Ops

Jos Boumans

Rohit yadav cloud stack internals

ShapeBlue

MySQL Monitoring Shoot Out

Kris Buytaert

Gerrit linuxtag2011

thkoch

A Fabric/Puppet Build/Deploy System

adrian_nye

Devoops: DoJ Annual Cybersecurity Training Symposium Edition 2015

Chris Gates

In a rare mash-up, DevOps is increasingly blending the work of both application and network security professionals. In a quest to move faster, organizations can end up creating security vulnerabilities using the tools and products meant to protect them. Both Chris Gates (carnal0wnage) and Ken Johnson (cktricky) will share their collaborative research into the technology driving DevOps as well as share their stories of what happens when these tools are used insecurely as well as when the tools are just insecure. Technologies discussed will encompass AWS Technology, Chef, Puppet, Hudson/Jenkins, Vagrant, Kickstart and much, much more. Everything from common misconfigurations to remote code execution will be presented. This is research to bring awareness to those responsible for securing a DevOps environment.

Introduction to node.js GDD

Sudar Muthu

Node js introduction

Joseph de Castelnau

Microservices Application Tracing Standards and Simulators - Adrians at OSCON

Adrian Cockcroft

Introducing resinOS: An Operating System Tailored for Containers and Built fo...

Balena

This presentation, from the Embedded Linux Conference Europe in October 2016, discusses how resinOS was built, highlights some of its key features, and shares a roadmap for future development and contribution. resinOS is the latest open-source tool built by resin.io to enable the future of hardware with the tools of modern software. resinOS is a simple yet powerful operating system that brings standard Docker containers to embedded devices and works on a wide variety of device types and architectures. resinOS was born from the team’s experience deploying embedded containers across device types and has been battle-tested in production environments. You can download resinOS at https://resinos.io

Nagios Conference 2014 - Spenser Reinhardt - Detecting Security Breaches With...

Nagios

Similar to Ganeti Web Manager: Cluster Management Made Simple (20)

ZendCon 2015 - DevOps for Small Teams

Madison PHP 2015 - DevOps For Small Teams

Midwest PHP 2017 DevOps For Small team

See Hudson Run, Run Hudson, Run [SELF 2010]

Slides from LAX & DEN usergroup meetings

Planning For High Performance Web Application

DevOps For Small Teams

Planning for-high-performance-web-application

OSDC 2016 - Continous Integration in Data Centers - Further 3 Years later by ...

One-Man Ops

Rohit yadav cloud stack internals

MySQL Monitoring Shoot Out

Gerrit linuxtag2011

A Fabric/Puppet Build/Deploy System

Devoops: DoJ Annual Cybersecurity Training Symposium Edition 2015

Introduction to node.js GDD

Node js introduction

Microservices Application Tracing Standards and Simulators - Adrians at OSCON

Introducing resinOS: An Operating System Tailored for Containers and Built fo...

Nagios Conference 2014 - Spenser Reinhardt - Detecting Security Breaches With...

More from OSCON Byrum

OSCON 2013 - Planning an OpenStack Cloud - Tom Fifield

Ganeti Web Manager: Cluster Management Made Simple

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Ganeti Web Manager: Cluster Management Made Simple

Similar to Ganeti Web Manager: Cluster Management Made Simple (20)

More from OSCON Byrum

More from OSCON Byrum (20)

Recently uploaded

Recently uploaded (20)

Ganeti Web Manager: Cluster Management Made Simple