Your SlideShare is downloading. ×
0
HOW WE USED RUBY TO
BUILD LOCAWEB'S CLOUD
WILLIAN MOLINARI
a.k.a PotHix
MOTIVATION
WORKING AT LOCAWEB
on the past 3 years
A LOT OF KNOWN PEOPLE
WHICH PROBLEM WE WANTED TO SOLVE
CLOUD SERVER
CLOUD SERVER TECHNOLOGIES
Perl
Java
VMware
VMWARE ARCHITECTURE
PROBLEMS
for both technical and product sides
CLOUD SERVER PRO
CLOUD SERVER PRO TECHNOLOGIES
Ruby
Xenserver
XEN ARCHITECTURE
XENSERVER, XENAPI AND XMLRPC
with a python example
RUBY + XENAPI
https://github.com/locaweb/xenapi-ruby
XENAPI EXAMPLE - GET DISKS
reference:
vm_ref = session.VM.get_by_uuid(virtual_machine.uuid)
vm_vbds_refs = session.VM.get_...
XENAPI-RUBY HELPER FOR THIS CODE
# Get all disks for a VM
disks = virtual_machine.vbds
XENAPI-RUBY PROVIDES SOME HELPERS
But the way is still validXenAPI
STARTING INFRASTRUCTURE
One pool, one firewall (providing NAT) and one DHCP server
SOMETHING LIKE THIS...
ASYNC ALL OVER THE PLACE
started with andActiveMQ Stomp
DIFFERENT QUEUE IMPLEMENTATIONS
XMPP, ActiveMQ (+ Stomp), Resque
RESQUE IMPLEMENTATION EXAMPLE
app/consumers/resque/consumer.rb
require "consumers/resque/configuration"
module Resque
clas...
XMPP IMPLEMENTATION EXAMPLE
app/consumers/xmpp/consumer.rb
require 'xmpp4r'
module XMPP
class Consumer
class << self
attr_...
EASILY CHANGEABLE
app/consumers/consumer.rb
# -*- coding: UTF-8 -*-
Consumer = Resque::Consumer
RESQUE WON
BTW
VIRTUAL MACHINE INSTALLATION PROCESS
http://github.com/locaweb/bpmachine
BPMACHINE EXAMPLE
process :of => :install do
must_be :machine_created
transition :select_ips,
:from => :machine_created,
:...
AND THE STEPS
app/steps/install_steps.rb
module InstallSteps
def select_ips
log_activity(:info, :id => id, :status => 'sta...
GET THE NAME AND IP
A different app to handle these
DHCP CONFIGURATION
isc-dhcp
DUMMY DHCP EXPLANATION
ISC DHCP FILE EXAMPLE
# cpro0007
host 3ea808bd-5ef0-4227-a617-f5b0694e408c{
hardware ethernet 00:25:22:bd:d1:20;
fixed-add...
ERB FILE
<% vms.each do |vm| %>
# <%= vm.name %>
host <%= vm.uuid %> {
hardware ethernet <%= vm.mac %>;
fixed-address <%= ...
SIMPLE SCRIPT FOR DHCP
# Prepare isc dhcp file
echo -e "<%= dhcp_config %>" > /etc/dhcp/<%= Config[:dhcp_conf_file] %>
# R...
NET::SSH AS TRANSPORT LAYER
Net::SSH.start(@host, @user, ssh_options) do |ssh|
ssh_result = ssh.open_channel do |channel|
...
FIREWALL CONFIGURATIONS
default DROP
ADDING SOME FIREWALL RULES
iptables -A <%= internal_address %>/32 
-p <%= rule.filter_protocol %> 
-s <%= rule.filter_addr...
CONFIGURING NAT
# Configuring NAT
...
iptables -t nat -A PREROUTING -d <%= rule.external_address %> 
-j DNAT --to-destinat...
VNC ACCESS
Provided by Xen VNC Proxy (a.k.a XVP)
EXAMPLE XVP FILE
POOL XENSERVERPOOL1
DOMAIN ""
MANAGER root proxy_password_encrypted_hash
HOST 10.20.30.40
VM 7890 4344dc8...
GENERATED FROM XVP BINARY
# for hosts
/bin/echo -e "proxy_password_here" | /usr/sbin/xvp -x
# for each console
/bin/echo -...
WAIT...XVP IS OPEN SOURCE
RUBY VERSION
coisa linda!
def encrypt_vnc(password)
key = [0xc1, 0x24, 0x08, 0x99, 0xc2, 0x26, 0x07, 0x05]
des = OpenSSL::...
POST INSTALL PROCESS
VM already shipped by Xen
WINDOWS? LINUX?
Specialized post install
DONE BY VIRTUAL MACHINE TYPE
def post_installers
{ :windows2003 => PostInstall::Windows,
:windows2008 => PostInstall::Wind...
SIMPLIFIED, OF COURSE... WE HAVE:
centos5, centos6, ubuntu9.04, ubuntu9.10, ubuntu10.04,
ubuntu10.10, debian5, debian6, wi...
AND MANAGED ONES!
linux + cpanel, linux + plesk, windows + plesk, ...
LINUX POST INSTALL
SSH and bash scripts
WINDOWS POST INSTALL
and powershell scriptsWinrm
PARTITIONING DISKS EXAMPLE
for Linux and for WindowsLVM Diskpart
MORE AND MORE STEPS
Added basic monitoring
Configure DNS
VM cleanup
and so on...
AND WE'RE DONE!
production ready
SHIPPED WITH A SIMPLE ARCHITECTURE
MULTI POOL
"all your pools are belong to us"
MULTI POOL
MATRIX MACHINES
and their maintenance
A WILD WINDOWS MATRIX MACHINE
APPEARED!
SECURITY FIX RELEASED
SECURITY FIX APPLIED
OK...BUT WE HAVE MORE!
WE CAN DEAL WITH IT!
BUT....WAT!
NUMBERS!
~ 7000 VMs
~ 25 pools
~ 300 hosts
XEN IMPORT AND EXPORT
not so well documented
USING HTTP!
def export(vm_uuid, options = {})
options = {:to => "/tmp/export_file"}.merge(options)
file = File.open(option...
MULTI STORAGE
or...datasets!
MULTI FIREWALL
avoiding package overflow
SO BASICALLY...WE HAVE THIS:
TIME TO BREAK RESPONSABILITIES
by improving the infrastructure
NEW NETWORK INFRASTRUCTURE
dedicated servers for firewall and dhcp
BEFORE
NOW
QUANTUM
from the openstack project
QUANTUM + OPENVSWITCH
to useLocaweb quantum plugin openvswitch
NOT SSH ANYMORE
Queueing and consuming using RabbitMQ
NEW FIREWALL AND DHCP ARCHITECTURE
NEW HYPERVISORS TO SUPPORT
KVM, VMware
SIMPLESTACK!
https://github.com/locaweb/simplestack
A PROXY FOR HYPERVISORS
SIMPLESTACK INSTEAD OF XENAPI DIRECTLY
Using
# Old xenapi code
session.VM.get_by_uuid(virtual_machine.uuid)
# Simplestack
...
FEATURES WORTH MENTIONING
AUTOSCALE AND ALERTS
driven by monitoring
LEELA
https://github.com/locaweb/leela
THE FLOW
JUST MESSAGING
BLATHER AND DAEMON TOOLS
require 'blather/client'
DaemonKit::Application.running!
message :chat?, :body do |m|
begin
if We...
HETEROGENEOUS ENVIRONMENT
new features and support for all versions!
XEN POOL KNOWS ITS NETWORK VERSION
# -*- coding: UTF-8 -*-
module Network
module Manager
extend self
def for(pool)
pool.ne...
DIFFERENT BEHAVIORS FROM VERSION 1
# -*- coding: UTF-8 -*-
module Network
class Ver1
def internal_vlan?
true
end
def insta...
TO NETWORK VERSION 2
# -*- coding: UTF-8 -*-
module Network
class Ver2
def internal_vlan?
false
end
def installation_ip(vm...
MULTIPLE IMPLEMENTATIONS FOR:
network configurations
firewall implementations
post install methods
LESSONS LEARNED
Start small and grow fast
OOP helps a lot to keep the code organized
Multiple projects
Multiple languages
THANKS!
BY WILLIAN MOLINARI (POTHIX)
Upcoming SlideShare
Loading in...5
×

How we used ruby to build locaweb's cloud (http://presentations.pothix.com/rubyconf2013/)

252

Published on

**The slides are not correctly rendered. The HTML/Javascript version is here: http://presentations.pothix.com/rubyconf2013/**

This presentation shows what we have done with Ruby to create Locaweb's cloud computing product.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
252
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "How we used ruby to build locaweb's cloud (http://presentations.pothix.com/rubyconf2013/)"

  1. 1. HOW WE USED RUBY TO BUILD LOCAWEB'S CLOUD WILLIAN MOLINARI a.k.a PotHix
  2. 2. MOTIVATION
  3. 3. WORKING AT LOCAWEB on the past 3 years
  4. 4. A LOT OF KNOWN PEOPLE
  5. 5. WHICH PROBLEM WE WANTED TO SOLVE
  6. 6. CLOUD SERVER
  7. 7. CLOUD SERVER TECHNOLOGIES Perl Java VMware
  8. 8. VMWARE ARCHITECTURE
  9. 9. PROBLEMS for both technical and product sides
  10. 10. CLOUD SERVER PRO
  11. 11. CLOUD SERVER PRO TECHNOLOGIES Ruby Xenserver
  12. 12. XEN ARCHITECTURE
  13. 13. XENSERVER, XENAPI AND XMLRPC with a python example
  14. 14. RUBY + XENAPI https://github.com/locaweb/xenapi-ruby
  15. 15. XENAPI EXAMPLE - GET DISKS reference: vm_ref = session.VM.get_by_uuid(virtual_machine.uuid) vm_vbds_refs = session.VM.get_record(vm_ref)["VBDs"] disks = vm_vbds_refs.inject({}) do |disks, vm_vbd_ref| vm_vbd_record = session.VBD.get_record(vm_vbd_ref) if vm_vbd_record["type"] == "Disk" disks[vm_vbd_ref] = vm_vbd_record end disks Xenserver API documentation
  16. 16. XENAPI-RUBY HELPER FOR THIS CODE # Get all disks for a VM disks = virtual_machine.vbds
  17. 17. XENAPI-RUBY PROVIDES SOME HELPERS But the way is still validXenAPI
  18. 18. STARTING INFRASTRUCTURE One pool, one firewall (providing NAT) and one DHCP server
  19. 19. SOMETHING LIKE THIS...
  20. 20. ASYNC ALL OVER THE PLACE started with andActiveMQ Stomp
  21. 21. DIFFERENT QUEUE IMPLEMENTATIONS XMPP, ActiveMQ (+ Stomp), Resque
  22. 22. RESQUE IMPLEMENTATION EXAMPLE app/consumers/resque/consumer.rb require "consumers/resque/configuration" module Resque class Consumer class << self def queue(name) @queue = name end ...
  23. 23. XMPP IMPLEMENTATION EXAMPLE app/consumers/xmpp/consumer.rb require 'xmpp4r' module XMPP class Consumer class << self attr_reader :queue_name, :queued_process, :timeout_value def queue(name) @queue_name = "#{name}@localhost" end ...
  24. 24. EASILY CHANGEABLE app/consumers/consumer.rb # -*- coding: UTF-8 -*- Consumer = Resque::Consumer
  25. 25. RESQUE WON BTW
  26. 26. VIRTUAL MACHINE INSTALLATION PROCESS http://github.com/locaweb/bpmachine
  27. 27. BPMACHINE EXAMPLE process :of => :install do must_be :machine_created transition :select_ips, :from => :machine_created, :to => :ips_selected transition :queue_dhcp_for_install, :from => :ips_selected, :to => :dhcp_synchronized ...
  28. 28. AND THE STEPS app/steps/install_steps.rb module InstallSteps def select_ips log_activity(:info, :id => id, :status => 'starting') create_ip(primary=true) unless primary_ip_pair log_activity(:info, :id => id, :status => 'done') end def queue_dhcp_for_install ... end
  29. 29. GET THE NAME AND IP A different app to handle these
  30. 30. DHCP CONFIGURATION isc-dhcp
  31. 31. DUMMY DHCP EXPLANATION
  32. 32. ISC DHCP FILE EXAMPLE # cpro0007 host 3ea808bd-5ef0-4227-a617-f5b0694e408c{ hardware ethernet 00:25:22:bd:d1:20; fixed-address 186.202.1.7; option host-name "cpro0007"; } # cpro0051 host 3ea808bd-5ef0-4227-a617-cf5b0694e408 { hardware ethernet 00:25:22:bd:d1:21; fixed-address 186.202.1.8; option host-name "cpro0051"; }
  33. 33. ERB FILE <% vms.each do |vm| %> # <%= vm.name %> host <%= vm.uuid %> { hardware ethernet <%= vm.mac %>; fixed-address <%= vm.private_ip.address %>; option host-name "<%= vm.name %>"; } <% end %>
  34. 34. SIMPLE SCRIPT FOR DHCP # Prepare isc dhcp file echo -e "<%= dhcp_config %>" > /etc/dhcp/<%= Config[:dhcp_conf_file] %> # Restart DHCP sudo /etc/init.d/isc-dhcp-server restart
  35. 35. NET::SSH AS TRANSPORT LAYER Net::SSH.start(@host, @user, ssh_options) do |ssh| ssh_result = ssh.open_channel do |channel| channel.request_pty do |chn, success| raise "Could not obtain pty from ssh" unless success chn[:out] = "" chn.exec command ...
  36. 36. FIREWALL CONFIGURATIONS default DROP
  37. 37. ADDING SOME FIREWALL RULES iptables -A <%= internal_address %>/32 -p <%= rule.filter_protocol %> -s <%= rule.filter_address %> -d <%= internal_address %> --dport <%= rule.filter_port %> -j ACCEPT
  38. 38. CONFIGURING NAT # Configuring NAT ... iptables -t nat -A PREROUTING -d <%= rule.external_address %> -j DNAT --to-destination <%= rule.internal_address %> iptables -t nat -A POSTROUTING -s <%= rule.internal_address %> -j SNAT --to-source <%= rule.external_address %> ...
  39. 39. VNC ACCESS Provided by Xen VNC Proxy (a.k.a XVP)
  40. 40. EXAMPLE XVP FILE POOL XENSERVERPOOL1 DOMAIN "" MANAGER root proxy_password_encrypted_hash HOST 10.20.30.40 VM 7890 4344dc8f-1bdd-4b65-812a-a0dc9b27256e console1_encrypted_pass VM 7891 c5b2d28e-4c11-492f-9d09-33670876cb4a console2_encrypted_pass
  41. 41. GENERATED FROM XVP BINARY # for hosts /bin/echo -e "proxy_password_here" | /usr/sbin/xvp -x # for each console /bin/echo -e "console_password_here" | /usr/sbin/xvp -e
  42. 42. WAIT...XVP IS OPEN SOURCE
  43. 43. RUBY VERSION coisa linda! def encrypt_vnc(password) key = [0xc1, 0x24, 0x08, 0x99, 0xc2, 0x26, 0x07, 0x05] des = OpenSSL::Cipher::Cipher.new("des-ecb") des.key = key.map(&:chr).join des.encrypt des.update(password).unpack('H*').first end
  44. 44. POST INSTALL PROCESS VM already shipped by Xen
  45. 45. WINDOWS? LINUX? Specialized post install
  46. 46. DONE BY VIRTUAL MACHINE TYPE def post_installers { :windows2003 => PostInstall::Windows, :windows2008 => PostInstall::Windows, :linux => PostInstall::Linux } end ... def post_install post_installers[code.to_sym].new end
  47. 47. SIMPLIFIED, OF COURSE... WE HAVE: centos5, centos6, ubuntu9.04, ubuntu9.10, ubuntu10.04, ubuntu10.10, debian5, debian6, windows2003, ...
  48. 48. AND MANAGED ONES! linux + cpanel, linux + plesk, windows + plesk, ...
  49. 49. LINUX POST INSTALL SSH and bash scripts
  50. 50. WINDOWS POST INSTALL and powershell scriptsWinrm
  51. 51. PARTITIONING DISKS EXAMPLE for Linux and for WindowsLVM Diskpart
  52. 52. MORE AND MORE STEPS Added basic monitoring Configure DNS VM cleanup and so on...
  53. 53. AND WE'RE DONE! production ready
  54. 54. SHIPPED WITH A SIMPLE ARCHITECTURE
  55. 55. MULTI POOL "all your pools are belong to us"
  56. 56. MULTI POOL
  57. 57. MATRIX MACHINES and their maintenance
  58. 58. A WILD WINDOWS MATRIX MACHINE APPEARED!
  59. 59. SECURITY FIX RELEASED
  60. 60. SECURITY FIX APPLIED
  61. 61. OK...BUT WE HAVE MORE!
  62. 62. WE CAN DEAL WITH IT!
  63. 63. BUT....WAT!
  64. 64. NUMBERS! ~ 7000 VMs ~ 25 pools ~ 300 hosts
  65. 65. XEN IMPORT AND EXPORT not so well documented
  66. 66. USING HTTP! def export(vm_uuid, options = {}) options = {:to => "/tmp/export_file"}.merge(options) file = File.open(options[:to], "wb") session_ref = self.key task_ref = self.task.create "export vm #{vm_uuid}", "export job" path = "/export?session_id=#{session_ref} ... " uri = URI.parse "http://#{master_address}#{path}" Net::HTTP.get_response(uri) do |res| res.read_body {|chunk| file.write chunk } end options[:to] ensure file.close rescue nil self.task.destroy(task_ref) rescue nil end
  67. 67. MULTI STORAGE or...datasets!
  68. 68. MULTI FIREWALL avoiding package overflow
  69. 69. SO BASICALLY...WE HAVE THIS:
  70. 70. TIME TO BREAK RESPONSABILITIES by improving the infrastructure
  71. 71. NEW NETWORK INFRASTRUCTURE dedicated servers for firewall and dhcp
  72. 72. BEFORE
  73. 73. NOW
  74. 74. QUANTUM from the openstack project
  75. 75. QUANTUM + OPENVSWITCH to useLocaweb quantum plugin openvswitch
  76. 76. NOT SSH ANYMORE Queueing and consuming using RabbitMQ
  77. 77. NEW FIREWALL AND DHCP ARCHITECTURE
  78. 78. NEW HYPERVISORS TO SUPPORT KVM, VMware
  79. 79. SIMPLESTACK! https://github.com/locaweb/simplestack
  80. 80. A PROXY FOR HYPERVISORS
  81. 81. SIMPLESTACK INSTEAD OF XENAPI DIRECTLY Using # Old xenapi code session.VM.get_by_uuid(virtual_machine.uuid) # Simplestack pool.on_simplestack.guests.find(virtual_machine.uuid) ruby simplestack client
  82. 82. FEATURES WORTH MENTIONING
  83. 83. AUTOSCALE AND ALERTS driven by monitoring
  84. 84. LEELA https://github.com/locaweb/leela
  85. 85. THE FLOW
  86. 86. JUST MESSAGING
  87. 87. BLATHER AND DAEMON TOOLS require 'blather/client' DaemonKit::Application.running! message :chat?, :body do |m| begin if Weatherman::Message.should_process?(m) message = JSON.parse(m.body, :allow_nan => true)["results"] unless message["event"].nil? Weatherman::Message.process(message["event"]) end end rescue => e DaemonKit.logger.error "Invalid message #{m.body}" DaemonKit.logger.error e.backtrace end end
  88. 88. HETEROGENEOUS ENVIRONMENT new features and support for all versions!
  89. 89. XEN POOL KNOWS ITS NETWORK VERSION # -*- coding: UTF-8 -*- module Network module Manager extend self def for(pool) pool.network_version.constantize.new end end end
  90. 90. DIFFERENT BEHAVIORS FROM VERSION 1 # -*- coding: UTF-8 -*- module Network class Ver1 def internal_vlan? true end def installation_ip(vm) vm.public_ip.address end end end
  91. 91. TO NETWORK VERSION 2 # -*- coding: UTF-8 -*- module Network class Ver2 def internal_vlan? false end def installation_ip(vm) vm.on_simplestack.ip end end end
  92. 92. MULTIPLE IMPLEMENTATIONS FOR: network configurations firewall implementations post install methods
  93. 93. LESSONS LEARNED Start small and grow fast OOP helps a lot to keep the code organized Multiple projects Multiple languages
  94. 94. THANKS! BY WILLIAN MOLINARI (POTHIX)
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×