Inside Sqale's Backend at Sapporo Ruby Kaigi 2012

Inside Sqale’s Backend Sapporo Ruby Kaigi 2012 Gosuke Miyashita paperboy&co. Inc.

A little bit about me

Technical Manager at paperboy&co.

https://github.com/mizzy http://mizzy.org/ @gosukenator

Inside Sqale’s Backend

http://www.facebook.com/sqalejp

WARNINGThere are little topicsabout Ruby in this talk

What is Sqale?

Cloud ApplicationPlatform like Heroku

Architecture Overview

SFTP Git over SSH HTTP/HTTPSAWS SSH Web Proxy SSH Router to Containers File Deploy Containers Repositories Servers

Containers

Virtual Environments Assigned To Users

Similar to Dynos of Heroku

Containers made byLXC (Linux Containers)

EC2 Instance (1 Virtual Machine)Container Container Container Container Container for for for for for user A user A user B user B user BContainer Container Container Container Container for for for for for user C user D user D user E user EContainer Container Container Container Container for for for for for user E user F user F user F user F

Nginx Unicorn sshd supervisrodon each container

Amazon Linux +Patched kernel(3.2.16)

grsecurity kernel patchfor various restrictions

original kernel patches to restrict tcp port bind and fork bomb

Anti fork bomb patchmakes some changes tocgroup and fork process

Seepaperboy-sqale/sqale-patches on GitHub

Web Proxy

HTTP/HTTPS ELB nginx nginxContainer Container Container Container Container Container for for for for for for user A user B user B user C user C user C

nginx lua-nginx-moduleredis2-nginx-module

http://lokka-mizzy.sqale.jp/ Which containers? Redis nginx host001:8083, host001:8084 orhost001 nginx port 8081 nginx port 8082 nginx port 8083 nginx port 8084 Container Container Container Container for for for for i4pc-mizzy i4pc-mizzy lokka-mizzy lokka-mizzy

nginx.conf (excerpt)location / { set $container ""; set $next_containers ""; error_page 502 = @failover; rewrite_by_lua_file dynamic-proxy.lua; proxy_pass http://$container;}

dynamic-proxy.lua (excerpt)local reply = ngx.location.capture("/redis")if reply.status ~= ngx.HTTP_OK then ngx.exit(503)endlocal containers, type = parser.parse_reply(reply.body)

dynamic-proxy.lua (excerpt)while #containers > 0 do tmp = table.remove( containers, math.random(#containers)) if ngx.shared.downed_containers:get(tmp) then ngx.log(ngx.DEBUG, tmp .. " is down") else container = tmp break endend

dynamic-proxy.lua (excerpt)ngx.var.container = containerngx.var.next_containers = luabins.save(containers)

nginx.conf (again)location / { set $container ""; set $next_containers ""; error_page 502 = @failover; rewrite_by_lua_file dynamic-proxy.lua; proxy_pass http://$container;}

nginx.conf (excerpt)location @failover { error_page 502 = @failover; rewrite_by_lua_file failover.lua; proxy_pass http://$container;}

failover.lua (excerpt)local downed_container = ngx.var.containerif downed_container then ngx.shared.downed_containers:set( downed_container, 1, sqale.NEGATIVE_CACHE_SECONDS )end

failover.lua (excerpt)while #containers > 0 do tmp = table.remove( containers, math.random(#containers)) if ngx.shared.downed_containers:get(tmp) then ngx.log(ngx.DEBUG, tmp .. " is down") else container = tmp break endend

failover.lua (excerpt)if not container then ngx.exit(503)endngx.var.container = containerngx.var.next_containers = luabins.save(containers)

nginx.conf (agin)location @failover { error_page 502 = @failover; rewrite_by_lua_file failover.lua; proxy_pass http://$container;}

Seehttp://bit.ly/UHbHIb by @hiboma

SSH Router

Git SFTP SSH Login SSH Router File FileRepositories Repositories Containers(Git Server) (File Server)

How implement this routing?

OpenSSH with scriptauthentication patch

Seemizzy/openssh-script-auth on GitHub

Change routes bySSH_ORIGNAL_COMMAND

In case ofSSH_ORIGINAL_COMMAND is “git-*”

git push (ssh sqale@gateway.sqale.jp git-recieve-pack ‘/mizzy/lokka.git’) Run AuthorizedKeys ScriptSSH Router MySQL Verify the public key and get the user’s git server command=“ssh sqale@git001.sqale.lan git-recieve-pack File ‘/var/repos/mizzy/lokka.git’”Repository(Git Server)

In case ofSSH_ORIGINAL_COMMAND is “sftp-server”

sftp sqale@gateway.sqale.jp (ssh sqale@gateway.sqale.jp sftp-server) Run AuthorizedKeys ScriptSSH Router MySQL Verify the public key and get the user’s file server command=“ssh sqale@file001.sqale.lan sftp-server” File git push File Repository Repository(File Server) (Git Server)

In case ofSSH_ORIGINAL_COMMAND is empty

ssh sqale@gateway.sqale.jp Run AuthorizedKeys ScriptSSH Router MySQL Verify the public key and get the user’s cotainers list Display the user’s containers list and wait the user’s selection command=“ssh sqale@Container users001.sqale.lan -p 8081”

Deploy Servers

Please ask to @kyanny

About Sqale’s Server Build Automationhttp://bit.ly/NBbj9F by @lamanotrama

Thanks

http://mizzy.org	1215
http://sapporo.rubykaigi.org	121
http://localhost	114
http://reader.freerss.net	16
https://twitter.com	14
http://www.freerss.net	6
https://si0.twimg.com	2
http://tweetedtimes.com	1
http://clipboard.com	1
http://freerss.net	1

Inside Sqale's Backend at Sapporo Ruby Kaigi 2012

by Gosuke Miyashita on Sep 14, 2012

Accessibility

Categories

Upload Details

Usage Rights

10 Embeds 1,491

Statistics