Inside Sqale's Backend at YAPC::Asia Tokyo 2012
Upcoming SlideShare
Loading in...5
×
 

Inside Sqale's Backend at YAPC::Asia Tokyo 2012

on

  • 3,879 views

 

Statistics

Views

Total Views
3,879
Views on SlideShare
2,746
Embed Views
1,133

Actions

Likes
16
Downloads
18
Comments
0

10 Embeds 1,133

http://mizzy.org 693
http://yapcasia.org 268
http://localhost 88
https://twitter.com 25
http://reader.freerss.net 25
http://freerss.net 18
http://www.freerss.net 10
https://si0.twimg.com 3
https://twimg0-a.akamaihd.net 2
http://tweetedtimes.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Inside Sqale's Backend at YAPC::Asia Tokyo 2012 Inside Sqale's Backend at YAPC::Asia Tokyo 2012 Presentation Transcript

  • Inside Sqale’s Backend YAPC::Asia Tokyo 2012 Gosuke Miyashita paperboy&co., Inc.
  • Technical Manager at paperboy&co.
  • cpan:mizzygithub.com/mizzy mizzy.org @gosukenator
  • Inside Sqale’s Backend
  • http://www.facebook.com/sqalejp
  • WARNING There are no topicsabout Perl in this talk
  • What is Sqale?
  • Cloud ApplicationPlatform like Heroku
  • Architecture Overview
  • SFTP Git over SSH HTTP/HTTPSAWS SSH Web Proxy SSH Router to Containers File Deploy Containers Repositories Servers
  • Containers
  • SFTP Git over SSH HTTP/HTTPSAWS SSH Web Proxy SSH Router to Containers File Deploy Containers Repositories Servers
  • Virtual Environments Assigned To Users
  • Similar to Dynos of Heroku
  • Containers made byLXC (Linux Containers)
  • EC2 Instance (1 Virtual Machine)Container Container Container Container Container for for for for for user A user A user B user B user BContainer Container Container Container Container for for for for for user C user D user D user E user EContainer Container Container Container Container for for for for for user E user F user F user F user F
  • Nginx Unicorn sshd supervisrodon each container
  • Amazon Linux +Patched kernel(3.2.16)
  • grsecurity kernel patchfor various restrictions
  • original kernel patches to restrict tcp port bind and fork bomb
  • Anti fork bomb patchmakes some changes tocgroup and fork process
  • Seepaperboy-sqale/sqale-patches on GitHub
  • Web Proxy
  • SFTP Git over SSH HTTP/HTTPSAWS SSH Web Proxy SSH Router to Containers File Deploy Containers Repositories Servers
  • HTTP/HTTPS ELB nginx nginxContainer Container Container Container Container Container for for for for for for user A user B user B user C user C user C
  • nginx lua-nginx-moduleredis2-nginx-module
  • http://www.i4pc.jp/ Which containers? Redis nginx host001:8083, host001:8084 orhost001 nginx port 8081 nginx port 8082 nginx port 8083 nginx port 8084 Container Container Container Container for for for for lokka-mizzy lokka-mizzy i4pc-mizzy i4pc-mizzy
  • nginx.conf (excerpt)location / { set $container ""; set $next_containers ""; error_page 502 = @failover; rewrite_by_lua_file dynamic-proxy.lua; proxy_pass http://$container;}
  • dynamic-proxy.lua (excerpt)local reply = ngx.location.capture("/redis")if reply.status ~= ngx.HTTP_OK then ngx.exit(503)endlocal containers, type = parser.parse_reply(reply.body)
  • dynamic-proxy.lua (excerpt)while #containers > 0 do tmp = table.remove( containers, math.random(#containers)) if ngx.shared.downed_containers:get(tmp) then ngx.log(ngx.DEBUG, tmp .. " is down") else container = tmp break endend
  • dynamic-proxy.lua (excerpt)ngx.var.container = containerngx.var.next_containers = luabins.save(containers)
  • nginx.conf (again)location / { set $container ""; set $next_containers ""; error_page 502 = @failover; rewrite_by_lua_file dynamic-proxy.lua; proxy_pass http://$container;}
  • nginx.conf (excerpt)location @failover { error_page 502 = @failover; rewrite_by_lua_file failover.lua; proxy_pass http://$container;}
  • failover.lua (excerpt)local downed_container = ngx.var.containerif downed_container then ngx.shared.downed_containers:set( downed_container, 1, sqale.NEGATIVE_CACHE_SECONDS )end
  • failover.lua (excerpt)while #containers > 0 do tmp = table.remove( containers, math.random(#containers)) if ngx.shared.downed_containers:get(tmp) then ngx.log(ngx.DEBUG, tmp .. " is down") else container = tmp break endend
  • failover.lua (excerpt)if not container then ngx.exit(503)endngx.var.container = containerngx.var.next_containers = luabins.save(containers)
  • nginx.conf (agin)location @failover { error_page 502 = @failover; rewrite_by_lua_file failover.lua; proxy_pass http://$container;}
  • Seehttp://bit.ly/UHbHIb by @hiboma
  • SSH Router
  • SFTP Git over SSH HTTP/HTTPSAWS SSH Web Proxy SSH Router to Containers File Deploy Containers Repositories Servers
  • Git SFTP SSH Login SSH Router File FileRepositories Repositories Containers(Git Server) (File Server)
  • How implement this routing?
  • OpenSSH with scriptauthentication patch
  • Seemizzy/openssh-script-auth on GitHub
  • Change routes bySSH_ORIGNAL_COMMAND
  • In case ofSSH_ORIGINAL_COMMAND is “git-*”
  • git push (ssh sqale@gateway.sqale.jp git-recieve-pack ‘/mizzy/lokka.git’) Run AuthorizedKeys ScriptSSH Router MySQL Verify the public key and get the user’s git server command=“ssh sqale@git001.sqale.lan git-recieve-pack File ‘/var/repos/mizzy/lokka.git’”Repository(Git Server)
  • In case ofSSH_ORIGINAL_COMMAND is “sftp-server”
  • sftp sqale@gateway.sqale.jp (ssh sqale@gateway.sqale.jp sftp-server) Run AuthorizedKeys ScriptSSH Router MySQL Verify the public key and get the user’s file server command=“ssh sqale@file001.sqale.lan sftp-server” File git push File Repository Repository(File Server) (Git Server)
  • In case ofSSH_ORIGINAL_COMMAND is empty
  • ssh sqale@gateway.sqale.jp Run AuthorizedKeys ScriptSSH Router MySQL Verify the public key and get the user’s cotainers list Display the user’s containers list and wait the user’s selection command=“ssh sqale@Container users001.sqale.lan -p 8081”
  • Deploy Servers
  • SFTP Git over SSH HTTP/HTTPSAWS SSH Web Proxy SSH Router to Containers File Deploy Containers Repositories Servers
  • Please ask to @kyanny
  • Other
  • About Sqale’s Server Build Automationhttp://bit.ly/NBbj9F by @lamanotrama
  • Thanks