Successfully reported this slideshow.

More Related Content

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Inside Sqale's Backend at Sapporo Ruby Kaigi 2012

  1. Inside Sqale’s Backend Sapporo Ruby Kaigi 2012 Gosuke Miyashita paperboy&co. Inc.
  2. A little bit about me
  3. Technical Manager at paperboy&co.
  4. https://github.com/mizzy http://mizzy.org/ @gosukenator
  5. Inside Sqale’s Backend
  6. http://www.facebook.com/sqalejp
  7. WARNING There are little topics about Ruby in this talk
  8. What is Sqale?
  9. Cloud Application Platform like Heroku
  10. Architecture Overview
  11. SFTP Git over SSH HTTP/HTTPS AWS SSH Web Proxy SSH Router to Containers File Deploy Containers Repositories Servers
  12. Containers
  13. SFTP Git over SSH HTTP/HTTPS AWS SSH Web Proxy SSH Router to Containers File Deploy Containers Repositories Servers
  14. Virtual Environments Assigned To Users
  15. Similar to Dynos of Heroku
  16. Containers made by LXC (Linux Containers)
  17. EC2 Instance (1 Virtual Machine) Container Container Container Container Container for for for for for user A user A user B user B user B Container Container Container Container Container for for for for for user C user D user D user E user E Container Container Container Container Container for for for for for user E user F user F user F user F
  18. Nginx Unicorn sshd supervisrod on each container
  19. Amazon Linux + Patched kernel(3.2.16)
  20. grsecurity kernel patch for various restrictions
  21. original kernel patches to restrict tcp port bind and fork bomb
  22. Anti fork bomb patch makes some changes to cgroup and fork process
  23. See paperboy-sqale/sqale-patches on GitHub
  24. Web Proxy
  25. SFTP Git over SSH HTTP/HTTPS AWS SSH Web Proxy SSH Router to Containers File Deploy Containers Repositories Servers
  26. HTTP/HTTPS ELB nginx nginx Container Container Container Container Container Container for for for for for for user A user B user B user C user C user C
  27. nginx lua-nginx-module redis2-nginx-module
  28. http://lokka-mizzy.sqale.jp/ Which containers? Redis nginx host001:8083, host001:8084 or host001 nginx port 8081 nginx port 8082 nginx port 8083 nginx port 8084 Container Container Container Container for for for for i4pc-mizzy i4pc-mizzy lokka-mizzy lokka-mizzy
  29. nginx.conf (excerpt) location / { set $container ""; set $next_containers ""; error_page 502 = @failover; rewrite_by_lua_file dynamic-proxy.lua; proxy_pass http://$container; }
  30. dynamic-proxy.lua (excerpt) local reply = ngx.location.capture("/redis") if reply.status ~= ngx.HTTP_OK then ngx.exit(503) end local containers, type = parser.parse_reply(reply.body)
  31. dynamic-proxy.lua (excerpt) while #containers > 0 do tmp = table.remove( containers, math.random(#containers)) if ngx.shared.downed_containers:get(tmp) then ngx.log(ngx.DEBUG, tmp .. " is down") else container = tmp break end end
  32. dynamic-proxy.lua (excerpt) ngx.var.container = container ngx.var.next_containers = luabins.save(containers)
  33. nginx.conf (again) location / { set $container ""; set $next_containers ""; error_page 502 = @failover; rewrite_by_lua_file dynamic-proxy.lua; proxy_pass http://$container; }
  34. nginx.conf (excerpt) location @failover { error_page 502 = @failover; rewrite_by_lua_file failover.lua; proxy_pass http://$container; }
  35. failover.lua (excerpt) local downed_container = ngx.var.container if downed_container then ngx.shared.downed_containers:set( downed_container, 1, sqale.NEGATIVE_CACHE_SECONDS ) end
  36. failover.lua (excerpt) while #containers > 0 do tmp = table.remove( containers, math.random(#containers)) if ngx.shared.downed_containers:get(tmp) then ngx.log(ngx.DEBUG, tmp .. " is down") else container = tmp break end end
  37. failover.lua (excerpt) if not container then ngx.exit(503) end ngx.var.container = container ngx.var.next_containers = luabins.save(containers)
  38. nginx.conf (agin) location @failover { error_page 502 = @failover; rewrite_by_lua_file failover.lua; proxy_pass http://$container; }
  39. See http://bit.ly/UHbHIb by @hiboma
  40. SSH Router
  41. SFTP Git over SSH HTTP/HTTPS AWS SSH Web Proxy SSH Router to Containers File Deploy Containers Repositories Servers
  42. Git SFTP SSH Login SSH Router File File Repositories Repositories Containers (Git Server) (File Server)
  43. How implement this routing?
  44. OpenSSH with script authentication patch
  45. See mizzy/openssh-script-auth on GitHub
  46. Change routes by SSH_ORIGNAL_COMMAND
  47. In case of SSH_ORIGINAL_COMMAND is “git-*”
  48. git push (ssh sqale@gateway.sqale.jp git-recieve-pack ‘/mizzy/lokka.git’) Run AuthorizedKeys Script SSH Router MySQL Verify the public key and get the user’s git server command=“ssh sqale@git001.sqale.lan git-recieve-pack File ‘/var/repos/mizzy/lokka.git’” Repository (Git Server)
  49. In case of SSH_ORIGINAL_COMMAND is “sftp-server”
  50. sftp sqale@gateway.sqale.jp (ssh sqale@gateway.sqale.jp sftp-server) Run AuthorizedKeys Script SSH Router MySQL Verify the public key and get the user’s file server command=“ssh sqale@file001.sqale.lan sftp-server” File git push File Repository Repository (File Server) (Git Server)
  51. In case of SSH_ORIGINAL_COMMAND is empty
  52. ssh sqale@gateway.sqale.jp Run AuthorizedKeys Script SSH Router MySQL Verify the public key and get the user’s cotainers list Display the user’s containers list and wait the user’s selection command=“ssh sqale@ Container users001.sqale.lan -p 8081”
  53. Deploy Servers
  54. SFTP Git over SSH HTTP/HTTPS AWS SSH Web Proxy SSH Router to Containers File Deploy Containers Repositories Servers
  55. Please ask to @kyanny
  56. Other
  57. About Sqale’s Server Build Automation http://bit.ly/NBbj9F by @lamanotrama
  58. Thanks

×