Inside Sqale's Backend at YAPC::Asia Tokyo 2012

4,196 views
4,098 views

Published on

Published in: Technology
0 Comments
16 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
4,196
On SlideShare
0
From Embeds
0
Number of Embeds
1,212
Actions
Shares
0
Downloads
20
Comments
0
Likes
16
Embeds 0
No embeds

No notes for slide

Inside Sqale's Backend at YAPC::Asia Tokyo 2012

  1. 1. Inside Sqale’s Backend YAPC::Asia Tokyo 2012 Gosuke Miyashita paperboy&co., Inc.
  2. 2. Technical Manager at paperboy&co.
  3. 3. cpan:mizzygithub.com/mizzy mizzy.org @gosukenator
  4. 4. Inside Sqale’s Backend
  5. 5. http://www.facebook.com/sqalejp
  6. 6. WARNING There are no topicsabout Perl in this talk
  7. 7. What is Sqale?
  8. 8. Cloud ApplicationPlatform like Heroku
  9. 9. Architecture Overview
  10. 10. SFTP Git over SSH HTTP/HTTPSAWS SSH Web Proxy SSH Router to Containers File Deploy Containers Repositories Servers
  11. 11. Containers
  12. 12. SFTP Git over SSH HTTP/HTTPSAWS SSH Web Proxy SSH Router to Containers File Deploy Containers Repositories Servers
  13. 13. Virtual Environments Assigned To Users
  14. 14. Similar to Dynos of Heroku
  15. 15. Containers made byLXC (Linux Containers)
  16. 16. EC2 Instance (1 Virtual Machine)Container Container Container Container Container for for for for for user A user A user B user B user BContainer Container Container Container Container for for for for for user C user D user D user E user EContainer Container Container Container Container for for for for for user E user F user F user F user F
  17. 17. Nginx Unicorn sshd supervisrodon each container
  18. 18. Amazon Linux +Patched kernel(3.2.16)
  19. 19. grsecurity kernel patchfor various restrictions
  20. 20. original kernel patches to restrict tcp port bind and fork bomb
  21. 21. Anti fork bomb patchmakes some changes tocgroup and fork process
  22. 22. Seepaperboy-sqale/sqale-patches on GitHub
  23. 23. Web Proxy
  24. 24. SFTP Git over SSH HTTP/HTTPSAWS SSH Web Proxy SSH Router to Containers File Deploy Containers Repositories Servers
  25. 25. HTTP/HTTPS ELB nginx nginxContainer Container Container Container Container Container for for for for for for user A user B user B user C user C user C
  26. 26. nginx lua-nginx-moduleredis2-nginx-module
  27. 27. http://www.i4pc.jp/ Which containers? Redis nginx host001:8083, host001:8084 orhost001 nginx port 8081 nginx port 8082 nginx port 8083 nginx port 8084 Container Container Container Container for for for for lokka-mizzy lokka-mizzy i4pc-mizzy i4pc-mizzy
  28. 28. nginx.conf (excerpt)location / { set $container ""; set $next_containers ""; error_page 502 = @failover; rewrite_by_lua_file dynamic-proxy.lua; proxy_pass http://$container;}
  29. 29. dynamic-proxy.lua (excerpt)local reply = ngx.location.capture("/redis")if reply.status ~= ngx.HTTP_OK then ngx.exit(503)endlocal containers, type = parser.parse_reply(reply.body)
  30. 30. dynamic-proxy.lua (excerpt)while #containers > 0 do tmp = table.remove( containers, math.random(#containers)) if ngx.shared.downed_containers:get(tmp) then ngx.log(ngx.DEBUG, tmp .. " is down") else container = tmp break endend
  31. 31. dynamic-proxy.lua (excerpt)ngx.var.container = containerngx.var.next_containers = luabins.save(containers)
  32. 32. nginx.conf (again)location / { set $container ""; set $next_containers ""; error_page 502 = @failover; rewrite_by_lua_file dynamic-proxy.lua; proxy_pass http://$container;}
  33. 33. nginx.conf (excerpt)location @failover { error_page 502 = @failover; rewrite_by_lua_file failover.lua; proxy_pass http://$container;}
  34. 34. failover.lua (excerpt)local downed_container = ngx.var.containerif downed_container then ngx.shared.downed_containers:set( downed_container, 1, sqale.NEGATIVE_CACHE_SECONDS )end
  35. 35. failover.lua (excerpt)while #containers > 0 do tmp = table.remove( containers, math.random(#containers)) if ngx.shared.downed_containers:get(tmp) then ngx.log(ngx.DEBUG, tmp .. " is down") else container = tmp break endend
  36. 36. failover.lua (excerpt)if not container then ngx.exit(503)endngx.var.container = containerngx.var.next_containers = luabins.save(containers)
  37. 37. nginx.conf (agin)location @failover { error_page 502 = @failover; rewrite_by_lua_file failover.lua; proxy_pass http://$container;}
  38. 38. Seehttp://bit.ly/UHbHIb by @hiboma
  39. 39. SSH Router
  40. 40. SFTP Git over SSH HTTP/HTTPSAWS SSH Web Proxy SSH Router to Containers File Deploy Containers Repositories Servers
  41. 41. Git SFTP SSH Login SSH Router File FileRepositories Repositories Containers(Git Server) (File Server)
  42. 42. How implement this routing?
  43. 43. OpenSSH with scriptauthentication patch
  44. 44. Seemizzy/openssh-script-auth on GitHub
  45. 45. Change routes bySSH_ORIGNAL_COMMAND
  46. 46. In case ofSSH_ORIGINAL_COMMAND is “git-*”
  47. 47. git push (ssh sqale@gateway.sqale.jp git-recieve-pack ‘/mizzy/lokka.git’) Run AuthorizedKeys ScriptSSH Router MySQL Verify the public key and get the user’s git server command=“ssh sqale@git001.sqale.lan git-recieve-pack File ‘/var/repos/mizzy/lokka.git’”Repository(Git Server)
  48. 48. In case ofSSH_ORIGINAL_COMMAND is “sftp-server”
  49. 49. sftp sqale@gateway.sqale.jp (ssh sqale@gateway.sqale.jp sftp-server) Run AuthorizedKeys ScriptSSH Router MySQL Verify the public key and get the user’s file server command=“ssh sqale@file001.sqale.lan sftp-server” File git push File Repository Repository(File Server) (Git Server)
  50. 50. In case ofSSH_ORIGINAL_COMMAND is empty
  51. 51. ssh sqale@gateway.sqale.jp Run AuthorizedKeys ScriptSSH Router MySQL Verify the public key and get the user’s cotainers list Display the user’s containers list and wait the user’s selection command=“ssh sqale@Container users001.sqale.lan -p 8081”
  52. 52. Deploy Servers
  53. 53. SFTP Git over SSH HTTP/HTTPSAWS SSH Web Proxy SSH Router to Containers File Deploy Containers Repositories Servers
  54. 54. Please ask to @kyanny
  55. 55. Other
  56. 56. About Sqale’s Server Build Automationhttp://bit.ly/NBbj9F by @lamanotrama
  57. 57. Thanks

×