Inside Sqale's Backend at RubyConf Taiwan 2012

  • 2,871 views
Uploaded on

 

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
2,871
On Slideshare
0
From Embeds
0
Number of Embeds
3

Actions

Shares
Downloads
10
Comments
0
Likes
7

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Inside Sqale’s Backend RubyConf Taiwan 2012 Gosuke Miyashita paperboy&co., Inc.,Japan
  • 2. cpan:mizzygithub.com/mizzy mizzy.org @gosukenator
  • 3. Kentaro KuribayashiLiving on the Edge Rails
  • 4. Kensuke NagaeIntroducing nonopaste-cli
  • 5. Shinya Tsunematsu Building production serverenvironment of ruby in modern way
  • 6. Hiroshi SHIBATAHow to discover the Rubys defects with web application
  • 7. Gosuke MiyashitaInside Sqale’s Backend
  • 8. We are frompaperboy&co., Japan
  • 9. What is Sqale?
  • 10. Cloud ApplicationPlatform like Heroku
  • 11. Architecture Overview
  • 12. SFTP Git over SSH HTTP/HTTPSAWS SSH Web Proxy SSH Router to Containers File Deploy Containers Repositories Servers
  • 13. Containers
  • 14. SFTP Git over SSH HTTP/HTTPSAWS SSH Web Proxy SSH Router to Containers File Deploy Containers Repositories Servers
  • 15. Virtual Environments Assigned To Users
  • 16. Similar to Dynos of Heroku
  • 17. Containers made byLXC (Linux Containers)
  • 18. EC2 Instance (1 Virtual Machine)Container Container Container Container Container for for for for for user A user A user B user B user BContainer Container Container Container Container for for for for for user C user D user D user E user EContainer Container Container Container Container for for for for for user E user F user F user F user F
  • 19. Nginx Unicorn sshd supervisrodon each container
  • 20. Amazon Linux +Patched kernel(3.2.16)
  • 21. grsecurity kernel patchfor various restrictions
  • 22. original kernel patches to restrict tcp port bind and fork bomb
  • 23. Anti fork bomb patchmakes some changes tocgroup and fork process
  • 24. Seepaperboy-sqale/sqale-patches on GitHub
  • 25. Web Proxy
  • 26. SFTP Git over SSH HTTP/HTTPSAWS SSH Web Proxy SSH Router to Containers File Deploy Containers Repositories Servers
  • 27. HTTP/HTTPS ELB nginx nginxContainer Container Container Container Container Container for for for for for for user A user B user B user C user C user C
  • 28. nginx lua-nginx-moduleredis2-nginx-module
  • 29. http://www.i4pc.jp/ Which containers? Redis nginx host001:8083, host001:8084 orhost001 nginx port 8081 nginx port 8082 nginx port 8083 nginx port 8084 Container Container Container Container for for for for lokka-mizzy lokka-mizzy i4pc-mizzy i4pc-mizzy
  • 30. nginx.conf (excerpt)location / { set $container ""; set $next_containers ""; error_page 502 = @failover; rewrite_by_lua_file dynamic-proxy.lua; proxy_pass http://$container;}
  • 31. dynamic-proxy.lua (excerpt)local reply = ngx.location.capture("/redis")if reply.status ~= ngx.HTTP_OK then ngx.exit(503)endlocal containers, type = parser.parse_reply(reply.body)
  • 32. dynamic-proxy.lua (excerpt)while #containers > 0 do tmp = table.remove( containers, math.random(#containers)) if ngx.shared.downed_containers:get(tmp) then ngx.log(ngx.DEBUG, tmp .. " is down") else container = tmp break endend
  • 33. dynamic-proxy.lua (excerpt)ngx.var.container = containerngx.var.next_containers = luabins.save(containers)
  • 34. nginx.conf (again)location / { set $container ""; set $next_containers ""; error_page 502 = @failover; rewrite_by_lua_file dynamic-proxy.lua; proxy_pass http://$container;}
  • 35. nginx.conf (excerpt)location @failover { error_page 502 = @failover; rewrite_by_lua_file failover.lua; proxy_pass http://$container;}
  • 36. failover.lua (excerpt)local downed_container = ngx.var.containerif downed_container then ngx.shared.downed_containers:set( downed_container, 1, sqale.NEGATIVE_CACHE_SECONDS )end
  • 37. failover.lua (excerpt)while #containers > 0 do tmp = table.remove( containers, math.random(#containers)) if ngx.shared.downed_containers:get(tmp) then ngx.log(ngx.DEBUG, tmp .. " is down") else container = tmp break endend
  • 38. failover.lua (excerpt)if not container then ngx.exit(503)endngx.var.container = containerngx.var.next_containers = luabins.save(containers)
  • 39. nginx.conf (agin)location @failover { error_page 502 = @failover; rewrite_by_lua_file failover.lua; proxy_pass http://$container;}
  • 40. SSH Router
  • 41. SFTP Git over SSH HTTP/HTTPSAWS SSH Web Proxy SSH Router to Containers File Deploy Containers Repositories Servers
  • 42. Git SFTP SSH Login SSH Router File FileRepositories Repositories Containers(Git Server) (File Server)
  • 43. How implement this routing?
  • 44. OpenSSH with scriptauthentication patch
  • 45. Seemizzy/openssh-script-auth on GitHub
  • 46. Change routes bySSH_ORIGNAL_COMMAND
  • 47. In case ofSSH_ORIGINAL_COMMAND is “git-*”
  • 48. git push (ssh sqale@gateway.sqale.jp git-recieve-pack ‘/mizzy/lokka.git’) Run AuthorizedKeys ScriptSSH Router MySQL Verify the public key and get the user’s git server command=“ssh sqale@git001.sqale.lan git-recieve-pack File ‘/var/repos/mizzy/lokka.git’”Repository(Git Server)
  • 49. In case ofSSH_ORIGINAL_COMMAND is “sftp-server”
  • 50. sftp sqale@gateway.sqale.jp (ssh sqale@gateway.sqale.jp sftp-server) Run AuthorizedKeys ScriptSSH Router MySQL Verify the public key and get the user’s file server command=“ssh sqale@file001.sqale.lan sftp-server” File git push File Repository Repository(File Server) (Git Server)
  • 51. In case ofSSH_ORIGINAL_COMMAND is empty
  • 52. ssh sqale@gateway.sqale.jp Run AuthorizedKeys ScriptSSH Router MySQL Verify the public key and get the user’s cotainers list Display the user’s containers list and wait the user’s selection command=“ssh sqale@Container users001.sqale.lan -p 8081”
  • 53. Ruby On Rails Puppet/Chef Unicorn ResqueAnd many gems
  • 54. 多謝