Inside Sqale's Backend at Sapporo Ruby Kaigi 2012

  • 8,921 views
Uploaded on

 

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
8,921
On Slideshare
0
From Embeds
0
Number of Embeds
9

Actions

Shares
Downloads
39
Comments
0
Likes
22

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Inside Sqale’s Backend Sapporo Ruby Kaigi 2012 Gosuke Miyashita paperboy&co. Inc.
  • 2. A little bit about me
  • 3. Technical Manager at paperboy&co.
  • 4. https://github.com/mizzy http://mizzy.org/ @gosukenator
  • 5. Inside Sqale’s Backend
  • 6. http://www.facebook.com/sqalejp
  • 7. WARNINGThere are little topicsabout Ruby in this talk
  • 8. What is Sqale?
  • 9. Cloud ApplicationPlatform like Heroku
  • 10. Architecture Overview
  • 11. SFTP Git over SSH HTTP/HTTPSAWS SSH Web Proxy SSH Router to Containers File Deploy Containers Repositories Servers
  • 12. Containers
  • 13. SFTP Git over SSH HTTP/HTTPSAWS SSH Web Proxy SSH Router to Containers File Deploy Containers Repositories Servers
  • 14. Virtual Environments Assigned To Users
  • 15. Similar to Dynos of Heroku
  • 16. Containers made byLXC (Linux Containers)
  • 17. EC2 Instance (1 Virtual Machine)Container Container Container Container Container for for for for for user A user A user B user B user BContainer Container Container Container Container for for for for for user C user D user D user E user EContainer Container Container Container Container for for for for for user E user F user F user F user F
  • 18. Nginx Unicorn sshd supervisrodon each container
  • 19. Amazon Linux +Patched kernel(3.2.16)
  • 20. grsecurity kernel patchfor various restrictions
  • 21. original kernel patches to restrict tcp port bind and fork bomb
  • 22. Anti fork bomb patchmakes some changes tocgroup and fork process
  • 23. Seepaperboy-sqale/sqale-patches on GitHub
  • 24. Web Proxy
  • 25. SFTP Git over SSH HTTP/HTTPSAWS SSH Web Proxy SSH Router to Containers File Deploy Containers Repositories Servers
  • 26. HTTP/HTTPS ELB nginx nginxContainer Container Container Container Container Container for for for for for for user A user B user B user C user C user C
  • 27. nginx lua-nginx-moduleredis2-nginx-module
  • 28. http://lokka-mizzy.sqale.jp/ Which containers? Redis nginx host001:8083, host001:8084 orhost001 nginx port 8081 nginx port 8082 nginx port 8083 nginx port 8084 Container Container Container Container for for for for i4pc-mizzy i4pc-mizzy lokka-mizzy lokka-mizzy
  • 29. nginx.conf (excerpt)location / { set $container ""; set $next_containers ""; error_page 502 = @failover; rewrite_by_lua_file dynamic-proxy.lua; proxy_pass http://$container;}
  • 30. dynamic-proxy.lua (excerpt)local reply = ngx.location.capture("/redis")if reply.status ~= ngx.HTTP_OK then ngx.exit(503)endlocal containers, type = parser.parse_reply(reply.body)
  • 31. dynamic-proxy.lua (excerpt)while #containers > 0 do tmp = table.remove( containers, math.random(#containers)) if ngx.shared.downed_containers:get(tmp) then ngx.log(ngx.DEBUG, tmp .. " is down") else container = tmp break endend
  • 32. dynamic-proxy.lua (excerpt)ngx.var.container = containerngx.var.next_containers = luabins.save(containers)
  • 33. nginx.conf (again)location / { set $container ""; set $next_containers ""; error_page 502 = @failover; rewrite_by_lua_file dynamic-proxy.lua; proxy_pass http://$container;}
  • 34. nginx.conf (excerpt)location @failover { error_page 502 = @failover; rewrite_by_lua_file failover.lua; proxy_pass http://$container;}
  • 35. failover.lua (excerpt)local downed_container = ngx.var.containerif downed_container then ngx.shared.downed_containers:set( downed_container, 1, sqale.NEGATIVE_CACHE_SECONDS )end
  • 36. failover.lua (excerpt)while #containers > 0 do tmp = table.remove( containers, math.random(#containers)) if ngx.shared.downed_containers:get(tmp) then ngx.log(ngx.DEBUG, tmp .. " is down") else container = tmp break endend
  • 37. failover.lua (excerpt)if not container then ngx.exit(503)endngx.var.container = containerngx.var.next_containers = luabins.save(containers)
  • 38. nginx.conf (agin)location @failover { error_page 502 = @failover; rewrite_by_lua_file failover.lua; proxy_pass http://$container;}
  • 39. Seehttp://bit.ly/UHbHIb by @hiboma
  • 40. SSH Router
  • 41. SFTP Git over SSH HTTP/HTTPSAWS SSH Web Proxy SSH Router to Containers File Deploy Containers Repositories Servers
  • 42. Git SFTP SSH Login SSH Router File FileRepositories Repositories Containers(Git Server) (File Server)
  • 43. How implement this routing?
  • 44. OpenSSH with scriptauthentication patch
  • 45. Seemizzy/openssh-script-auth on GitHub
  • 46. Change routes bySSH_ORIGNAL_COMMAND
  • 47. In case ofSSH_ORIGINAL_COMMAND is “git-*”
  • 48. git push (ssh sqale@gateway.sqale.jp git-recieve-pack ‘/mizzy/lokka.git’) Run AuthorizedKeys ScriptSSH Router MySQL Verify the public key and get the user’s git server command=“ssh sqale@git001.sqale.lan git-recieve-pack File ‘/var/repos/mizzy/lokka.git’”Repository(Git Server)
  • 49. In case ofSSH_ORIGINAL_COMMAND is “sftp-server”
  • 50. sftp sqale@gateway.sqale.jp (ssh sqale@gateway.sqale.jp sftp-server) Run AuthorizedKeys ScriptSSH Router MySQL Verify the public key and get the user’s file server command=“ssh sqale@file001.sqale.lan sftp-server” File git push File Repository Repository(File Server) (Git Server)
  • 51. In case ofSSH_ORIGINAL_COMMAND is empty
  • 52. ssh sqale@gateway.sqale.jp Run AuthorizedKeys ScriptSSH Router MySQL Verify the public key and get the user’s cotainers list Display the user’s containers list and wait the user’s selection command=“ssh sqale@Container users001.sqale.lan -p 8081”
  • 53. Deploy Servers
  • 54. SFTP Git over SSH HTTP/HTTPSAWS SSH Web Proxy SSH Router to Containers File Deploy Containers Repositories Servers
  • 55. Please ask to @kyanny
  • 56. Other
  • 57. About Sqale’s Server Build Automationhttp://bit.ly/NBbj9F by @lamanotrama
  • 58. Thanks