Your SlideShare is downloading. ×
Inside Sqale's Backend at YAPC::Asia Tokyo 2012
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Inside Sqale's Backend at YAPC::Asia Tokyo 2012

3,531
views

Published on

Published in: Technology

0 Comments
16 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,531
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
18
Comments
0
Likes
16
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Inside Sqale’s Backend YAPC::Asia Tokyo 2012 Gosuke Miyashita paperboy&co., Inc.
  • 2. Technical Manager at paperboy&co.
  • 3. cpan:mizzygithub.com/mizzy mizzy.org @gosukenator
  • 4. Inside Sqale’s Backend
  • 5. http://www.facebook.com/sqalejp
  • 6. WARNING There are no topicsabout Perl in this talk
  • 7. What is Sqale?
  • 8. Cloud ApplicationPlatform like Heroku
  • 9. Architecture Overview
  • 10. SFTP Git over SSH HTTP/HTTPSAWS SSH Web Proxy SSH Router to Containers File Deploy Containers Repositories Servers
  • 11. Containers
  • 12. SFTP Git over SSH HTTP/HTTPSAWS SSH Web Proxy SSH Router to Containers File Deploy Containers Repositories Servers
  • 13. Virtual Environments Assigned To Users
  • 14. Similar to Dynos of Heroku
  • 15. Containers made byLXC (Linux Containers)
  • 16. EC2 Instance (1 Virtual Machine)Container Container Container Container Container for for for for for user A user A user B user B user BContainer Container Container Container Container for for for for for user C user D user D user E user EContainer Container Container Container Container for for for for for user E user F user F user F user F
  • 17. Nginx Unicorn sshd supervisrodon each container
  • 18. Amazon Linux +Patched kernel(3.2.16)
  • 19. grsecurity kernel patchfor various restrictions
  • 20. original kernel patches to restrict tcp port bind and fork bomb
  • 21. Anti fork bomb patchmakes some changes tocgroup and fork process
  • 22. Seepaperboy-sqale/sqale-patches on GitHub
  • 23. Web Proxy
  • 24. SFTP Git over SSH HTTP/HTTPSAWS SSH Web Proxy SSH Router to Containers File Deploy Containers Repositories Servers
  • 25. HTTP/HTTPS ELB nginx nginxContainer Container Container Container Container Container for for for for for for user A user B user B user C user C user C
  • 26. nginx lua-nginx-moduleredis2-nginx-module
  • 27. http://www.i4pc.jp/ Which containers? Redis nginx host001:8083, host001:8084 orhost001 nginx port 8081 nginx port 8082 nginx port 8083 nginx port 8084 Container Container Container Container for for for for lokka-mizzy lokka-mizzy i4pc-mizzy i4pc-mizzy
  • 28. nginx.conf (excerpt)location / { set $container ""; set $next_containers ""; error_page 502 = @failover; rewrite_by_lua_file dynamic-proxy.lua; proxy_pass http://$container;}
  • 29. dynamic-proxy.lua (excerpt)local reply = ngx.location.capture("/redis")if reply.status ~= ngx.HTTP_OK then ngx.exit(503)endlocal containers, type = parser.parse_reply(reply.body)
  • 30. dynamic-proxy.lua (excerpt)while #containers > 0 do tmp = table.remove( containers, math.random(#containers)) if ngx.shared.downed_containers:get(tmp) then ngx.log(ngx.DEBUG, tmp .. " is down") else container = tmp break endend
  • 31. dynamic-proxy.lua (excerpt)ngx.var.container = containerngx.var.next_containers = luabins.save(containers)
  • 32. nginx.conf (again)location / { set $container ""; set $next_containers ""; error_page 502 = @failover; rewrite_by_lua_file dynamic-proxy.lua; proxy_pass http://$container;}
  • 33. nginx.conf (excerpt)location @failover { error_page 502 = @failover; rewrite_by_lua_file failover.lua; proxy_pass http://$container;}
  • 34. failover.lua (excerpt)local downed_container = ngx.var.containerif downed_container then ngx.shared.downed_containers:set( downed_container, 1, sqale.NEGATIVE_CACHE_SECONDS )end
  • 35. failover.lua (excerpt)while #containers > 0 do tmp = table.remove( containers, math.random(#containers)) if ngx.shared.downed_containers:get(tmp) then ngx.log(ngx.DEBUG, tmp .. " is down") else container = tmp break endend
  • 36. failover.lua (excerpt)if not container then ngx.exit(503)endngx.var.container = containerngx.var.next_containers = luabins.save(containers)
  • 37. nginx.conf (agin)location @failover { error_page 502 = @failover; rewrite_by_lua_file failover.lua; proxy_pass http://$container;}
  • 38. Seehttp://bit.ly/UHbHIb by @hiboma
  • 39. SSH Router
  • 40. SFTP Git over SSH HTTP/HTTPSAWS SSH Web Proxy SSH Router to Containers File Deploy Containers Repositories Servers
  • 41. Git SFTP SSH Login SSH Router File FileRepositories Repositories Containers(Git Server) (File Server)
  • 42. How implement this routing?
  • 43. OpenSSH with scriptauthentication patch
  • 44. Seemizzy/openssh-script-auth on GitHub
  • 45. Change routes bySSH_ORIGNAL_COMMAND
  • 46. In case ofSSH_ORIGINAL_COMMAND is “git-*”
  • 47. git push (ssh sqale@gateway.sqale.jp git-recieve-pack ‘/mizzy/lokka.git’) Run AuthorizedKeys ScriptSSH Router MySQL Verify the public key and get the user’s git server command=“ssh sqale@git001.sqale.lan git-recieve-pack File ‘/var/repos/mizzy/lokka.git’”Repository(Git Server)
  • 48. In case ofSSH_ORIGINAL_COMMAND is “sftp-server”
  • 49. sftp sqale@gateway.sqale.jp (ssh sqale@gateway.sqale.jp sftp-server) Run AuthorizedKeys ScriptSSH Router MySQL Verify the public key and get the user’s file server command=“ssh sqale@file001.sqale.lan sftp-server” File git push File Repository Repository(File Server) (Git Server)
  • 50. In case ofSSH_ORIGINAL_COMMAND is empty
  • 51. ssh sqale@gateway.sqale.jp Run AuthorizedKeys ScriptSSH Router MySQL Verify the public key and get the user’s cotainers list Display the user’s containers list and wait the user’s selection command=“ssh sqale@Container users001.sqale.lan -p 8081”
  • 52. Deploy Servers
  • 53. SFTP Git over SSH HTTP/HTTPSAWS SSH Web Proxy SSH Router to Containers File Deploy Containers Repositories Servers
  • 54. Please ask to @kyanny
  • 55. Other
  • 56. About Sqale’s Server Build Automationhttp://bit.ly/NBbj9F by @lamanotrama
  • 57. Thanks