1. Inside Sqale’s Backend
YAPC::Asia Tokyo 2012
Gosuke Miyashita
paperboy&co., Inc.

2. Technical Manager
at
paperboy&co.

3. cpan:mizzy
github.com/mizzy
mizzy.org
@gosukenator

4. Inside Sqale’s Backend

5. http://www.facebook.com/sqalejp

6. WARNING
There are no topics
about Perl in this talk

7. What is Sqale?

9. Cloud Application
Platform like Heroku

11. Architecture Overview

12. SFTP
Git over SSH HTTP/HTTPS
AWS SSH
Web Proxy
SSH Router
to Containers
File Deploy
Containers
Repositories Servers

13. Containers

14. SFTP
Git over SSH HTTP/HTTPS
AWS SSH
Web Proxy
SSH Router
to Containers
File Deploy
Containers
Repositories Servers

15. Virtual Environments
Assigned To Users

16. Similar to Dynos of
Heroku

17. Containers made by
LXC (Linux Containers)

18. EC2 Instance (1 Virtual Machine)
Container Container Container Container Container
for for for for for
user A user A user B user B user B
Container Container Container Container Container
for for for for for
user C user D user D user E user E
Container Container Container Container Container
for for for for for
user E user F user F user F user F

19. Nginx
Unicorn
sshd
supervisrod
on each container

20. Amazon Linux
+
Patched kernel(3.2.16)

21. grsecurity kernel patch
for various restrictions

22. original kernel patches
to restrict tcp port
bind and fork bomb

25. Anti fork bomb patch
makes some changes to
cgroup and fork process

26. See
paperboy-sqale/sqale-patches
on GitHub

27. Web Proxy

28. SFTP
Git over SSH HTTP/HTTPS
AWS SSH
Web Proxy
SSH Router
to Containers
File Deploy
Containers
Repositories Servers

29. HTTP/HTTPS
ELB
nginx nginx
Container Container Container Container Container Container
for for for for for for
user A user B user B user C user C user C

30. nginx
lua-nginx-module
redis2-nginx-module

31. http://www.i4pc.jp/
Which containers?
Redis nginx
host001:8083, host001:8084
or
host001
nginx port 8081 nginx port 8082 nginx port 8083 nginx port 8084
Container Container Container Container
for for for for
lokka-mizzy lokka-mizzy i4pc-mizzy i4pc-mizzy

32. nginx.conf (excerpt)
location / {
set $container "";
set $next_containers "";
error_page 502 = @failover;
rewrite_by_lua_file dynamic-proxy.lua;
proxy_pass http://$container;
}

33. dynamic-proxy.lua (excerpt)
local reply = ngx.location.capture("/redis")
if reply.status ~= ngx.HTTP_OK then
ngx.exit(503)
end
local containers, type =
parser.parse_reply(reply.body)

34. dynamic-proxy.lua (excerpt)
while #containers > 0 do
tmp = table.remove(
containers,
math.random(#containers))
if ngx.shared.downed_containers:get(tmp) then
ngx.log(ngx.DEBUG, tmp .. " is down")
else
container = tmp
break
end
end

35. dynamic-proxy.lua (excerpt)
ngx.var.container = container
ngx.var.next_containers
= luabins.save(containers)

36. nginx.conf (again)
location / {
set $container "";
set $next_containers "";
error_page 502 = @failover;
rewrite_by_lua_file dynamic-proxy.lua;
proxy_pass http://$container;
}

37. nginx.conf (excerpt)
location @failover {
error_page 502 = @failover;
rewrite_by_lua_file failover.lua;
proxy_pass http://$container;
}

38. failover.lua (excerpt)
local downed_container = ngx.var.container
if downed_container then
ngx.shared.downed_containers:set(
downed_container,
1,
sqale.NEGATIVE_CACHE_SECONDS
)
end

39. failover.lua (excerpt)
while #containers > 0 do
tmp = table.remove(
containers,
math.random(#containers))
if ngx.shared.downed_containers:get(tmp) then
ngx.log(ngx.DEBUG, tmp .. " is down")
else
container = tmp
break
end
end

40. failover.lua (excerpt)
if not container then
ngx.exit(503)
end
ngx.var.container = container
ngx.var.next_containers
= luabins.save(containers)

41. nginx.conf (agin)
location @failover {
error_page 502 = @failover;
rewrite_by_lua_file failover.lua;
proxy_pass http://$container;
}

42. See
http://bit.ly/UHbHIb
by @hiboma

43. SSH Router

44. SFTP
Git over SSH HTTP/HTTPS
AWS SSH
Web Proxy
SSH Router
to Containers
File Deploy
Containers
Repositories Servers

45. Git SFTP SSH Login
SSH Router
File File
Repositories Repositories Containers
(Git Server) (File Server)

46. How implement this
routing?

47. OpenSSH with script
authentication patch

49. See
mizzy/openssh-script-auth
on GitHub

50. Change routes by
SSH_ORIGNAL_COMMAND

52. In case of
SSH_ORIGINAL_COMMAND
is “git-*”

53. git push
(ssh sqale@gateway.sqale.jp git-recieve-pack
‘/mizzy/lokka.git’)
Run AuthorizedKeys
Script
SSH Router MySQL
Verify the public key
and get the user’s git
server
command=“ssh sqale@git001.sqale.lan
git-recieve-pack
File ‘/var/repos/mizzy/lokka.git’”
Repository
(Git Server)

54. In case of
SSH_ORIGINAL_COMMAND
is “sftp-server”

55. sftp sqale@gateway.sqale.jp
(ssh sqale@gateway.sqale.jp sftp-server)
Run AuthorizedKeys
Script
SSH Router MySQL
Verify the public key
and get the user’s file
server
command=“ssh sqale@file001.sqale.lan
sftp-server”
File git push File
Repository Repository
(File Server) (Git Server)

56. In case of
SSH_ORIGINAL_COMMAND
is empty

57. ssh sqale@gateway.sqale.jp
Run AuthorizedKeys
Script
SSH Router MySQL
Verify the public key
and get the user’s
cotainers list
Display the user’s containers list and
wait the user’s selection
command=“ssh sqale@
Container users001.sqale.lan -p 8081”

60. Deploy Servers

61. SFTP
Git over SSH HTTP/HTTPS
AWS SSH
Web Proxy
SSH Router
to Containers
File Deploy
Containers
Repositories Servers

62. Please ask to
@kyanny

63. Other

64. About Sqale’s Server
Build Automation
http://bit.ly/NBbj9F
by @lamanotrama

65. Thanks