0
/ 30
Using ngx_lua in pixiv
Harukasan
a.k.a MICHII Shunsuke
2013-06-24
/ 30
はるかさん
インフラ部所属 2年目
Nginx / Apache / MySQL / MongoDB
Kyototycoon / Solr / Traffic Server / Fluentd
Capistrano / Ruby on...
/ 303
• about pixiv Image Cluster
• ngx_lua
• directives in nginx.conf
• nginx api in lua scripts
• ex. code from pixiv Im...
/ 304
pixiv Image Cluster
/ 305

Front
Cache Cluster
Image Store

SSD-RAID0
   
HDD SSD
Thumbnailbig size
Memory
DEVICEpeak: 1k req/sec
10Gbp...
/ 306

Front
Cache Cluster
Image Store
PRODUCT
Nginx with tmpfs
Traffic Server
Apache / NginxNginx
/ 307
ngx_lua
github.com/chaoslawful/lua-nginx-module
/ 308
• embeds Lua/LuaJIT into Nginx
• has 100% non-blocking API
• has many libraries provided by OpenResty
• memcached, m...
/ 309
# load from /etc/nginx/scripts/?.lua and default root
lua_package_path "/etc/nginx/scripts/?.lua;;";
server {
listen...
/ 3010
Directives in nginx.conf
• *_by_lua, *_by_lua_file
• init_by_lua
• rewrite_by_lua
• access_by_lua
• content_by_lua
設...
/ 3011
init_by_lua <code>
masterが設定をロードするタイミングで実行する
init_by_lua " memcached = require 'resty.memcached' ";
server {
! list...
/ 3012
server {
listen 80;
# GET /sum?a=2&b=3 -> 5
location /sum {
set_by_lua $sum "
return tonumber(ngx.var.arg_a) + tonu...
/ 3013
server rewrite
find config
rewrite
post rewrite
pre-access
access
post-access
try files
content
logging
Request
rewrit...
/ 3014
server {
listen 8888;
location /rewrite_by {
rewrite_by_lua "ngx.var.toggle = true";
if ($toggle) {
rewrite / /inde...
/ 3015
server {
listen 80;
# GET /access_by?status=404 => 404 Not Found
location /access_by {
access_by_lua "ngx.exit(ngx....
/ 3016
# GET /hello
location /hello {
content_by_lua "
ngx.say('Hello, from lua')
";
}
content_by_lua <code>
responseを生成する...
/ 3017
Directives in nginx.conf
• LuaスクリプトでNginxの各動作を制御できる
• HTTPで必要な各フェーズでフックできる
• rewrite
• access control
• content gen...
/ 3018
Nginx API in lua scripts
• generate content
• variables and shared tables
• access memcached / MySQL
/ 3019
location /var {
content_by_lua " ngx.say('Hello, from lua!') ";
}
ngx.say content bodyに文字列を出力する
location /var {
acc...
/ 3020
location /var {
set $a "alpha";
rewrite_by_lua "ngx.var.b = ngx.var.a .. 'beta'";
echo $b;
}
ngx.var.*
ngx.ctx.*
lo...
/ 3021
ngx.shared.* shared memoryテーブル
http {
lua_shared_dict foo 10m;
server {
location /shared-set {
content_by_lua "
loc...
/ 3022
Regular Expressions
• ngx.re.match
• ngx.re.gmatch
• ngx.re.sub
• ngx.re.gsub
/ 3023
local matches, err = ngx.re.match("key=val", "([^=]+)=(.+)")
ngx.re.match
ngx.re.sub
matches[0] => "key=val"
matche...
/ 3024
• OpenResty
• lua-resty-memcached
github.com/agentzh/lua-resty-memcached
• lua-resty-mysql
github.com/agentzh/lua-r...
/ 3025
Nginx API in lua scripts
• 簡単な処理を書くには十分な関数が
提供されている
• 正規表現
• ハッシュ(md5, sha1)
• ソケット
• OpenRestyを使うとmemcachedへのアクセス
...
/ 3026
Example code from
pixiv Image Cluster
/ 3027
Kyototycoon
Logical Delete
Nginx
200
404
/img01.png 200
/img02.png 200
/img03.png 404
/img04.png 200


GET /img0...
/ 3028
Logical Delete
local memcached = require "resty.memcached"
local uri = ngx.var.request_uri
local memc = memcached:n...
/ 3029
Conclusion
• Nginxだけで簡単な処理を実現できる
• アクセス制御とかインプレッション計測
とかには便利そう
• 高速で省メモリ
• メンテナンスコストを抑えられる
/ 3030
References
HTTPLuaModule - Nginx Wiki
http://wiki.nginx.org/HttpLuaModule
chaoslawful/lua-nginx-module - Github
htt...
Upcoming SlideShare
Loading in...5
×

Using ngx_lua / lua-nginx-module in pixiv

5,437

Published on

ドリコム、リブセンスさんとの合同勉強会で話したスライドです。

Published in: Technology
0 Comments
30 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
5,437
On Slideshare
0
From Embeds
0
Number of Embeds
11
Actions
Shares
0
Downloads
51
Comments
0
Likes
30
Embeds 0
No embeds

No notes for slide

Transcript of "Using ngx_lua / lua-nginx-module in pixiv"

  1. 1. / 30 Using ngx_lua in pixiv Harukasan a.k.a MICHII Shunsuke 2013-06-24
  2. 2. / 30 はるかさん インフラ部所属 2年目 Nginx / Apache / MySQL / MongoDB Kyototycoon / Solr / Traffic Server / Fluentd Capistrano / Ruby on Rails / Macbook Air / Vim 2
  3. 3. / 303 • about pixiv Image Cluster • ngx_lua • directives in nginx.conf • nginx api in lua scripts • ex. code from pixiv Image Cluster Agenda
  4. 4. / 304 pixiv Image Cluster
  5. 5. / 305  Front Cache Cluster Image Store  SSD-RAID0     HDD SSD Thumbnailbig size Memory DEVICEpeak: 1k req/sec 10Gbps x20 x20
  6. 6. / 306  Front Cache Cluster Image Store PRODUCT Nginx with tmpfs Traffic Server Apache / NginxNginx
  7. 7. / 307 ngx_lua github.com/chaoslawful/lua-nginx-module
  8. 8. / 308 • embeds Lua/LuaJIT into Nginx • has 100% non-blocking API • has many libraries provided by OpenResty • memcached, mysql, redis, etc. ngx_lua
  9. 9. / 309 # load from /etc/nginx/scripts/?.lua and default root lua_package_path "/etc/nginx/scripts/?.lua;;"; server { listen 80; # GET /fact?a=4 -> 24 (4! = 4 * 3 * 2 * 1) location /fact { content_by_lua " local val = 1 for i = 2, ngx.var.arg_a do val *= i; end "; } # same code in file location /fact_script { content_by_lua_file "fact.lua"; } }
  10. 10. / 3010 Directives in nginx.conf • *_by_lua, *_by_lua_file • init_by_lua • rewrite_by_lua • access_by_lua • content_by_lua 設定読み込み rewrite処理 access処理 bodyの生成処理
  11. 11. / 3011 init_by_lua <code> masterが設定をロードするタイミングで実行する init_by_lua " memcached = require 'resty.memcached' "; server { ! listen 8888; ! default_type 'text/plain'; ! location /init_by { ! ! content_by_lua " ! ! local hello = memcached::get('Hello'); ... ! ! "; ! } }
  12. 12. / 3012 server { listen 80; # GET /sum?a=2&b=3 -> 5 location /sum { set_by_lua $sum " return tonumber(ngx.var.arg_a) + tonumber(ngx.var.arg_b) "; echo $sum; } } set_by_lua $res <code> 実行したコードの返値を$resにセットする
  13. 13. / 3013 server rewrite find config rewrite post rewrite pre-access access post-access try files content logging Request rewrite/access/content_by_lua サーバーコンテキストのrewrite 該当するlocation configを探す locationのrewrite処理 ファイルへのアクセス try_fileの処理 bodyの生成 ログ書き込み access_by_lua content_by_lua rewrite_by_lua rewriteに関連する処理 アクセス制御 bodyの操作
  14. 14. / 3014 server { listen 8888; location /rewrite_by { rewrite_by_lua "ngx.var.toggle = true"; if ($toggle) { rewrite / /index_second.html; } } } rewrite_by_lua <code> rewriteの前に指定したコードを実行する
  15. 15. / 3015 server { listen 80; # GET /access_by?status=404 => 404 Not Found location /access_by { access_by_lua "ngx.exit(ngx.var.arg_status)"; } } access_by_lua <code> access phaseに指定したコードを実行する
  16. 16. / 3016 # GET /hello location /hello { content_by_lua " ngx.say('Hello, from lua') "; } content_by_lua <code> responseを生成するコードを実行する
  17. 17. / 3017 Directives in nginx.conf • LuaスクリプトでNginxの各動作を制御できる • HTTPで必要な各フェーズでフックできる • rewrite • access control • content generation ここまで8分
  18. 18. / 3018 Nginx API in lua scripts • generate content • variables and shared tables • access memcached / MySQL
  19. 19. / 3019 location /var { content_by_lua " ngx.say('Hello, from lua!') "; } ngx.say content bodyに文字列を出力する location /var { access_by_lua " ngx.exit(418) "; # i'm a teapot } ngx.exit 指定したステータスコードを返す
  20. 20. / 3020 location /var { set $a "alpha"; rewrite_by_lua "ngx.var.b = ngx.var.a .. 'beta'"; echo $b; } ngx.var.* ngx.ctx.* location /ctx { access_by_lua "ngx.ctx.foo = 74"; rewrite_by_lua "ngx.ctx.foo = ngx.ctx.foo + 5"; content_by_lua "ngx.say('foo: ' .. ngx.ctx.foo)"; } Nginxの変数にアクセスする リクエスト中に共有されるテーブル
  21. 21. / 3021 ngx.shared.* shared memoryテーブル http { lua_shared_dict foo 10m; server { location /shared-set { content_by_lua " local foo = ngx.shared.foo foo:set(ngx.var.arg_key, ngx.var.arg_val) "; } location /shared-get { content_by_lua " local foo = ngx.shared.foo ngx.say(foo:get(ngx.var.arg_key)) "; } } }
  22. 22. / 3022 Regular Expressions • ngx.re.match • ngx.re.gmatch • ngx.re.sub • ngx.re.gsub
  23. 23. / 3023 local matches, err = ngx.re.match("key=val", "([^=]+)=(.+)") ngx.re.match ngx.re.sub matches[0] => "key=val" matches[1] => "key" matches[2] => "val" local newstr,n,err = ngx.re.sub("Hello,lua","lua", "nginx") newstr = "Hello,nginx"
  24. 24. / 3024 • OpenResty • lua-resty-memcached github.com/agentzh/lua-resty-memcached • lua-resty-mysql github.com/agentzh/lua-resty-mysql • ngx.socketを使って実装されている Access to memcached/MySQL
  25. 25. / 3025 Nginx API in lua scripts • 簡単な処理を書くには十分な関数が 提供されている • 正規表現 • ハッシュ(md5, sha1) • ソケット • OpenRestyを使うとmemcachedへのアクセス とかも簡単
  26. 26. / 3026 Example code from pixiv Image Cluster
  27. 27. / 3027 Kyototycoon Logical Delete Nginx 200 404 /img01.png 200 /img02.png 200 /img03.png 404 /img04.png 200   GET /img01.png GET /img03.png 404
  28. 28. / 3028 Logical Delete local memcached = require "resty.memcached" local uri = ngx.var.request_uri local memc = memcached:new() local val, flags, err = memc:get(request_uri) if val and val ~= "200" then exit(tonumber(val)) end logical_delete.lua location / { access_by_lua_file logical_delete.lua; }
  29. 29. / 3029 Conclusion • Nginxだけで簡単な処理を実現できる • アクセス制御とかインプレッション計測 とかには便利そう • 高速で省メモリ • メンテナンスコストを抑えられる
  30. 30. / 3030 References HTTPLuaModule - Nginx Wiki http://wiki.nginx.org/HttpLuaModule chaoslawful/lua-nginx-module - Github https://github.com/chaoslawful/lua-nginx-module
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×