Using ngx_lua / lua-nginx-module in pixiv

  • 4,070 views
Uploaded on

ドリコム、リブセンスさんとの合同勉強会で話したスライドです。

ドリコム、リブセンスさんとの合同勉強会で話したスライドです。

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
4,070
On Slideshare
0
From Embeds
0
Number of Embeds
10

Actions

Shares
Downloads
39
Comments
0
Likes
19

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. / 30 Using ngx_lua in pixiv Harukasan a.k.a MICHII Shunsuke 2013-06-24
  • 2. / 30 はるかさん インフラ部所属 2年目 Nginx / Apache / MySQL / MongoDB Kyototycoon / Solr / Traffic Server / Fluentd Capistrano / Ruby on Rails / Macbook Air / Vim 2
  • 3. / 303 • about pixiv Image Cluster • ngx_lua • directives in nginx.conf • nginx api in lua scripts • ex. code from pixiv Image Cluster Agenda
  • 4. / 304 pixiv Image Cluster
  • 5. / 305  Front Cache Cluster Image Store  SSD-RAID0     HDD SSD Thumbnailbig size Memory DEVICEpeak: 1k req/sec 10Gbps x20 x20
  • 6. / 306  Front Cache Cluster Image Store PRODUCT Nginx with tmpfs Traffic Server Apache / NginxNginx
  • 7. / 307 ngx_lua github.com/chaoslawful/lua-nginx-module
  • 8. / 308 • embeds Lua/LuaJIT into Nginx • has 100% non-blocking API • has many libraries provided by OpenResty • memcached, mysql, redis, etc. ngx_lua
  • 9. / 309 # load from /etc/nginx/scripts/?.lua and default root lua_package_path "/etc/nginx/scripts/?.lua;;"; server { listen 80; # GET /fact?a=4 -> 24 (4! = 4 * 3 * 2 * 1) location /fact { content_by_lua " local val = 1 for i = 2, ngx.var.arg_a do val *= i; end "; } # same code in file location /fact_script { content_by_lua_file "fact.lua"; } }
  • 10. / 3010 Directives in nginx.conf • *_by_lua, *_by_lua_file • init_by_lua • rewrite_by_lua • access_by_lua • content_by_lua 設定読み込み rewrite処理 access処理 bodyの生成処理
  • 11. / 3011 init_by_lua <code> masterが設定をロードするタイミングで実行する init_by_lua " memcached = require 'resty.memcached' "; server { ! listen 8888; ! default_type 'text/plain'; ! location /init_by { ! ! content_by_lua " ! ! local hello = memcached::get('Hello'); ... ! ! "; ! } }
  • 12. / 3012 server { listen 80; # GET /sum?a=2&b=3 -> 5 location /sum { set_by_lua $sum " return tonumber(ngx.var.arg_a) + tonumber(ngx.var.arg_b) "; echo $sum; } } set_by_lua $res <code> 実行したコードの返値を$resにセットする
  • 13. / 3013 server rewrite find config rewrite post rewrite pre-access access post-access try files content logging Request rewrite/access/content_by_lua サーバーコンテキストのrewrite 該当するlocation configを探す locationのrewrite処理 ファイルへのアクセス try_fileの処理 bodyの生成 ログ書き込み access_by_lua content_by_lua rewrite_by_lua rewriteに関連する処理 アクセス制御 bodyの操作
  • 14. / 3014 server { listen 8888; location /rewrite_by { rewrite_by_lua "ngx.var.toggle = true"; if ($toggle) { rewrite / /index_second.html; } } } rewrite_by_lua <code> rewriteの前に指定したコードを実行する
  • 15. / 3015 server { listen 80; # GET /access_by?status=404 => 404 Not Found location /access_by { access_by_lua "ngx.exit(ngx.var.arg_status)"; } } access_by_lua <code> access phaseに指定したコードを実行する
  • 16. / 3016 # GET /hello location /hello { content_by_lua " ngx.say('Hello, from lua') "; } content_by_lua <code> responseを生成するコードを実行する
  • 17. / 3017 Directives in nginx.conf • LuaスクリプトでNginxの各動作を制御できる • HTTPで必要な各フェーズでフックできる • rewrite • access control • content generation ここまで8分
  • 18. / 3018 Nginx API in lua scripts • generate content • variables and shared tables • access memcached / MySQL
  • 19. / 3019 location /var { content_by_lua " ngx.say('Hello, from lua!') "; } ngx.say content bodyに文字列を出力する location /var { access_by_lua " ngx.exit(418) "; # i'm a teapot } ngx.exit 指定したステータスコードを返す
  • 20. / 3020 location /var { set $a "alpha"; rewrite_by_lua "ngx.var.b = ngx.var.a .. 'beta'"; echo $b; } ngx.var.* ngx.ctx.* location /ctx { access_by_lua "ngx.ctx.foo = 74"; rewrite_by_lua "ngx.ctx.foo = ngx.ctx.foo + 5"; content_by_lua "ngx.say('foo: ' .. ngx.ctx.foo)"; } Nginxの変数にアクセスする リクエスト中に共有されるテーブル
  • 21. / 3021 ngx.shared.* shared memoryテーブル http { lua_shared_dict foo 10m; server { location /shared-set { content_by_lua " local foo = ngx.shared.foo foo:set(ngx.var.arg_key, ngx.var.arg_val) "; } location /shared-get { content_by_lua " local foo = ngx.shared.foo ngx.say(foo:get(ngx.var.arg_key)) "; } } }
  • 22. / 3022 Regular Expressions • ngx.re.match • ngx.re.gmatch • ngx.re.sub • ngx.re.gsub
  • 23. / 3023 local matches, err = ngx.re.match("key=val", "([^=]+)=(.+)") ngx.re.match ngx.re.sub matches[0] => "key=val" matches[1] => "key" matches[2] => "val" local newstr,n,err = ngx.re.sub("Hello,lua","lua", "nginx") newstr = "Hello,nginx"
  • 24. / 3024 • OpenResty • lua-resty-memcached github.com/agentzh/lua-resty-memcached • lua-resty-mysql github.com/agentzh/lua-resty-mysql • ngx.socketを使って実装されている Access to memcached/MySQL
  • 25. / 3025 Nginx API in lua scripts • 簡単な処理を書くには十分な関数が 提供されている • 正規表現 • ハッシュ(md5, sha1) • ソケット • OpenRestyを使うとmemcachedへのアクセス とかも簡単
  • 26. / 3026 Example code from pixiv Image Cluster
  • 27. / 3027 Kyototycoon Logical Delete Nginx 200 404 /img01.png 200 /img02.png 200 /img03.png 404 /img04.png 200   GET /img01.png GET /img03.png 404
  • 28. / 3028 Logical Delete local memcached = require "resty.memcached" local uri = ngx.var.request_uri local memc = memcached:new() local val, flags, err = memc:get(request_uri) if val and val ~= "200" then exit(tonumber(val)) end logical_delete.lua location / { access_by_lua_file logical_delete.lua; }
  • 29. / 3029 Conclusion • Nginxだけで簡単な処理を実現できる • アクセス制御とかインプレッション計測 とかには便利そう • 高速で省メモリ • メンテナンスコストを抑えられる
  • 30. / 3030 References HTTPLuaModule - Nginx Wiki http://wiki.nginx.org/HttpLuaModule chaoslawful/lua-nginx-module - Github https://github.com/chaoslawful/lua-nginx-module