Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Implementing a Fileserver with Nginx and Lua

637 views

Published on

Using the power of Nginx it is easy to implement quite complex logic of file upload with metadata and authorization support, and without need of any heavy application server. In this article you can find the basic implementation of such Fileserver using Nginx and Lua only.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Implementing a Fileserver with Nginx and Lua

  1. 1. Implementing a Fileserver with Nginx and Lua techtalk@ferret Andrii Gakhov 27.02.2017
  2. 2. Current Architecture Restful API server that communicates with clients using application/json Every client is authenticated by a dedicated 3rd party authentication server and authorized by the API server End users get benefits from the API via client-side frontend application
  3. 3. The Problem At some point, end users request the ability to manage files using the same client-side application and, consequently, same API. allow to process multipart/form-data requests (that will be proxied from the form on the client-side application) extract and handle file metadata provide file storage and access
  4. 4. The Solution Obviously, such functionality is out of the scope for the API and the natural decision is to split it across 2 applications: API is used for meta file management Fileserver takes care about actual files upload/download Instead of using Flask/Django etc., we have decided to implements such functionality in Nginx with Lua scripting
  5. 5. Nginx and Lua Nginx is a free, open-source, high-performance HTTP server and reverse proxy, as well as an IMAP/POP3 proxy server. Lua is a powerful, efficient, lightweight, embeddable scripting language. It supports procedural programming, object-oriented programming, functional programming, data-driven programming, and data description. The Lua module for Nginx ngx_http_lua_module embeds Lua into Nginx and by leveraging Nginx's subrequests, allows the integration of Lua threads into the Nginx event model.
  6. 6. Nginx and Lua Nginx object is available in Lua as ngx and server variables are accessible as ngx.var.{name}, requested GET arguments - as ngx.var.arg_{name}, but unfortunately POST variables Nginx doesn't parse into variables automatically. The various *_by_lua_* configuration directives serve as gateways to the Lua API within the nginx.conf. We use *_by_lua_block to embed Lua code into Nginx as the most notable way to do so.
  7. 7. Nginx and Lua header_filter_by_lua_block - uses Lua code to define an output header filter (e.g. for overriding or adding a response header). access_by_lua_block - acts as an access phase handler and executes Lua code for every request. as with other access phase handlers, access_by_lua will not run in subrequests. content_by_lua_block - acts as a "content handler" and executes Lua code for every request.
  8. 8. Architecture / Upload location = /_upload { limit_except POST { deny all; } if ($http_content_type !~ "multipart/form-data") { return 406; } lua_need_request_body off; client_body_buffer_size 200M; client_max_body_size 200M; default_type 'application/json'; ... }
  9. 9. Architecture / Download location ~ ^/_download/(.*)$ { set $path /$1; default_type 'application/json'; limit_except GET { deny all; } ... }
  10. 10. User Authentication function authenticate_user(access_token) local params = { method = ngx.HTTP_GET, args = { access_token = access_token } } local res = ngx.location.capture( "/_auth/access/check", params) if not res or res.status ~= 200 then return nil end return cjson.decode(res.body) end access_by_lua_block { local cjson = require("cjson") local access_token = ngx.var.arg_access_token if not access_token then return_http_forbidden("Forbidden") end -- authenticate user local credentials = authenticate_user(access_token) -- if user can't be resolved, return 403 Forbidden if not credentials or not credentials.data.user.id then return_http_forbidden("Forbidden") end }
  11. 11. Upload -- Extract POST variables in a streaming way -- and store file object in a temporary -- file (form_data.file.filepath) -- All.other form variables are in form_data -- (we expect them to be small) local helpers = require("utils") form_data, err = helpers.streaming_multipart_form() if form_data == nil then return_http_internal_server_error(err) end data = { status = "not_ready", title = form.title.value, filename = form.file.filename, } local params = { method = ngx.HTTP_POST, args = {access_token = access_token}, body = cjson.encode(data) } local res = ngx.location.capture("/_api/v1/file", params) if not res then return_http_bad_gateway("No metadata") end local create_metadata_resp = res.body if res and res.status ~= 201 then ngx.status = res.status ngx.print(create_metadata_resp) ngx.exit(res.status) end local file_metadata = cjson.decode(create_metadata_resp) Important part to make it work is to specify client_body_buffer_size equal to client_max_body_size. Otherwise, if the client body is bigger than client_body_buffer_size, the nginx variable $request_body will be empty.
  12. 12. Upload local filex = require("pl.file") local file_fullpath = "/storage/files/" .. file_metadata.hash .. file_metadata.path -- ensure that subdirectories exist (if not, create) -- store tmp file to its permanent position is_moved, err = filex.move( form.file.fullpath, file_fullpath) -- make it available for download if is_moved then local params = { method = ngx.HTTP_PUT, args = {access_token = access_token}, body = cjson.encode({status = "ready"}) } ngx.location.capture( "/_api/v1/file/" .. file_metadata.id, params) end -- provide some headers with metadata ngx.header["X-File-ID"] = file_metadata.id ngx.status = ngx.HTTP_CREATED ngx.print(create_metadata_resp) ngx.exit(ngx.HTTP_CREATED) Since we need to manage access to the file, we use access policy of files’ metadata instead of the files themselves. We create metadata in any case (“register the file”), but allow to download it only if file has status “ready”, meaning it was successfully created at the specified location.
  13. 13. Download local search_by_path = { filters = { path = ngx.var.path, status = "ready" }, size = 1 } local params = { method = ngx.HTTP_POST, args = {access_token = access_token}, body = cjson.encode(search_by_path) } local res = ngx.location.capture( "/_api/v1/search/files", params) if not res then return_http_bad_gateway("Search error") end local found = cjson.decode(res.body) if found.total < 1 then return_http_not_found("File Not Found") end local file_metadata = found.results[1] ngx.header["X-File-Name"] = file_metadata.name ngx.header["X-File-Path"] = file_metadata.path ngx.header["X-File-ID"] = file_metadata.id ngx.req.set_uri( "/" .. file_metadata.hash .. file_metadata.path) ngx.exec("@download_file", download_params) As soon as the file has been found, its metadata provides us with all necessary information about the location and we are ready to respond it to the user. Sometimes people recommend to read such file in Lua and return it to the user with ngx.print that is a bad idea for big files (Lua virtual machine will just crash).
  14. 14. Download location @download_file { internal; root /storage/files/; try_files $uri =404; header_filter_by_lua_block { ngx.header["Cache-Control"] = "no-cache" ngx.header["Content-Disposition"] = "attachment; filename="" .. ngx.header["X-File-Name"] .. """ } } The @download_file location is quite simple, but additionally we want to play with response headers to provide a real filename for download (on our filesystem all files are stored with unique generated names). It is an internal named location (to prevent unauthorized access) that just serves requested static files from the desired directory.
  15. 15. How to Use Upload curl -XPOST https://files.example.com/_upload?access_token={SOME_TOKEN} --form file=@/tmp/file.pdf --form title="Example title" -H "Content-Type: multipart/form-data" Download curl -XGET https://files.example.com/_download/723533/2338342189083057604.pdf?access_token={SOME_TOKEN}
  16. 16. Thank you Read it in the web: https://www.datacrucis.com/research/implementing-api-based- fileserver-with-nginx-and-lua.html

×