Implementing a Fileserver with Nginx and Lua

Andrii Gakhov
Andrii GakhovPh.D., Senior Software Engineer at Ferret-Go
Implementing a Fileserver
with Nginx and Lua
techtalk@ferret
Andrii Gakhov
27.02.2017
Current Architecture
Restful API server that communicates with clients using
application/json
Every client is authenticated by a dedicated 3rd party
authentication server and authorized by the API server
End users get benefits from the API via client-side
frontend application
The Problem
At some point, end users request the ability to manage files
using the same client-side application and, consequently,
same API.
allow to process multipart/form-data requests (that will be
proxied from the form on the client-side application)
extract and handle file metadata
provide file storage and access
The Solution
Obviously, such functionality is out of the scope for the API
and the natural decision is to split it across 2
applications:
API is used for meta file management
Fileserver takes care about actual files upload/download
Instead of using Flask/Django etc., we have decided to
implements such functionality in Nginx with Lua scripting
Nginx and Lua
Nginx is a free, open-source, high-performance HTTP server
and reverse proxy, as well as an IMAP/POP3 proxy server.
Lua is a powerful, efficient, lightweight, embeddable
scripting language. It supports procedural programming,
object-oriented programming, functional programming,
data-driven programming, and data description.
The Lua module for Nginx ngx_http_lua_module embeds Lua into
Nginx and by leveraging Nginx's subrequests, allows the
integration of Lua threads into the Nginx event model.
Nginx and Lua
Nginx object is available in Lua as ngx and server variables
are accessible as ngx.var.{name}, requested GET arguments -
as ngx.var.arg_{name}, but unfortunately POST variables Nginx
doesn't parse into variables automatically.
The various *_by_lua_* configuration directives serve as
gateways to the Lua API within the nginx.conf. We use
*_by_lua_block to embed Lua code into Nginx as the most
notable way to do so.
Nginx and Lua
header_filter_by_lua_block - uses Lua code to define an
output header filter (e.g. for overriding or adding a
response header).
access_by_lua_block - acts as an access phase handler and
executes Lua code for every request. as with other access
phase handlers, access_by_lua will not run in subrequests.
content_by_lua_block - acts as a "content handler" and
executes Lua code for every request.
Architecture / Upload
location = /_upload {
limit_except POST { deny all; }
if ($http_content_type !~ "multipart/form-data") {
return 406;
}
lua_need_request_body off;
client_body_buffer_size 200M;
client_max_body_size 200M;
default_type 'application/json';
...
}
Architecture / Download
location ~ ^/_download/(.*)$ {
set $path /$1;
default_type 'application/json';
limit_except GET { deny all; }
...
}
User Authentication
function authenticate_user(access_token)
local params = {
method = ngx.HTTP_GET,
args = {
access_token = access_token
}
}
local res = ngx.location.capture(
"/_auth/access/check", params)
if not res or res.status ~= 200 then
return nil
end
return cjson.decode(res.body)
end
access_by_lua_block {
local cjson = require("cjson")
local access_token = ngx.var.arg_access_token
if not access_token then
return_http_forbidden("Forbidden")
end
-- authenticate user
local credentials = authenticate_user(access_token)
-- if user can't be resolved, return 403 Forbidden
if not credentials or not credentials.data.user.id
then
return_http_forbidden("Forbidden")
end
}
Upload
-- Extract POST variables in a streaming way
-- and store file object in a temporary
-- file (form_data.file.filepath)
-- All.other form variables are in form_data
-- (we expect them to be small)
local helpers = require("utils")
form_data, err = helpers.streaming_multipart_form()
if form_data == nil then
return_http_internal_server_error(err)
end
data = {
status = "not_ready",
title = form.title.value,
filename = form.file.filename,
}
local params = {
method = ngx.HTTP_POST,
args = {access_token = access_token},
body = cjson.encode(data)
}
local res = ngx.location.capture("/_api/v1/file", params)
if not res then
return_http_bad_gateway("No metadata")
end
local create_metadata_resp = res.body
if res and res.status ~= 201 then
ngx.status = res.status
ngx.print(create_metadata_resp)
ngx.exit(res.status)
end
local file_metadata = cjson.decode(create_metadata_resp)
Important part to make it work is to specify
client_body_buffer_size equal to
client_max_body_size.
Otherwise, if the client body is bigger than
client_body_buffer_size, the nginx variable
$request_body will be empty.
Upload
local filex = require("pl.file")
local file_fullpath = "/storage/files/" ..
file_metadata.hash .. file_metadata.path
-- ensure that subdirectories exist (if not, create)
-- store tmp file to its permanent position
is_moved, err = filex.move(
form.file.fullpath, file_fullpath)
-- make it available for download
if is_moved then
local params = {
method = ngx.HTTP_PUT,
args = {access_token = access_token},
body = cjson.encode({status = "ready"})
}
ngx.location.capture(
"/_api/v1/file/" .. file_metadata.id, params)
end
-- provide some headers with metadata
ngx.header["X-File-ID"] = file_metadata.id
ngx.status = ngx.HTTP_CREATED
ngx.print(create_metadata_resp)
ngx.exit(ngx.HTTP_CREATED)
Since we need to manage access to the file, we
use access policy of files’ metadata instead of the
files themselves.
We create metadata in any case (“register the
file”), but allow to download it only if file has status
“ready”, meaning it was successfully created at the
specified location.
Download
local search_by_path = {
filters = {
path = ngx.var.path,
status = "ready"
},
size = 1
}
local params = {
method = ngx.HTTP_POST,
args = {access_token = access_token},
body = cjson.encode(search_by_path)
}
local res = ngx.location.capture(
"/_api/v1/search/files", params)
if not res then
return_http_bad_gateway("Search error")
end
local found = cjson.decode(res.body)
if found.total < 1 then
return_http_not_found("File Not Found")
end
local file_metadata = found.results[1]
ngx.header["X-File-Name"] = file_metadata.name
ngx.header["X-File-Path"] = file_metadata.path
ngx.header["X-File-ID"] = file_metadata.id
ngx.req.set_uri(
"/" .. file_metadata.hash .. file_metadata.path)
ngx.exec("@download_file", download_params)
As soon as the file has been found, its metadata
provides us with all necessary information about
the location and we are ready to respond it to the
user.
Sometimes people recommend to read such file in
Lua and return it to the user with ngx.print that is a
bad idea for big files (Lua virtual machine will just
crash).
Download
location @download_file {
internal;
root /storage/files/;
try_files $uri =404;
header_filter_by_lua_block {
ngx.header["Cache-Control"] = "no-cache"
ngx.header["Content-Disposition"] = "attachment; filename="" .. ngx.header["X-File-Name"] .. """
}
}
The @download_file location is quite simple, but additionally we want to play with response headers to provide
a real filename for download (on our filesystem all files are stored with unique generated names).
It is an internal named location (to prevent unauthorized access) that just serves requested static files from the
desired directory.
How to Use
Upload
curl -XPOST https://files.example.com/_upload?access_token={SOME_TOKEN} 
--form file=@/tmp/file.pdf
--form title="Example title"
-H "Content-Type: multipart/form-data"
Download
curl -XGET https://files.example.com/_download/723533/2338342189083057604.pdf?access_token={SOME_TOKEN}
Thank you
Read it in the web:
https://www.datacrucis.com/research/implementing-api-based-
fileserver-with-nginx-and-lua.html
1 of 16

Recommended

Play Framework: async I/O with Java and Scala by
Play Framework: async I/O with Java and ScalaPlay Framework: async I/O with Java and Scala
Play Framework: async I/O with Java and ScalaYevgeniy Brikman
108.6K views184 slides
Learn nginx in 90mins by
Learn nginx in 90minsLearn nginx in 90mins
Learn nginx in 90minsLarry Cai
8.5K views20 slides
Multi threading by
Multi threadingMulti threading
Multi threadinggndu
3.3K views14 slides
Servlets by
ServletsServlets
ServletsSasidhar Kothuru
1.5K views32 slides
Java Networking by
Java NetworkingJava Networking
Java NetworkingSunil OS
409.5K views52 slides
Introduction to package in java by
Introduction to package in javaIntroduction to package in java
Introduction to package in javaPrognoz Technologies Pvt. Ltd.
2.4K views21 slides

More Related Content

What's hot

Android Data Persistence by
Android Data PersistenceAndroid Data Persistence
Android Data PersistenceJussi Pohjolainen
9.4K views32 slides
JS Event Loop by
JS Event LoopJS Event Loop
JS Event LoopSaai Vignesh P
399 views46 slides
jQuery by
jQueryjQuery
jQueryVishwa Mohan
3.3K views75 slides
Introduction à la sécurité des WebServices by
Introduction à la sécurité des WebServicesIntroduction à la sécurité des WebServices
Introduction à la sécurité des WebServicesConFoo
11.5K views50 slides
SQLITE Android by
SQLITE AndroidSQLITE Android
SQLITE AndroidSourabh Sahu
3.5K views22 slides
JMeter & ColdFusion by
JMeter & ColdFusion JMeter & ColdFusion
JMeter & ColdFusion isummation
3.8K views30 slides

What's hot(20)

Introduction à la sécurité des WebServices by ConFoo
Introduction à la sécurité des WebServicesIntroduction à la sécurité des WebServices
Introduction à la sécurité des WebServices
ConFoo11.5K views
JMeter & ColdFusion by isummation
JMeter & ColdFusion JMeter & ColdFusion
JMeter & ColdFusion
isummation3.8K views
NGINX Installation and Tuning by NGINX, Inc.
NGINX Installation and TuningNGINX Installation and Tuning
NGINX Installation and Tuning
NGINX, Inc.10.3K views
jQuery Tutorial For Beginners | Developing User Interface (UI) Using jQuery |... by Edureka!
jQuery Tutorial For Beginners | Developing User Interface (UI) Using jQuery |...jQuery Tutorial For Beginners | Developing User Interface (UI) Using jQuery |...
jQuery Tutorial For Beginners | Developing User Interface (UI) Using jQuery |...
Edureka!478 views
Hibernate architecture by Anurag
Hibernate architectureHibernate architecture
Hibernate architecture
Anurag 5.9K views
Introduction to rest.li by Joe Betz
Introduction to rest.liIntroduction to rest.li
Introduction to rest.li
Joe Betz30.3K views
Packages in java by jamunaashok
Packages in javaPackages in java
Packages in java
jamunaashok499 views
Chapter 3 servlet & jsp by Jafar Nesargi
Chapter 3 servlet & jspChapter 3 servlet & jsp
Chapter 3 servlet & jsp
Jafar Nesargi3.7K views
Atomicity In Redis: Thomas Hunter by Redis Labs
Atomicity In Redis: Thomas HunterAtomicity In Redis: Thomas Hunter
Atomicity In Redis: Thomas Hunter
Redis Labs4.5K views
Improved alerting with Prometheus and Alertmanager by Julien Pivotto
Improved alerting with Prometheus and AlertmanagerImproved alerting with Prometheus and Alertmanager
Improved alerting with Prometheus and Alertmanager
Julien Pivotto4.5K views
enums by teach4uin
enumsenums
enums
teach4uin2.3K views

Viewers also liked

Pecha Kucha: Ukrainian Food Traditions by
Pecha Kucha: Ukrainian Food TraditionsPecha Kucha: Ukrainian Food Traditions
Pecha Kucha: Ukrainian Food TraditionsAndrii Gakhov
839 views20 slides
Recurrent Neural Networks. Part 1: Theory by
Recurrent Neural Networks. Part 1: TheoryRecurrent Neural Networks. Part 1: Theory
Recurrent Neural Networks. Part 1: TheoryAndrii Gakhov
14.2K views36 slides
Open Source Software You Can Use by
Open Source Software You Can UseOpen Source Software You Can Use
Open Source Software You Can UseMaxwell Pearl
1.1K views31 slides
14 Skip Lists by
14 Skip Lists14 Skip Lists
14 Skip ListsAndres Mendez-Vazquez
3.7K views211 slides
Вероятностные структуры данных by
Вероятностные структуры данныхВероятностные структуры данных
Вероятностные структуры данныхAndrii Gakhov
1.3K views48 slides
Time series predictions using LSTMs by
Time series predictions using LSTMsTime series predictions using LSTMs
Time series predictions using LSTMsSetu Chokshi
2K views20 slides

Viewers also liked(20)

Pecha Kucha: Ukrainian Food Traditions by Andrii Gakhov
Pecha Kucha: Ukrainian Food TraditionsPecha Kucha: Ukrainian Food Traditions
Pecha Kucha: Ukrainian Food Traditions
Andrii Gakhov839 views
Recurrent Neural Networks. Part 1: Theory by Andrii Gakhov
Recurrent Neural Networks. Part 1: TheoryRecurrent Neural Networks. Part 1: Theory
Recurrent Neural Networks. Part 1: Theory
Andrii Gakhov14.2K views
Open Source Software You Can Use by Maxwell Pearl
Open Source Software You Can UseOpen Source Software You Can Use
Open Source Software You Can Use
Maxwell Pearl1.1K views
Вероятностные структуры данных by Andrii Gakhov
Вероятностные структуры данныхВероятностные структуры данных
Вероятностные структуры данных
Andrii Gakhov1.3K views
Time series predictions using LSTMs by Setu Chokshi
Time series predictions using LSTMsTime series predictions using LSTMs
Time series predictions using LSTMs
Setu Chokshi2K views
Bloom filter by feng lee
Bloom filterBloom filter
Bloom filter
feng lee3.1K views
Probabilistic data structures. Part 3. Frequency by Andrii Gakhov
Probabilistic data structures. Part 3. FrequencyProbabilistic data structures. Part 3. Frequency
Probabilistic data structures. Part 3. Frequency
Andrii Gakhov1.7K views
Probabilistic data structures. Part 2. Cardinality by Andrii Gakhov
Probabilistic data structures. Part 2. CardinalityProbabilistic data structures. Part 2. Cardinality
Probabilistic data structures. Part 2. Cardinality
Andrii Gakhov1.7K views
Probabilistic data structures. Part 4. Similarity by Andrii Gakhov
Probabilistic data structures. Part 4. SimilarityProbabilistic data structures. Part 4. Similarity
Probabilistic data structures. Part 4. Similarity
Andrii Gakhov2.4K views
Nginx Conference 2016 - Learnings and State of the Industry by Benjamin Scholler
Nginx Conference 2016 - Learnings and State of the IndustryNginx Conference 2016 - Learnings and State of the Industry
Nginx Conference 2016 - Learnings and State of the Industry
Benjamin Scholler315 views
Microservices designing deploying by Suresh Kumar
Microservices designing deployingMicroservices designing deploying
Microservices designing deploying
Suresh Kumar993 views
Machine Learning 101 by Setu Chokshi
Machine Learning 101Machine Learning 101
Machine Learning 101
Setu Chokshi2.6K views
Trello介紹&操作說明 by 妤璇 林
Trello介紹&操作說明Trello介紹&操作說明
Trello介紹&操作說明
妤璇 林14.9K views
skip list by iammutex
skip listskip list
skip list
iammutex9.4K views
Microservices with Spring Cloud by Daniel Eichten
Microservices with Spring CloudMicroservices with Spring Cloud
Microservices with Spring Cloud
Daniel Eichten751 views
New File Server Features Of Windows Server 2008 by Microsoft TechNet
New File Server Features Of Windows Server 2008New File Server Features Of Windows Server 2008
New File Server Features Of Windows Server 2008
Microsoft TechNet5.8K views
Roll Your Own API Management Platform with nginx and Lua by Jon Moore
Roll Your Own API Management Platform with nginx and LuaRoll Your Own API Management Platform with nginx and Lua
Roll Your Own API Management Platform with nginx and Lua
Jon Moore47.4K views
Accelerating Your Web Application with NGINX by Kevin Jones
Accelerating Your Web Application with NGINXAccelerating Your Web Application with NGINX
Accelerating Your Web Application with NGINX
Kevin Jones731 views
Apache Big Data Europe 2015: Selected Talks by Andrii Gakhov
Apache Big Data Europe 2015: Selected TalksApache Big Data Europe 2015: Selected Talks
Apache Big Data Europe 2015: Selected Talks
Andrii Gakhov716 views

Similar to Implementing a Fileserver with Nginx and Lua

nodejs_at_a_glance.ppt by
nodejs_at_a_glance.pptnodejs_at_a_glance.ppt
nodejs_at_a_glance.pptWalaSidhom1
7 views66 slides
Language Resource Processing Configuration and Run by
Language Resource Processing Configuration and RunLanguage Resource Processing Configuration and Run
Language Resource Processing Configuration and Runmario_munoz
3.2K views14 slides
Rapid java backend and api development for mobile devices by
Rapid java backend and api development for mobile devicesRapid java backend and api development for mobile devices
Rapid java backend and api development for mobile devicesciklum_ods
1.9K views18 slides
Using and scaling Rack and Rack-based middleware by
Using and scaling Rack and Rack-based middlewareUsing and scaling Rack and Rack-based middleware
Using and scaling Rack and Rack-based middlewareAlona Mekhovova
1.1K views38 slides
Airflow tutorials hands_on by
Airflow tutorials hands_onAirflow tutorials hands_on
Airflow tutorials hands_onpko89403
91 views24 slides
Building Web Apps with Express by
Building Web Apps with ExpressBuilding Web Apps with Express
Building Web Apps with ExpressAaron Stannard
10.3K views23 slides

Similar to Implementing a Fileserver with Nginx and Lua(20)

nodejs_at_a_glance.ppt by WalaSidhom1
nodejs_at_a_glance.pptnodejs_at_a_glance.ppt
nodejs_at_a_glance.ppt
WalaSidhom17 views
Language Resource Processing Configuration and Run by mario_munoz
Language Resource Processing Configuration and RunLanguage Resource Processing Configuration and Run
Language Resource Processing Configuration and Run
mario_munoz3.2K views
Rapid java backend and api development for mobile devices by ciklum_ods
Rapid java backend and api development for mobile devicesRapid java backend and api development for mobile devices
Rapid java backend and api development for mobile devices
ciklum_ods1.9K views
Using and scaling Rack and Rack-based middleware by Alona Mekhovova
Using and scaling Rack and Rack-based middlewareUsing and scaling Rack and Rack-based middleware
Using and scaling Rack and Rack-based middleware
Alona Mekhovova1.1K views
Airflow tutorials hands_on by pko89403
Airflow tutorials hands_onAirflow tutorials hands_on
Airflow tutorials hands_on
pko8940391 views
Building Web Apps with Express by Aaron Stannard
Building Web Apps with ExpressBuilding Web Apps with Express
Building Web Apps with Express
Aaron Stannard10.3K views
JDD 2017: Nginx + Lua = OpenResty (Marcin Stożek) by PROIDEA
JDD 2017: Nginx + Lua = OpenResty (Marcin Stożek)JDD 2017: Nginx + Lua = OpenResty (Marcin Stożek)
JDD 2017: Nginx + Lua = OpenResty (Marcin Stożek)
PROIDEA91 views
Introducing Node.js in an Oracle technology environment (including hands-on) by Lucas Jellema
Introducing Node.js in an Oracle technology environment (including hands-on)Introducing Node.js in an Oracle technology environment (including hands-on)
Introducing Node.js in an Oracle technology environment (including hands-on)
Lucas Jellema2.4K views
Cloudbase.io MoSync Reload Course by cloudbase.io
Cloudbase.io MoSync Reload CourseCloudbase.io MoSync Reload Course
Cloudbase.io MoSync Reload Course
cloudbase.io1.1K views
Reverse proxies & Inconsistency by GreenD0g
Reverse proxies & InconsistencyReverse proxies & Inconsistency
Reverse proxies & Inconsistency
GreenD0g4.8K views
Nginx internals by liqiang xu
Nginx internalsNginx internals
Nginx internals
liqiang xu4.1K views
WebTalk - Implementing Web Services with a dedicated Java daemon by Geert Van Pamel
WebTalk - Implementing Web Services with a dedicated Java daemonWebTalk - Implementing Web Services with a dedicated Java daemon
WebTalk - Implementing Web Services with a dedicated Java daemon
Geert Van Pamel6.9K views
The Magic Revealed: Four Real-World Examples of Using the Client Object Model... by SPTechCon
The Magic Revealed: Four Real-World Examples of Using the Client Object Model...The Magic Revealed: Four Real-World Examples of Using the Client Object Model...
The Magic Revealed: Four Real-World Examples of Using the Client Object Model...
SPTechCon3K views
Overview of RESTful web services by nbuddharaju
Overview of RESTful web servicesOverview of RESTful web services
Overview of RESTful web services
nbuddharaju4.2K views
Asynchronous I/O in NodeJS - new standard or challenges? by Dinh Pham
Asynchronous I/O in NodeJS - new standard or challenges?Asynchronous I/O in NodeJS - new standard or challenges?
Asynchronous I/O in NodeJS - new standard or challenges?
Dinh Pham2.4K views
Scalable network applications, event-driven - Node JS by Cosmin Mereuta
Scalable network applications, event-driven - Node JSScalable network applications, event-driven - Node JS
Scalable network applications, event-driven - Node JS
Cosmin Mereuta1.2K views
Everything you wanted to know about writing async, concurrent http apps in java by Baruch Sadogursky
Everything you wanted to know about writing async, concurrent http apps in java Everything you wanted to know about writing async, concurrent http apps in java
Everything you wanted to know about writing async, concurrent http apps in java
Baruch Sadogursky5.4K views
Node.js Workshop - Sela SDP 2015 by Nir Noy
Node.js Workshop  - Sela SDP 2015Node.js Workshop  - Sela SDP 2015
Node.js Workshop - Sela SDP 2015
Nir Noy937 views

More from Andrii Gakhov

Let's start GraphQL: structure, behavior, and architecture by
Let's start GraphQL: structure, behavior, and architectureLet's start GraphQL: structure, behavior, and architecture
Let's start GraphQL: structure, behavior, and architectureAndrii Gakhov
423 views90 slides
Exceeding Classical: Probabilistic Data Structures in Data Intensive Applicat... by
Exceeding Classical: Probabilistic Data Structures in Data Intensive Applicat...Exceeding Classical: Probabilistic Data Structures in Data Intensive Applicat...
Exceeding Classical: Probabilistic Data Structures in Data Intensive Applicat...Andrii Gakhov
735 views39 slides
Too Much Data? - Just Sample, Just Hash, ... by
Too Much Data? - Just Sample, Just Hash, ...Too Much Data? - Just Sample, Just Hash, ...
Too Much Data? - Just Sample, Just Hash, ...Andrii Gakhov
386 views23 slides
DNS Delegation by
DNS DelegationDNS Delegation
DNS DelegationAndrii Gakhov
902 views15 slides
Swagger / Quick Start Guide by
Swagger / Quick Start GuideSwagger / Quick Start Guide
Swagger / Quick Start GuideAndrii Gakhov
7.6K views26 slides
API Days Berlin highlights by
API Days Berlin highlightsAPI Days Berlin highlights
API Days Berlin highlightsAndrii Gakhov
787 views25 slides

More from Andrii Gakhov(20)

Let's start GraphQL: structure, behavior, and architecture by Andrii Gakhov
Let's start GraphQL: structure, behavior, and architectureLet's start GraphQL: structure, behavior, and architecture
Let's start GraphQL: structure, behavior, and architecture
Andrii Gakhov423 views
Exceeding Classical: Probabilistic Data Structures in Data Intensive Applicat... by Andrii Gakhov
Exceeding Classical: Probabilistic Data Structures in Data Intensive Applicat...Exceeding Classical: Probabilistic Data Structures in Data Intensive Applicat...
Exceeding Classical: Probabilistic Data Structures in Data Intensive Applicat...
Andrii Gakhov735 views
Too Much Data? - Just Sample, Just Hash, ... by Andrii Gakhov
Too Much Data? - Just Sample, Just Hash, ...Too Much Data? - Just Sample, Just Hash, ...
Too Much Data? - Just Sample, Just Hash, ...
Andrii Gakhov386 views
Swagger / Quick Start Guide by Andrii Gakhov
Swagger / Quick Start GuideSwagger / Quick Start Guide
Swagger / Quick Start Guide
Andrii Gakhov7.6K views
API Days Berlin highlights by Andrii Gakhov
API Days Berlin highlightsAPI Days Berlin highlights
API Days Berlin highlights
Andrii Gakhov787 views
ELK - What's new and showcases by Andrii Gakhov
ELK - What's new and showcasesELK - What's new and showcases
ELK - What's new and showcases
Andrii Gakhov938 views
Apache Spark Overview @ ferret by Andrii Gakhov
Apache Spark Overview @ ferretApache Spark Overview @ ferret
Apache Spark Overview @ ferret
Andrii Gakhov1.2K views
Data Mining - lecture 8 - 2014 by Andrii Gakhov
Data Mining - lecture 8 - 2014Data Mining - lecture 8 - 2014
Data Mining - lecture 8 - 2014
Andrii Gakhov1.3K views
Data Mining - lecture 7 - 2014 by Andrii Gakhov
Data Mining - lecture 7 - 2014Data Mining - lecture 7 - 2014
Data Mining - lecture 7 - 2014
Andrii Gakhov2.5K views
Data Mining - lecture 6 - 2014 by Andrii Gakhov
Data Mining - lecture 6 - 2014Data Mining - lecture 6 - 2014
Data Mining - lecture 6 - 2014
Andrii Gakhov1.1K views
Data Mining - lecture 5 - 2014 by Andrii Gakhov
Data Mining - lecture 5 - 2014Data Mining - lecture 5 - 2014
Data Mining - lecture 5 - 2014
Andrii Gakhov816 views
Data Mining - lecture 4 - 2014 by Andrii Gakhov
Data Mining - lecture 4 - 2014Data Mining - lecture 4 - 2014
Data Mining - lecture 4 - 2014
Andrii Gakhov1K views
Data Mining - lecture 3 - 2014 by Andrii Gakhov
Data Mining - lecture 3 - 2014Data Mining - lecture 3 - 2014
Data Mining - lecture 3 - 2014
Andrii Gakhov1K views
Decision Theory - lecture 1 (introduction) by Andrii Gakhov
Decision Theory - lecture 1 (introduction)Decision Theory - lecture 1 (introduction)
Decision Theory - lecture 1 (introduction)
Andrii Gakhov1.4K views
Data Mining - lecture 2 - 2014 by Andrii Gakhov
Data Mining - lecture 2 - 2014Data Mining - lecture 2 - 2014
Data Mining - lecture 2 - 2014
Andrii Gakhov824 views
Data Mining - lecture 1 - 2014 by Andrii Gakhov
Data Mining - lecture 1 - 2014Data Mining - lecture 1 - 2014
Data Mining - lecture 1 - 2014
Andrii Gakhov2.1K views
Buzzwords 2014 / Overview / part2 by Andrii Gakhov
Buzzwords 2014 / Overview / part2Buzzwords 2014 / Overview / part2
Buzzwords 2014 / Overview / part2
Andrii Gakhov634 views
Buzzwords 2014 / Overview / part1 by Andrii Gakhov
Buzzwords 2014 / Overview / part1Buzzwords 2014 / Overview / part1
Buzzwords 2014 / Overview / part1
Andrii Gakhov902 views

Recently uploaded

NTGapps NTG LowCode Platform by
NTGapps NTG LowCode Platform NTGapps NTG LowCode Platform
NTGapps NTG LowCode Platform Mustafa Kuğu
287 views30 slides
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ... by
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...ShapeBlue
97 views28 slides
KVM Security Groups Under the Hood - Wido den Hollander - Your.Online by
KVM Security Groups Under the Hood - Wido den Hollander - Your.OnlineKVM Security Groups Under the Hood - Wido den Hollander - Your.Online
KVM Security Groups Under the Hood - Wido den Hollander - Your.OnlineShapeBlue
154 views19 slides
Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda... by
Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda...Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda...
Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda...ShapeBlue
93 views13 slides
Future of AR - Facebook Presentation by
Future of AR - Facebook PresentationFuture of AR - Facebook Presentation
Future of AR - Facebook PresentationRob McCarty
54 views27 slides
DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti... by
DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti...DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti...
DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti...ShapeBlue
69 views29 slides

Recently uploaded(20)

NTGapps NTG LowCode Platform by Mustafa Kuğu
NTGapps NTG LowCode Platform NTGapps NTG LowCode Platform
NTGapps NTG LowCode Platform
Mustafa Kuğu287 views
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ... by ShapeBlue
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...
ShapeBlue97 views
KVM Security Groups Under the Hood - Wido den Hollander - Your.Online by ShapeBlue
KVM Security Groups Under the Hood - Wido den Hollander - Your.OnlineKVM Security Groups Under the Hood - Wido den Hollander - Your.Online
KVM Security Groups Under the Hood - Wido den Hollander - Your.Online
ShapeBlue154 views
Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda... by ShapeBlue
Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda...Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda...
Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda...
ShapeBlue93 views
Future of AR - Facebook Presentation by Rob McCarty
Future of AR - Facebook PresentationFuture of AR - Facebook Presentation
Future of AR - Facebook Presentation
Rob McCarty54 views
DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti... by ShapeBlue
DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti...DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti...
DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti...
ShapeBlue69 views
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ... by ShapeBlue
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...
ShapeBlue48 views
Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ... by ShapeBlue
Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ...Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ...
Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ...
ShapeBlue114 views
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas... by Bernd Ruecker
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...
Bernd Ruecker50 views
What’s New in CloudStack 4.19 - Abhishek Kumar - ShapeBlue by ShapeBlue
What’s New in CloudStack 4.19 - Abhishek Kumar - ShapeBlueWhat’s New in CloudStack 4.19 - Abhishek Kumar - ShapeBlue
What’s New in CloudStack 4.19 - Abhishek Kumar - ShapeBlue
ShapeBlue191 views
"Surviving highload with Node.js", Andrii Shumada by Fwdays
"Surviving highload with Node.js", Andrii Shumada "Surviving highload with Node.js", Andrii Shumada
"Surviving highload with Node.js", Andrii Shumada
Fwdays49 views
Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit... by ShapeBlue
Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit...Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit...
Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit...
ShapeBlue86 views
Updates on the LINSTOR Driver for CloudStack - Rene Peinthor - LINBIT by ShapeBlue
Updates on the LINSTOR Driver for CloudStack - Rene Peinthor - LINBITUpdates on the LINSTOR Driver for CloudStack - Rene Peinthor - LINBIT
Updates on the LINSTOR Driver for CloudStack - Rene Peinthor - LINBIT
ShapeBlue138 views
Extending KVM Host HA for Non-NFS Storage - Alex Ivanov - StorPool by ShapeBlue
Extending KVM Host HA for Non-NFS Storage -  Alex Ivanov - StorPoolExtending KVM Host HA for Non-NFS Storage -  Alex Ivanov - StorPool
Extending KVM Host HA for Non-NFS Storage - Alex Ivanov - StorPool
ShapeBlue56 views
Keynote Talk: Open Source is Not Dead - Charles Schulz - Vates by ShapeBlue
Keynote Talk: Open Source is Not Dead - Charles Schulz - VatesKeynote Talk: Open Source is Not Dead - Charles Schulz - Vates
Keynote Talk: Open Source is Not Dead - Charles Schulz - Vates
ShapeBlue178 views
VNF Integration and Support in CloudStack - Wei Zhou - ShapeBlue by ShapeBlue
VNF Integration and Support in CloudStack - Wei Zhou - ShapeBlueVNF Integration and Support in CloudStack - Wei Zhou - ShapeBlue
VNF Integration and Support in CloudStack - Wei Zhou - ShapeBlue
ShapeBlue134 views
CloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlue by ShapeBlue
CloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlueCloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlue
CloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlue
ShapeBlue63 views

Implementing a Fileserver with Nginx and Lua

  • 1. Implementing a Fileserver with Nginx and Lua techtalk@ferret Andrii Gakhov 27.02.2017
  • 2. Current Architecture Restful API server that communicates with clients using application/json Every client is authenticated by a dedicated 3rd party authentication server and authorized by the API server End users get benefits from the API via client-side frontend application
  • 3. The Problem At some point, end users request the ability to manage files using the same client-side application and, consequently, same API. allow to process multipart/form-data requests (that will be proxied from the form on the client-side application) extract and handle file metadata provide file storage and access
  • 4. The Solution Obviously, such functionality is out of the scope for the API and the natural decision is to split it across 2 applications: API is used for meta file management Fileserver takes care about actual files upload/download Instead of using Flask/Django etc., we have decided to implements such functionality in Nginx with Lua scripting
  • 5. Nginx and Lua Nginx is a free, open-source, high-performance HTTP server and reverse proxy, as well as an IMAP/POP3 proxy server. Lua is a powerful, efficient, lightweight, embeddable scripting language. It supports procedural programming, object-oriented programming, functional programming, data-driven programming, and data description. The Lua module for Nginx ngx_http_lua_module embeds Lua into Nginx and by leveraging Nginx's subrequests, allows the integration of Lua threads into the Nginx event model.
  • 6. Nginx and Lua Nginx object is available in Lua as ngx and server variables are accessible as ngx.var.{name}, requested GET arguments - as ngx.var.arg_{name}, but unfortunately POST variables Nginx doesn't parse into variables automatically. The various *_by_lua_* configuration directives serve as gateways to the Lua API within the nginx.conf. We use *_by_lua_block to embed Lua code into Nginx as the most notable way to do so.
  • 7. Nginx and Lua header_filter_by_lua_block - uses Lua code to define an output header filter (e.g. for overriding or adding a response header). access_by_lua_block - acts as an access phase handler and executes Lua code for every request. as with other access phase handlers, access_by_lua will not run in subrequests. content_by_lua_block - acts as a "content handler" and executes Lua code for every request.
  • 8. Architecture / Upload location = /_upload { limit_except POST { deny all; } if ($http_content_type !~ "multipart/form-data") { return 406; } lua_need_request_body off; client_body_buffer_size 200M; client_max_body_size 200M; default_type 'application/json'; ... }
  • 9. Architecture / Download location ~ ^/_download/(.*)$ { set $path /$1; default_type 'application/json'; limit_except GET { deny all; } ... }
  • 10. User Authentication function authenticate_user(access_token) local params = { method = ngx.HTTP_GET, args = { access_token = access_token } } local res = ngx.location.capture( "/_auth/access/check", params) if not res or res.status ~= 200 then return nil end return cjson.decode(res.body) end access_by_lua_block { local cjson = require("cjson") local access_token = ngx.var.arg_access_token if not access_token then return_http_forbidden("Forbidden") end -- authenticate user local credentials = authenticate_user(access_token) -- if user can't be resolved, return 403 Forbidden if not credentials or not credentials.data.user.id then return_http_forbidden("Forbidden") end }
  • 11. Upload -- Extract POST variables in a streaming way -- and store file object in a temporary -- file (form_data.file.filepath) -- All.other form variables are in form_data -- (we expect them to be small) local helpers = require("utils") form_data, err = helpers.streaming_multipart_form() if form_data == nil then return_http_internal_server_error(err) end data = { status = "not_ready", title = form.title.value, filename = form.file.filename, } local params = { method = ngx.HTTP_POST, args = {access_token = access_token}, body = cjson.encode(data) } local res = ngx.location.capture("/_api/v1/file", params) if not res then return_http_bad_gateway("No metadata") end local create_metadata_resp = res.body if res and res.status ~= 201 then ngx.status = res.status ngx.print(create_metadata_resp) ngx.exit(res.status) end local file_metadata = cjson.decode(create_metadata_resp) Important part to make it work is to specify client_body_buffer_size equal to client_max_body_size. Otherwise, if the client body is bigger than client_body_buffer_size, the nginx variable $request_body will be empty.
  • 12. Upload local filex = require("pl.file") local file_fullpath = "/storage/files/" .. file_metadata.hash .. file_metadata.path -- ensure that subdirectories exist (if not, create) -- store tmp file to its permanent position is_moved, err = filex.move( form.file.fullpath, file_fullpath) -- make it available for download if is_moved then local params = { method = ngx.HTTP_PUT, args = {access_token = access_token}, body = cjson.encode({status = "ready"}) } ngx.location.capture( "/_api/v1/file/" .. file_metadata.id, params) end -- provide some headers with metadata ngx.header["X-File-ID"] = file_metadata.id ngx.status = ngx.HTTP_CREATED ngx.print(create_metadata_resp) ngx.exit(ngx.HTTP_CREATED) Since we need to manage access to the file, we use access policy of files’ metadata instead of the files themselves. We create metadata in any case (“register the file”), but allow to download it only if file has status “ready”, meaning it was successfully created at the specified location.
  • 13. Download local search_by_path = { filters = { path = ngx.var.path, status = "ready" }, size = 1 } local params = { method = ngx.HTTP_POST, args = {access_token = access_token}, body = cjson.encode(search_by_path) } local res = ngx.location.capture( "/_api/v1/search/files", params) if not res then return_http_bad_gateway("Search error") end local found = cjson.decode(res.body) if found.total < 1 then return_http_not_found("File Not Found") end local file_metadata = found.results[1] ngx.header["X-File-Name"] = file_metadata.name ngx.header["X-File-Path"] = file_metadata.path ngx.header["X-File-ID"] = file_metadata.id ngx.req.set_uri( "/" .. file_metadata.hash .. file_metadata.path) ngx.exec("@download_file", download_params) As soon as the file has been found, its metadata provides us with all necessary information about the location and we are ready to respond it to the user. Sometimes people recommend to read such file in Lua and return it to the user with ngx.print that is a bad idea for big files (Lua virtual machine will just crash).
  • 14. Download location @download_file { internal; root /storage/files/; try_files $uri =404; header_filter_by_lua_block { ngx.header["Cache-Control"] = "no-cache" ngx.header["Content-Disposition"] = "attachment; filename="" .. ngx.header["X-File-Name"] .. """ } } The @download_file location is quite simple, but additionally we want to play with response headers to provide a real filename for download (on our filesystem all files are stored with unique generated names). It is an internal named location (to prevent unauthorized access) that just serves requested static files from the desired directory.
  • 15. How to Use Upload curl -XPOST https://files.example.com/_upload?access_token={SOME_TOKEN} --form file=@/tmp/file.pdf --form title="Example title" -H "Content-Type: multipart/form-data" Download curl -XGET https://files.example.com/_download/723533/2338342189083057604.pdf?access_token={SOME_TOKEN}
  • 16. Thank you Read it in the web: https://www.datacrucis.com/research/implementing-api-based- fileserver-with-nginx-and-lua.html