Best Practices for Getting Started with NGINX Open Source You already use NGINX, but can you take its performance "over 9000"? Alessandro Fael Garcia, Senior Solution Engineer for NGINX, describes best practices for configuring NGINX as a reverse proxy, load balancer, and cache, and how small adjustments with NGINX directives can improve performance. Speakers: Alessandro Fael Garcia

1. Best Practices for Getting
Started with NGINX Open
Source
Alessandro Fael Garcia
Senior Solutions Engineer – Community & Alliances

2. ©2022 F5
2 Source: https://news.netcraft.com/archives/2022/06/30/june-2022-web-server-survey.html

3. ©2022 F5
3

4. ©2022 F5
4
Installing NGINX
Best practices

5. ©2022 F5
5
Use the NGINX Open Source official repository!
https://nginx.org/en/linux_packages.html

6. ©2022 F5
6
TIL
• nginx –t → Check if NGINX configuration is valid
• nginx –T → Dump full NGINX configuration
• nginx –v → Print NGINX version
• nginx –V → Print NGINX package config arguments
• nginx –s <start/stop/reload> → Start NGINX; stop (kill) NGINX; reload NGINX configuration (gracefully)
Key NGINX Commands

7. ©2022 F5
7
/etc/nginx/nginx.conf
• Main NGINX configuration file
• Global settings
• Contains sensible defaults (when installing NGINX from our
official repositories)
• Avoid modifying unless you know what you are doing
(defaults will work out of the box for >80% of use cases)
• Includes HTTP block (adding a Stream block is one of the
few cases where you’d want to modify the file)
/etc/nginx/conf.d/*.conf
• Default directory for additional NGINX configuration files
• By default, files here are contained within the HTTP context
• default.conf includes sample configuration with the NGINX
default landing page
• Start with a single configuration file, split your configuration
into further files as necessary
Recommended NGINX Directory Structure
Defaults? What defaults?!

8. ©2022 F5
8
Use Let’s Encrypt and Certbot for easy certs!
https://certbot.eff.org/instructions?ws=nginx&os=ubuntufocal

9. ©2022 F5
9
Tuning NGINX
One step at a time

10. ©2022 F5
10
nginx.conf
nginx.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
...
worker_processes auto;
worker_rlimit_nofile 2048;
...
events {
worker_connections 1024;
}
http {
access_log off;
sendfile on;
tcp_nopush on;
proxy_cache_lock on;
...
upstream app {
server w.x.y.z;
keepalive 2;
...
}
server {
access_log /var/log/nginx/access.log main buffer=512k
flush=5m;
ssl_session_cache shared:SSL:10m;
...
location / {
proxy_http_version 1.1;
proxy_set_header Connection “”;
proxy_pass http://app;
...
}
}
}

11. ©2022 F5
11
worker_processes
nginx.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
...
worker_processes auto;
worker_rlimit_nofile 2048;
...
events {
worker_connections 1024;
}
http {
access_log off;
sendfile on;
tcp_nopush on;
...
upstream app {
server w.x.y.z;
keepalive 2;
...
}
server {
access_log /var/log/nginx/access.log main buffer=512k
flush=5m;
ssl_session_cache shared:SSL:10m;
...
location / {
proxy_http_version 1.1;
proxy_set_header Connection “”;
proxy_pass http://app;
proxy_cache_lock on;
...
}
}
}
Make sure you spawn one NGINX worker process per CPU
core (default: 1)

12. ©2022 F5
12
worker_connections & worker_rlimit_nofile
nginx.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
...
worker_processes auto;
worker_rlimit_nofile 2048;
...
events {
worker_connections 1024;
}
http {
access_log off;
sendfile on;
tcp_nopush on;
...
upstream app {
server w.x.y.z;
keepalive 2;
...
}
server {
access_log /var/log/nginx/access.log main buffer=512k
flush=5m;
ssl_session_cache shared:SSL:10m;
...
location / {
proxy_http_version 1.1;
proxy_set_header Connection “”;
proxy_pass http://app;
proxy_cache_lock on;
...
}
}
}
a) Increase the worker connections to >1024 (default: 512)
b) Increase the limit on the maximum number of open files
to at least twice the number of worker connections
(default: system limit)

13. ©2022 F5
13
access_log
nginx.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
...
worker_processes auto;
worker_rlimit_nofile 2048;
...
events {
worker_connections 1024;
}
http {
access_log off;
sendfile on;
tcp_nopush on;
...
upstream app {
server w.x.y.z;
keepalive 2;
...
}
server {
access_log /var/log/nginx/access.log main buffer=512k
flush=5m;
ssl_session_cache shared:SSL:10m;
...
location / {
proxy_http_version 1.1;
proxy_set_header Connection “”;
proxy_pass http://app;
proxy_cache_lock on;
...
}
}
}
• Turn off the access log for extra performance (default: on)
or
• Set a buffer or a time to only write logs at an interval
(default: off)

14. ©2022 F5
14
keepalive
nginx.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
...
worker_processes auto;
worker_rlimit_nofile 2048;
...
events {
worker_connections 1024;
}
http {
access_log off;
sendfile on;
tcp_nopush on;
...
upstream app {
server w.x.y.z;
keepalive 2;
...
}
server {
access_log /var/log/nginx/access.log main buffer=512k
flush=5m;
ssl_session_cache shared:SSL:10m;
...
location / {
proxy_http_version 1.1;
proxy_set_header Connection “”;
proxy_pass http://app;
proxy_cache_lock on;
...
}
}
}
Use keepalives to keep connections to upstream servers
open (default: 0) → You will need to set HTTP to 1.1 and
rewrite the Connection header

15. ©2022 F5
15
ssl_session_cache
nginx.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
...
worker_processes auto;
worker_rlimit_nofile 2048;
...
events {
worker_connections 1024;
}
http {
access_log off;
sendfile on;
tcp_nopush on;
...
upstream app {
server w.x.y.z;
keepalive 2;
...
}
server {
access_log /var/log/nginx/access.log main buffer=512k
flush=5m;
ssl_session_cache shared:SSL:10m;
...
location / {
proxy_http_version 1.1;
proxy_set_header Connection “”;
proxy_pass http://app;
proxy_cache_lock on;
...
}
}
}
Cache and share your SSL sessions between all your NGINX
processes (default: disabled)

16. ©2022 F5
16
proxy_cache_lock
nginx.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
...
worker_processes auto;
worker_rlimit_nofile 2048;
...
events {
worker_connections 1024;
}
http {
access_log off;
sendfile on;
tcp_nopush on;
...
upstream app {
server w.x.y.z;
keepalive 2;
...
}
server {
access_log /var/log/nginx/access.log main buffer=512k
flush=5m;
ssl_session_cache shared:SSL:10m;
...
location / {
proxy_http_version 1.1;
proxy_set_header Connection “”;
proxy_pass http://app;
proxy_cache_lock on;
...
}
}
}
Send only one request to the upstream server when there
are multiple cache misses for the same file (default: off)

17. ©2022 F5
17
Recap
nginx.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
...
worker_processes auto;
worker_rlimit_nofile 2048;
...
events {
worker_connections 1024;
}
http {
access_log off;
sendfile on;
tcp_nopush on;
...
upstream app {
server w.x.y.z;
keepalive 2;
...
}
server {
access_log /var/log/nginx/access.log main buffer=512k
flush=5m;
ssl_session_cache shared:SSL:10m;
...
location / {
proxy_http_version 1.1;
proxy_set_header Connection “”;
proxy_pass http://app;
proxy_cache_lock on;
...
}
}
}
• Make sure you spawn one NGINX worker process per
CPU core (default: 1)
• Increase the worker connections to >1024 (default: 512)
• Increase the limit on the maximum number of open files to
at least twice the number of worker connections (default:
system limit)
• Turn off the access log for extra performance (default: on)
• Set a buffer or a time to only write logs at an interval
(default: off)
• Use keepalives to keep connections to upstream servers
open (default: 0) → You will need to set HTTP to 1.1 and
rewrite the Connection header
• Cache and share your SSL sessions between all your
NGINX processes (default: disabled)
• Send only one request to the upstream server when there
are multiple cache misses for the same file (default: off)

18. ©2022 F5
18
Common NGINX Mistakes
That we’ve all made at some stage

19. ©2022 F5
19
error_log
nginx.conf
1
2
3
...
error_log off;
...
nginx.conf
1
2
3
...
error_log /dev/null emerg;
...
Creates an error log named off
Redirects error log data to /dev/null

20. ©2022 F5
20
Directive inheritance is not additive
nginx.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
http {
add_header HTTP_HEADER;
...
server {
add_header HTTP_HEADER;
...
location / {
add_header HTTP_HEADER;
add_header LOCATION_HEADER:
...
}
}
}
Sets directive
Inherits directive
Overrides directive

21. ©2022 F5
21
ip_hash
nginx.conf
1
2
3
4
5
6
7
8
9
10
11
12
http {
...
upstream {
ip_hash;
server 10.10.20.105:8080;
server 10.10.20.106:8080;
server 10.10.20.108:8080;
}
server {
...
}
}
If all your traffic comes from the same CIDR block,
use hash or any other load balancing algorithm instead

22. ©2022 F5
22
proxy_buffering
nginx.conf
1
2
3
4
http {
proxy_buffering off;
...
}
Avoiding buffers might speed up the initial response to your client,
but it might also saturate your open connections

23. ©2022 F5
23
stub_status
nginx.conf
1
2
3
4
5
6
server {
...
location = /status {
stub_status;
}
}
nginx.conf
1
2
3
4
5
6
7
8
9
10
11
server {
...
location = /status {
satisfy any;
auth_basic “closed site”;
auth_basic_user_file conf.d/.htpasswd;
allow 192.168.1.0/24;
deny all;
stub_status;
}
}
Everyone can access your data
Secure access to your data

24. ©2022 F5
24
proxy_pass
nginx.conf
1
2
3
4
5
6
7
8
9
10
http {
...
server {
...
location / {
...
proxy_pass http://localhost:3000/;
}
}
}
nginx.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
http {
...
upstream node_backend {
zone upstreams 64K;
hash;
server 127.0.0.1:3000 max_fails=1 fail_timeout=2s;
server 127.0.0.1:5000 max_fails=1 fail_timeout=2s;
keepalive 4;
}
server {
...
location / {
...
proxy_next_upstream error timeout http_500;
proxy_pass http://node_backend/;
}
}
}
Proxy to an upstream server directly
• Load balance
• Upstream stats
• Keepalives
• Passive health checks
• Define behavior if the upstream servers go down

25. ©2022 F5
25
If is Evil
Much Computationally Expensive!
Very Segfaults 😱
If only works as intended if you use return or rewrite inside your if block

26. ©2022 F5
26
• error_log off != turn off the error log
• Directive inheritance is not additive
• ip_hash does not work for addresses under the same CIDR block
• proxy_buffering off might lead unexpected saturated connections
• Beware of not properly securing your stat locations
• It’s better to proxy_pass to upstream groups than directly to an upstream server
• If. Is. Evil.
Recap

27. ©2022 F5
27
Thankyouforattending!
a.faelgarcia@f5.com
alessfg
@alessfg
Alessandro Fael Garcia

29. ©2022 F5
29
Further Resources
• Performance-Tuning NGINX https://www.youtube.com/watch?v=YEdhuC2muOE
• Best Practices for NGINX https://www.youtube.com/watch?v=pkHQCPXaimU
• Avoiding the Top 10 NGINX Configuration Mistakes https://www.nginx.com/blog/avoiding-top-10-nginx-configuration-mistakes
• Tuning NGINX for Performance https://www.nginx.com/blog/tuning-nginx/