Advanced nginx in mercari - How to handle over 1,200,000 HTTPS Reqs/Min 「nginx実践入門」出版記念！執筆者らが語る nginx Tech Talks

1. Advanced nginx in Mercari
「nginx実践入門」出版記念！執筆者らが語る nginx Tech Talks
Masahiro Nagano
How to handle over 1,200,000 HTTPS Reqs/Min

2. Me
• Masahiro Nagano @kazeburo
• Mercari, inc.
• Principal Engineer, Site Reliability Engineering Team
• BASE, inc. Technical Advisor

3. Mercari
Your Friendly Mobile MarketPlace
JP US

4. Mercari KPI
DOWNLOAD
GMV
ITEMS
3100万DL(JP+US)
月間数十億円
1日数十万品以上

5. API Traffic
1,200,000 reqs/min
(HTTPS only)

6. nginx in mercari
©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
Users Client Multimedia Corporate
data center
Trad
se
Mobile Client
AWS Management
Console
IAM Add-on Example:
IAM Add-on
Tasks (HIT) TaskTurk
ice Specific
©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
User
Users
Client
Multimedia
Corporatedata center
Mobile Client
net
AWS Management
Console
IAM Add-on
Example:IAM Add-on
Requester
azon
W
eb Services LLC
or its affiliates. All rights reserved.
ser
Users
Client
Multim
edia
Corporate
data
center
Tradit
server
Mobile
Client
AW
S
Managem
ent
Console
IAM
Add-on
Exam
ple:
IAM
Add-on
ence
HIT)
Assignm
ent/
Task
W
orkers
L7 load balancer
Reverse Proxy TLS Terminator
Contents Distribution©2011
Am
azon
W
eb
Services LLC
or its affiliates. All rights reserve
Users
Client
M
ultim
edia
Corporate
data
center
Tradition
server
M
obile
Client
Internet
AW
S
M
anagem
ent
Console
IAM
Add-on
Exam
ple:
IAM
Add-on
m
ent/
Task
Requester
W
orkers

7. nginx Extension in mercari
• ngx_dynamic_upstream
• Operate upsteam servers Up/Down dynamically
• OpenResty (ngx_lua)
• More powerful L7 load balancer
• Application log collector

8. Agenda
• Continuous updating nginx
• TLS Optimization
• Automated TLS session ticket key updating
• SPDY to HTTP/2

9. Continuous updating nginx

10. Keep nginx the latest
• Use nginx mainline version
• stableは安定しているという意味ではなく新機能が追加さ
れず、バグ修正も緊急な修正を除いて行われないというこ
と。すべての新機能やバグ修正はmainlineにて行われるの
で結果的にmainlineが安定している
https://www.nginx.com/blog/nginx-1-6-1-7-released/
• nginxは無停止でのアップデートが可能
• Periodic OpenSSL vulnerability report 😓

11. nginx-build
• nginx builder command written in Go
https://github.com/cubicdaiya/nginx-build
• Automate download, configure and make
• Download OpenSSL, zlib and PCRE. build them
statically ( independent from OS’s library version)
• Support Tengine and OpenResty

12. nginx-build
$ EXPORT $VERSION=1.9.10
$ nginx-build
-clear
-d work
-v $VERSION
-c nginx-build-conf/configure
-m nginx-build-conf/modules3rd.ini
-zlib
-zlibversion=1.2.8
-verbose
-pcre
-pcreversion=8.38
-openssl
-opensslversion=1.0.2f
$ cd work/$VERSION/nginx-$VERSION
$ sudo make install

13. TLS Optimization

14. TLS Optimization
• PFS - Perfect Forward Secrecy
• TTFB - Time To First Byte

15. Perfect Forward Secrecy
• Mozilla Wiki
https://wiki.mozilla.org/Security/Server_Side_TLS
• Mozilla SSL Configuration Generator
https://mozilla.github.io/server-side-tls/ssl-config-
generator/
• *最新の情報を参照し、プロダクション環境に投
入する前にテストを行ってください

16. TTFB
• TLS Session Cache
• TLS Session Tickets
• OCSP Stapling
• SSL buffer size

17. TLS Session Cache
• Server side session cache
• Client software does not matter
• nginx could’t share session cache between servers.
ssl_session_cache shared:SSL:120m;
ssl_session_timeout 5m;
CONS
PROS

18. TLS Session Cache
• OpenResty will support it?
https://twitter.com/agentzh/status/
686655229828403201

19. TLS Session Tickets
• Client Side Session Cache
(like a CookieStore)
• No need to share cache between servers
• It’s required share ssl_session_ticket_key file
• Client support required
ssl_session_tickets on;
ssl_session_ticket_key /path/to/ssl_session_ticket;
PROS
CONS

20. TLS Session Tickets
User Agent Session Tickets
Android 2.3.7 No
Android 4.0.4 YES
Android 5 YES
Chrome 47 YES
Firefox 42 YES
IE11/Win7 No
IE11/Win10 YES
Edge/Win10 YES
Safari 9/iOS9 No
Safari 9/MacOSX10.11 No
!!!!
!!!!
https://www.ssllabs.com/ssltest/clients.html

21. OCSP Stapling
https://wiki.mozilla.org/Security/Server_Side_TLS
Client required to checking the
revocation status. If OCSP
Responder is down, page load is
failed.
Server retrieves OCSP record,
caches it and servers it to client
directly. Fast!

22. OSCP Stapling
• All Android version don’t support yet
• iOS < 9 doesn’t support it
CONS
CONS
ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /path/to/server.crt;
resolver 8.8.8.8 valid=30s;
resolver_timeout 5s;

23. TLS buffer size
• TLS record size. Default 16KB, it’s too large. That’s
effect to latency
https://www.igvita.com/2013/12/16/optimizing-nginx-tls-time-to-first-byte/
ssl_buffer_size 1k;

24. Automated
TLS session tickets key
updating

25. TLS Session Tickets
• Should share ssl_session_ticket_key file between
servers
• Mozilla recommends replace it every day for
forward secrecy
• Previously we changed the key file almost every
month in manual operation😓

26. Update Tickets with Consul
• Inspired by Stretcher (fujiwara👍)
• Key_file is very small (48byte), It can store into a payload
of consul event
• We’re updating key_file twice a day by using Consul

27. Update Tickets with Consul
JOB
Consul
Servers
nginx
Consul
Agent
$ consul event
-name=“ssl-session-ticket-refresh"
$(openssl rand 48 | base64)
nginx
Consul
Agent
nginx
Consul
Agent
cat | jq -r '.Payload' |
base64 -d | base64 -d
> /path/to/ticket.key
service nginx reload
❶
❷ ❷ ❷
❸

28. SPDY to HTTP/2

29. SPDY is alive
Request to a webview
728
1,009,268
270,527
HTTP/1.1 SPDY HTTP2

30. SPDY to HTTP/2
• In Mobile App, transition to HTTP/2 from SPDY is not
so quickly
• Android 4.x, iOS 8...
• nginx dropped SPDY in 1.9.4, but we need to
transition gracefully in a few and more years
• We need a server that speak both HTTP/2 and SPDY

31. Tengine
Now we’re trying Tengine (at a previous slide)

32. cloudflare patch?
https://blog.cloudflare.com/introducing-http2/

33. End