TLS Cross-host Session
Resumption via ngx_lua
forward secrecy, best practices and more
24/09/2015
Zi Lin zi@cloudflare.com
@lziest
#nginx #nginxconf
About Me
• System Engineer at CloudFlare Inc.
• Touch everything about TLS
• Nginx/OpenSSL
• CFSSL
• Internal system security infrastructure
2
Agenda
#nginx #nginxconf3
1
2
3
4
TLS Handshake 101
Brief introduction on TLS handshake
and session resumption
Cross-host session resumption
Why it poses an engineering problem
Session resumption by ngx_lua
designs for both session id and session
ticket
Solve session resumption at
CloudFlare
How we achieve session resumptions
with performance and forward secrecy
CC BY 2.0 image by Brenda Clarke
#nginx #nginxconf
TLS Handshake
4
source: https://youtu.be/KvxCv_yrcCY
#nginx #nginxconf
TLS Handshake
RSA
5
ClientHello
ServerHello
ClientKeyExchange
Server RandomClient Random
Encrypted
premaster secret
Premaster secret
= + +Session Key
Finished
RSA Private Key
#nginx #nginxconf
TLS Handshake
Diffie-Hellman
6
ClientHello
ServerHello
ServerKeyExchange
ClientKeyExchange
Server RandomClient Random
6
= + +Session Key
+=+=
Server DHClient DH
Finished
#nginx #nginxconf
TLS Handshake is expensive
7
#nginx #nginxconf
TLS Handshake
Performance
8
RSA Decryption ECDH
ops/sec, NIST p-curve
0
150
300
450
600
2048 4096
0
650
1300
1950
2600
256 384 521
ops/sec
#nginx #nginxconf9
TLS Handshake
Latency
Network Round-trip Latency
0
40ms
80ms
120ms
160ms
Local Coast to Coast
#nginx #nginxconf
TLS Session Resumption
ClientHello
ServerHello
Hi Server,
Remember me?

Yup, let’s
reuse the
cipher
Session Key
#nginx #nginxconf
Resumed by Session ID
11
Hi Server,
Remember me?

session #2
ClientHello
ServerHello
Yup, let’s
reuse the
cipher
lookup
session idSession Key
#nginx #nginxconf
Resumed by Session ID
12
Hi Server,
Remember me?

session #100K
ClientHello
ServerHello
Yup, let’s
reuse the
cipher
lookup
session id
10K session/sec * 3600 sec * 100 B/session = 3.6GB
Session Key
#nginx #nginxconf
Let Clients Do Storage:
Session Ticket
13
Finished
Session ticket
Keep this!
lookup
ticket key
=
#nginx #nginxconf
Resumed by Session Ticket
14
Hi Server,
Remember this?
ClientHello
ServerHello
Yup, let’s
reuse the
cipher
lookup
ticket key
=Session Key
Session Ticket
#nginx #nginxconf15
Much better
#nginx #nginxconf
Nginx supports both mechanisms!
But what if there are >1 servers?
16
#nginx #nginxconf
If each server maintains its
own states
17
TLS Handshake
Session Resumption Fail
0
0.25
0.5
0.75
1
1 2 5 10 100
Success rate
The More, The Less Merrier
#nginx #nginxconf18
#nginx #nginxconf
CloudFlare must support
both session resumption
mechanisms across hosts
19
Have to, because some IEs and most Safaris
don’t support session ticket yet.
https://www.howsmyssl.com
#nginx #nginxconf
CloudFlare must support
both session resumption
mechanisms across hosts
20
SSL sessions %
0 15 30 45 60
ID Support Ticket Support
#nginx #nginxconf
Cross-host Session Resumption
by Id
21
ClientHello
ServerHello
lookup
session id
store session
by id
#nginx #nginxconf
Cross-host Session Ticket
Resumption
22
ClientHello
ServerHello
lookup
ticket key
=
#nginx #nginxconf
Forward Secrecy
23
Hi Server,
Remember this
know ticket key,
knows all
hours later
Hi Server,
Remember this
#nginx #nginxconf
Shared ticket key rotation
24
update
ticket key
timestamp key
T_0
T_1
T_2
T_3
key schedule
#nginx #nginxconf
Resumed by session ticket
25
Yup, reuse
the cipher
Hi Server,
Remember this
btw, use this
next time
=re_encrypt( )
ticket key
rotated
#nginx #nginxconf
Forward Secrecy
26
Hi Server,
Remember this
???
hours later
Hi Server,
Remember this
#nginx #nginxconf
Cross-host Session
Resumption
27
ClientHello
ServerHello
lookup
ticket key
=
time key
T_0
T_1
T_2
T_3
update
ticket key
#nginx #nginxconf
Requirements for Cross-Host
Session Resumption
• Shared key store: memcached interface
• Performance - Non-blocking I/O
• Performance - multi-tiered caching
• Security - Session encryption; Ticket key encryption;
• Usability - Easy-to-maintain Configuration
28
#nginx #nginxconf
A new nginx conf module?
29
Shared Memcached
session store
#nginx #nginxconf30
Sample solution: Augmented nginx conf
Set "ssl_session_cache memcached:<name>[:<host>[:<port>]]"
• Blocking I/O :(
• Need to add session encryption :(
• No tests. :(
#nginx #nginxconf
• Shared session store: memcached interface
• Performance - Non-blocking I/O
• Performance - multi-tiered caching
• Security - Session encryption
• Usability - Script as Configuration
• Have Tests!
31
Let’s do it in ngx_lua

a.k.a. Openresty
#nginx #nginxconf32
ngx_lua in a nutshell
Write C callbacks in Lua
with almost no performance hit
#nginx #nginxconf33
Follow the path of ssl-cert-by-lua @agentzh
#nginx #nginxconf
OpenSSL TLS server-side
state machines
34
• OpenSSL state machine needs to have a state for non-blocking session I/O
WaitForReceive
WaitForSend
ProcessClientMessage WaitForSession
WaitForCertCallback*
*Need Nginx patch
#nginx #nginxconf
Non-blocking session I/O
with Nginx/OpenSSL
35
OpenSSL TLS
Handshake
State Machine
• Event Handler needs to know the handshake is ongoing with session I/O
#nginx #nginxconf
Minimal changes in
Nginx/OpenSSL
• OpenSSL patch, ported from BoringSSL

• Nginx patch (on top of ssl-cert-by-lua patch)
36
#nginx #nginxconf
https://github.com/
openresty/lua-nginx-module
37
branch ssl-session-by-lua on top of ssl-cert-by-lua
Development in ngx_lua
Still under internal review/testing
#nginx #nginxconf
Lua scripting for Session I/O
38
cache.lua: 250 loc, 7.5kb
memc.lua: 298 loc, 8kb
shdict.lua: 81 loc, 1.8kb
#nginx #nginxconf
Cross-host Session Ticket
Resumption
39
ClientHello
ServerHello
lookup
ticket key
=
#nginx #nginxconf
Ticket key rotation
• Rotate the ticket encryption key once a while
• Nginx embeds a key array as OpenSSL ex_data
40
[key_0, key_1, key_2, … key_n]
encryption decryption
[new-key, key_0, key_1, key_2, … key_n]
#nginx #nginxconf41
Ticket key rotation
• Lock-free key synchronization
[key_0, key_{n}, key_{n-1}, … key_1, key_0]
[key_0, new-key, key_{n-1}, …, key_1, key_0]
[new-key, new-key, key_{n-1}, …, key_1, key_0]
[new-key, key_{n-1}, new-key, …, key_1, key_0]
[new-key, key_{n-1}, key_{n-2}, …, key_0, new-key]
Key update loop
#nginx #nginxconf
Let’s modify Nginx?
• Shared ticket key store: memcached interface
• Non-blocking I/O
• Implementation of a timer
• Ticket key encryption
• Configuration
42
#nginx #nginxconf
Let’s do it in ngx_lua
43
• Shared ticket key store: memcached interface
• Non-blocking I/O
• Implementation of a timer
• Ticket key encryption
• Configuration
#nginx #nginxconf44
There is no need to modify Nginx/OpenSSL!
Only scripting in nginx.conf
Let’s do it in ngx_lua
#nginx #nginxconf45
periodically
poll
periodically
poll
HKG
SIN
Master
timestamp key
T_0
T_1
T_2
T_3
timestamp key
T_0
T_1
T_2
T_3
timestamp key
T_0
T_1
T_2
T_3
Going further at CloudFlare
#nginx #nginxconf
A Sneak Peek into
Session resumption in TLSv1.3
46
Finished
Session ticket
Keep this!
lookup
ticket key
= Session ID
#nginx #nginxconf
Cross-host Session
Resumption
47
ClientHello
ServerHello
lookup
ticket key
=
time key
T_0
T_1
T_2
T_3
update
ticket key
id
id
#nginx #nginxconf
Team
• Yichun Zhang @agentzh
• Nick Sullivan @grittygrease
• Shuxin Yang
• Jiale Zhi @_calio
• Guanlan Dai
48
#nginx #nginxconf
Recap
• Cross-host TLS session resumption by id
• New ngx_lua directives:

ssl_session_store_by_lua

ssl_session_fetch_by_lua
• Small patches in Nginx/OpenSSL
• TLSv1.3 compatible
• Cross-host TLS session ticket resumption with forward
secrecy
• Scripting in init_worker_by_lua and init_by_lua
• No need to touch Nginx or OpenSSL
• TLSv1.3 compatible
49
#nginx #nginxconf
Caching is necessary
• Can’t spend 100ms for each session retrieval
• Shared memory cache - workers
• Lua cache - single worker
• Worse case is pretty rare
50

Zi nginx conf_2015