Secure Redis Cluster At Box: Vova Galchenko, Ravitej Sistla
1. PRESENTED BY
Secure Redis Cluster at Box
Vova Galchenko & Ravitej Sistla
Box Inc., Database and Cache Infrastructure Team
2. PRESENTED BY
1 Redis Cluster at Box
What is Box and how we use Redis
2 Security features within Redis Cluster
What’s available in Redis Cluster out of the box
3 Introducing Secure Redis Proxy
How we secure our Redis Clusters
Agenda:
4 Secure Redis Proxy in practice
Operational characteristics, open issues and learnings
5. PRESENTED BY
Database & Cache Infra Team
Provide highly available, consistent, performant and easy-
to-use infrastructure for online transaction processing
6. PRESENTED BY
Redis Cluster at Box
CACHE SOURCE OF TRUTH
• Transient
• Fast
• Synchronized with external
source of truth
• Persistent
• Redundant
• Capable of handling diverse
access patterns
7. PRESENTED BY
Redis Cluster at Box: Cache Case Study
PHP Clients
Data Access Service
Java Clients Scala Clients
Data Access Service
8. PRESENTED BY
Redis Cluster at Box: Cache Case Study
Basic APIs Rich APIs
Consistent
Performance
Spiky
Performance
Operationally
Simple
Operationally
Complex
9. PRESENTED BY
Redis Cluster at Box: Cache Case Study
Cache relationships between objects:
• For each relationship differentiate cache values by relationship facet
• Be able to clear all facets of a single relationship quickly
D
A B C
03/2019 02/2019 01/2019
Folder D’s files ordered by file name: A, B, C
Folder D’s files ordered by last modified date: C, B, A
Folder D’s files larger than 100KB ordered by size: C, B
10KB 12.5MB 7MB
X
01/2019
7MB
10. PRESENTED BY
Redis Cluster at Box: Cache Case Study
Cache relationships between objects with the following requirements:
• For each relationship differentiate cache values by relationship facet
• Be able to clear all facets of a single relationship quickly
field value
Key: [“folder”, “folder_id”, “D”, “files”]
11. PRESENTED BY
Redis Cluster at Box: Cache Case Study
Cache relationships between objects with the following requirements:
• For each relationship differentiate cache values by relationship facet
• Be able to clear all facets of a single relationship quickly
field value
{“where”: ”none”, “orderby”: [“file_name”, “ASC”]} A, B, C
Key: [“folder”, “folder_id”, “D”, “files”]
12. PRESENTED BY
Redis Cluster at Box: Cache Case Study
Cache relationships between objects with the following requirements:
• For each relationship differentiate cache values by relationship facet
• Be able to clear all facets of a single relationship quickly
field value
{“where”: ”none”, “orderby”: [“file_name”, “ASC”]} A, B, C
{“where”: “none”, “orderby”: [“mod_date”, “ASC”]} C, B, A
Key: [“folder”, “folder_id”, “D”, “files”]
13. PRESENTED BY
Redis Cluster at Box: Cache Case Study
Cache relationships between objects with the following requirements:
• For each relationship differentiate cache values by relationship facet
• Be able to clear all facets of a single relationship quickly
field value
{“where”: ”none”, “orderby”: [“file_name”, “ASC”]} A, B, C
{“where”: “none”, “orderby”: [“mod_date”, “ASC”]} C, B, A
{“where”: “size > 100”, “orderby”: [“size”, “ASC”]} C, B
Key: [“folder”, “folder_id”, ”D”, “files”]
14. PRESENTED BY
Redis Cluster at Box: Security & Compliance Requirements
• Access to every data store must be authenticated
• Interactions with every data store must be encrypted
• Credentials must be rotated regularly
16. PRESENTED BY
Redis Philosophy on Security
Redis is designed to be accessed by trusted clients inside trusted environments.
“
https://redis.io/topics/security
17. PRESENTED BY
Redis Philosophy on Security
Redis is not optimized for maximum security but for
maximum performance and simplicity
“
https://redis.io/topics/security
18. PRESENTED BY
Client-side Encryption
• Key rotation problematic in use-cases where Redis is used as a system of record
• Doesn’t protect against a malicious actor corrupting the dataset
• Prevents effective use of ordered data types, i.e. Sorted Sets
• Can work well for simple Redis-as-a-cache use-cases
Plain TCP
19. PRESENTED BY
Security options within Redis: Encryption
• No native solution exists
• Tunneling solutions, such as stunnel and spiped are generally recommended
20. PRESENTED BY
Security options within Redis: Authentication
The requirepass and masterauth configuration directives allow setting a single
password to protect access to the cluster
• Cluster members must rotate passwords synchronously
• Cluster clients must rotate passwords synchronously
21. PRESENTED BY
Security options within Redis: Authentication
The requirepass and masterauth configuration directives allows setting a single
password to protect access to the cluster
• Cluster members must rotate passwords synchronously
• Cluster clients must rotate passwords synchronously
23. PRESENTED BY
• Add authentication to Redis Cluster with online credential rotation support
• Add encryption to Redis Cluster connections with online key rotation support
• Minimize operational impact to the rest of the infrastructure
− Minimize latency impact (< 2ms per operation)
− No downtime when proxy deploys
− Minimize changes for existing clients
Goals
30. PRESENTED BY
Clients Secure Redis Proxy Redis Instance
Send Auth Request
Authenticated Access using Secure Redis Proxy
31. PRESENTED BY
Clients Secure Redis Proxy Redis Instance
Send Auth Request
Auth Success
Authenticated Access using Secure Redis Proxy
32. PRESENTED BY
Clients Secure Redis Proxy Redis Instance
Send Auth Request
Auth Success
GET Command
Authenticated Access using Secure Redis Proxy
33. PRESENTED BY
Clients Secure Redis Proxy Redis Instance
Send Auth Request
Auth Success
GET Command GET Command
Authenticated Access using Secure Redis Proxy
34. PRESENTED BY
Clients Secure Redis Proxy Redis Instance
Send Auth Request
Auth Success
GET Command GET Command
GET Response
Authenticated Access using Secure Redis Proxy
35. PRESENTED BY
Clients Secure Redis Proxy Redis Instance
Send Auth Request
Auth Success
GET Command GET Command
GET Response
GET Response
Authenticated Access using Secure Redis Proxy
36. PRESENTED BY
• Secure Redis Proxy can read and validate multiple passwords provided by Vault
• Passwords are periodically rotated and recycled using the following procedure
1. Add new password to Vault
2. Secure Redis Proxy accepts both old and new passwords
3. Clients migrate to use the new password
4. Old password is retired from Vault
Password Rotation
37. PRESENTED BY
• Sudden restart of Secure Proxy might cause a flood of reconnections from clients to
Redis leading to thundering herd problem
• We deploy two Secure Proxies on each system and swing the traffic between the
processes using iptables rules.
• To handle long living connections to Secure proxy, we issue a drain connections
command to the retiring Secure Proxy process that gradually disconnects the
connections
Upgrades
41. PRESENTED BY
• Cluster bus communications are on a different port and they fly under the radar
• No sensitive information is exchanged on this channel
• User data is exchanged between cluster nodes via replication commands on the
standard Redis port
• The design presented so far will block this traffic preventing the cluster from operating effectively
• Potential solution
• Pass all replication commands originating within the cluster through Secure Redis Proxy without
enforcing authentication or encryption
What about Cluster Communication?
65. PRESENTED BY
• Written in Golang
• Currently in production:
• 20k new client connections per second per Redis host
• 15k concurrent connections per Redis host
• 5 GB/s bytes per Redis host at peaks
• Memory stats
− < 200 MB Heap size
− 50 MB/s allocation rate
− < 0.5ms per GC pause
• Latency Stats
− ~0.5ms latency overhead per plain text request
• We are currently working on enabling TLS on all proxy connections
Operational Characteristics
72. PRESENTED BY
Work in progress for TLS support
• Use golang 1.12.1 for better memory performance for Encryption and decryption
• Use TLS 1.3 for better handshake performance
• If using TLS 1.2
• enable TLS False Start.
• configure session resumption (not so helpful in our case)
• enable forward secrecy
73. PRESENTED BY
TLS (Transport Layer Security)
Ref: http://blog.fourthbit.com/2014/12/23/traffic-analysis-of-an-ssl-slash-tls-session
74. PRESENTED BY
TLS (Transport Layer Security)
Ref: http://blog.fourthbit.com/2014/12/23/traffic-analysis-of-an-ssl-slash-tls-session
75. PRESENTED BY
Secure Redis Proxy on TLS
Ref: http://blog.fourthbit.com/2014/12/23/traffic-analysis-of-an-ssl-slash-tls-session
Yes
No
New connection from
client on RPX
Read First 2 bytes
Initiate TLS connection
Complete HandshakeRead bytes
Continue
Handshake
