3. 1/27/2017 3
•Written in: C
•Main point: Blazing fast
•License: BSD
•Protocol: Telnet-like, binary safe
•Disk-backed in-memory database,
4. 4
in-memory data structure store
Redis is an advanced key-value store, where keys can contain data
structures such as strings, hashes, lists, sets, and sorted sets. Supporting
a set of atomic operations on these data types.
Redis can persist data to the disk Redis is not only a key-value
store
Can be used as Database, a Caching layer or a Message broker.
Redis is fast
Official webpage: http://redis.io
Source Code: https://github.com/antirez/redis
Online Demo: http://try.redis.io
5. Redis is not a replacement for Relational Databases nor Document
Stores.
It might be used complementary to a SQL relational store, and/or
NoSQL document store.
Best Used: For rapidly changing data with a foreseeable database
size (should fit mostly in memory).
5
What Redis is not...
6. Written in ANSI C by Salvatore Sanfilippo (@antirez).
Works in most POSIX systems like Linux, BSD and OS X.
Linux is the recommended
No official support for Windows, but Microsoft develops and
maintains an open source Win-64 port of Redis*
Redis is a single-threaded server, not designed to benefit from
multiple CPU cores. Several Redis instances can be launched to
scale out on several cores.
All operations are atomic (no two commands can run at the same
time).
It executes most commands in O(1) complexity and with minimal
lines of code.
1/27/2017 6
8. Caching
Pub/sub
Blocking/Delayed Queues
o Such as https://github.com/resque/resque
o http://techblog.netflix.com/2016/08/distributed-delay-queues-based-
on.html
Short lived items, with ttl
o Fraud detection
o User Sessions
o Request Filtering Service
Counting Reviews most bought items
Real time Analysis
Storing Unique items over time.
8
Use cases for Redis
9. Speed is key
More than just key-value pairs
Redis is an in memory data store, so there are memory
limitations.
o An empty instance uses ~ 1MB of memory.
o 1 Million small Keys -> String Value pairs use ~ 100MB of
memory.
o 1 Million Keys -> Hash value, representing an object with 5
fields, use ~ 200 MB of memory.
Dataset is not critical
1/27/2017 9
12. Stat’s From Twitter(2014)
Timeline Service for one datacenter using Hybrid List:
o ~40TB allocated heap
o ~30MM qps
o > 6,000 instances
Use of BTree in one datacenter:
o ~65TB allocated heap
o ~9MM qps
o >4,000 instances
1/27/2017 12
13. 1/27/2017 13
Redis Data Type Contains Read/write ability
String
Binary-safe strings (up to 512 MB),
Integers or Floating point values,
Bitmaps.
Operate on the whole string, parts,
increment/decrement the integers and floats,
get/set bits by position.
Hash
Unordered hash table of keys to
string values
Add, fetch, or remove individual ítems by key,
fetch the whole hash.
List Doubly linked list of strings
Push or pop items from both ends, trim based
on offsets, read individual or multiple items,
find or remove items by value.
Set
Unordered collection of unique
strings
Add, fetch, or remove individual items, check
membership, intersect, union, difference, fetch
random items.
Sorted Set
Ordered mapping of string members
to floating-point scores, ordered by
score
Add, fetch, or remove individual items, fetch
items based on score ranges or member value.
Geospatial
index
Sorted set implementation using
geospatial information as the score
Add, fetch or remove individual items, search
by coordinates and radius, calculate distance.
HyperLogLog
Probabilistic data structure to count
unique things using 12Kb of
memory
Add individual or multiple items, get the
cardinality.(http://antirez.com/news/75)
14. Persistence
Redis provides two mechanisms to deal with persistence: Redis database snapshots (RDB) and
append-only files (AOF).
Replication
A Redis instance known as the master, ensures that one or more instances kwown as the slaves, become
exact copies of the master. Clients can connect to the master or to the slaves. Slaves are read only by
default.
Partitioning
Breaking up data and distributing it across different hosts in a cluster.
Can be implemented in different layers:
o Client: Partitioning on client-side code.
o Proxy: An extra layer that proxies all redis queries and performs partitioning (i.e.
Twemproxy,Dynomite).
o Query Router: instances will make sure to forward the query to the right node. (i.e. Redis Cluster).
o Redis Cluster is a mix between query routing and client side partitioning.
Failover
o Manual
o Automatic with Redis Sentinel (for master-slave topology)
o Automatic with Redis Cluster (for cluster topology)
1/27/2017 14
15. 15
Redis Persistence(Read more about it at
https://redis.io/topics/persistence )
• RDB
o performs point-in-time snapshots of your dataset at specified intervals
• AOF
o AOF persistence logs every write operation received by the serve
• If you wish, you can disable persistence at all, if you want your data to just exist as
long as the server is running
Master not persisting, Slave Persisting to disk.
• It is possible to combine both AOF and RDB in the same instance.
16. 16
RDB advantages
o RDB is a very compact single-file point-in-time representation of your Redis data.
o RDB is very good for disaster recovery, being a single compact file can be transferred
to far data centers
o RDB maximizes Redis performances since the only work the Redis parent process
needs to do in order to persist is forking a child that will do all the rest. The parent
instance will never perform disk I/O or alike
o RDB allows faster restarts with big datasets compared to AOF.
RDB disadvantages
• RDB is NOT good if you need to minimize the chance of data loss in case Redis stops
working
o Fork() can be time consuming if the dataset is big, and may result in Redis to stop
serving clients for some millisecond or even for one second if the dataset is very big
and the CPU performance not great
17. o AOF advantages
• Using AOF Redis is much more durable: you can have different fsync policies: no fsync at all,
fsync every second, fsync at every query.
• The AOF log is an append only log, so there are no seeks, nor corruption problems if there is a
power outage
• Redis is able to automatically rewrite the AOF in background when it gets too big
• AOF contains a log of all the operations one after the other in an easy to understand and parse
format
o AOF Disadvantages
• AOF files are usually bigger than the equivalent RDB files for the same dataset
• AOF can be slower than RDB depending on the exact fsync policy. In general with fsync set
to every second performances are still very high.
1/27/2017 17
18. Partitioning in Redis serves two main goals:
It allows for much larger databases, using the sum of the memory of many computers. Without partitioning
you are limited to the amount of memory a single computer can support.
It allows scaling the computational power to multiple cores and multiple computers, and the network
bandwidth to multiple computers and network adapters.
Client side partitioning means that the clients directly select the right node where to
write or read a given key. Many Redis clients implement client side partitioning.e.g
JedisSharding
Proxy assisted partitioning means that our clients send requests to a proxy that is able to
speak the Redis protocol, instead of sending requests directly to the right Redis instance.
The proxy will make sure to forward our request to the right Redis
proxy Twemproxy implements proxy assisted partitioning.
Query routing means that you can send your query to a random instance, and the
instance will make sure to forward your query to the right node
1/27/2017Hystrix 18
20. The key space is split into 16384 slots,
Effectively setting an upper limit for the cluster size of 16384
master nodes (however the suggested max size of nodes is in
the order of ~ 1000 nodes).
HASH_SLOT = CRC16(key) mod 16384
When the cluster is stable, a single hash slot will be served by
a single node
https://redis.io/topics/cluster-spec#keys-distribution-model
1/27/2017 20
21. Redis Cluster supports the ability to add and remove nodes while
the cluster is running. (https://redis.io/topics/cluster-spec)
Adding or removing a node is abstracted into the same operation:
moving a hash slot from one node to another.
o To add a new node to the cluster an empty node is added to the cluster
and some set of hash slots are moved from existing nodes to the new
node.
o To remove a node from the cluster the hash slots assigned to that node
are moved to other existing nodes.
o To rebalance the cluster a given set of hash slots are moved between
nodes.
1/27/2017 21
22. 1/27/2017 22
Twemproxy (single)
Twemproxy* works as a proxy between the clients and many Redis instances.
Is able to automatically distribute data among different standalone Redis instances.
Supports consistent hashing with different strategies and hashing functions.
Multi-key commands and transactions are not supported.
Twemproxy (load balanced)
*Twemproxy is a project from Twitter and is not part of redis: https://github.com/twitter/twemproxy
**Illustrations from Redis Essentials book by Maxwell Dayvson Da Silva and Hugo Lopes Tavares
23. 1/27/2017Dynomite 23
In the age of high scalability and big data,
Dynomite’s design goal is to turn those
single-server datastore solutions into peer-
to-peer, linearly scalable, clustered
systems while still preserving the native
client/server protocols of the datastores,
e.g., Redis protocol.
24. o Manual
o Automatic with Redis Sentinel (for master-slave topology)
o Automatic with Redis Cluster (for cluster topology)
1/27/2017 24
30. Spring Data Redis project supports Jedis and Lettuce, two of
the most popular java clients for Redis.
Other Java Redis client of note is Redisson.
1/27/2017 30
31. A Request/Response server can be implemented so that it is able
to process new requests even if the client didn't already read the
old responses. This way it is possible to send multiple
commands to the server without waiting for the replies at all, and
finally read the replies in a single step.
Lettuce client supports pipelining on redis cluster
Jedis does not.
Without pipelining 1.185238 seconds with pipelining 0.250783
seconds
https://redis.io/topics/pipelining
1/27/2017 31