In this talk I make an introduction to Redis, then I explain how some big names (twitter, pinterest...) are using it, then I describe some pitfalls, then I explain how we are using redis at teowaki
9. In the beginner's
mind there are many
possibilities, in the
expert's mind there
are few.
Shunryu Suzuki
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
11. Intel(R) Xeon(R) CPU E5520 @ 2.27GHz (with pipelining)
$ ./redis-benchmark -r 1000000 -n 2000000 -t get,set,lpush,lpop -P 16 -q
SET: 552,028 requests per second
GET: 707,463 requests per second
LPUSH: 767,459 requests per second
LPOP: 770,119 requests per second
Intel(R) Xeon(R) CPU E5520 @ 2.27GHz (without pipelining)
$ ./redis-benchmark -r 1000000 -n 2000000 -t get,set,lpush,lpop -q
SET: 122,556 requests per second
GET: 123,601 requests per second
LPUSH: 136,752 requests per second
LPOP: 132,424 requests per second
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
12. myth: arrrrRedis is just
like memcached
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
13. myth: rrrrRedis is for
queues
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
14. open source, BSD licensed, advanced
key-value store. It is often referred to as a
data structure server since keys can contain
strings, hashes, lists, sets and sorted sets.
http://redis.io
started in 2009 by Salvatore Sanfilippo @antirez
100 contributors at
https://github.com/antirez/redis
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
16. The arrrrRedis Manifesto
1.A DSL for Abstract Data Types
2.Memory storage is #1
3.Fundamental data structures for a
fundamental API
4.Two levels of API
5.Code is like a poem; it's not just something
we write to reach some practical result
6.We're against complexity
7.We optimize for joy
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
17. basics: setting and getting
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
21. basics: sets
cool example about sunionstore
http://robots.thoughtbot.com/post/46335890055/redis-set-intersection-using-sets-to-filter-data
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
23. Perrrrrsistance: RDB
Compact binary format
Saves snapshots every few minutes
Good for backups and synchronizing
If Redis crashes, a few minutes worth of
data will be lost
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
24. Durrrrrability: AOF
Log text format
Configurable durability
Large file, can slow down startup
If Redis crashes, typically one second of
data could be lost
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
26. atomicity
single threaded, so no
concurrency problems
transactions and lua
scripts to run multiple
operations atomicallyjavier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
27. Scrrrrripting with lua
You can use Lua for scripting Redis when you need to
atomically execute a sequence of commands in which
the output of a command is used as input for another
It reduces the need to use complex lock mechanisms
and simplifies dependencies between clients
You can even extend the functionality of Redis by using
Lua scripts
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
28. what's being used forrrrr
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
29. twitterrrrr
Every time line (800 tweets
per user) is on redis
5000 writes per second avg
300K reads per second
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
30. twitter
user info from
gizmoduck
(memcached)
user id tweet id metadata
write API (from browser or client app)
rpushx to Redis
tweet info from
tweetypie
(memcached + mysql)
your twitter
timeline
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
fanout (flockDB)
one per follower
31. pinterest object grrrrraph
from mysql+memcached
to redis
30% save on IOps
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
32. per userrrrr
Redis SortedSet, with timestamp as the score, is used to
store the users followed explicitly.
Redis SortedSet, with timestamp as the score, is used to
store the users followed implicitly
Redis SortedSet, with timestamp as the score, is used to
store the user’s explicit followers
Redis SortedSet, with timestamp as the score, is used to
store the user’s implicit followers
Redis Set is used to store boards followed explicitly
Redis Set is used to store boards unfollowed explicitly
per boarrrrrd
Redis Hash is used to store a board’s explicit followers
Redis Set is used to store a board’s explicit unfollowers
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
33. viacom
Object dependency graph. Cache on steroids
Redis as a queue for background jobs
Activity tracking and view counts buffering before saving
to mysql
Lua script working on a slave to recalculate ranking and
popularity of contents, then send the data to master.
The new process takes 1/60th less than the old one in
mysql
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
34. openredis
redis as a service on AWS
serving over 1000
customers with a single
machine
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
35. when things go wrong
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
36. the instagrrrrram case
moving from redis to
cassandra: 75% savings on
servers
lesson learnt:
know when redis is not appropriate
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
37. the twilio case
credit card hell
lesson learnt:
know what you are doing.
Don't change config on the fly
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
38. how teowaki is using redis
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
39. Abusing sidekiq/resque
keep track of every activity in the system, even if you
don't need them all right now:
- every page view
- every API request
- every time a record is created/updated/deleted
benefits:
- highly decoupled system
- easier to divide into services
- you can add behaviour without changing your app
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
40. Some of our uses of Lua
Expiring attributes inside a Redis hash
Inserting notifications into a list only if there are not
pending notifications from the same user for the
same scope
Paginating a list by an attribute
Manipulating JSON directly at the Redis layer
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
41. counterrrrrs
Atomic counters can be safely invoked concurrently
from anywhere, so you can implement “like”
features, global sequences or usage monitoring
systems in highly concurrent applications for free.
You can share your counters with any other internal
application and still be sure they won't collide.
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
42. Temporarrrrry data
Redis allows us to self expire keys after a time has
passed. You can use this mechanism for a simple
cache
If you operate on a key with expiration time, you
can change its value and still keep the expiration
going. By combining a decrementing counter with
an expiration time, implementint usage quotas is
trivial
We are controlling our API usage quota with a lua
script following this pattern
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
43. bloom filterrrrrs
bloom filter: space-efficient probabilistic data
structure that is used to test whether an element is
a member of a set. False positive matches are
possible, but false negatives are not.
Redis bit operations make easy to implement bloom
filters
We are using bloom filters for checking uniqueness
of user names without going to postgresql
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
44. nginx + lua + redis
apache + mruby + redis
Multiple levels of cache by using Redis on the webserver/
middleware layer
http://wiki.nginx.org/HttpRedis
https://github.com/jodosha/redis-store
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
45. summarrrrrizing
* Redis is more powerful than it seems
* Very fast, easy to use, simple, good documentation
* In-memory data structures, distributed, shared and persisted
* Good as data store, intermediate data store, cache or queue
* Lots of use cases, both in huge and smaller systems
You should probably use it a lot more
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
46. If you enjoyed this presentation, please thank
me by registering on http://teowaki.com
It's a site for developers, you can hang around for
free, and I think it's quite cool
<3 <3 <3
Javier Ramírez
@supercoco9
arrrrcamp 2013