The document discusses Azure Cache and Redis. It provides an overview of Redis, including its data types, transactions, pub/sub capabilities, scripting, and sharding/partitioning. It then discusses common patterns for using Redis, such as caching, counting likes on Facebook, getting the latest reviews, rate limiting, and autocompletion. The document emphasizes that Redis is very flexible and can be used for more than just caching, acting as a general datastore. It concludes by recommending a Redis reference book for further learning.
Get more than a cache back! The Microsoft Azure Redis Cache (NDC Oslo)
1. Get more than a cache back!
The Microsoft Azure (Redis) Cache
Maarten Balliauw
@maartenballiauw
2. Who am I?
Maarten Balliauw
Antwerp, Belgium
Software Engineer, Microsoft
Founder, MyGet
AZUG
Focus on web
ASP.NET MVC, Azure, SignalR, ...
Former MVP Azure & ASPInsider
Big passion: Azure
http://blog.maartenballiauw.be
@maartenballiauw
Shameless self promotion: Pro NuGet -
http://amzn.to/pronuget2
5. Caching on Azure
Windows Server: AppFabric (“Velocity”) (2009-2010)
Azure
(ASP.NET cache)
“Shared Cache” (AppFabric cache, multi-tenant)
Self-Hosted Dedicated Cache
Managed Cache (the above, but managed by Microsoft and with an SLA)
A bit of history...
6. Caching on Azure
1920: Cache used to be a cache
2012: Cache became the go-to datastore
Twitter stores 800 tweets / timeline in Redis
New application types on Azure: PHP, Node, Java, ...
Too much work to write AppFabric clients for all!
http://www.redis.io/clients
Way easier: just make it work on Windows and Azure
New ways to develop applications
8. Redis
“Redis is an open source, BSD licensed,
networked, single-threaded, in-memory key-
value cache and store. It is often referred to as a data
structure server since keys can contain strings, hashes, lists, sets,
sorted sets, bitmaps and hyperloglogs.”
REmote DIctionary Server
9. Redis
Key-value cache and store (value can be a couple of things)
In-memory (no persistence, but you can)
Single-threaded (atomic operations & transactions)
Networked (it’s a server and it does master/slave)
Some other stuff (scripting, pub/sub, Sentinel, snapshot, …)
Things to remember
10. So no persistence?
Possible using a lot of I/O
AOF (append-only files)
RDB (Redis DB snapshots)
With or without: all your data must fit in memory
12. The Little Redis Book
http://openmymind.net/redis.pdf
By Karl Seguin
13. Data types
Type Example Key Example value
String cache:/home <html><head><title>Home page</title>
Hash categories:1 Field
name
description
numproducts
Member
Books
Books for sale
50000
Set categories:1:products 20 11 56 89 32 4
List products:20:comments [0] -> “...”
[1] -> “...”
[2] -> “...”
Sorted set products:bestsellers Field
Eye Patch
Parrot
Score
84632
82120
+ Bitmap, Hyperloglog
15. Data types
Type Example Key Example value
String cache:/home
user:nextid
<html><head><title>Home page</title>
2
Hash user:1 Field
name
handle
Member
Maarten
@maartenballiauw
Set user:1:followers 10 42 99 85
List user:1:timeline [0] -> { ... }
[1] -> { ... }
[2] -> { ... }
Sorted set trending:no:tags Field
#ndcoslo
#redis
Score
84632
82120
+ Bitmap, Hyperloglog
16. Keys
Good practice: use a pattern for keys
E.g. users:1:maarten
Get by id:
KEYS users:1:*
GET ......
Get by name:
KEYS users:*:maarten
GET ......
O(N) operation – use with caution!
19. But... Azure?
“It’s just Redis!”
Main differences: auth mechanism and SSL support
Use local for development from GitHub or NuGet
(Redis-32 or Redis-64)
22. Transactions
MULTI, EXEC, DISCARD, WATCH
No rollbacks (just discard queue)
Failures are because of you, not Redis
Optimistic locking with WATCH
Only execute transaction queue if watched keys did not change
26. Scripting
Run Lua scripts on Redis
http://www.redis.io/commands/eval
EVAL ‘return “Hello, World!”’ 0
Scripts are cached (SCRIPT FLUSH)
Scripts can be used as functions
Although we must pass the body all the time (or use SCRIPT LOAD +
EVALSHA)
Helper functions available!
Base, table, string, math, debug, struct, cjson, cmsgpack, redis.sha1hex
28. What if I need more memory?
Sharding!
On client (consistent hashing)
On server (Redis Cluster, not on Azure)
Using a proxy (Twemproxy - https://github.com/twitter/twemproxy)
var options = new ConfigurationOptions
{
EndPoints = { "my-server" },
Proxy = Proxy.Twemproxy
};
30. When can I use Redis?
ASP.NET Output Caching (RedisAspNetProviders)
ASP.NET Session State
General-purpose cache
Table Storage secondary index
“Cache with benefits”
Pub/sub
Generally: when use case maps and data fits in RAM
31. When should I avoid Redis?
More data than can fit in RAM
Can use multiple, but even then: SUM(RAM) == max. data size
Data and query model fits well in a relational model
Materialize certain queries in Redis if needed
Avoid Redis for large objects...
...with lots of concurrent read/write operations
32. Counting stuff
How would you count “likes” if you were Facebook?
Table Storage
1 entity per likeable entity + per server instance for writes, sum entities for read
I/O and CPU time...
SQL
UPDATE ...
Locking...
Redis
INCR post:12345
(Every once in a while, check keys to persist)
33. Counting stuff (2)
How would you count “likes” if you were Facebook?
And how would you get the 5 most popular posts?
Use a sorted set
Elements are scored (= # of likes)
We can use ZREVRANGE to get the top X posts
ZREVRANGE posts:likes 0 5 withscores
34. Get the latest 5 product reviews
No need to go to the database
Use a Redis List, truncated at 5 items
Need more? Query the database
35. Rate limiting
API should only allows 5 requests per 60 seconds
Redis List
Record 5 latest requests
If we’re at 5, check the leftmost item versus current time
1 2 3 4 5
12:20:25 12:20:22 12:20:21 12:20:18 12:19:50
36. What’sUp?
Sending short messages between parties, how to
implement?
Pub/Sub:
Every user gets a subscription, others can send to that subscription
Server listens on all subscriptions to archive/persist messages
When user comes online, server can publish messages since last visit from
archive/a Redis list
37. Autocompletion
How to provide autocompletion over our million-
records product catalog?
SQL “LIKE”
Lucene.NET / ElasticSearch
Redis Sorted Set
38. Intersecting collections
“Which friends do we have in common?”
Algorithm could be:
Find friends that we have in common
Find friends they have in common
Score the # of connections
39. Sort
We have a series of bugs
Priority, Status, Title, ...
A user follows a few of these bugs
How to store?
How to query? (give me my top 5 bugs by priority)
41. Conclusion
Azure Cache
Redis is not just a cache
Data types
Transactions
Pub/sub
Scripting
Sharding/partitioning
Patterns
1 thing to remember: http://openmymind.net/redis.pdf
Start server
Show config file and some of the options in there
Connect client, mention on localhost it autoconnects as I’m using default ports and such
Run some commands:
get name
set name “Maarten”
get name
getrange name 2 4
expire name 5
get name
Explain simple commands, yet powerful (e.g. substring)
Check config with “info” (also cache misses/hits etc.)
Demo some of the data types. Here’s a bunch to choose from:
String:
get cache:/home
set cache:/home "Home page"
get cache:/home
getrange cache:/home 2 5
expire cache:/home 5
get cache:/home
incr site:visits
incrby site:visits 6
Hash:
hmset categories:1 name Books numproducts 4
hgetall categories:1
hincrby categories:1 numproducts 1
hget categories:1 name
hset categories:1 numproducts 500
hgetall categories:1
Set:
sadd categories:1:products 20 11 56 89 32 4
sadd categories:1:products 99
scard categories:1:products
sadd categories:2:products 20 11
sinter categories:1:products categories:2:products
sdiff categories:1:products categories:2:products
List:
rpush products:20:comments "Awesome product!"
rpush products:20:comments "This sucks."
rpush products:20:comments "Yarr!"
lindex products:20:comments 1
llen products:20:comments
brpop products:20:comments queue 0 #blocking at the end! pub/sub?
Sorted Set:
zadd products:bestsellers 100 "Eye Patch" 10 "Parrot" 1 "Ship"
zrevrange products:bestsellers 0 1
zrevrank products:bestsellers "Ship"
zscore products:bestsellers "Ship"
Create a quick console app
Install StackExchange.Redis
ConnectionMultiplexer redis = ConnectionMultiplexer.Connect("localhost:6379"); // explain: keep this one!
var database = redis.GetDatabase(); // explain: can be disposed andall
database.StringSet("test", "Just a quick test");
Console.WriteLine(database.StringGet("test"));
Console.ReadLine();
Create a Redis cache through portal.azure.com
Explain Standard vs. Basic (replica, but data loss still possible)
Update previous sample
ConnectionMultiplexer.Connect("maartenba.redis.cache.windows.net:6379,password=/tvkIbDBTyhu8vJueOt150iP3kzIXTb/u2dTB5PxJsM=");
In the portal, show configuration of maxmemory
In the portal, show alerts and notifications
In the portal, show stats / hits / misses etc
Show this connecting using the redis-cli, too (redis-cli –h host –a pass)
And there is also... www.redsmin.com (login, explain some of the things we can see here)
Connect to localhost, perhaps flushall firstSET where “Sweden”SET conference “Unknown”SET attendees 0
Connect a second clientMULTIINCRBY attendees 100SET conference “CloudBurst”EXEC
Worked fine! Now let’s do this on client 1:WATCH attendeesMULTIINCRBY attendees 100And on client 2:SET attendees 99And on client 1:EXEC
.NET:
var transaction = redis.GetDatabase().CreateTransaction();
transaction.AddCondition(Condition.KeyExists("foo"));
transaction.StringSetAsync("foo", "bar");
bool committed = transaction.Execute();
Use two clients
Client 1: SUBSCRIBE news
Client 2: PUBLISH news “This just in”
Let’s do that in .NET:
ConnectionMultiplexer redis = ConnectionMultiplexer.Connect("localhost:6379");
ISubscriber subscriber = redis.GetSubscriber();
subscriber.Subscribe("news", (channel, message) =>
{
Console.WriteLine("News: {0}", (string)message);
});
ISubscriber publisher = redis.GetSubscriber();
publisher.Publish("news", "This just in!");
Console.ReadLine();
Send a message from one of the clients, too
Keyspace notifications:CONFIG SET notify-keyspace-events KEAPSUBSCRIBE '__key*__:*'