0
REDIS
EVERYWHERE
Ricard Clau (@ricardclau)
PHP London - 1st August 2013
HELLO WORLD
• Ricard Clau, born and grown up in Barcelona
• Software engineer at Hailo
• Symfony2 lover and PHP believer
•...
WHAT IS REDIS?
• REmote DIctionary Server
• Created in 2009 by Salvatore Sanfilipo (@antirez)
• Open source
• Advanced in-m...
TO ME REDIS IS LIKE...
AGENDA
• REDIS introduction
• Getting started with PHP / Symfony2: sessions, caching
• Cookbooks: queues, statistics, lead...
INTRODUCTIONTO REDIS
And some demystifications
DATATYPES
• Strings up to 512 Mb
• Lists of elements sorted by insertion order (max 2^32 - 1)
• Sets unordered collection ...
ONLY IN-MEMORY?
• Your data needs to fit in your memory
• It has configurable persistance: RDB snapshots,AOF
persistence log...
INTERESTING FEATURES
• Master-slave replication
• Pipelining to improve performance
• Transactions (kind of with MULTI ......
SHARDING DATA
• Redis cluster will be available in 3.0 (delayed every year...)
• Project Twemproxy can help sharding memca...
HARDWARETIPS
• Redis is single-threaded, so a 2 core machine is enough
• If using EC2, maybe m1.large (7.5Gb) is the most ...
GETTING STARTED WITH PHP
It is easier than you may think!
PHP CLIENTS
• http://redis.io/clients
• Predis (PHP) vs phpredis (PHP extension)
• Predis is very mature, actively maintai...
REDIS-CLI AND MONITOR
• Redis-cli to connect to a Redis instance
• With Monitor we can watch live activity!
SESSIONS, CACHING
• Super-easy to implement with commercial frameworks
• Commands used: GET <session-id>, SETEX <session-i...
SYMFONY2 INTEGRATION
• There must be a bundle for that...
• https://github.com/snc/SncRedisBundle
• Out of the box: Sessio...
REDIS EVERYWHERE
SOME COOKBOOKS
And comparisons with other technologies
QUEUE PROCESSING
• Using Lists and LPUSH / RPOP instructions
• We can have different languages accessing data
• We used th...
REAL-TIME ANALYTICS
• We can use hashes to group many stats under one key
• Most of the times we have counters:
HINCRBY “s...
LEADERBOARDS
• Super-easy with Redis, heavy consuming with other systems
• Each board with a single key, using sorted sets...
WHO IS ONLINE? (I)
• We can use SETS to achieve it!
• One set for every minute, SADD <minute> <userId>
Time +1m +2m +3m
2
...
WHO IS ONLINE? (II)
• Online users == everyone active in the last N minutes
• SUNION <now> <now-1> <now-2> ...
• SUNIONSTO...
FRIENDS ONLINE?
• If we store user´s friends in a SET with key friends-<user-id>
• Friends online: SINTERSTORE “friends-<u...
DISTRIBUTED LOCKING (I)
• Problem in heavy-write applications: 2 almost concurrent
requests trying to update same records
...
DISTRIBUTED LOCKING (II)
• Locking can be solved with Redis: SETNX returns true only
if key does not exist.
• PseudoCode: ...
WAR STORIES
http://apps.facebook.com/dragoncity/
(~7.5M DAU Facebook Game)
PROGRESSIVELY
• DragonCity had ~2M DAU when we introduced Redis
• First usages: Queues PHP-Python and Temporary storage
• ...
REDIS IS SUPER-FAST
• Your network gets killed much before Redis is down
• Always increase your ulimit values in Redis ser...
SURPRISES WITH MEMORY
• Redis documentation:“Sets of binary-safe strings”so,
we changed values in sets from being just num...
FINALTHOUGHTS
When and when not to consider Redis
REDIS IS PERFECT FOR...
• Intensive read-write data applications
• Temporary stored data
• Data that fits in memory
• Probl...
BUT IS NOT SUITABLE WHEN..
• Big data sets, archive data
• Relational data (RDBMS are absolutely fine to scale)
• We don´t ...
SPECIALTHANKS
• Ronny López (@ronnylt) & AlonsoVidales (@alonsovidales)
• All backend and systems engineers at @socialpoin...
QUESTIONS?
• Twitter: @ricardclau
• E-mail: ricard.clau@gmail.com
• Github: https://github.com/ricardclau
• Blog about PHP...
Upcoming SlideShare
Loading in...5
×

Redis everywhere - PHP London

20,751

Published on

2 Comments
16 Likes
Statistics
Notes
No Downloads
Views
Total Views
20,751
On Slideshare
0
From Embeds
0
Number of Embeds
25
Actions
Shares
0
Downloads
85
Comments
2
Likes
16
Embeds 0
No embeds

No notes for slide

Transcript of "Redis everywhere - PHP London"

  1. 1. REDIS EVERYWHERE Ricard Clau (@ricardclau) PHP London - 1st August 2013
  2. 2. HELLO WORLD • Ricard Clau, born and grown up in Barcelona • Software engineer at Hailo • Symfony2 lover and PHP believer • Open-source contributor, sometimes I give talks • Twitter @ricardclau / Gmail ricard.clau@gmail.com
  3. 3. WHAT IS REDIS? • REmote DIctionary Server • Created in 2009 by Salvatore Sanfilipo (@antirez) • Open source • Advanced in-memory key-value data-structure server • Stop thinking SQL, think back to structures!
  4. 4. TO ME REDIS IS LIKE...
  5. 5. AGENDA • REDIS introduction • Getting started with PHP / Symfony2: sessions, caching • Cookbooks: queues, statistics, leaderboards, users online, friends online, locking • War Stories from a 7.5M DAU project • Final thoughts
  6. 6. INTRODUCTIONTO REDIS And some demystifications
  7. 7. DATATYPES • Strings up to 512 Mb • Lists of elements sorted by insertion order (max 2^32 - 1) • Sets unordered collection of elements supporting unions, intersections and differences (max 2^32 - 1) • Hashes key-value pairs (max 2^32 - 1) • Sorted sets automatically ordered by score (max 2^32 - 1)
  8. 8. ONLY IN-MEMORY? • Your data needs to fit in your memory • It has configurable persistance: RDB snapshots,AOF persistence logs, combine both or no persistance at all • Think like memcached on steroids with persistance • http://redis.io/topics/persistence • “Memory is the new disk, disk is the new tape”
  9. 9. INTERESTING FEATURES • Master-slave replication • Pipelining to improve performance • Transactions (kind of with MULTI ... EXEC) • Publisher / Subscriber • Scripting with LUA (~stored procedures) • Predictable performance (check commands complexity)
  10. 10. SHARDING DATA • Redis cluster will be available in 3.0 (delayed every year...) • Project Twemproxy can help sharding memcached / redis • Client-side partitioning is enough for most cases • Range partitioningVS hash partitioning • Presharding can help scaling horizontally
  11. 11. HARDWARETIPS • Redis is single-threaded, so a 2 core machine is enough • If using EC2, maybe m1.large (7.5Gb) is the most suitable, but if it is not enough you might consider any m2 memory optimized type • CPU is rarely the bottleneck with Redis • Read carefully the doc to maximize performance!
  12. 12. GETTING STARTED WITH PHP It is easier than you may think!
  13. 13. PHP CLIENTS • http://redis.io/clients • Predis (PHP) vs phpredis (PHP extension) • Predis is very mature, actively maintained, feature complete, extendable, composer friendly and supports all Redis versions • Phpredis is faster, but not backwards compatible • Network latency is the biggest performance killer
  14. 14. REDIS-CLI AND MONITOR • Redis-cli to connect to a Redis instance • With Monitor we can watch live activity!
  15. 15. SESSIONS, CACHING • Super-easy to implement with commercial frameworks • Commands used: GET <session-id>, SETEX <session-id> <expires> <serialized-data> • Much better performance than PDO and also persistent! • Caching and temp-data storage work exactly equal • Be careful with temp-data storage and memory usage!
  16. 16. SYMFONY2 INTEGRATION • There must be a bundle for that... • https://github.com/snc/SncRedisBundle • Out of the box: Session management, Monolog handler, Swiftmailer spooling, Doctrine caching • Full integration with Symfony2 profiler
  17. 17. REDIS EVERYWHERE
  18. 18. SOME COOKBOOKS And comparisons with other technologies
  19. 19. QUEUE PROCESSING • Using Lists and LPUSH / RPOP instructions • We can have different languages accessing data • We used that to send OpenGraph requests to Facebook (REALLY slow API) with Python daemons • ~80k messages/minute processed with 1 m1.large EC2 • If intensive, consider using RabbitMQ,ActiveMQ, ZeroMQ...
  20. 20. REAL-TIME ANALYTICS • We can use hashes to group many stats under one key • Most of the times we have counters: HINCRBY “stats" <key> <increment> • HMGET “stats” <key1> <key2> ... to retrieve values and HMSET “stats”“stat1” <value1> “stat2” <value2> to set them • HGETALL to get the full hash
  21. 21. LEADERBOARDS • Super-easy with Redis, heavy consuming with other systems • Each board with a single key, using sorted sets • ZINCRBY “rankXXX” <increment> <userId> • Top N ZREVRANGE “rankXXX” 0 N [WITHSCORES] • We stored several milions of members under 1 key
  22. 22. WHO IS ONLINE? (I) • We can use SETS to achieve it! • One set for every minute, SADD <minute> <userId> Time +1m +2m +3m 2 2 7 2 3 1 4 1 1 1 3 5 SUNION 2 1 3 5 7 4
  23. 23. WHO IS ONLINE? (II) • Online users == everyone active in the last N minutes • SUNION <now> <now-1> <now-2> ... • SUNIONSTORE “online” <now> <now-1> <now-2> ... • Number of users: SCARD “online” • Obtain online users with SMEMBERS “online”
  24. 24. FRIENDS ONLINE? • If we store user´s friends in a SET with key friends-<user-id> • Friends online: SINTERSTORE “friends-<userid>-online” “online”“friends-<userid>” • Imagine how you would do it without Redis! 7 1 3 5 2 4 15 9 3 10 7 1 ONLINE FRIENDS-12 SINTER FRIENDS-12-ONLINE
  25. 25. DISTRIBUTED LOCKING (I) • Problem in heavy-write applications: 2 almost concurrent requests trying to update same records • SELECT FOR UPDATE? Easy to create deadlocks! timeline R1 R2 R1read R2save R2read R1save R1 updates are lost!!!!!
  26. 26. DISTRIBUTED LOCKING (II) • Locking can be solved with Redis: SETNX returns true only if key does not exist. • PseudoCode: try SETNX <lock> <time+expires>, if false check if existing lock has expired and overwrite if so. Otherwise, sleep some milliseconds and try to acquire again. Remember to release lock at the end! • 8 m1.xlarge instances to lock 7.5M Daily Active Users • Alternatives:Apache Zookeeper, perhaps better for scaling
  27. 27. WAR STORIES http://apps.facebook.com/dragoncity/ (~7.5M DAU Facebook Game)
  28. 28. PROGRESSIVELY • DragonCity had ~2M DAU when we introduced Redis • First usages: Queues PHP-Python and Temporary storage • Introduced Locking with Redis -> End of deadlocks • Used for temporary contests involving rankings • Session tracking, cheat detection and many others • About 7.5M DAU and similar amount of servers
  29. 29. REDIS IS SUPER-FAST • Your network gets killed much before Redis is down • Always increase your ulimit values in Redis servers • We had Gigabit network connectivity between machines but it was not enough to handle the data sent and retrieved • Redis was just at 20% of CPU • Do some heavy-load tests before launching!
  30. 30. SURPRISES WITH MEMORY • Redis documentation:“Sets of binary-safe strings”so, we changed values in sets from being just numbers to number + :<char> to add some info to values • We expected ~20% memory increase but in was 500% • https://github.com/antirez/redis/blob/unstable/src/t_set.c#L55 • If value can be represented as long, it is stored more efficiently • http://alonso-vidales.blogspot.co.uk/2013/06/not-all-redis- values-are-strings.html
  31. 31. FINALTHOUGHTS When and when not to consider Redis
  32. 32. REDIS IS PERFECT FOR... • Intensive read-write data applications • Temporary stored data • Data that fits in memory • Problems that fit Redis built-in data types • Predictability in performance needed (all commands complexity documented)
  33. 33. BUT IS NOT SUITABLE WHEN.. • Big data sets, archive data • Relational data (RDBMS are absolutely fine to scale) • We don´t know how we will access data • Reporting applications (no where clauses) • ALWAYS choose the right tool for the job!
  34. 34. SPECIALTHANKS • Ronny López (@ronnylt) & AlonsoVidales (@alonsovidales) • All backend and systems engineers at @socialpoint • Of course, to all of you for coming the 1st of August! • Check our open-source project Redtrine, PR welcome: https://github.com/redtrine/redtrine
  35. 35. QUESTIONS? • Twitter: @ricardclau • E-mail: ricard.clau@gmail.com • Github: https://github.com/ricardclau • Blog about PHP and Symfony2: http://www.ricardclau.com • Hailo is hiring! If you are interested, talk to us or apply!
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×