Your SlideShare is downloading. ×
Redis/Lessons learned
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Redis/Lessons learned

4,762
views

Published on

An insight into NoSQL solutions implemented at RTV Slovenia and elsewhere, what problems we are trying to solve and an introduction to solving them with Redis. …

An insight into NoSQL solutions implemented at RTV Slovenia and elsewhere, what problems we are trying to solve and an introduction to solving them with Redis.

Talk given at #wwwh @ Ljubljana, 30.1.2013 by me, Tit Petric


1 Comment
8 Likes
Statistics
Notes
No Downloads
Views
Total Views
4,762
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
90
Comments
1
Likes
8
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Lessons learned: NoSQL/Redis Tit Petric / Jan 2013 Monotek d.o.o.
  • 2. NoSQL: why it’s hereSQL:- Slow query performance- Concurrency / locking- Hard to scale (even harder for writes, storage)Typical problems- Session storage- Statistics (high write to read ratio)- Modifying schema on large data setsTit Petric / Twitter @titpetric
  • 3. NoSQL: memcacheMemcache (RTV 2008-Present)- Pro: stability, speed- Pro: simple text protocol (added binary to fuck with us)- Love/Hate: Scaling out reads/writes- Con: Persistence- Con: Replication- Con: Key eviction not based on LRU/TTL but slab allocationTit Petric / Twitter @titpetric
  • 4. NoSQL: sharedanceSharedance (2009-2011)- Pro: Persistent KV storage- Pro: Simple text protocol (wrote a client in LUA)- Con: Had to patch daemon to handle eviction load (1 key = 1 file, filesystems can’t handle this)- Con: Had to use special ReiserFS filesystem on deploymentTit Petric / Twitter @titpetric
  • 5. NoSQL: redisRedis (2011-Present)- Pro: stability, speed- Pro: simple text protocol, documentation, clients- Love: Scaling out reads/writes- Pro: Persistence- Pro: Replication- Con/Feature request: key segmentation / grouping.Tit Petric / Twitter @titpetric
  • 6. NoSQL: uses at RTV SloveniaTo-do (2013-?)- Memcached protocol translator to Redis- Look at twemcached, avoid client based sharding- Implement webdis deployment- Redis scripting with LUATit Petric / Twitter @titpetric
  • 7. NoSQL: Redis data typesRedis is still a Key/Value store!Only we can have different values: - Strings (essentialy the same as memcache) - Hashes (nested k/v pairs) - Lists (simple arrays) - Sets (unique arrays) - Sorted sets (weighted unique arrays)Tit Petric / Twitter @titpetric
  • 8. NoSQL: Redis data types LETS SEE SOME EXAMPLES! Tit Petric / Twitter @titpetric
  • 9. Redis data types: StringsLimiting Google bot crawl rate “A 503 (Service Unavailable) error will result in fairly frequent retrying. To temporarily suspend crawling, it is recommended to serve a 503 HTTP result code.”Tit Petric / Twitter @titpetric
  • 10. Redis data types: StringsLimiting Google bot crawl rateSETNX – set a key if it doesn’t existEXPIRE – expire a key after TTL secondsINCR – increment value by oneTit Petric / Twitter @titpetric
  • 11. Redis data types: StringsLimiting Google bot crawl rateTit Petric / Twitter @titpetric
  • 12. Redis data types: HashesNews ratingsHMSET – set multiple hash fields / valuesHGETALL – get all fields and valuesHINCRBY – increment integer value of a hashfieldTit Petric / Twitter @titpetric
  • 13. Redis data types: HashesNews ratingsTit Petric / Twitter @titpetric
  • 14. Redis data types: HashesNews ratingsVote data Why Expire? Race condition. We need HMSETNX Could be better.Tit Petric / Twitter @titpetric
  • 15. Redis data types: HashesOther use cases include:- User data, partial data retrieval - select username, realname, birthday from users where id=? - HMGET users:$id username realname birthday- Using SORT (list|set) BY hash values- Don’t use HASHes to store session. Eviction policy works on KEYS not on hash values!Tit Petric / Twitter @titpetric
  • 16. Redis data types: ListsAny kind of information log (statistics,…)LPUSH – push values to the beginning of the listRPUSH – push values to the end of the listLRANGE – get a range of valuesLTRIM – trim a list to the specified rangeLLEN – get the length of the listTit Petric / Twitter @titpetric
  • 17. Redis data types: ListsCollecting statisticsWe can skip the database completelyTit Petric / Twitter @titpetric
  • 18. Redis data types: ListsProcess dataInto SQLdatabaseTit Petric / Twitter @titpetric
  • 19. Redis data types: ListsWell, it’s a way to scale writes to SQL Processing job can DIE for ages, because:- Back of the envelope calculation for redis memory use: 100M keys use 16 GB ram- Logs get processed in small chunks (200 items), avoiding memory limits. Could increase this by a lot.- We also use sharding so writes are distributed per $tableTit Petric / Twitter @titpetric
  • 20. Redis data types: SetsSet values are UNIQUESADD – Add one or more members to a setPerfect use case: set insersection withSINTERSTORE, find duplicates.MySQL is too slow for this, even withindexes…Tit Petric / Twitter @titpetric
  • 21. Redis data types: SetsSET Intersection in MySQLList1 = first table of dataList2 = second table of dataTit Petric / Twitter @titpetric
  • 22. Redis data types: SetsBulk transfer MySQL data to redisVia: http://dcw.ca/blog/2013/01/02/mysql-to-redis-in-one-step/Tit Petric / Twitter @titpetric
  • 23. Redis data types: SetsSET Intersection in RedisMuch faster, without indexes!0.118 seconds vs. mysql 1.35 (+0.36 for index)15x speed increase!Tit Petric / Twitter @titpetric
  • 24. Redis data types: SetsOther possible uses for sets:• Common friends between A and B• Friend suggestions (You might know…)• People currently online …Tit Petric / Twitter @titpetric
  • 25. Redis data types: Sets vs. Sorted setsOk, typical use case in sql>select title, content from news order bystamp desc limit 0,10#1) Use SORT from redis + HMGET#2) Use sorted sets (ZSET type)Tit Petric / Twitter @titpetric
  • 26. Redis data types: Sorted setsSorted sets by time with a PKauto_increment? NO!• Most read news items (sort by views)• Order comments by comment rating• Friends by most friends in commonTit Petric / Twitter @titpetric
  • 27. Redis data types: Sorted setsOrder comments by ratingZINCRBY – increase/decrease score of itemZRANGE – return portion of sorted set, ASCZREVRANGE – portion of sorted set, DESCTit Petric / Twitter @titpetric
  • 28. Redis data types: Sorted setsSort comments by rating! With pagination!ZRANGE – return portion of sorted set, ASCZREVRANGE – portion of sorted set, DESCTit Petric / Twitter @titpetric
  • 29. Scaling Redis deploymentSLAVEOF [host] [port]Starts replicating from [host]:[port], making this instance a slaveSLAVEOF NO ONEPromote instance to MASTER roleTit Petric / Twitter @titpetric
  • 30. Scaling Redis deploymentPhpredis client does not implementsharding by itself! But …- Master / Multi-slave scaling is easy to do- Failover for reads is easy, node ejection possible- Client deploys still take time – twemproxy is an option- Twemproxy also provides sharding support, & Memcached - Want to see what Redis is doing? Issue “MONITOR” command.- Stale data is better than no data, we still consider Redis volatile- FlushDB = rebuild cache, we tolerate data lossTit Petric / Twitter @titpetric
  • 31. Redis: Q & A section Questions and answers! Follow me on Twitter: @titpetric Read our tech blog: http://foreach.orgTit Petric / Twitter @titpetric