RedisConf17- Redis as a Primary Data Store
•Download as PPTX, PDF•
2 likes•4,009 views
RedisConf17 breakout session
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to RedisConf17- Redis as a Primary Data Store
Similar to RedisConf17- Redis as a Primary Data Store (20)
More from Redis Labs
More from Redis Labs (20)
Recently uploaded
Recently uploaded (20)
RedisConf17- Redis as a Primary Data Store
- 1. Redis as a Primary Data Store John Loehrer Joya Communications
- 2. Isn’t Redis just a cache? often compared to memcached
- 3. Wait, not so fast …
- 4. Data Modeling I can haz schema?
- 5. “You must unlearn what you have learned.”
- 7. Simple Data Types can build complex systems
- 8. Foriegn Key Constraints Don’t do it! They don’t scale.
- 9. Persistence Redis writes to disk
- 10. Replication Easy synchronization for failover
- 11. HA many options for high availability
- 12. BUT, RAM IS EXPENSIVE! still less than other hidden costs
- 13. Hidden Costs Time is money
- 14. • Query Tuning • System Tuning • Complex Backup/Restore • Still need to add Caching + invalidation • Sharding Logic Scaling Databases
- 15. RAM doesn’t have to be cheap just affordable
- 16. Easy Prototyping
- 17. Hashes For Records Build more complex data types on top of simple key-value pairs
- 18. More Complex Data • consider flattening the data • Use LUA scripting to safely modify json • New Json Module is exciting
- 19. Sorted Sets for Indexes for things where the sort order of a list changes
- 21. Pack Multiple Records into one Hash given a record id like 12345678, do: HSET MyKeyspace{1234} 5678:field1 value1
- 22. Fast Unit Testing using redislite python package
- 23. TDD really is possible run tests instantly
- 24. Write Lots of Tests 100k+ redis queries
- 25. Lean Stack fewer things for OPS to manage
- 26. Redis Just Works but still expect failures from underlying hardware
- 27. No Query Tuning But you may need to pipeline
- 28. OK to use GUIDs we use a 22-character url-safe base64 variant
- 29. Map GUIDS to an internal id? when you reference GUIDs in other fields, it adds up
- 30. Scaling Redis
- 31. Where are the bottlenecks? we are more constrained by RAM than CPU
- 32. Partition Vertically keyspaces mapped to different redis backends
- 33. Partition Horizontally use redis-cluster, twemproxy, etc
- 34. Partition Geographically Keys prefixed with region for intelligent routing
- 35. Monitor your Keyspaces scan, sample, get stats
- 37. Time Series Data move old data into compressed files on S3
- 38. Infrequent Access shuttle cold keys out of redis and repopulate on demand
- 39. Plan for Failure practice makes perfect (kinda)
- 40. Test How app responds to Failures expect incomplete data occasionally
- 41. Don’t Expect Transactionality expect cases where data and index are inconsistent
- 42. Restore 800 GB of data in 15 mins check out redisimp!
- 43. Inappropriate Uses • Banking • Archiving of TB data sets • Nuclear Missile Silo Security
- 44. Questions?
Editor's Notes
- Story of meeting the Lolcats creator
- story about simplicity and building complex systems with simple tools.
- Story of my first big database project bill of materials into a label for vitamin supplement company
- Story from days at gaia trying to rebuild mySQL and getting it to restart after a crash.
- Story of fixing MySQL master-master replication, snapshotting, advancing the binlog skipping over conflicts AGHHHH! Redis replication allows you to just point at the source and it does the right thing.
- Tell story of project I RoR project consulted on. Unit tests took 5 minutes to spin up and run Couldn’t easily run one test Required setup/teardown of MySQL which was slow Schema was brittle, required lots of setup
- I wish we could say we started out this way. The first big code push we wrote code frantically and had no unit tests. I remember just trying to eyeball api requests and responses from our mobile client to determine if we got the logic right. Took hours. We thought we were saving time by not writing tests but actually we wasted so much more. Don’t write tests just to write tests. Use it to do all the things that you would have to check by hand each time otherwise.
- One-man-band kind of ops team. Until last november I was the only person running ops. We were able to keep that kind of lean operational budget because the architecture was so simple and because it didn’t cause problems.
- on dev they can map to same redis db, and fan it out on production as you scale. easy to replicate a database to another then swap one keyspace over and drop the unused keyspaces on each new database afterward.
- Story about an old keyspace no longer being used: 10% of redis memory Another cleanup case: realized we didn’t need a certain data structure after it was 3 months old 30% savings Being strategic about what data you expire can make a big difference Don’t store data you don’t need Do log it or store in another mechanism where you can rehydrate it back in if you need We use our analytics service to capture all event data that is important, and we can crawl that in redshift to build reports that can be scripted to re-hydrate and backfill data when needed