Key-value databases in practice Redis @ DotNetToscana

1,086 views
944 views

Published on

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,086
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
39
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Key-value databases in practice Redis @ DotNetToscana

  1. 1. Matteo Baglini www.dotnettoscana.orgSoftware Developer, Freelancematteo.baglini@gmail.comhttp://it.linkedin.com/in/matteobaglinihttp://github.cpom/bmatte
  2. 2. «Advanced key-value store. It is often referred to as a data structure server» 2
  3. 3. Key Valuepage:index <html><head>[...]user:123:session xDrSdEwd4dSlZkEkj+user:123:avatar 77u/PD94bWwgdm+ Everything is a «blob»Commands, primarily, can GET and SET the values 3
  4. 4. Key Value Typepage:index <html><head>[...] Stringevents:timeline { «Joe logged», «File X Uploaded», …} Listlogged:today { 1, 2, 3, 4, 5 } Set time => 10927353user:123:profile Hash username => bmatte joe ~ 1.3483game:leaderboard smith ~ 293.45 Sorted Set fred ~ 83.22 Different «data type/structure» Rich set of specialized commands 4
  5. 5.  Everything is stored in memory Screamingly fast performance Persistent via snapshot or append-only log file Replication (only Master/Slave) Extensible via embedded scripting engine (Lua) Rich set of client libraries High availability (In progress) ◦ Cluster (Fault tolerance, Multi-Node consistence) ◦ Sentinel (Monitoring, Notification, Automatic failover) 5
  6. 6.  Created by Salvatore Sanfilippo (@antirez) First «public release» in March 2009. Since 2010 sponsored by VMware.Initially written to improve performance of Web Analytics product LLOOGG out of his startup 6
  7. 7.  Written in ANSI C No external dependencies Single thread (asynchronous evented I/O) Works on all POSIX-like system Exist unofficial build for Windows Open-source BSD licensed Community (list, IRC & wiki) 7
  8. 8. 1. A DSL for Abstract Data Types.2. Memory storage is #1.3. Fundamental data structures for a fundamental API.4. Code is like a poem.5. Were against complexity.6. Two levels of API.7. We optimize for joy. 8
  9. 9. GettingStarted 9
  10. 10. Latest stable version (2.6.*) 10
  11. 11. Latest unstable version (2.9.7) 11
  12. 12. 12
  13. 13. 13
  14. 14. 14
  15. 15. 15
  16. 16. Data Types 16
  17. 17. Strings 17
  18. 18. Any blob will do(A value can be at max 512MB) 18
  19. 19. Operations on strings holding an integer 19
  20. 20. 20
  21. 21.  Sharing state across processes ◦ Distribute lock, Incremental ID, Time series, User session. Web Analytics ◦ User visit (day, week, month), Feature Tracking. Caching ◦ String values can hold arbitrary data. Rate limiting ◦ Limit number of API calls/minute. 21
  22. 22. Keys 22
  23. 23. Any item in can be made to expire after or at a certain time. 23
  24. 24. 24
  25. 25. Lists 25
  26. 26. Sequence of string values 26
  27. 27. Sequence of string values (Max length is 232 - 1 elements) 27
  28. 28. Prevent indefinite growth 28
  29. 29. 29
  30. 30.  Events Store or Notification ◦ Logs, Social Network Timelines, Notifications. Fixed Data ◦ Last N activity. Message Passing ◦ Durable MQ, Job Queue. Circular list 30
  31. 31. Sets 31
  32. 32. Unordered set of unique values 32
  33. 33. Unordered set of unique values (Max number of members is 232 – 1) 33
  34. 34. You can do unions, intersections,differences of sets in very short time. 34
  35. 35. 35
  36. 36.  Web Analytics ◦ Unique Page View, IP addresses visiting. Relations ◦ Friends, Followers, Tags. Caching Result ◦ Store result of expensive intersection of data. 36
  37. 37. Sorted Set 37
  38. 38. Ordered set of unique values 38
  39. 39. Access by rank 39
  40. 40. Access by score 40
  41. 41. 41
  42. 42.  Web Analytics ◦ Online users, Most visited pages. Leaderbord ◦ Show top N. Order by data ◦ Maintain a set of ordered data like user by age. 42
  43. 43. Hashes 43
  44. 44. Key → Value map (as value) 44
  45. 45. Set attributes(Store up to 232 - 1 field-value pairs) 45
  46. 46. Get attributes 46
  47. 47. 47
  48. 48.  Storing Objects ◦ Hashes are maps between string fields and string values, so they are the perfect data type to represent objects. 48
  49. 49. Persistence 49
  50. 50. Dump data to disk after certain conditions are met 50
  51. 51.  Pro: ◦ RDB is a very compact single-file. ◦ RDB files are perfect for backups. ◦ RDB is very good for disaster recovery. ◦ RDB allows faster restarts with big datasets. ◦ RDB maximizes performances (backgr. I/O via fork(2)). Contro: ◦ RDB is NOT good if you need to minimize the chance of data loss in case Redis stops working (for example after a power outage). ◦ Fork can be time consuming if the dataset is very big. 51
  52. 52. Append all write operations to a log 52
  53. 53. Durability depends on fsync(2) policy 53
  54. 54.  Pro: ◦ AOF is much more durable. ◦ AOF is an append only log, no seeks, nor corruption problems (for example after a power outage). ◦ AOF contains a log of all the operations one after the other in an easy to understand and parse format. Contro: ◦ AOF files are usually bigger than the equivalent RDB. ◦ AOF can be slower then RDB depending on the exact fsync policy. 54
  55. 55.  Use both persistence methods if you want a degree of data safety comparable to what any RDBMS can provide you. If you care a lot about your data, but still can live with a few minutes of data lose in case of disasters, you can simply use RDB alone. There are many users using AOF alone, but we discourage it since to have an RDB snapshot from time to time is a great idea for doing database backups, for faster restarts. 55
  56. 56. C#Clients 56
  57. 57. Rich set of clients 57
  58. 58. 58
  59. 59. 59
  60. 60. Code 60
  61. 61. Transactions 61
  62. 62. Multiple commands (ACID) 62
  63. 63. 63
  64. 64.  Classic scenario ◦ Multi atomic commands. Optimistic locking ◦ Check and Set (CAS Pattern) write only if not changed. 64
  65. 65. PublishSubscribe 65
  66. 66. Provide 1-N messaging 66
  67. 67. Subscribe multi channels decoupled from the key space 67
  68. 68. Publish on some channel 68
  69. 69. Subscriber getting notified 69
  70. 70. 70
  71. 71.  Message Passing ◦ Distribute message-oriented system, Event- Driven Architecture, Service Bus. 71
  72. 72. Code 72
  73. 73. Replication 73
  74. 74. One master replicate to multiple slaves 74
  75. 75. Slave send SYNC command and master transfers the database file to the slave 75
  76. 76. Slaves can perform only read operation 76
  77. 77.  Scalability ◦ Multiple slaves for read-only queries. Redundancy ◦ Data replication. Slave of Slave ◦ Graph-like structure for more scalability e redundancy. 77
  78. 78. Performance 78
  79. 79. Screamingly fast performance ~50K read/write operations per seconds. ~100K read/write ops per second on a regular EC2 instance. 79
  80. 80. redis-benchmark tool on a Ubuntu virtual machine ~36K rps 80
  81. 81. ApplicationArchitecture 81
  82. 82. Application Server SQL RedisServer 82
  83. 83. 83
  84. 84. Finally 84
  85. 85. «I see Redis definitely more as a flexible tool than as a solution specialized to solve a specific problem: his mixed soul of cache, store, and messaging server shows this very well» Salvatore Sanfilippo 85
  86. 86.  http://redis.io/ http://github.com/antirez/redis http://groups.google.com/group/redis-db 86

×