Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Front Range PHP NoSQL Databases

6,057 views

Published on

The presentation I did for the FrontRange PHP User Group on 3/10/2010.

Published in: Technology
  • Be the first to comment

Front Range PHP NoSQL Databases

  1. 1. NoSQL Databases Jon Meredith [email_address]
  2. 2. What isn't NoSQL? <ul><li>NOT a standard.
  3. 3. NOT a product.
  4. 4. NOT a single technology. </li></ul>
  5. 5. Well, what is it? <ul>It's a buzzword . <li>A banner for non-relational databases to organize under.
  6. 6. Mostly created in response to scaling and reliability problems.
  7. 7. Huge differences between 'NoSQL' systems – but have elements in common. </li></ul>
  8. 8. Where did it come from? <ul><li>They've been around for a while </li><ul><li>Local key/value stores
  9. 9. Object databases
  10. 10. Graph databases
  11. 11. XML databases </li></ul><li>New problems are emerging </li><ul><li>Internet search
  12. 12. e-commerce
  13. 13. Social networking </li></ul></ul>
  14. 14. Where did it come from? <ul><li>Some efforts came from scaling the web...
  15. 15. Several papers published </li><ul><li>2006 – Google BigTable
  16. 16. 2007 – Dynamo Paper </li></ul><li>In 2008 - explosion of data storage projects
  17. 17. All shambling under the NoSQL banner. </li></ul>
  18. 18. Really, why not use RDBMs? <ul><li>I need to perform arbitrary queries
  19. 19. My application needs transactions
  20. 20. Data needs to be nicely normalized
  21. 21. I have replication for scalabilty/reliability </li></ul>
  22. 22. Data Mapping Woes <ul><li>Relational databases divide data into columns made up of tables.
  23. 23. Programmers use complex nested data structures </li><ul><li>Hashes
  24. 24. Sets
  25. 25. Arrays
  26. 26. Things of things </li></ul><li>Have to map between the two </li></ul>
  27. 27. Data Mapping Woes (2) <ul><li>Data in systems evolve over time … which means changes to the schema.
  28. 28. Upgrade/rollback scripts have to operate on the whole database – could be millions of rows.
  29. 29. Doing phased rollouts is hard … the application needs to do work </li></ul>
  30. 30. Alternative! <ul><li>Let the application do it
  31. 31. Use convenient language features </li><ul><li>PHP serialize/unserialize </li></ul><li>… or use standards for mixed platforms </li><ul><li>JSON very popular and well supported
  32. 32. Google's protocol buffers
  33. 33. … even XML </li></ul><li>Design for forward compatibility </li><ul><li>Preserve unknown fields
  34. 34. Version objects </li></ul></ul>
  35. 35. Scalability and Availability <ul><li>Scalability </li><ul><li>How many requests you can process </li></ul><li>Availability </li><ul><li>How does your service degrade as things break. </li></ul><li>RDBMS solutions - replication and sharding </li></ul>
  36. 36. Scaling RDBMs - Replication <ul><li>Master-Slave replication is easiest
  37. 37. Every change on the master happens on the slave.
  38. 38. Slaves are read-only. Does not scale INSERT, UPDATE, DELETE queries.
  39. 39. Application responsible for distributing queries to correct server. </li></ul>
  40. 40. Scaling RDBMs - Replication <ul><li>Multi-master ring replication </li><ul><li>Can update any master
  41. 41. Updates travel around the ring
  42. 42. What happens when it fails? </li><ul><li>Reconfigure the ring </li></ul><li>What happens on return </li><ul><li>Synchronize the master
  43. 43. Add back in to the ring </li></ul></ul></ul>
  44. 44. Replication <ul><li>Replication is usually asynchronous for performance – you don't want to wait for the slowest slave on each update.
  45. 45. Replication takes time – there is time lag between the first and last server to see an update.
  46. 46. You may not read your writes – not getting aCid properties any more. </li></ul>
  47. 47. Scaling RDBMS – Sharding <ul><li>Do application level splitting of data </li><ul><li>Split large table into N smaller tables
  48. 48. Use Id modulo N to find the right table </li></ul><li>Tables could be spread across multiple database servers </li><ul><li>But the application needs to know where to query </li></ul></ul>
  49. 49. Availability <ul><li>If you want availability you need multiple servers – maybe even multiple sites.
  50. 50. In the real world you get network partitions </li><ul><li>Just because you can't see your other data center doesn't mean users can't. </li></ul><li>What should you do if you can't see the other data center? </li></ul>
  51. 51. Availability <ul><li>Degrade one site to read-only </li><ul><li>Defeats availability </li></ul><li>If you allow both sites to operate </li><ul><li>There's a chance two users could modify the same data.
  52. 52. The application needs to know how to resolve it </li></ul></ul>
  53. 53. The bottom line... <ul><li>Building systems that are </li><ul><li>...Scalable...
  54. 54. ...Available...
  55. 55. ...Maintainable...
  56. 56. with an RDBMs requires large efforts by application developers and operational staff </li></ul></ul>
  57. 57. It's hard because... <ul><li>Significant work for developers. </li><ul><li>App needs to convert data to table/columns
  58. 58. App needs to know data location
  59. 59. App needs to handle failover
  60. 60. App needs to handle inconsistency </li></ul><li>Work for operational staff </li><ul><li>Fixing replication topologies and synchronizing servers is fiddly work. </li></ul></ul>
  61. 61. Last decades bleeding edge is here <ul><li>Organizations with big problems started experimenting with alternatives
  62. 62. Developed internal systems during the mid 2000s </li><ul><li>Distributed by design
  63. 63. Different data models </li></ul><li>Published details in 2006/2007 </li></ul>
  64. 64. Amazon <ul><li>Huge e-commerce vendor.
  65. 65. Amazon cares about customer experience </li><ul><li>Availabilty
  66. 66. Latency at the 99 th percentile </li></ul><li>Built as an SOA – pages built from hundreds of services.
  67. 67. Amazon runs multiple data centers. </li><ul><li>Hardware failure is their normal state
  68. 68. Network partitions common </li></ul></ul>
  69. 69. Amazon Requirements <ul><li>Shopping cart service must always be available
  70. 70. Customers should be able to view and add to their carts (in their words) </li><ul><li>If disks are failing
  71. 71. Network routes are flapping
  72. 72. Data centers are being destroyed by tornadoes </li></ul></ul>
  73. 73. Amazon Observations <ul><li>Many services just stored state. </li><ul><li>Access by primary key
  74. 74. No queries </li></ul><li>Examples </li><ul><li>Shopping carts
  75. 75. Best seller lists
  76. 76. Customer profiles </li></ul><li>Hard to scale out relational databases </li></ul>
  77. 77. Amazon Solution: Dynamo <ul><li>Primary key access only
  78. 78. Fault tolerant: Keeps N copies of the data
  79. 79. Designed for inconsistency
  80. 80. Totally decentralized – nodes 'gossip' state
  81. 81. Self-healing </li></ul>
  82. 82. Eventual Consistency 1 <ul><li>Brewer's CAP Theorem </li><ul><li>Consistency
  83. 83. Availability
  84. 84. Partition tolerance </li></ul><li>Pick two out of three!
  85. 85. Amazon chose A-P over C </li></ul>
  86. 86. Eventual Consistency 2 <ul><li>N copies of each value
  87. 87. Read operations (get) require 'R' nodes to respond
  88. 88. Write operations (put) require 'W' nodes to respond
  89. 89. If R+W > N nodes will read their writes (if no failure)
  90. 90. NRW tunes the cluster – typically (3,2,2) </li></ul>
  91. 91. Eventual Consistency 3 <ul><li>Consequence of availability: Conflicts
  92. 92. Conflicts can come from </li><ul><li>Network partitions
  93. 93. Applications themselves – no transactions or locking </li></ul><li>Applications must handle conflicts
  94. 94. Dynamo minimizes with vector clocks </li></ul>
  95. 95. Vector Clocks
  96. 96. Partitioning
  97. 97. Example: Shopping Cart <ul><li>User browses site – adds 3 widgets </li></ul>
  98. 98. Shopping Cart - Conflict Network Failure
  99. 99. Shopping Cart - Merge
  100. 100. Open Source Dynamo <ul><li>Dynamo is internal to Amazon
  101. 101. Open source options </li><ul><li>Riak from Basho
  102. 102. Project Voldemort </li></ul></ul>
  103. 103. Google BigTables <ul><li>Used internally at Google </li><ul><li>Indexing the web
  104. 104. Google Earth
  105. 105. Finance </li></ul><li>Distributed storage system for structured data </li></ul>
  106. 106. Data representation <ul><li>Data stored in tables.
  107. 107. Table indexed by {key,timestamp} and a variable number of sparse columns
  108. 108. Columns are grouped into column families. Columns in a family are stored together.
  109. 109. Each table is broken into tablets.
  110. 110. Tablets are stored on a cluster file system (GFS). </li></ul>
  111. 111. BigTable – Column Families Copyright Google
  112. 112. Map/Reduce <ul><li>Processing framework that sits on top of BigTable.
  113. 113. Programmers write two functions map() and reduce().
  114. 114. Table is mapped, then reduced.
  115. 115. Job control system monitors and resubmits. </li></ul>
  116. 116. Map/Reduce Source: institutes.lanl.gov
  117. 117. BigTable has inspired... <ul><li>Hadoop/Hbase
  118. 118. Cassandra </li></ul><ul><li>Riak
  119. 119. CouchDB </li></ul>Map/Reduce
  120. 120. Explosion of NoSQL Dbs <ul><li>Too many projects
  121. 121. Two good resources </li><ul><li>http://nosql.mypopescu.com/
  122. 122. http://www.vineetgupta.com/ 2010/01/nosql-databases-part-1-landscape.html </li></ul></ul>
  123. 123. So many projects! Dynamo, BigTables, Redis Riak, Voldemort, CouchDb, Peanuts Hadoop/Hbase, Cassandra, Hypertable MongoDb, Terrastore, Scalaris, BerkleyDB MemcacheDB, Dynomite, Neo4J, TokyoCabinet … and more
  124. 124. NoSQL Characteristics <ul><li>Broad types </li><ul><li>Key/Value
  125. 125. Sparse Column Family
  126. 126. Document oriented </li></ul><li>Persistence </li><ul><li>In memory
  127. 127. On disk </li></ul><li>Distribution </li><ul><li>Replicated
  128. 128. Decentralized </li></ul></ul>
  129. 129. Riak from Basho http://riak.basho.com <ul><li>Dynamo clone written in Erlang
  130. 130. RESTful HTTP interface
  131. 131. Fully distributed
  132. 132. Clients for multiple languages
  133. 133. Multiple storage backends </li><ul><li>In-memory
  134. 134. Filesystem
  135. 135. Embedded InnoDB </li></ul><li>I work there now! </li></ul>
  136. 136. Redis 1.2 <ul><li>http://code.google.com/p/redis/
  137. 137. Key/Value Store with structured values
  138. 138. Written in C
  139. 139. Memcache-like protocol
  140. 140. In use at </li><ul><li>Github
  141. 141. Engine Yard
  142. 142. VideoWiki </li></ul></ul>
  143. 143. Redis 1.2 (cont) <ul><li>Values can be strings, sets, ordered sets, lists
  144. 144. Operations like increment, decrement, intersection, push, pop
  145. 145. In-memory (can be backed by disk)
  146. 146. Auto-sharding in client libraries
  147. 147. No fault tolerance (coming after 2.0)
  148. 148. Example: retwis – Twitter clone in PHP </li></ul>
  149. 149. Cassandra <ul><li>http://incubator.apache.org/cassandra/
  150. 150. BigTable ColumnFamily data model
  151. 151. Dynamo data distribution
  152. 152. Written in Java
  153. 153. Thrift based interface
  154. 154. In use at </li><ul><li>Facebook
  155. 155. Twitter </li></ul></ul>
  156. 156. CouchDB <ul><li>Document oriented database </li><ul><li>All JSON documents </li></ul><li>Written in Erlang
  157. 157. Used by Ubuntu One
  158. 158. HTTP interface
  159. 159. Uses Javascript for indexing/mapreduce
  160. 160. Incremental replication </li></ul>
  161. 161. BerkleyDB <ul><li>Sleepycat now owned by Oracle
  162. 162. Key/Value Store </li><ul><li>Multi-threaded
  163. 163. Multi-process
  164. 164. Replicated
  165. 165. Tranactional </li></ul><li>Alternative: Tokyo Cabinet </li></ul>
  166. 166. I'm out of time <ul><li>MongoDB
  167. 167. Neo4J – Graph Database
  168. 168. Peanuts – Yahoo </li></ul>
  169. 169. This is all great but... <ul><li>Relational databases provide a lot of functionality. </li><ul><li>Giving up queries
  170. 170. Even range queries are hard for distributed hash systems.
  171. 171. No transactions – rules out some classes of applications.
  172. 172. Space is still evolving </li></ul></ul>
  173. 173. Conclusion <ul><li>NoSQL systems give applications the tools they need for scalability/availability
  174. 174. They force you to think about distributed design issues like consistency.
  175. 175. Play with them! </li></ul>

×