NoSQL Databases Jon Meredith [email_address]
What isn't NoSQL? <ul><li>NOT  a standard.
NOT  a product.
NOT  a single technology. </li></ul>
Well, what  is  it? <ul>It's a  buzzword . <li>A banner for non-relational databases to organize under.
Mostly created in response to scaling and reliability problems.
Huge differences between 'NoSQL' systems – but have elements in common. </li></ul>
Where did it come from? <ul><li>They've been around for a while </li><ul><li>Local key/value stores
Object databases
Graph databases
XML databases </li></ul><li>New problems are emerging </li><ul><li>Internet search
e-commerce
Social networking </li></ul></ul>
Where did it come from? <ul><li>Some efforts came from scaling the web...
Several papers published  </li><ul><li>2006 – Google BigTable
2007 – Dynamo Paper </li></ul><li>In 2008 - explosion of data storage projects
All shambling under the NoSQL banner. </li></ul>
Really, why not use RDBMs? <ul><li>I need to perform arbitrary queries
My application needs transactions
Data needs to be nicely normalized
I have replication for scalabilty/reliability </li></ul>
Data Mapping Woes <ul><li>Relational databases divide data into columns made up of tables.
Programmers use complex nested data structures </li><ul><li>Hashes
Sets
Arrays
Things of things </li></ul><li>Have to map between the two </li></ul>
Data Mapping Woes (2) <ul><li>Data in systems evolve over time … which means changes to the schema.
Upgrade/rollback scripts have to operate on the whole database – could be millions of rows.
Doing phased rollouts is hard … the application needs to do work </li></ul>
Alternative! <ul><li>Let the application do it
Use convenient language features </li><ul><li>PHP serialize/unserialize </li></ul><li>… or use standards for mixed platfor...
Google's protocol buffers
… even XML </li></ul><li>Design for forward compatibility </li><ul><li>Preserve unknown fields
Version objects </li></ul></ul>
Scalability and Availability <ul><li>Scalability </li><ul><li>How many requests you can process </li></ul><li>Availability...
Scaling RDBMs - Replication <ul><li>Master-Slave replication is easiest
Every change on the master happens on the slave.
Slaves are read-only. Does not scale INSERT, UPDATE, DELETE queries.
Application responsible for distributing queries to correct server. </li></ul>
Scaling RDBMs - Replication <ul><li>Multi-master ring replication </li><ul><li>Can update any master
Updates travel around the ring
What happens when it fails? </li><ul><li>Reconfigure the ring </li></ul><li>What happens on return </li><ul><li>Synchroniz...
Add back in to the ring </li></ul></ul></ul>
Replication <ul><li>Replication is usually asynchronous for performance – you don't want to wait for the slowest slave on ...
Replication takes time – there is time lag between the first and last server to see an update.
You may not read your writes – not getting aCid properties any more. </li></ul>
Scaling RDBMS – Sharding <ul><li>Do application level splitting of data </li><ul><li>Split large table into N smaller tables
Use Id modulo N to find the right table </li></ul><li>Tables could be spread across multiple database servers </li><ul><li...
Availability <ul><li>If you want availability you need multiple servers – maybe even multiple sites.
In the real world you get network partitions </li><ul><li>Just because you can't see your other data center doesn't mean u...
Upcoming SlideShare
Loading in...5
×

Front Range PHP NoSQL Databases

5,564

Published on

The presentation I did for the FrontRange PHP User Group on 3/10/2010.

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
5,564
On Slideshare
0
From Embeds
0
Number of Embeds
16
Actions
Shares
0
Downloads
102
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide
  • Introduce Disclose work for Basho Working on Dynamo clone for the last couple of years
  • Front Range PHP NoSQL Databases

    1. 1. NoSQL Databases Jon Meredith [email_address]
    2. 2. What isn't NoSQL? <ul><li>NOT a standard.
    3. 3. NOT a product.
    4. 4. NOT a single technology. </li></ul>
    5. 5. Well, what is it? <ul>It's a buzzword . <li>A banner for non-relational databases to organize under.
    6. 6. Mostly created in response to scaling and reliability problems.
    7. 7. Huge differences between 'NoSQL' systems – but have elements in common. </li></ul>
    8. 8. Where did it come from? <ul><li>They've been around for a while </li><ul><li>Local key/value stores
    9. 9. Object databases
    10. 10. Graph databases
    11. 11. XML databases </li></ul><li>New problems are emerging </li><ul><li>Internet search
    12. 12. e-commerce
    13. 13. Social networking </li></ul></ul>
    14. 14. Where did it come from? <ul><li>Some efforts came from scaling the web...
    15. 15. Several papers published </li><ul><li>2006 – Google BigTable
    16. 16. 2007 – Dynamo Paper </li></ul><li>In 2008 - explosion of data storage projects
    17. 17. All shambling under the NoSQL banner. </li></ul>
    18. 18. Really, why not use RDBMs? <ul><li>I need to perform arbitrary queries
    19. 19. My application needs transactions
    20. 20. Data needs to be nicely normalized
    21. 21. I have replication for scalabilty/reliability </li></ul>
    22. 22. Data Mapping Woes <ul><li>Relational databases divide data into columns made up of tables.
    23. 23. Programmers use complex nested data structures </li><ul><li>Hashes
    24. 24. Sets
    25. 25. Arrays
    26. 26. Things of things </li></ul><li>Have to map between the two </li></ul>
    27. 27. Data Mapping Woes (2) <ul><li>Data in systems evolve over time … which means changes to the schema.
    28. 28. Upgrade/rollback scripts have to operate on the whole database – could be millions of rows.
    29. 29. Doing phased rollouts is hard … the application needs to do work </li></ul>
    30. 30. Alternative! <ul><li>Let the application do it
    31. 31. Use convenient language features </li><ul><li>PHP serialize/unserialize </li></ul><li>… or use standards for mixed platforms </li><ul><li>JSON very popular and well supported
    32. 32. Google's protocol buffers
    33. 33. … even XML </li></ul><li>Design for forward compatibility </li><ul><li>Preserve unknown fields
    34. 34. Version objects </li></ul></ul>
    35. 35. Scalability and Availability <ul><li>Scalability </li><ul><li>How many requests you can process </li></ul><li>Availability </li><ul><li>How does your service degrade as things break. </li></ul><li>RDBMS solutions - replication and sharding </li></ul>
    36. 36. Scaling RDBMs - Replication <ul><li>Master-Slave replication is easiest
    37. 37. Every change on the master happens on the slave.
    38. 38. Slaves are read-only. Does not scale INSERT, UPDATE, DELETE queries.
    39. 39. Application responsible for distributing queries to correct server. </li></ul>
    40. 40. Scaling RDBMs - Replication <ul><li>Multi-master ring replication </li><ul><li>Can update any master
    41. 41. Updates travel around the ring
    42. 42. What happens when it fails? </li><ul><li>Reconfigure the ring </li></ul><li>What happens on return </li><ul><li>Synchronize the master
    43. 43. Add back in to the ring </li></ul></ul></ul>
    44. 44. Replication <ul><li>Replication is usually asynchronous for performance – you don't want to wait for the slowest slave on each update.
    45. 45. Replication takes time – there is time lag between the first and last server to see an update.
    46. 46. You may not read your writes – not getting aCid properties any more. </li></ul>
    47. 47. Scaling RDBMS – Sharding <ul><li>Do application level splitting of data </li><ul><li>Split large table into N smaller tables
    48. 48. Use Id modulo N to find the right table </li></ul><li>Tables could be spread across multiple database servers </li><ul><li>But the application needs to know where to query </li></ul></ul>
    49. 49. Availability <ul><li>If you want availability you need multiple servers – maybe even multiple sites.
    50. 50. In the real world you get network partitions </li><ul><li>Just because you can't see your other data center doesn't mean users can't. </li></ul><li>What should you do if you can't see the other data center? </li></ul>
    51. 51. Availability <ul><li>Degrade one site to read-only </li><ul><li>Defeats availability </li></ul><li>If you allow both sites to operate </li><ul><li>There's a chance two users could modify the same data.
    52. 52. The application needs to know how to resolve it </li></ul></ul>
    53. 53. The bottom line... <ul><li>Building systems that are </li><ul><li>...Scalable...
    54. 54. ...Available...
    55. 55. ...Maintainable...
    56. 56. with an RDBMs requires large efforts by application developers and operational staff </li></ul></ul>
    57. 57. It's hard because... <ul><li>Significant work for developers. </li><ul><li>App needs to convert data to table/columns
    58. 58. App needs to know data location
    59. 59. App needs to handle failover
    60. 60. App needs to handle inconsistency </li></ul><li>Work for operational staff </li><ul><li>Fixing replication topologies and synchronizing servers is fiddly work. </li></ul></ul>
    61. 61. Last decades bleeding edge is here <ul><li>Organizations with big problems started experimenting with alternatives
    62. 62. Developed internal systems during the mid 2000s </li><ul><li>Distributed by design
    63. 63. Different data models </li></ul><li>Published details in 2006/2007 </li></ul>
    64. 64. Amazon <ul><li>Huge e-commerce vendor.
    65. 65. Amazon cares about customer experience </li><ul><li>Availabilty
    66. 66. Latency at the 99 th percentile </li></ul><li>Built as an SOA – pages built from hundreds of services.
    67. 67. Amazon runs multiple data centers. </li><ul><li>Hardware failure is their normal state
    68. 68. Network partitions common </li></ul></ul>
    69. 69. Amazon Requirements <ul><li>Shopping cart service must always be available
    70. 70. Customers should be able to view and add to their carts (in their words) </li><ul><li>If disks are failing
    71. 71. Network routes are flapping
    72. 72. Data centers are being destroyed by tornadoes </li></ul></ul>
    73. 73. Amazon Observations <ul><li>Many services just stored state. </li><ul><li>Access by primary key
    74. 74. No queries </li></ul><li>Examples </li><ul><li>Shopping carts
    75. 75. Best seller lists
    76. 76. Customer profiles </li></ul><li>Hard to scale out relational databases </li></ul>
    77. 77. Amazon Solution: Dynamo <ul><li>Primary key access only
    78. 78. Fault tolerant: Keeps N copies of the data
    79. 79. Designed for inconsistency
    80. 80. Totally decentralized – nodes 'gossip' state
    81. 81. Self-healing </li></ul>
    82. 82. Eventual Consistency 1 <ul><li>Brewer's CAP Theorem </li><ul><li>Consistency
    83. 83. Availability
    84. 84. Partition tolerance </li></ul><li>Pick two out of three!
    85. 85. Amazon chose A-P over C </li></ul>
    86. 86. Eventual Consistency 2 <ul><li>N copies of each value
    87. 87. Read operations (get) require 'R' nodes to respond
    88. 88. Write operations (put) require 'W' nodes to respond
    89. 89. If R+W > N nodes will read their writes (if no failure)
    90. 90. NRW tunes the cluster – typically (3,2,2) </li></ul>
    91. 91. Eventual Consistency 3 <ul><li>Consequence of availability: Conflicts
    92. 92. Conflicts can come from </li><ul><li>Network partitions
    93. 93. Applications themselves – no transactions or locking </li></ul><li>Applications must handle conflicts
    94. 94. Dynamo minimizes with vector clocks </li></ul>
    95. 95. Vector Clocks
    96. 96. Partitioning
    97. 97. Example: Shopping Cart <ul><li>User browses site – adds 3 widgets </li></ul>
    98. 98. Shopping Cart - Conflict Network Failure
    99. 99. Shopping Cart - Merge
    100. 100. Open Source Dynamo <ul><li>Dynamo is internal to Amazon
    101. 101. Open source options </li><ul><li>Riak from Basho
    102. 102. Project Voldemort </li></ul></ul>
    103. 103. Google BigTables <ul><li>Used internally at Google </li><ul><li>Indexing the web
    104. 104. Google Earth
    105. 105. Finance </li></ul><li>Distributed storage system for structured data </li></ul>
    106. 106. Data representation <ul><li>Data stored in tables.
    107. 107. Table indexed by {key,timestamp} and a variable number of sparse columns
    108. 108. Columns are grouped into column families. Columns in a family are stored together.
    109. 109. Each table is broken into tablets.
    110. 110. Tablets are stored on a cluster file system (GFS). </li></ul>
    111. 111. BigTable – Column Families Copyright Google
    112. 112. Map/Reduce <ul><li>Processing framework that sits on top of BigTable.
    113. 113. Programmers write two functions map() and reduce().
    114. 114. Table is mapped, then reduced.
    115. 115. Job control system monitors and resubmits. </li></ul>
    116. 116. Map/Reduce Source: institutes.lanl.gov
    117. 117. BigTable has inspired... <ul><li>Hadoop/Hbase
    118. 118. Cassandra </li></ul><ul><li>Riak
    119. 119. CouchDB </li></ul>Map/Reduce
    120. 120. Explosion of NoSQL Dbs <ul><li>Too many projects
    121. 121. Two good resources </li><ul><li>http://nosql.mypopescu.com/
    122. 122. http://www.vineetgupta.com/ 2010/01/nosql-databases-part-1-landscape.html </li></ul></ul>
    123. 123. So many projects! Dynamo, BigTables, Redis Riak, Voldemort, CouchDb, Peanuts Hadoop/Hbase, Cassandra, Hypertable MongoDb, Terrastore, Scalaris, BerkleyDB MemcacheDB, Dynomite, Neo4J, TokyoCabinet … and more
    124. 124. NoSQL Characteristics <ul><li>Broad types </li><ul><li>Key/Value
    125. 125. Sparse Column Family
    126. 126. Document oriented </li></ul><li>Persistence </li><ul><li>In memory
    127. 127. On disk </li></ul><li>Distribution </li><ul><li>Replicated
    128. 128. Decentralized </li></ul></ul>
    129. 129. Riak from Basho http://riak.basho.com <ul><li>Dynamo clone written in Erlang
    130. 130. RESTful HTTP interface
    131. 131. Fully distributed
    132. 132. Clients for multiple languages
    133. 133. Multiple storage backends </li><ul><li>In-memory
    134. 134. Filesystem
    135. 135. Embedded InnoDB </li></ul><li>I work there now! </li></ul>
    136. 136. Redis 1.2 <ul><li>http://code.google.com/p/redis/
    137. 137. Key/Value Store with structured values
    138. 138. Written in C
    139. 139. Memcache-like protocol
    140. 140. In use at </li><ul><li>Github
    141. 141. Engine Yard
    142. 142. VideoWiki </li></ul></ul>
    143. 143. Redis 1.2 (cont) <ul><li>Values can be strings, sets, ordered sets, lists
    144. 144. Operations like increment, decrement, intersection, push, pop
    145. 145. In-memory (can be backed by disk)
    146. 146. Auto-sharding in client libraries
    147. 147. No fault tolerance (coming after 2.0)
    148. 148. Example: retwis – Twitter clone in PHP </li></ul>
    149. 149. Cassandra <ul><li>http://incubator.apache.org/cassandra/
    150. 150. BigTable ColumnFamily data model
    151. 151. Dynamo data distribution
    152. 152. Written in Java
    153. 153. Thrift based interface
    154. 154. In use at </li><ul><li>Facebook
    155. 155. Twitter </li></ul></ul>
    156. 156. CouchDB <ul><li>Document oriented database </li><ul><li>All JSON documents </li></ul><li>Written in Erlang
    157. 157. Used by Ubuntu One
    158. 158. HTTP interface
    159. 159. Uses Javascript for indexing/mapreduce
    160. 160. Incremental replication </li></ul>
    161. 161. BerkleyDB <ul><li>Sleepycat now owned by Oracle
    162. 162. Key/Value Store </li><ul><li>Multi-threaded
    163. 163. Multi-process
    164. 164. Replicated
    165. 165. Tranactional </li></ul><li>Alternative: Tokyo Cabinet </li></ul>
    166. 166. I'm out of time <ul><li>MongoDB
    167. 167. Neo4J – Graph Database
    168. 168. Peanuts – Yahoo </li></ul>
    169. 169. This is all great but... <ul><li>Relational databases provide a lot of functionality. </li><ul><li>Giving up queries
    170. 170. Even range queries are hard for distributed hash systems.
    171. 171. No transactions – rules out some classes of applications.
    172. 172. Space is still evolving </li></ul></ul>
    173. 173. Conclusion <ul><li>NoSQL systems give applications the tools they need for scalability/availability
    174. 174. They force you to think about distributed design issues like consistency.
    175. 175. Play with them! </li></ul>
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×