NoSQL
Implementations
Real-world designs explored by
Matt Brender
Aging
Buckets
Emphasis on persisting data.
Not a lot of data.
Possibility of conflict.
3
Aging Buckets
Problem
Difficult to know what user viewed who’s profile
• Don’t want to heavily edit current database design
• Want a simple, scalable solution
Solution
Buckets of relationships using sets (CRDT)
• Allows for simultaneous writes without siblings
• Denormalize and flatten data as JSON
• Store in Riak, index with Solr
4
Aging Buckets
{
[“Beth” : “Tom”],
[“Beth” : “Jim”],
[“Beth” : “George”]
}
2015:05:27
{
[“George” : “Tom”],
[“Beth” : “Jim”],
[“George” : “Jim”]
}
2015:05:26
{
[“Tom” : “Beth”],
[“Beth” : “Tom”],
[“George” : “Jim”]
}
2015:05:25
5
Aging Buckets
Riak
6
Client
Client
Client
Aging Buckets
Riak
7
Client
Client
Client
{
[“Tom” : “Beth”],
[“Beth” : “Tom”],
[“George” : “Jim”]
}
{
[“Tom” : “Beth”],
[“Beth” : “Tom”],
[“George” : “Jim”]
}
Aging Buckets
Riak
8
Client
Client
Client
{
[“Tom” : “Beth”],
[“Beth” : “Tom”],
[“George” : “Jim”]
}
{
[“Jane”: “Tom”],
[“Tom” : “Beth”],
[“Beth” : “Tom”],
[“George” : “Jim”]
}
Aging Buckets
Riak
9
Client
Client
Client
{
[“Jane”: “Tom”],
[“Tom” : “Beth”],
[“Beth” : “Tom”],
[“George” : “Jim”]
}
{
[“Jane”: “Tom”],
[“Tom” : “Beth”],
[“Beth” : “Tom”],
[“George” : “Jim”]
}
Aging Buckets
Riak
10
Client
Client
Client
{
[“Jane”: “Tom”],
[“Tom” : “Beth”],
[“Beth” : “Tom”],
[“George” : “Jim”]
}
{
[“Jane”: “Tom”],
[“Tom” : “Beth”],
[“Beth” : “Tom”],
[“George” : “Jim”]
} {
[“Jane”: “Tom”],
[“Tom” : “Beth”],
[“Beth” : “Tom”],
[“George” : “Jim”]
}
Aging Buckets
Riak
11
Client
Client
Client
{
[“Jane”: “Tom”],
[“Tom” : “Beth”],
[“Beth” : “Tom”],
[“George” : “Jim”]
}
{
[“Jane”: “Tom”],
[“Tom” : “Beth”],
[“Beth” : “Tom”],
[“George” : “Jim”]
} {
[“Jane”: “Tom”],
[“Tom” : “Beth”],
[“Beth” : “Tom”],
[“George” : “Jim”]
}
Aging Buckets
Riak
12
Client
Client
Client
{
[“Jane”: “Tom”],
[“Tom” : “Beth”],
[“Beth” : “Tom”],
[“George” : “Jim”],
[“Tom”: “Jane”]
}
{
[“Jane”: “Tom”],
[“Tom” : “Beth”],
[“Beth” : “Tom”],
[“George” : “Jim”]
} {
[“Jane”: “Tom”],
[“Tom” : “Beth”],
[“Beth” : “Tom”],
[“George” : “Jim”]
}
Aging Buckets
Riak
13
ClientClient
Client
{
[“Jane”: “Tom”],
[“Tom” : “Beth”],
[“Beth” : “Tom”],
[“George” : “Jim”],
[“Tom”: “Jane”]
}
{
[“Jane”: “Tom”],
[“Tom” : “Beth”],
[“Beth” : “Tom”],
[“George” : “Jim”],
[“Beth”, “Jane”]
} {
[“Jane”: “Tom”],
[“Tom” : “Beth”],
[“Beth” : “Tom”],
[“George” : “Jim”]
}
Aging Buckets
Riak
14
ClientClient
Client
{
[“Jane”: “Tom”],
[“Tom” : “Beth”],
[“Beth” : “Tom”],
[“George” : “Jim”],
[“Tom”: “Jane”]
}
{
[“Jane”: “Tom”],
[“Tom” : “Beth”],
[“Beth” : “Tom”],
[“George” : “Jim”],
[“Beth”, “Jane”]
}
{
[“Jane”: “Tom”],
[“Tom” : “Beth”],
[“Beth” : “Tom”],
[“George” : “Jim”],
[“Beth”, “Jane”]
}
15
Bucket properties
an important step in this case
➜ bin/riak-admin bucket-type create sets
'{"props":{"datatype":"set"}}'
sets created
➜ bin/riak-admin bucket-type activate sets
sets has been activated
require ‘riak’
Riak::Crdt::DEFAULT_BUCKET_TYPES[:set] = 'sets'
Aging Buckets
{ [“Jane”: “Tom”],
[“Tom” : “Beth”],
[“Beth” : “Tom”],
[“George” : “Jim”],
[“Tom”: “Jane”],
[“Beth”: “Jane”] }
16
{
[“Jane”: “Tom”],
[“Tom” : “Beth”],
[“Beth” : “Tom”],
[“George” : “Jim”],
[“Tom”: “Jane”]
}
{
[“Jane”: “Tom”],
[“Tom” : “Beth”],
[“Beth” : “Tom”],
[“George” : “Jim”],
[“Beth”, “Jane”]
}
}set CRDT behavior

NoSQL Implementation - Part 1 (Velocity 2015)

  • 1.
  • 2.
    Aging Buckets Emphasis on persistingdata. Not a lot of data. Possibility of conflict. 3
  • 3.
    Aging Buckets Problem Difficult toknow what user viewed who’s profile • Don’t want to heavily edit current database design • Want a simple, scalable solution Solution Buckets of relationships using sets (CRDT) • Allows for simultaneous writes without siblings • Denormalize and flatten data as JSON • Store in Riak, index with Solr 4
  • 4.
    Aging Buckets { [“Beth” :“Tom”], [“Beth” : “Jim”], [“Beth” : “George”] } 2015:05:27 { [“George” : “Tom”], [“Beth” : “Jim”], [“George” : “Jim”] } 2015:05:26 { [“Tom” : “Beth”], [“Beth” : “Tom”], [“George” : “Jim”] } 2015:05:25 5
  • 5.
  • 6.
    Aging Buckets Riak 7 Client Client Client { [“Tom” :“Beth”], [“Beth” : “Tom”], [“George” : “Jim”] } { [“Tom” : “Beth”], [“Beth” : “Tom”], [“George” : “Jim”] }
  • 7.
    Aging Buckets Riak 8 Client Client Client { [“Tom” :“Beth”], [“Beth” : “Tom”], [“George” : “Jim”] } { [“Jane”: “Tom”], [“Tom” : “Beth”], [“Beth” : “Tom”], [“George” : “Jim”] }
  • 8.
    Aging Buckets Riak 9 Client Client Client { [“Jane”: “Tom”], [“Tom”: “Beth”], [“Beth” : “Tom”], [“George” : “Jim”] } { [“Jane”: “Tom”], [“Tom” : “Beth”], [“Beth” : “Tom”], [“George” : “Jim”] }
  • 9.
    Aging Buckets Riak 10 Client Client Client { [“Jane”: “Tom”], [“Tom”: “Beth”], [“Beth” : “Tom”], [“George” : “Jim”] } { [“Jane”: “Tom”], [“Tom” : “Beth”], [“Beth” : “Tom”], [“George” : “Jim”] } { [“Jane”: “Tom”], [“Tom” : “Beth”], [“Beth” : “Tom”], [“George” : “Jim”] }
  • 10.
    Aging Buckets Riak 11 Client Client Client { [“Jane”: “Tom”], [“Tom”: “Beth”], [“Beth” : “Tom”], [“George” : “Jim”] } { [“Jane”: “Tom”], [“Tom” : “Beth”], [“Beth” : “Tom”], [“George” : “Jim”] } { [“Jane”: “Tom”], [“Tom” : “Beth”], [“Beth” : “Tom”], [“George” : “Jim”] }
  • 11.
    Aging Buckets Riak 12 Client Client Client { [“Jane”: “Tom”], [“Tom”: “Beth”], [“Beth” : “Tom”], [“George” : “Jim”], [“Tom”: “Jane”] } { [“Jane”: “Tom”], [“Tom” : “Beth”], [“Beth” : “Tom”], [“George” : “Jim”] } { [“Jane”: “Tom”], [“Tom” : “Beth”], [“Beth” : “Tom”], [“George” : “Jim”] }
  • 12.
    Aging Buckets Riak 13 ClientClient Client { [“Jane”: “Tom”], [“Tom”: “Beth”], [“Beth” : “Tom”], [“George” : “Jim”], [“Tom”: “Jane”] } { [“Jane”: “Tom”], [“Tom” : “Beth”], [“Beth” : “Tom”], [“George” : “Jim”], [“Beth”, “Jane”] } { [“Jane”: “Tom”], [“Tom” : “Beth”], [“Beth” : “Tom”], [“George” : “Jim”] }
  • 13.
    Aging Buckets Riak 14 ClientClient Client { [“Jane”: “Tom”], [“Tom”: “Beth”], [“Beth” : “Tom”], [“George” : “Jim”], [“Tom”: “Jane”] } { [“Jane”: “Tom”], [“Tom” : “Beth”], [“Beth” : “Tom”], [“George” : “Jim”], [“Beth”, “Jane”] } { [“Jane”: “Tom”], [“Tom” : “Beth”], [“Beth” : “Tom”], [“George” : “Jim”], [“Beth”, “Jane”] }
  • 14.
    15 Bucket properties an importantstep in this case ➜ bin/riak-admin bucket-type create sets '{"props":{"datatype":"set"}}' sets created ➜ bin/riak-admin bucket-type activate sets sets has been activated require ‘riak’ Riak::Crdt::DEFAULT_BUCKET_TYPES[:set] = 'sets'
  • 15.
    Aging Buckets { [“Jane”:“Tom”], [“Tom” : “Beth”], [“Beth” : “Tom”], [“George” : “Jim”], [“Tom”: “Jane”], [“Beth”: “Jane”] } 16 { [“Jane”: “Tom”], [“Tom” : “Beth”], [“Beth” : “Tom”], [“George” : “Jim”], [“Tom”: “Jane”] } { [“Jane”: “Tom”], [“Tom” : “Beth”], [“Beth” : “Tom”], [“George” : “Jim”], [“Beth”, “Jane”] } }set CRDT behavior

Editor's Notes

  • #5 This is brilliant in its simplicity. Buckets are formatted on a per-day syntax with a simple appending strategy in the client. Common question here: what’s the limit to number of buckets you have? The answer is a bucket is nothing more space consuming than a key, so its affectively limitless, or practically up to the hundreds of millions. For relationship data. The first question you should ask yourself is “Why not store it in a relational database.” Good question! THIS IS NOT, however, WHY RIAK was the best choice for this application.
  • #6 THIS is why Riak was chosen. The example I’m thinking of has a 12 node Riak cluster. You could have 12. You could have 200. With load balancers in place, your code can be identical. Data in their environment is moving at an incredibly high velocity with up-to-dozens of different clients writing data to the same Riak cluster. This access pattern is completely valid for Riak – clusters will always accept writes when a quorum of nodes is reachable. What could happen, of course, is that we could write similar data to the same key/value pair. We’re going to walk through what happens here (and I’m going to go back to abstracting the riak nodes into a big old cluster). I’m abstracting the number of nodes since it is the behavior that you’ll care most about as you code.
  • #7 If the far right client writes, you want the left client to read that exact value. Riak is made up of a dozen plus values.
  • #8 If the far right client writes, you want the left client to read that exact value. Riak is made up of a dozen plus values.
  • #9 But eventually --- which is generally ms, not seconds, the client would see the updated value.
  • #10 But eventually --- which is generally ms, not seconds, the client would see the updated value.
  • #11 But eventually --- which is generally ms, not seconds, the client would see the updated value.
  • #12 The portion of the cluster on one side has updated the key/value pair in today’s bucket, adding another relationship. That’s something we’ll have to worry about once our network partition resolves.
  • #13 But wait, there’s more! The client on the OTHER side of the partition ALSO updates its value.
  • #14 Due to read repair or AAE, the third client will receive the updates. What happens?!? when the partition resolves itself though??
  • #15 We’re
  • #16 Now the conflict needs resolution.