CouchConf Tokyo Developing with Couchbase Part II


Published on

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Modeling data from the real world to software is nothing new. In the case of Couchbase, as a document oriented system, data modeling is pretty easy. We aren’t constrained by schemas and needing to fit things into relational algebra. Instead, we only need to think about
  • Tribal crossing set out to designa new game. They planned for a large audience, running in a cloud environment. With a previous experiment they’d deployed as a Facebook app, my polls (based on an RDBMS), they had planned for the traditional sharding approach. The problem was, at the time Tribal was only a few engineers, no real operations staff. If the app took off and became popular, they would have to reshard the database to keep up with the load. It turned out that my polls started to become very quickly popular over a weekend. They spent the entire weekend adding nodes, resharding, repeating just to try to keep up with the new users. Once sharded out across those system, shrinking the system would also be an issue.
  • No easy way to query: How often do you need to run complex join query? When data is denormalized for speed, how much complex query are you really running? “ Stop thinking in terms of joins and queries is ticket to speed ” Not handling bank transactions: We can live with small percentage of concurrency issue. Err on the side of making player happy.
  • To represent game data in our system, we simply represent objects as JSON. We will then determine the key for an object using the class name or type of the object and a unique ID. In fact, Couchbase Server can serve up sequence numbers pretty easily by using it’s built in increment function. To represent a one to many relationship, we can have a small list that shows the relationships. This allows us to be closer to normalized, but be slightly denormalized. The code for building out our graph of related items will be quite simple, and because it’s distributed and Couchbase caches hot items, it should be very fast.
  • TODO: add artwork from screenshots.
  • Here we see the three different objects in their JSON document form. These are very simple documents, but show the concept. Each document’s key (also known as the _id) is the object’s class, followed by a serial number. Since each player has a plant list, or we can simply create one if the player does not yet have plants, we create the plant list as an array
  • On this slide, we see a sample blog post in JSON. It has most of the fields you’d expect to have in a blog entry. The one field that is a little different is the comments field. One approach here would be to store all comments on this blog in the blog. This is simple, denormalized and lets us get the data in one shot. There are a coupledownsides though. One is that we may not want to display all of the comments. If I’m showing multiple plogs, maybe blog summaries on a given page, I don’t want to display the comments. The other is that some popular blogs, from popular bloggers, may have 100s or 1000s of commments. Of course, the challenge with this is that we don’t want to display them all at once, and may not want to have to grab such a large amount of data. We can reapply the same denormalization technique we’d encountered earlier.
  • As you see here, rather than storing comments inline, we can separate them to a comment list, and then from there to individual comments. Comments in this case can be threaded. You may wonder about the performance of such an arrangement because of all of the traffic across the wire. First off, in a distributed system the data may not be local anyway, so we’ll just make it easier by having the client system fetch the data from the server.
  • If you’re expecting a very large number of comments, or want to display them threaded, you can easily imagine doing so by extending the list technique discussed earlier. This allows us to very easily build very complex arrangements of the data across various keys. Since they distribute throughout the cluster, we spread load out among the cluster nodes. In contrast, with a typical relational model, you may have to have the comments and blogs colocated on a single shard system so you can use join queries. This creates hotspots in the system, and resharding to redistribute the data becomes a manual process. Because of the active cache management in Couchbase Server, the hottest data will be in memory and served very quickly, so the data items may be served very quickly if they’re popular.
  • First we’ll do a demonstration of finding all of the items owned by a particular player through a view. Then we will do a demonstration of showing a leaderboard from the gamesim data previously shown.
  • Using a view over all transactions, say they’re in a separate bucket or have type information on them, we can easily query for individual balances.
  • CouchConf Tokyo Developing with Couchbase Part II

    1. 1. Advanced Document Design J Chris Anderson Mobile
    2. 2. <ul><li>When considering how to model data for a given application, you should… </li></ul><ul><li>Think of a logical container for the data </li></ul><ul><li>Think of how data groups together </li></ul><ul><li>You may notice that… </li></ul><ul><li>From a software development standpoint, this maps better to the way you structure the data </li></ul><ul><li>It’s a more natural way of representing the entities the application will be handling </li></ul>Modeling Data
    3. 3. Example of A User Profile
    4. 4. <ul><li>Key selection: start with the username </li></ul><ul><ul><li>Use this as a common prefix for related data </li></ul></ul><ul><ul><li>When building user’s session information, fetch these items </li></ul></ul><ul><li>Create related documents by prefixing with the same username </li></ul><ul><ul><li>Can extend this concept further </li></ul></ul><ul><ul><ul><li>Other data records for this user, building out object graph, etc. </li></ul></ul></ul>Example: Data Profile for a User { “ _id”: “ auser_profile ” , “ user_id ” : 7778 “ password”: “ a1004cdcaa3191b7 ” , ” common_name ” : ” Alan User ” , ” nicknames ” : [ ” Al ” , ” Buddy ” ], &quot;sign_up_timestamp&quot;: 1224612317, &quot;last_login_timestamp&quot;: 1245613101 } { “ _id”: “ auser_friends ” , “ friends”: [ “ joe ” , “ ingenthr ” , “ toru ” ] }
    5. 5. Social Game Example
    6. 6. <ul><li>As they prepped their new game in 2011, the developers at Tribal Crossing recognized they needed to plan for this game a bit differently: </li></ul><ul><li>Planned to scale </li></ul><ul><li>Deploys to cloud servers </li></ul><ul><li>Be able to easily scale and monitor the overall system </li></ul><ul><li>Previous experience: </li></ul><ul><li>My polls went from 100 users to 1M users in a very long weekend of resharding! </li></ul>Tribal Crossing
    7. 7. <ul><li>But, there are some differences in using Couchbase to handle the game data: </li></ul><ul><ul><ul><li>Direct key or _id queries only </li></ul></ul></ul><ul><ul><ul><li>Transactions do not span objects </li></ul></ul></ul><ul><li>add speed and distribution </li></ul><ul><li>Tribal Questioned: Can this work for an online game? </li></ul><ul><ul><ul><li>Decided they could, just needed to tackle the modeling issues </li></ul></ul></ul><ul><ul><ul><li>Decided that for their deployment, they could relax some constraints to allow for higher concurrency and “repair” issues when encountered by erring to player happyness </li></ul></ul></ul>Modeling Data
    8. 8. <ul><li>Representing Objects </li></ul><ul><li>Simply treat an object as a JSON document </li></ul><ul><ul><li>Could also represent it with serialized objects from the higher level language </li></ul></ul><ul><li>Determine the key for an object using the class name (or type) of the object and an unique ID </li></ul><ul><li>Representing Object Lists </li></ul><ul><li>Denormalization </li></ul><ul><li>Save an array of object IDs in this list </li></ul>Tribal Crossing: Representing Game Data in Couchbase
    9. 9. <ul><li>Three entities (for this simple example) </li></ul><ul><ul><li>Players </li></ul></ul><ul><ul><ul><li>Hold the player’s profile, enough to instantiate the needed data for a session </li></ul></ul></ul><ul><ul><li>Plant lists </li></ul></ul><ul><ul><ul><li>A simple list of plants owned by a given player– semi normalized </li></ul></ul></ul><ul><ul><li>Plants </li></ul></ul><ul><ul><ul><li>The plants owned by a given player </li></ul></ul></ul>Modeling the Example Game Data
    10. 10. <ul><li>Player Object </li></ul><ul><li>Key : 'Player1' </li></ul><ul><li>JSON </li></ul><ul><li>{ </li></ul><ul><li>“ _id ” : “ Player1 ” , </li></ul><ul><li>“ nid ” : 1, </li></ul><ul><li>“ name ” : “ Shawn ” </li></ul><ul><li>} </li></ul>Social Game Data in Couchbase Plant Object Key : 'Plant201' JSON { “ _id ” : “ Plant201 ” , “ nid ” : 201, “ player_id ” : 1 “ name ” : “ Starflower ” } PlayerPlant List Key : 'Player1_PlantList' JSON { “ _id ” : “ Player1_Plantlist ” , “ plants ” : [201, 202, 204] }
    11. 11. <ul><li>No need to “ ALTER TABLE ” </li></ul><ul><li>Add new “ fields ” all objects at any time </li></ul><ul><ul><ul><li>Specify default value for missing fields </li></ul></ul></ul><ul><ul><ul><li>Increased development speed </li></ul></ul></ul><ul><li>Using JSON for data objects </li></ul><ul><ul><li>This will allow future capabilities with Couchbase Server 2.0 </li></ul></ul><ul><ul><li>Offers the ability to query and analyze arbitrary fields with views </li></ul></ul>Tribal Crossing: Schema-less Modeling of Data
    12. 12. <ul><li>Give a player a new plant </li></ul><ul><li>// Create the new plant </li></ul><ul><li>Plant givenPlant = new Plant(100, &quot;Mushroom&quot;); </li></ul><ul><li>cbclient.set(&quot;Plant100&quot;, givenPlant); </li></ul><ul><li>// Update the player plant list </li></ul><ul><li>Player thePlayer = player.fetch(cbclient.get(&quot;Player1&quot;); </li></ul><ul><li>// Add the plant to the player </li></ul><ul><li>thePlayer.receivePlant(givenPlant); </li></ul><ul><li>// Store the player's new plant list </li></ul><ul><li>cbclient.set(&quot;Player1_PlantList&quot;, </li></ul><ul><li>thePlayer.getPlantsArray()); </li></ul>Modifying Game Data in Couchbase
    13. 13. When to break up a document: blog example
    14. 14. <ul><li>User profile </li></ul><ul><ul><li>Main pointer into the user data: </li></ul></ul><ul><ul><ul><li>Blog entries </li></ul></ul></ul><ul><ul><ul><li>Badge settings, like a twitter badge </li></ul></ul></ul><ul><li>Blog posts </li></ul><ul><ul><li>Contains the blogs themselves </li></ul></ul><ul><li>Blog comments </li></ul><ul><ul><li>Comments from other users </li></ul></ul>Entities for a Blog
    15. 15. <ul><li>{ </li></ul><ul><li>“ _id”: “ jchris_Hello_World ” , </li></ul><ul><li>“ author”: “ jchris ” , </li></ul><ul><li>“ type”: “post” </li></ul><ul><li>“ title”: “Hello World”, </li></ul><ul><li>“ format”: “markdown”, </li></ul><ul><li>“ body”: “Hello from [Couchbase]( ).”, </li></ul><ul><li>“ html”: “<p>Hello from <a href=“http: … </li></ul><ul><li>“ comments”: [ </li></ul><ul><li>[ “format”: “markdown”, “ body ” : ” Awesome post! ” ], </li></ul><ul><li>[ “format”: “markdown”, “ body ” : ” Like it. ” ] </li></ul><ul><li>] </li></ul><ul><li>} </li></ul>Blog Document Sample
    16. 16. Blog Document Sample, Broken up { “ _id”: “ jchris_Hello_World ” , “ author”: “ jchris ” , “ type”: “post” “ title”: “Hello World”, “ format”: “markdown”, “ body”: “Hello from [Couchbase]( ).”, “ html”: “<p>Hello from <a href=“http: … “ comments”: [ “ comment1_jchris_Hello_world” ] } { “ _id”: “comment1_jchris_Hello_World”, “ format”: “markdown”, “ body ” : ” Awesome post! ” }
    17. 17. <ul><li>You can imagine how to take this to a threaded list </li></ul>Threaded Comments Blog First comment Reply to comment More Comments List List <ul><li>Advantages: </li></ul><ul><li>Only fetch the data when you need it </li></ul><ul><ul><li>For example, rendering part of the page, jQuery style </li></ul></ul><ul><li>Spread the data (and thus the load) across the entire cluster </li></ul>
    18. 18. Couchbase Server 2.0: Querying and Aggregation with Views
    19. 19. <ul><li>Player items </li></ul><ul><ul><li>Find all the weapons owned by an individual player </li></ul></ul><ul><li>Leader board view </li></ul><ul><ul><li>Showing who has the highest level in the system </li></ul></ul>Demo: Game Simulator Views
    20. 20. <ul><li>Q: How can I implement transactions across multiple documents? </li></ul><ul><li>A: Transactions in this distributed system can be re-imagined with views . (example to follow…) </li></ul><ul><li>Q: Can I write more complex aggregation logic? For example apply a transform or a filter when reducing? </li></ul><ul><li>A: While the built in reduce functions are often enough and highly tuned, Couchbase Server can execute arbitrary javascript functions. </li></ul>Common Questions
    21. 21. <ul><li>Simulate the way federated systems work: </li></ul><ul><ul><li>Examples: checking accounts, credit card transactions </li></ul></ul><ul><li>Create a record per transaction </li></ul><ul><ul><li>Leverage views to reconcile the results of the transaction </li></ul></ul><ul><ul><li>If results don’t reconcile, there is a missing transaction or a flaw in the business logic </li></ul></ul>Exchanging Virtual Currency { “From”: “matt”, “ to”: “ jchris ” , “ coins ” : 30 } { “From”: “matt”, “ to”: “ perry ” , “ coins ” : 30 } { “From”: “ jchris ” , “ to”: “ perry ” , “ coins ” : 30 } { “From”: “matt”, “ to”: “ jchris ” , “ coins ” : 30 } { “From”: “matt”, “ to”: “ perry ” , “ coins ” : 30 } { “From”: “ jchris ” , “ to”: “ perry ” , “ coins ” : 30 }
    22. 22. Exchanging Virtual Currency { “From”: “matt”, “to”: “ jchris ” , “ coins ” : 30 } { “From”: “matt”, “to”: “ perry ” , “ coins ” : 30 } { “From”: “ jchris ” , “ to ” : “ perry ” , “ coins ” : 30 } Mapper function(transaction) { emit(transaction.from, transaction.amount *-1); emit (, transaction.amount); } Reduce function(keys, values) { return sum(values); } Query balance view with key == user, get the balance for the user. Query the sum of the entire view, value should be 0.
    23. 23. Questions?