Open Source Mobile Backend on Cassandra


Published on

Video online on YouTube at

Published in: Technology, Education
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • \n
  • Hi, I’m Ed Anuff, I do our mobile platform over at Apigee\nI’m also part of the Hector project\nI spoke here last year about indexing techniques, some of you may have caught that\n
  • Quick Apigee plug, we do API management and analytics\nand we also do a backend as a service - Usergrid\nA lot of this stuff is powered by Cassandra, we’ve got a few guys from the Hector team at the company and a lot of Java devs using Cassandra for various stuff (yes, we’re hiring)\nAs I said, I work on our mobile platform, Usergrid\nIt's actually something that became part of Apigee at the start of the year,\nbut it's been under development for a couple of years\n\n
  • Usergrid is a backend-as-a-service\nYou may have heard that term, I don't love it, but you don't always get to name the category\nThe idea is to power mobile apps or web apps via web services, no direct access to DB, all via APIs\nNow, this is not a product pitch, remember everything I talk about today is open source, Apache licensed\nbut I’m going to show a couple of slides to talk about how we envisioned this being used\n\n
  • So, the idea is a backend data platform that you can build apps against,\nmobile apps, rich client apps, apps running devices\nand have all the data on demand via HTTP REST APIs\n\n\n
  • No monolithic server web stacks, LAMP, Ruby on Rails. etc.\nNo MVC, no sessions, no cookies, no page flows\n\n
  • Just cool, interactive, responsive apps doing their thing\ngetting the data they need when they need it from the cloud\n\n
  • So, for us, our priorities in building this are are:\nWe want to run this in the cloud as a service, so it’s a platform\nWe want to offer a very flexible data model that we can build things like user management on top of\nWe want to make it very accessible from any language\n
  • Now, a lot of this is designed to appeal to people doing stuff in Javascript or Objective-C or Android\nalthough we’re finding that people using dynamic languages like Ruby or Python like it a lot too\nBut even if you’re a Java developer, there’s a lot of important stuff here\nIn fact, especially if you’re a Java developer, because if you look under the hood, you’ll find some really interesting things about doing some complicated stuff in Cassandra\n
  • So, lets take a look at the important things for Usergrid for you folks\nOpen source - everything I’m talking about is on GitHub\nBut, if you’re trying to build on Hector, if you want to implement multi-tenancy, work with JSON data, figure out permissions, or use REST, then this might be a good starting point\n
  • So the main thing I’m going to talk about today is the foundation of everything that Usergrid does\nwhat lets us provide it as a service and a way to power lots of apps\n
  • The idea is a platform service with lots of teams, each with lots of devs\nThink something like the GitHub model\nThis is the basic foundation, lots of software-as-a-service needs this\n
  • And the idea is that each team can build as many apps as they want\neach app is going to get a dataspace of its own\n
  • and those apps could have a lot of users, the people who actually interact with the app\nand 10 millions users has to be as easy as ten thousand\n
  • and every user has a device and a bunch of data \n
  • So this is the goal\nNow that’s a lot of data, and it’s “big data”, but it’s also kind of a hard data model\nan inherently relational data model\nso lots of people start with relational DBs\n
  • Now this is the textbook guide to database multi-tenancy\nLots of pros and cons to each approach with scalability tradeoffs\n
  • Well, we’re not going to do separate databases\nand we’re not going to do separate schemas\nI’ve been down this road before\n
  • And even if you get it working, the assumption is that you’re going to end up sharding your database\nIf I’m Salesforce, I can write a big check to Oracle, but it’s still a hell of a lot of work\n
  • Most of the NoSQL solutions don’t help\nthey’re still at the sweet spot of “fire up a copy of node and mongo and power my app”, sort of LAMP reloaded\n
  • So, is Cassandra any better\nWell, at least it gets rid of the sharding issue, but it’s constructs in this area are still pretty heavy\n
  • Lets look at it for Cassandra\nI want to be clear here, I’m not talking about one of those situations where you’re saying “can I do 10 apps” on a cluster. You can, honestly, I’ve had issues with unit tests that run out of memory doing that. This is really about unbound scale, meaning, can you honestly with a straight face say, I don’t really care how many apps I have on this\nLike if your business guys are saying they want to do a freemium business model or whatever\nOr you’re an IT group that wants to run a centralized internal platform as a service and don’t want to be running a hundred separate Mongo servers for every team that needs a datastore\nSo that’s the lens I’m looking at this through\nand when I do that, I realize that there are some limitations here\nKeyspaces are not flyweight nor are CFs in the keyspace, they will add up and use a lot of memory\nI know some people are doing this, doing hosted Cassandra this way, but then again a lot of people are providing shared MySQL hosting using the same mechanisms\n\n\n
  • \n
  • So, lets go with a typical set of rows\n
  • and your key is typically just a UUID\n
  • And you’ll have a bunch of them as rows in your keyspace\n
  • What we do is just used a single shared keyspace and prefix the rows with a tenant ID.\nThis is how we create the virtual keyspace.\n
  • \n
  • So, Cassandra’s built in options are probably not great.\nSo, we introduced something into Hector some time back, a feature called Virtual Keyspaces\nThis is in the code, we’ve used it in production for a while\n
  • This is Hector’s call for creating a Virtual Keyspace. Once you do this, all normal operations will act on the virtual keyspace.\nNote that “createKeyspace” and “createVirtualKeyspace” in Hector don’t actually create the Keyspace in Cassandra, they create an access object for operating on the Keyspace\n
  • The magic here, though, is how we do this behind the scenes in Hector\nSo the app calls Hector and it’s basically a 1 line change of code to switch from actual keyspaces to virtual keyspaces\n
  • So what’s the catch\nwell, the first casualty of this is native secondary indexes\nso if you’re doing this, Hector currently doesn’t filter the results to discard the ones not in the current virtual keyspace\nThat’s actually something we should do and will probably get to\nBut the way we did this was to use app-tier indexes, using the CF-based indexes\n
  • Now, what about CQL?\nWell, Hector doesn’t support this yet either\nNow, if you’re using the UG code, it doesn’t matter, UG has a query language that’s a superset of CQL\n
  • So, that’s the multi-tenancy\n
  • \n
  • \n
  • \n
  • One more thing\n
  • Surprise!\nThis one is kind of fun, bit of a skunkworks project, but we can actually connect to this thing using the Mongo binary wire protocol and do queries and stuff\n
  • So, lots of good stuff in here\n
  • remember, it’s all open source, check it out\n
  • \n
  • Open Source Mobile Backend on Cassandra

    1. UsergridOpen Source Mobile Backend on Cassandra
    2. Hi I’mEd Anuff
    3. API ManagementAPI AnalyticsAPI Tools
    4. UsergridBackend-as-a-Service& Open SourceMobile andRich client apps
    5. App, Cloud, & HTTP
    6. No Web Stack Page MVC Logic Database
    7. Just Apps and APIs User Management! Datastore! Analytics! Location! Social Graph! Activity Streams! Messaging!
    8. Key CapabilitiesA platform not a single appFlexible data modelEasy to use from any language
    9. Why should Cassandradevelopers care aboutUsergrid?
    10. Open Source (APLv2)Built on HectorMulti-tenant (Virtual Keyspaces)JSON and full-text indexingRich permissions modelREST API
    11. Multi-tenant data forApps
    12. Lots of Teams... Team Team Team
    13. Lots of Apps... App App App Team
    14. And App Users... App Team
    15. And data and devices App Team
    16. Multi-tenancy at scaleApp App App App App App Team Team Team
    17. Schema Separate Database Shared Database - Scalability + ScalabilityShared + Isolation - No Isolation + Simple - ComplicatedSeparateSchema - Scalability - Scalability + Isolation - No Isolation + Not Complicated - Complicated
    19. Sharding SQL is toomuch work.Oracle is too muchmoney.
    20. What about NoSQL?Most NoSQL works atthe wrong abstractionlevel.
    21. Does Cassandraprovide any moreoptions?Sort of
    22. Schema Separate Keyspace Shared KeyspaceShared Expensive InterestingSeparateSchema Very Unwieldy Expensive
    23. How would SharedSchema/SharedKeyspace work?
    24. Users = { "4e3c0423-…" : { username: "edanuff", email: "", … : … }, "e5d61f2b-…" : { username: "jdoe", email: "", … : … }}
    25. Users = { "4e3c0423-…" : { username: "edanuff", email: "", … : … }, "e5d61f2b-…" : { username: "jdoe", email: "", … : … }}
    26. Conventional Row Keys In Single Keyspace Row UUID Row UUID Row UUID Row UUID Row UUID Row UUID
    27. Multi-tenant Row Keys In Shared Keyspace Tenant ID Row UUID Tenant ID Row UUID Tenant ID Row UUID Tenant ID Row UUID Tenant ID Row UUID Tenant ID Row UUID
    28. “Wonderful, morethings for me to buildmyself in the app tier...”Hold that thought...
    29. Shared Schema/Shared Keyspace is“Virtual Keyspaces”Part of Hector Client
    30. public static <E> Keyspace createVirtualKeyspace( String keyspace, E keyPrefix, Serializer<E> kps, Cluster cluster);
    31. Hector hides the Tenant Prefixes Prepend Tenant UUID Hector Cassandra Remove Tenant UUID
    32. What about indexes?Hector doesn’t filter resultsDon’t add a tenant columnApp-tier CF indexespreferable
    33. What about CQL?See previous slide for indexlimitationsHector doesn’t currently supportVirtual Keyspaces via CQLI don’t actually ever use CQL
    34. So what else is inthere?
    35. JSONFull JSON values, fullyindexed, using Jacksonand stuff from Lucene
    36. PermissionsFine grained, per user,built on Shiro
    37. RESTFull REST API, built onJersey
    38. Anything else?
    39. Mongo Compatible APIExperimental MongoAPI emulation via NettyMongo binary wireprotocol compatible
    40. That’s whatUsergrid is.
    41. Use the ideasUse the codeUse the cloud