Wrong confirmation ID

Email
Favorite
Favorited ×
Download
Embed
Private Content
Copy and paste this code into your blog or website Copy Customize… Close
We have emailed the verification/download link to "".
Login to your email and click the link to download the file directly.

To request the link at a different email address, update it here. Close
Validation messages. Success message. Fail message.

Check your bulk/spam folders if you can't find our mail.

Loading…

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

+ Follow

Ruby APIs for NoSQL - Polyglot Persistence - GoGaRuCo 2010

by Sarah Mei on Sep 17, 2010

2,036 views

Presentation from GoGaRuCo 2010 on Polyglot Persistence and Ruby APIs for NoSQL databases.

More…

Presentation from GoGaRuCo 2010 on Polyglot Persistence and Ruby APIs for NoSQL databases.

Less

1 Embed 7

http://lanyrd.com

Statistics

Favorites: 0
Downloads: 19
Comments: 0
Embed Views: 7
Views on SlideShare: 2,029
Total Views: 2,036

I’m Sarah Mei, and I’m a Ruby developer at Pivotal Labs here in San Francisco. We’re an agile consulting company, and we do a lot of Rails. I’m here to talk to you today about Ruby APIs for NoSQL. I also like to call this talk....
Polyglot Persistence. We’re going to talk about how you store data in Ruby when you’re writing a system that uses more than just a relational database. And I’d like to start with a little bit of audience participation... so get ready....
Show of hands: who’s written an application in Ruby that uses a relational database to store data?

Who’s written a Ruby app that uses a relational database AND some kind of alternative datastore?

Who’s written a Ruby app that uses ONLY non-relational datastores?
So I’m going to start off by showing you this diagram that’s been in hundreds of “teach yourself Rails in 25 seconds” type blog posts. Here we have a vanilla Rails application - requests come in, they go through routes to a controller, which cedes to the models, which use ActiveRecord to access MySQL. And then we go back the way we came, except on the way out we go through views instead of routes, and return a response.

I was showing this diagram to a friend, and he said, “you know, if you have a diagram like this, you are legally required to have a little drawing of a cloud to represent the internet.” So...
There you go. Anyway, I would be surprised if anyone in this room has written a real application with real users in which these are the only boxes that appear on the system diagram, and in particular, in which MySQL is the only way data is persisted. Certainly you don’t set out to create a poly-persistant system - it just happens. It starts innocently enough ...
You add solr for free-text search. Pretty soon though you have more than one app server, so you start...
...uploading your assets to s3. Pretty soon you realize you need to do some...
...background jobs, so you install a little resque in there. Then since you’re a social network you realize you need to cache your friend lists, so you toss those in redis too...
And pretty soon you decide you need an object cache, so you stick in cache-money...and let me just pause to say that I’m drawing all these arrows into and out of the models. That’s what you should do. I know some of you put this logic in your controllers, but I’m ignoring you. Let’s pretend we put it all in the right place.

So, at this point, you have five different datastores, and you haven’t even done anything particularly crazy. In fact I would say that this type of setup is a lot more common in real applications than my original undecorated diagram of what a Rails app looks like. This is really what a basic Rails app looks like these days.
It’s the new normal. It’s pretty amazing! In the last year, we’ve come to recognize that most applications have models with data that doesn’t fit relational persistence very well.

These models have relationships, which are hard to model in SQL - those joins get nasty pretty quick. They have blocks of text that need to be searched semantically. They have frequently-read, rarely-written objects that need to be retrieved very very quickly. And they have associated assets that go along with them.

And I would like to point out, before I go on, that until recently, we didn’t even think of all of these things as alternative datastores. Cassandra and redis, certainly, but s3? solr? memcached? Are all of these things datastores?
Of course you know what I think. One of the really interesting things about the NoSQL movement is that they’ve expanded our definition of a datastore. Does anyone remember those guys in the 90s that were writing big Java apps and persisting stuff in XML text files? Was anyone here one of those guys?

Those guys were doing NoSQL...before it was coool.
But big problem #1 with a system like this is....
How do you encapsulate all of these disparate data stores into a model class that makes sense? What does it look like?
So that’s question 1, keep that in mind.

There’s another question, too...
How do you go from this...
...to this? How do you replace your primary datastore with something else? All of these data stores we’ve been looking at so far have been satellites to the primary which was SQL. But if you’re lucky, and very successful, you’ll need to replace it with something else.

Digg, for example, just moved over to Cassandra! For everything! Of course this is not always the right answer, because shortly after Digg rolled out their new architecture, they fired their VP of engineering, ... so...

But if you want to do it, and sometimes you do, historically, this has been pretty difficult to do in Rails.
That’s question 2 - how do you replace your primary datastore?
In reality, rather than this kind of thing, where all you have is Cassandra and no SQL-based storage at all, ...
Most apps use a hybrid approach. They use standard SQL persistance for some things, because there IS data that’s suited for, such as basic CRUD stuff.

Then they use non-SQL persistence for other things, where it fits. I mean, you can keep cassandra for your voluminous sparse data...if you have that, and a surprising number of apps do!
So here are the two interesting questions we have when we’re building a system that uses multiple datastores:
So just to sum up this section, this idea that multiple datastores are a fact of life if you’re building even simple applications is called polyglot persistence.

It’s not a new idea - Ben Scofield has been talking about it for more than a year. But a lot of Ruby developers think of “the data storage layer” and they think it’s a choice between MySQL and Postgres, when in reality, it’s going to end up with an assortment of different technologies. Whether or not they want that, and whether or not they explicitly plan for it.

So let’s take a look at what that’s going to look like.
Let’s start with the base case - I decided for my sample application I would build a cephalopod social network, mostly because I like the word cephalopod. So here we’re starting out with a basic Squid class. We’re inheriting from ActiveRecord::Base so we get the relational introspection and stuff for free.
Then our product owner comes along and tells us that customers want to free-text search the set of squids. So, we install sunspot, and add solr free-text search by adding a “searchable” block describing which attributes get indexed in solr.
And then of course since it’s a web application, we have to have a friend graph. So we decide to store a denormalized list of each squid’s friends in redis. This probably wouldn’t be the first solution I settled on, but pretend with me that I’ve already tried storing this in MySQL and encountered the pain of horrendous joins. I’ve realized that this type of relationship data just doesn’t fit into a row-centered relational datastore very well, so I’m trying something else.

So I add a “follow” method that uses the astonishingly-descriptively-named “redis” gem to insert stuff into a redis keyspace.
And then, of course, since this is a social network centered around cephalopods and the novels they write, I want to allow them to upload a finished manuscript to s3. So I install the, again, very usefully-named s3 gem, and add an upload_novel method that puts the tex file into s3.

Current datastore count: 4

Now at this point, the Squid class is started to look a little...
...mismatched.
You’re descending from ActiveRecord::Base, but you’ve got this searchable method from solr, plus you’ve got this follow method from redis, plus an upload method that uses s3, plus the magic hidden ActiveRecord methods. Each of these things persists a little piece of the whole squid.

You know what I would like? Some kind of consistent interface.
Now you may laugh, but I originally thought it would be awesome to be able to do everything through ActiveRecord. It’s very tempting! Most of us who use Rails think of ActiveRecord as synonymous with “model.”
It’d be awesome to have everything go through one interface. Active Record - it’s my model of all my data stores! My code lives happily on top!
But, sadly, it just doesn’t work that way. And it’s not going to. This is not because Rails core is trying to make our lives difficult. It’s because ActiveRecord is specifically built to model a relational database. It’s an ORM, which means Object-Relational Mapper. In fact, in Rails 3, ActiveRecord moves even more in that direction with Arel, which is, conceptually, an implementation of the theory of a relational database. This is called relational algebra. There’s a whole talk on it tomorrow! It’s awesome! It’s cool. It’s conceptually very consistent, easy to understand, and very tidy, all of which appeal to me.

Unfortunately, relational algebra is totally useless with a non-relational datastore where you don’t structure your data into rows, and columns, and tables. It just doesn’t map to something like a key-value store like redis, or column-oriented document database like mongo.
But once I started looking at Rails 3, I got very happy. The folks who rewrote it realized, correctly, that ActiveRecord in Rails 2 was modelling two very different types of behavior.
Here’s what we have in Rails 2: a big, blobby ActiveRecord that does both the communication with the database, and lots of other useful stuff like validations and lifecycle callbacks - stuff your calling code uses, but that isn’t directly related to how the data is persisted.
So in Rails 3 they’ve split this up into two separate libraries: ActiveRecord that handles persistance, and ActiveModel that handles validations and callbacks.

A lot of the blog posts and stuff about ActiveModel have talked about how this makes it easier to take validations and serialization and so on from ActiveModel and use them in plain old Ruby objects outside of Rails.

But as someone who does mostly Rails development, I actually don’t really care about that. What I’m excited about is that it’s now much easier to extract ActiveRecord from your models in Rails. It makes it much easier to write an adapter for a non-relational datastore and be able to use it in your models in a Rails app. All you have to do is present the same API to ActiveModel that ActiveRecord does, and that’s actually pretty easy. And in fact,
There are already several ActiveModel-compliant persistance libraries out there for accessing these things, like mongoid by Durran Jordan. So, you actually can, sort of, have a similar interface for different persistance models in Rails 3. Now, this doesn’t really solve the problem of multiple stores in one model, but it does make it possible to keep a lot of the niceties of what was previously “ActiveRecord” without actually persisting to a relational database. And that’s pretty hot!

Now that it’s getting easier to use a nonrelational store, I think we’re going to start seeing a lot more applications take advantage of them.
Here are a few of the extant ActiveModel-compliant libraries. Most of them are quite alpha and experimental. But expect to see more, and see them mature.
Here’s the basic setup. In Rails 3 we no longer inherit from ActiveRecord::Base, so the models are nicely decoupled from any default datastore. Here we’re defining Mongo as our primary store.

Sadly, ActiveModel doesn’t solve all of our problems. We still have to work at it to use more than one datastore within the same model.

For instance,
to add solr search we still need a searchable block. However, UNLIKE in Rails 2, this will still work no matter what primary persistance we use. It uses on the lifecycle methods that are part of ActiveModel. In Rails 2 they were so tightly coupled with ActiveRecord that you pretty much couldn’t use them separately.
So coming back to our 2 questions from the beginning of the talk, which were: how do you encapsulate a model that has data scattered across multiple stores, and how do you replace the primary store...
Sadly, the first one - not really solved yet. The best you can do is break stuff into modules where possible, and try to extract the common bits. No silver bullet here, just general good software engineering.

But the second one
...is fully solved in Rails 3. I’m actually really excited by that and it’ll be interesting to see which ActiveModel-compliant libraries gain traction.
And I’d like to close with this thought: if you do is read all the blogs and twitter, you might think SQL vs. NoSQL is some kind of standoff, and that you have to be on one side or the other. Either you’re the guys in yellow, or you’re the guys in blue.
But that’s a false dichotomy. Any application with more than a hundred users will use both. So please: quit it with the FUD, and look for ways they can place nice together.

AND, if you’re going to do this, upgrade to Rails 3!

Ruby APIs for NoSQL - Polyglot Persistence - GoGaRuCo 2010 — Presentation Transcript

Ruby APIs for NoSQL Sarah Mei Pivotal Labs github/twitter: sarahmei
Ruby APIs for NoSQL or Polyglot Persistence Sarah Mei Pivotal Labs github/twitter: sarahmei
The Baseline
The New Baseline
YES http://www.flickr.com/photos/gluemoon/180019580
The New Baseline
http://www.flickr.com/photos/frostnova/2039732274
1 http://www.flickr.com/photos/frostnova/2039732274
The New Baseline
2 http://www.ﬂickr.com/photos/faceme/1777843915
2 Questions • How do you unify a model with data scattered across multiple stores? • How do you replace the primary store?
http://www.flickr.com/photos/wwworks/2607036664
ActiveRecord datastore count: 1
gem: sunspot-rails datastore count: 2
gem: redis datastore count: 3
gem: s3 datastore count: 4
http://www.flickr.com/photos/foundphotoslj/536912809
http://www.flickr.com/photos/dbloete/368519992/
My Co MySQL image © Oracle (sigh) PostgreSQL image © The PostgreSQL Global Development Group Unicorn image: http://www.ﬂickr.com/photos/schoschie/3480868625
X My Co
progress http://www.flickr.com/photos/swanksalot/1419724568
Rails 2
Rails 3
ActiveModel mongoid mongo_mapper dm-active_model supermodel neo4j-rails couchrails DataMapper image © Dan Kubb Neo4j image © neo4j.org CouchDB image © The Apache Software Foundation
gem: mongoid
gem: mongoid
2 Questions • How do you unify a model with data scattered across multiple stores? • How do you replace the primary store?
2 Questions not solved yet • How do you unify a model with data scattered across multiple stores? • How do you replace the primary store?
2 Questions not solved yet • How do you unify a model with data scattered across multiple stores? • solved inyou replace the primary store? How do Rails 3
http://www.flickr.com/photos/kashklick/3406972544
http://www.flickr.com/photos/argenberg/478299028/
Questions http://www.flickr.com/photos/anselmhook/1101211907
Hack Day Sunday! postGogarucoHackDay.eventbrite.com

Ruby APIs for NoSQL - Polyglot Persistence - GoGaRuCo 2010

by Sarah Mei on Sep 17, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

1 Embed 7

Statistics

Ruby APIs for NoSQL - Polyglot Persistence - GoGaRuCo 2010 — Presentation Transcript