• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Rapid and Scalable Development with MongoDB, PyMongo, and Ming
 

Rapid and Scalable Development with MongoDB, PyMongo, and Ming

on

  • 1,796 views

This intermediate-level talk will teach you techniques using the popular NoSQL database MongoDB and the Python library Ming to write maintainable, high-performance, and scalable applications. We will ...

This intermediate-level talk will teach you techniques using the popular NoSQL database MongoDB and the Python library Ming to write maintainable, high-performance, and scalable applications. We will cover everything you need to become an effective Ming/MongoDB developer from basic PyMongo queries to high-level object-document mapping setups in Ming.

Statistics

Views

Total Views
1,796
Views on SlideShare
1,692
Embed Views
104

Actions

Likes
1
Downloads
13
Comments
0

4 Embeds 104

http://www.arborian.com 100
http://feeds.feedburner.com 2
http://fiberization26.katinia.com 1
https://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Who’s using mongodb? In production?Who’s using python? In production?Who’s using both together? In Production?I’m a consultant, but I *was* an engineer at SourceForgeI’ve been using MongoDB at SourceForge since 0.8, Python since 2002 or so, love them bothWe extracted our MongoDB libraries into a separate OSS project ‘Ming’ (play off of the planet Mongo from Flash Gordon)We’re going to have lots of code here. Hope you like Python!
  • Someone is going to put weird data in your database.Or they will try.Catch them in the act.

Rapid and Scalable Development with MongoDB, PyMongo, and Ming Rapid and Scalable Development with MongoDB, PyMongo, and Ming Presentation Transcript

  • Rick Copeland @rick446Arborian Consulting, LLC
  • http://www.flickr.com/photos/fazen/9079179/
  • - Get started with PyMongo- Sprinkle in some Ming schemas- ODM: When a dict just won’t do
  • >>>importpymongo>>>conn = pymongo.Connection()>>>connConnection(localhost, 27017)>>>conn.testDatabase(Connection(localhost, 27017), utest)>>>conn.test.fooCollection(Database(Connection(localhost, 27017), utest), ufoo)>>>conn[test-db]Database(Connection(localhost, 27017), utest-db)>>>conn[test-db][foo-collection]Collection(Database(Connection(localhost, 27017), utest-db), ufoo- collection)>>>conn.test.foo.bar.bazCollection(Database(Connection(localhost, 27017), utest), ufoo.bar.baz)
  • >>>db = conn.test>>>id = db.foo.insert({bar:1, baz:[ 1, 2, {k:5} ] })>>>idObjectId(4e712e21eb033009fa000000) Auto-Generated _id>>>db.foo.find()<pymongo.cursor.Cursor object at 0x29c7d50>>>>list(db.foo.find()) Cursors are python[{ubar: 1, u_id: ObjectId(4e712e21eb033009fa000000), ubaz: [1, 2, {k: 5}]}] generators>>>db.foo.update({_id:id}, {$set: { bar:2}})>>>db.foo.find().next(){ubar: 2, u_id: ObjectId(4e712e21eb033009fa000000), ubaz: [1, 2, {k: 5}]}>>>db.foo.remove({_id:id})>>>list(db.foo.find())[] Remove uses same query language as find()
  • >>>db.foo.insert([ dict(x=x) forxinrange(10) ])[ObjectId(4e71313aeb033009fa00000b), … ]>>>list(db.foo.find({ x: {$gt: 3} })) Range Query[{ux: 4, u_id: ObjectId(4e71313aeb033009fa00000f)}, {ux: 5, u_id: ObjectId(4e71313aeb033009fa000010)}, {ux: 6, u_id: ObjectId(4e71313aeb033009fa000011)}, …]>>>list(db.foo.find({ x: {$gt: 3} }, { _id:0 } ))[{ux: 4}, {ux: 5}, {ux: 6}, {ux: 7}, {ux: 8}, {ux: 9}]>>>list(db.foo.find({ x: {$gt: 3} }, { _id:0 } )....skip(1).limit(2))[{ux: 5}, {ux: 6}]>>>db.foo.ensure_index([ Partial Retrieval...(x’,pymongo.ASCENDING),(y’,pymongo.DESCENDING)])ux_1_y_-1’ Compound Indexes
  • One Rule (for now): Avoid Javascript http://www.flickr.com/photos/lizjones/295567490/
  •  You gotta write Javascript (for now) It’s pretty slow (single-threaded JS engine)  MongoDB 2.2 with New Javascript is used by Aggregation Framework  $where in a query Coming Real Soon Now ™  .group(key, condition, initial, reduce, finalize=None)  .map_reduce(map, reduce, out, finalize=None, …) Sharding gives some parallelism with .map_reduce() (and possibly ‘$where’). Otherwise you’re single threaded.
  • >>>importgridfs>>>fs = gridfs.GridFS(db) Python context>>>withfs.new_file() asfp: manager... fp.write(The file)...>>>fp<gridfs.grid_file.GridIn object at 0x2cae910>>>>fp._idObjectId(4e727f64eb03300c0b000003) Retrieve file by _id>>>fs.get(fp._id).read()The file Arbitrary data can be stored in the ‘fp’ object – it’s just a Document (but please put it in ‘fp.metadata’)  Mime type, links to other docs, etc.
  • >>>file_id = fs.put(Moar data!, filename=foo.txt)>>>fs.get_last_version(foo.txt).read()Moar data!’ Create file by filename>>>file_id = fs.put(Evenmoar data!, filename=foo.txt)>>>fs.get_last_version(foo.txt).read()Even moar data!’>>>fs.get_version(foo.txt, -2).read()Moar data!’>>>fs.list() “2nd from the last”[ufoo.txt]>>>fs.delete(fs.get_last_version(foo.txt)._id)>>>fs.list()[ufoo.txt]>>>fs.delete(fs.get_last_version(foo.txt)._id)>>>fs.list()[]
  • - Get started with PyMongo- Sprinkle in some Ming schemas- ODM: When a dict just won’t do
  •  Your data has a schema  Your database can define and enforce it  It can live in your application (as with MongoDB)  Nice to have the schema defined in one place in the code Sometimes you need a “migration”  Changing the structure/meaning of fields  Adding indexes, particularly unique indexes  Sometimes lazy, sometimes eager “Unit of work:” Queuing up all your updates can be handy
  • Model Datastore(schema) (database) Session
  • >>>importming.datastore>>>ds = ming.datastore.DataStore(mongodb://localhost:27017, database=test)>>>ds.db Connection +Database(Connection(localhost, 27017), utest) Database>>>session = ming.Session(ds)>>>session.dbDatabase(Connection(localhost, 27017), utest)>>>ming.configure(**{ Optimized for config... ming.main.master:mongodb://localhost:27017, files... ming.main.database:test})>>>Session.by_name(main).dbDatabase(Connection(ulocalhost, 27017), utest)
  • http://www.flickr.com/photos/pictureclara/5333266789/
  • frommingimport schema, FieldWikiDoc= collection(‘wiki_page, session,Field(_id, schema.ObjectId()), Index created onField(title, str, index=True), importField(text, str))CommentDoc= collection(‘comment, session,Field(_id, schema.ObjectId()),Field(page_id, schema.ObjectId(), index=True),Field(text, str)) Shorthand for schema.String
  • frommingimport Document, Session, FieldclassWikiDoc(Document):class__mongometa__:session=Session.by_name(’main)name=wiki_page’indexes=[ (title) ]title = Field(str)text = Field(str) Old declarative syntax continues to exist and be supported, but it’s not being actively improved Sometimes nice when you want additional methods/attrs on your document class
  • >>>doc = WikiDoc(dict(title=Cats, text=I can hazcheezburger?))>>>doc.m.save()>>>WikiDoc.m.find()<ming.base.Cursor object at 0x2c2cd90>>>>WikiDoc.m.find().all() Documents are dict subclasses[{text: uI can hazcheezburger?, _id: ObjectId(4e727163eb03300c0b000001), title: uCats}]>>>WikiDoc.m.find().one().textuIcan hazcheezburger?’>>>doc = WikiDoc(dict(tietul=LOL, text=Invisible bicycle))>>>doc.m.save()Traceback(most recent call last):File "<stdin>", line 1, … Exception pinpointsming.schema.Invalid: <class ming.metadata.Document<wiki_page>>: problemExtra keys: set([tietul])
  • >>>ming.datastore.DataStore(mim://, database=test).dbmim.Database(test) MongoDB is (generally) fast  … except when creating databases  … particularly when you preallocate Unit tests like things to be isolated MIM gives you isolation at the expense of speed & scaling
  • - Get started with PyMongo- Sprinkle in some Ming schemas- ODM: When a dict just won’t do
  • frommingimport schema, Fieldfromming.odmimport (mapper, Mapper, RelationProperty,ForeignIdProperty) Plain Old PythonWikiDoc= collection(wiki_page,session, … ) ClassesCommentDoc= collection(’comment’, session, … )classWikiPage(object): pass Map classes toclassComment(object): pass collection + sessionodmsession.mapper(WikiPage, WikiDoc, properties=dict(comments=RelationProperty(WikiComment)))odmsession.mapper(Comment, CommentDoc, properties=dict(page_id=ForeignIdProperty(WikiPage),page=RelationProperty(WikiPage))) “Relations”
  • classWikiPage(MappedClass):class__mongometa__:session = main_odm_sessionname=wiki_page’indexes = [ title] _id = FieldProperty(S.ObjectId)title = FieldProperty(str)text = FieldProperty(str)comments = RelationProperty(’Comment’)
  •  Session ODMSession My_collection.m… My_mapped_class.query… ODMSessionactually does stuff  Track object identity  Track object modifications  Unit of work flushing all changes at once >>>pg = WikiPage(title=MyPage, text=is here) >>>session.db.wiki_page.count() 0 >>>main_orm_session.flush() >>>session.db.wiki_page.count() 1
  • http://www.flickr.com/photos/39747297@N05/5229733647/
  •  Various plug points in the session  before_flush  after_flush Some uses  Logging changes to sensitive data or for analytics  Full-text search indexing  “last modified” fields  Performance instrumentation
  •  Various plug points in the mapper  before_/after_:  Insert  Update  Delete  Remove Some uses  Collection/model-specific logging (user creation, etc.)  Anything you might want a SessionExtension for but would rather do per-model
  • PyMongohttp://api.mongodb.org/pythonApache LicenseMinghttp://sf.net/projects/merciless/MIT License
  • Rick Copeland @rick446 Arborian Consulting, LLCFeedback? http://www.surveymonkey.com/s/5DLCYKN http://www.flickr.com/photos/f-oxymoron/5005673112/