0
Rick Copeland @rick446Arborian Consulting, LLC
http://www.flickr.com/photos/fazen/9079179/
-   Get started with PyMongo-   Sprinkle in some Ming schemas-   ODM: When a dict just won’t do
>>>importpymongo>>>conn = pymongo.Connection()>>>connConnection(localhost, 27017)>>>conn.testDatabase(Connection(localhost...
>>>db = conn.test>>>id = db.foo.insert({bar:1, baz:[ 1, 2, {k:5} ] })>>>idObjectId(4e712e21eb033009fa000000)              ...
>>>db.foo.insert([ dict(x=x) forxinrange(10) ])[ObjectId(4e71313aeb033009fa00000b), … ]>>>list(db.foo.find({ x: {$gt: 3} }...
One Rule (for now): Avoid Javascript                                       http://www.flickr.com/photos/lizjones/295567490/
   You gotta write Javascript (for now)   It’s pretty slow (single-threaded JS engine)                                ...
>>>importgridfs>>>fs = gridfs.GridFS(db)                           Python context>>>withfs.new_file() asfp:               ...
>>>file_id = fs.put(Moar data!, filename=foo.txt)>>>fs.get_last_version(foo.txt).read()Moar data!’                        ...
-   Get started with PyMongo-   Sprinkle in some Ming schemas-   ODM: When a dict just won’t do
   Your data has a schema       Your database can define and enforce it       It can live in your application (as with ...
Model              Datastore(schema)             (database)           Session
>>>importming.datastore>>>ds = ming.datastore.DataStore(mongodb://localhost:27017, database=test)>>>ds.db                 ...
http://www.flickr.com/photos/pictureclara/5333266789/
frommingimport schema, FieldWikiDoc= collection(‘wiki_page, session,Field(_id, schema.ObjectId()),                        ...
frommingimport Document, Session, FieldclassWikiDoc(Document):class__mongometa__:session=Session.by_name(’main)name=wiki_p...
>>>doc = WikiDoc(dict(title=Cats, text=I can hazcheezburger?))>>>doc.m.save()>>>WikiDoc.m.find()<ming.base.Cursor object a...
>>>ming.datastore.DataStore(mim://, database=test).dbmim.Database(test)   MongoDB is (generally) fast       … except whe...
-   Get started with PyMongo-   Sprinkle in some Ming schemas-   ODM: When a dict just won’t do
frommingimport schema, Fieldfromming.odmimport (mapper, Mapper, RelationProperty,ForeignIdProperty)                       ...
classWikiPage(MappedClass):class__mongometa__:session = main_odm_sessionname=wiki_page’indexes = [ title]    _id = FieldPr...
       Session ODMSession       My_collection.m… My_mapped_class.query…       ODMSessionactually does stuff         ...
http://www.flickr.com/photos/39747297@N05/5229733647/
   Various plug points in the session       before_flush       after_flush   Some uses       Logging changes to sensi...
   Various plug points in the mapper       before_/after_:           Insert           Update           Delete        ...
PyMongohttp://api.mongodb.org/pythonApache LicenseMinghttp://sf.net/projects/merciless/MIT License
Rick Copeland @rick446   Arborian Consulting, LLCFeedback? http://www.surveymonkey.com/s/5DLCYKN                          ...
Upcoming SlideShare
Loading in...5
×

Rapid and Scalable Development with MongoDB, PyMongo, and Ming

1,776

Published on

This intermediate-level talk will teach you techniques using the popular NoSQL database MongoDB and the Python library Ming to write maintainable, high-performance, and scalable applications. We will cover everything you need to become an effective Ming/MongoDB developer from basic PyMongo queries to high-level object-document mapping setups in Ming.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,776
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
15
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Who’s using mongodb? In production?Who’s using python? In production?Who’s using both together? In Production?I’m a consultant, but I *was* an engineer at SourceForgeI’ve been using MongoDB at SourceForge since 0.8, Python since 2002 or so, love them bothWe extracted our MongoDB libraries into a separate OSS project ‘Ming’ (play off of the planet Mongo from Flash Gordon)We’re going to have lots of code here. Hope you like Python!
  • Someone is going to put weird data in your database.Or they will try.Catch them in the act.
  • Transcript of "Rapid and Scalable Development with MongoDB, PyMongo, and Ming"

    1. 1. Rick Copeland @rick446Arborian Consulting, LLC
    2. 2. http://www.flickr.com/photos/fazen/9079179/
    3. 3. - Get started with PyMongo- Sprinkle in some Ming schemas- ODM: When a dict just won’t do
    4. 4. >>>importpymongo>>>conn = pymongo.Connection()>>>connConnection(localhost, 27017)>>>conn.testDatabase(Connection(localhost, 27017), utest)>>>conn.test.fooCollection(Database(Connection(localhost, 27017), utest), ufoo)>>>conn[test-db]Database(Connection(localhost, 27017), utest-db)>>>conn[test-db][foo-collection]Collection(Database(Connection(localhost, 27017), utest-db), ufoo- collection)>>>conn.test.foo.bar.bazCollection(Database(Connection(localhost, 27017), utest), ufoo.bar.baz)
    5. 5. >>>db = conn.test>>>id = db.foo.insert({bar:1, baz:[ 1, 2, {k:5} ] })>>>idObjectId(4e712e21eb033009fa000000) Auto-Generated _id>>>db.foo.find()<pymongo.cursor.Cursor object at 0x29c7d50>>>>list(db.foo.find()) Cursors are python[{ubar: 1, u_id: ObjectId(4e712e21eb033009fa000000), ubaz: [1, 2, {k: 5}]}] generators>>>db.foo.update({_id:id}, {$set: { bar:2}})>>>db.foo.find().next(){ubar: 2, u_id: ObjectId(4e712e21eb033009fa000000), ubaz: [1, 2, {k: 5}]}>>>db.foo.remove({_id:id})>>>list(db.foo.find())[] Remove uses same query language as find()
    6. 6. >>>db.foo.insert([ dict(x=x) forxinrange(10) ])[ObjectId(4e71313aeb033009fa00000b), … ]>>>list(db.foo.find({ x: {$gt: 3} })) Range Query[{ux: 4, u_id: ObjectId(4e71313aeb033009fa00000f)}, {ux: 5, u_id: ObjectId(4e71313aeb033009fa000010)}, {ux: 6, u_id: ObjectId(4e71313aeb033009fa000011)}, …]>>>list(db.foo.find({ x: {$gt: 3} }, { _id:0 } ))[{ux: 4}, {ux: 5}, {ux: 6}, {ux: 7}, {ux: 8}, {ux: 9}]>>>list(db.foo.find({ x: {$gt: 3} }, { _id:0 } )....skip(1).limit(2))[{ux: 5}, {ux: 6}]>>>db.foo.ensure_index([ Partial Retrieval...(x’,pymongo.ASCENDING),(y’,pymongo.DESCENDING)])ux_1_y_-1’ Compound Indexes
    7. 7. One Rule (for now): Avoid Javascript http://www.flickr.com/photos/lizjones/295567490/
    8. 8.  You gotta write Javascript (for now) It’s pretty slow (single-threaded JS engine)  MongoDB 2.2 with New Javascript is used by Aggregation Framework  $where in a query Coming Real Soon Now ™  .group(key, condition, initial, reduce, finalize=None)  .map_reduce(map, reduce, out, finalize=None, …) Sharding gives some parallelism with .map_reduce() (and possibly ‘$where’). Otherwise you’re single threaded.
    9. 9. >>>importgridfs>>>fs = gridfs.GridFS(db) Python context>>>withfs.new_file() asfp: manager... fp.write(The file)...>>>fp<gridfs.grid_file.GridIn object at 0x2cae910>>>>fp._idObjectId(4e727f64eb03300c0b000003) Retrieve file by _id>>>fs.get(fp._id).read()The file Arbitrary data can be stored in the ‘fp’ object – it’s just a Document (but please put it in ‘fp.metadata’)  Mime type, links to other docs, etc.
    10. 10. >>>file_id = fs.put(Moar data!, filename=foo.txt)>>>fs.get_last_version(foo.txt).read()Moar data!’ Create file by filename>>>file_id = fs.put(Evenmoar data!, filename=foo.txt)>>>fs.get_last_version(foo.txt).read()Even moar data!’>>>fs.get_version(foo.txt, -2).read()Moar data!’>>>fs.list() “2nd from the last”[ufoo.txt]>>>fs.delete(fs.get_last_version(foo.txt)._id)>>>fs.list()[ufoo.txt]>>>fs.delete(fs.get_last_version(foo.txt)._id)>>>fs.list()[]
    11. 11. - Get started with PyMongo- Sprinkle in some Ming schemas- ODM: When a dict just won’t do
    12. 12.  Your data has a schema  Your database can define and enforce it  It can live in your application (as with MongoDB)  Nice to have the schema defined in one place in the code Sometimes you need a “migration”  Changing the structure/meaning of fields  Adding indexes, particularly unique indexes  Sometimes lazy, sometimes eager “Unit of work:” Queuing up all your updates can be handy
    13. 13. Model Datastore(schema) (database) Session
    14. 14. >>>importming.datastore>>>ds = ming.datastore.DataStore(mongodb://localhost:27017, database=test)>>>ds.db Connection +Database(Connection(localhost, 27017), utest) Database>>>session = ming.Session(ds)>>>session.dbDatabase(Connection(localhost, 27017), utest)>>>ming.configure(**{ Optimized for config... ming.main.master:mongodb://localhost:27017, files... ming.main.database:test})>>>Session.by_name(main).dbDatabase(Connection(ulocalhost, 27017), utest)
    15. 15. http://www.flickr.com/photos/pictureclara/5333266789/
    16. 16. frommingimport schema, FieldWikiDoc= collection(‘wiki_page, session,Field(_id, schema.ObjectId()), Index created onField(title, str, index=True), importField(text, str))CommentDoc= collection(‘comment, session,Field(_id, schema.ObjectId()),Field(page_id, schema.ObjectId(), index=True),Field(text, str)) Shorthand for schema.String
    17. 17. frommingimport Document, Session, FieldclassWikiDoc(Document):class__mongometa__:session=Session.by_name(’main)name=wiki_page’indexes=[ (title) ]title = Field(str)text = Field(str) Old declarative syntax continues to exist and be supported, but it’s not being actively improved Sometimes nice when you want additional methods/attrs on your document class
    18. 18. >>>doc = WikiDoc(dict(title=Cats, text=I can hazcheezburger?))>>>doc.m.save()>>>WikiDoc.m.find()<ming.base.Cursor object at 0x2c2cd90>>>>WikiDoc.m.find().all() Documents are dict subclasses[{text: uI can hazcheezburger?, _id: ObjectId(4e727163eb03300c0b000001), title: uCats}]>>>WikiDoc.m.find().one().textuIcan hazcheezburger?’>>>doc = WikiDoc(dict(tietul=LOL, text=Invisible bicycle))>>>doc.m.save()Traceback(most recent call last):File "<stdin>", line 1, … Exception pinpointsming.schema.Invalid: <class ming.metadata.Document<wiki_page>>: problemExtra keys: set([tietul])
    19. 19. >>>ming.datastore.DataStore(mim://, database=test).dbmim.Database(test) MongoDB is (generally) fast  … except when creating databases  … particularly when you preallocate Unit tests like things to be isolated MIM gives you isolation at the expense of speed & scaling
    20. 20. - Get started with PyMongo- Sprinkle in some Ming schemas- ODM: When a dict just won’t do
    21. 21. frommingimport schema, Fieldfromming.odmimport (mapper, Mapper, RelationProperty,ForeignIdProperty) Plain Old PythonWikiDoc= collection(wiki_page,session, … ) ClassesCommentDoc= collection(’comment’, session, … )classWikiPage(object): pass Map classes toclassComment(object): pass collection + sessionodmsession.mapper(WikiPage, WikiDoc, properties=dict(comments=RelationProperty(WikiComment)))odmsession.mapper(Comment, CommentDoc, properties=dict(page_id=ForeignIdProperty(WikiPage),page=RelationProperty(WikiPage))) “Relations”
    22. 22. classWikiPage(MappedClass):class__mongometa__:session = main_odm_sessionname=wiki_page’indexes = [ title] _id = FieldProperty(S.ObjectId)title = FieldProperty(str)text = FieldProperty(str)comments = RelationProperty(’Comment’)
    23. 23.  Session ODMSession My_collection.m… My_mapped_class.query… ODMSessionactually does stuff  Track object identity  Track object modifications  Unit of work flushing all changes at once >>>pg = WikiPage(title=MyPage, text=is here) >>>session.db.wiki_page.count() 0 >>>main_orm_session.flush() >>>session.db.wiki_page.count() 1
    24. 24. http://www.flickr.com/photos/39747297@N05/5229733647/
    25. 25.  Various plug points in the session  before_flush  after_flush Some uses  Logging changes to sensitive data or for analytics  Full-text search indexing  “last modified” fields  Performance instrumentation
    26. 26.  Various plug points in the mapper  before_/after_:  Insert  Update  Delete  Remove Some uses  Collection/model-specific logging (user creation, etc.)  Anything you might want a SessionExtension for but would rather do per-model
    27. 27. PyMongohttp://api.mongodb.org/pythonApache LicenseMinghttp://sf.net/projects/merciless/MIT License
    28. 28. Rick Copeland @rick446 Arborian Consulting, LLCFeedback? http://www.surveymonkey.com/s/5DLCYKN http://www.flickr.com/photos/f-oxymoron/5005673112/
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×