EuroPython 2016 : A Deep Dive into the Pymongo Driver
•Download as PPTX, PDF•
0 likes•405 views
A detailed overview of how the Pymongo MongoDB driver handles replica set connection and fail over.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (19)
Similar to EuroPython 2016 : A Deep Dive into the Pymongo Driver
Similar to EuroPython 2016 : A Deep Dive into the Pymongo Driver (20)
More from Joe Drumgoole
More from Joe Drumgoole (11)
Recently uploaded
Recently uploaded (20)
EuroPython 2016 : A Deep Dive into the Pymongo Driver
- 1. A Deep Dive into the Pymongo Driver Joe Drumgoole Director of Developer Advocacy, EMEA 21-July-2016 V1.0
- 2. 2 MongoDB MongoDB Query Language (MQL) + Native Drivers MongoDB Document/JSON Data Model Wired Tiger MMAP Management Security In-memory Encrypted 3rd party Shared Clusters Replica Sets
- 4. 4 BSON Side Bar • MongoDB uses a binary format of JSON called BSON (Binary, jSON) • Adds type and size information • Allows efficient parsing and skipping • You can use MongoDB Drivers without every knowing that BSON exists • Open standard (http://bsonspec.org/, licensed under the Creative Commons) • There are BSON libraries in every driver if you fancy trying it out • Similar to google protocol buffers
- 7. 7 Replica Set Primary Failure Driver Secondary Secondary
- 8. 8 Replica Set Election Driver Secondary Secondary
- 9. 9 Replica Set New Primary Driver Primary Secondary
- 10. 10 Replica Set Recovery Driver Primary Secondary Secondary
- 11. 11 Sharded Cluster Driver Mongod Mongod Mongod Mongod Mongod Mongod Mongod Mongod Mongod mongos mongos
- 12. 12 Driver Responsibilities https://github.com/mongodb/mongo-python-driver Driver Authentication & Security Python<->BSON Error handling & Recovery Wire Protocol Topology Management Connection Pool
- 13. 13 Driver Responsibilities https://github.com/mongodb/mongo-python-driver Driver Authentication & Security Python<->BSON Error handling & Recovery Wire Protocol Topology Management Connection Pool
- 14. 14 Example API Calls import pymongo client = pymongo.MongoClient( host=“localhost”, port=27017) database = client[ ‘test_database’ ] collection = database[ ‘test_collection’ ] collection.insert_one({ "hello" : "world" , "goodbye" : "world" } ) collection.find_one( { "hello" : "world" } ) collection.update({ "hello" : "world" }, { "$set" : { "buenos dias" : "world" }} ) collection.delete_one({ "hello" : "world" } )
- 15. 15 Start MongoClient c = MongoClient( "host1, host2", replicaSet="replset" )
- 16. 16 Client Side View Secondary host2 Secondary host3 Primary host1 Mongo Client MongoClient( "host1, host2", replicaSet="replset" )
- 17. 17 Client Side View Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 { ismaster : False, secondary: True, hosts : [ host1, host2, host3 ] }
- 18. 18 What Does ismaster show? >>> pprint.pprint( db.command( "ismaster" )) {u'hosts': [u'JD10Gen-old.local:27017', u'JD10Gen-old.local:27018', u'JD10Gen-old.local:27019'], u'ismaster' : False, u'secondary': True, u'setName' : u'replset', …} >>>
- 20. 20 Client Side View Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2
- 21. 21 Client Side View Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3
- 22. 22 Client Side View Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code
- 23. 23 Next Is Insert c = MongoClient( "host1, host2", replicaSet="replset" ) client.db.col.insert_one( { "a" : "b" } )
- 24. 24 Insert Will Block Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code Insert
- 25. 25 ismaster response from Host 1 Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code Insert ismaster
- 26. 26 Now Write Can Proceed Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code Insert Insert
- 27. 27 Later Host 3 Responds Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code
- 28. 28 Steady State Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code
- 29. 29 Life Intervenes Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code ✖
- 30. 30 Monitor may not detect Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code ✖ Insert ConnectionFailure
- 31. 31 So Retry Secondary host2 Secondary host3 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code ✖ Insert
- 32. 32 Check for Primary Secondary host2 Secondary host3 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code ✖ Insert
- 33. 33 Host 2 Is Primary Primary host2 Secondary host3 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code ✖ Insert
- 34. 34 Steady State Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code
- 35. 35 What Does This Mean? - Connect import pymongo client = pymongo.MongoClient() try: client.admin.command( "ismaster" ) except pymongo.errors.ConnectionFailure, e : print( "Cannot connect: %s" % e )
- 36. 36 What Does This Mean? - Queries import pymongo def find_with_recovery( collection, query ) : try: return collection.find_one( query ) except pymongo.errors.ConnectionFailure, e : logging.info( "Connection failure : %s" e ) return collection.find_one( query )
- 37. 37 What Does This Mean? - Inserts def insert_with_recovery( collection, doc ) : doc[ "_id" ] = ObjectId() try: collection.insert_one( doc ) except pymongo.errors.ConnectionFailure, e: logging.info( "Connection error: %s" % e ) collection.insert_one( doc ) except DuplicateKeyError: pass
- 38. 38 What Does This Mean? - Updates collection.update( { "_id" : 1 }, { "$inc" : { "counter" : 1 }})
- 40. 40 connectTimeoutMS Secondary host2 Secondary host3 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code ✖ Insert connectTimeoutMS serverTimeoutMS
- 41. 41 More Reading • The spec author Jess Jiryu Davis has a collection of links and his better version of this talk https://emptysqua.re/blog/server-discovery-and-monitoring-in-mongodb- drivers/ • The full server discovery and monitoring spec is on GitHub https://github.com/mongodb/specifications/blob/master/source/server- discovery-and-monitoring/server-discovery-and-monitoring.rst
- 43. 43 insert_one • Stages – Parse the parameters – Get a socket to write data on – Add the object Id – Convert the whole insert command and parameters to a SON object – Apply the writeConcern to the command – Encode the message into a BSON object – Send the message to the server via the socket (TCP/IP) – Check for writeErrors (e.g. DuplicateKeyError) – Check for writeConcernErrors (e.g.writeTimeout) – Return Result object
- 44. 44 Bulk Insert bulker = collection.initialize_ordered_bulk_op() bulker.insert( { "a" : "b" } ) bulker.insert( { "c" : "d" } ) bulker.insert( { "e" : "f" } ) try: bulker.execute() except pymongo.errors.BulkWriteError as e : print( "Bulk write error : %s" % e.detail )
- 45. 45 Bulk Write • Create Bulker object • Accumulate operations • Each operation is created as a SON object • The operations are accumulated in a list • Once execute is called – For ordered execute in order added – For unordered execute INSERT, UPDATEs then DELETE • Errors will abort the whole batch unless no write concern specified
Editor's Notes
- Primary secondary, secondary
- Primary secondary, secondary
- Primary secondary, secondary
- Primary secondary, secondary
- Primary secondary, secondary
- Present a native language interface - converts python types to BSON objects Convert the JSON query language into commands for the database Convert JSON data into BSON data and vice-versa Handles interfacing to different MongoDB topologies Helps recover from server side outages/network errors Manages the client side connection pool The pymongo driver code is on Github (Apache License)
- Present a native language interface - converts python types to BSON objects Convert the JSON query language into commands for the database Convert JSON data into BSON data and vice-versa Handles interfacing to different MongoDB topologies Helps recover from server side outages/network errors Manages the client side connection pool The pymongo driver code is on Github (Apache License)
- Calls i
- Calls i
- State machine, full set of states defined in spec.
- Calls i
- Calls i
- Calls i
- Needs a primary to complete a write.
- Needs a primary to complete a write.
- Needs a primary to complete a write.
- Needs a primary to complete a write.
- Each thread wakes every 10 seconds. Runs ismaster, sleeps. We use ismaster to check latency. Keep topology description up to date.
- Each thread wakes every 10 seconds. Runs ismaster, sleeps. We use ismaster to check latency. Keep topology description up to date.
- Each thread wakes every 10 seconds. Runs ismaster, sleeps. We use ismaster to check latency. Keep topology description up to date.
- Primary is marked as unknown Wakes up all monitor threads to check for a primary.
- Primary is marked as unknown Wakes up all monitor threads to check for a primary every half second.
- Primary is marked as unknown Wakes up all monitor threads to check for a primary every half second.
- Each thread wakes every 10 seconds. Runs ismaster, sleeps. We use ismaster to check latency. Keep topology description up to date.
- Try once. This will accomdate elections. Other errore should be propagated.
- Try once. This will accomdate elections. Other errore should be propagated.
- Can you afford to over or under count. Operations need to be idempotent. Turn an update into a write of a document, cf EventSourcing. Then aggregate on the server.
- How long should a connection wait before timing out and sleeping for 10 seconds.