Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Server discovery and monitoring with MongoDB

215 views

Published on

Overview of how the drivers discover servers and recover from server failures.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Server discovery and monitoring with MongoDB

  1. 1. Server Discovery and Monitoring Joe Drumgoole Director of Developer Advocacy, EMEA 21-July-2016 V1.0
  2. 2. 2 MongoDB MongoDB Query Language (MQL) + Native Drivers MongoDB Document/JSON Data Model Wired Tiger MMAP Management Security In-memory Encrypted 3rd party Shared Clusters Replica Sets
  3. 3. 3 Drivers and Frameworks Morphia MEAN Stack
  4. 4. 4 Single Server Driver Mongod
  5. 5. 5 Replica Set Driver Secondary Secondary Primary
  6. 6. 6 Replica Set Primary Failure Driver Secondary Secondary
  7. 7. 7 Replica Set Election Driver Secondary Secondary
  8. 8. 8 Replica Set New Primary Driver Primary Secondary
  9. 9. 9 Replica Set Recovery Driver Primary Secondary Secondary
  10. 10. 10 Sharded Cluster Driver Mongod Mongod Mongod Mongod Mongod Mongod Mongod Mongod Mongod mongos mongos
  11. 11. 11 Driver Responsibilities https://github.com/mongodb/mongo-python-driver Driver Authentication & Security Python<->BSON Error handling & Recovery Wire Protocol Topology Management Connection Pool
  12. 12. 12 Driver Responsibilities https://github.com/mongodb/mongo-python-driver Driver Authentication & Security Python<->BSON Error handling & Recovery Wire Protocol Topology Management Connection Pool
  13. 13. 13 Example API Calls import pymongo client = pymongo.MongoClient( host=“localhost”, port=27017) database = client[ ‘test_database’ ] collection = database[ ‘test_collection’ ] collection.insert_one({ "hello" : "world" , "goodbye" : "world" } ) collection.find_one( { "hello" : "world" } ) collection.update({ "hello" : "world" }, { "$set" : { "buenos dias" : "world" }} ) collection.delete_one({ "hello" : "world" } )
  14. 14. 14 Start MongoClient c = MongoClient( "host1, host2", replicaSet="replset" )
  15. 15. 15 Client Side View Secondary host2 Secondary host3 Primary host1 Mongo Client MongoClient( "host1, host2", replicaSet="replset" )
  16. 16. 16 Client Side View Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 { ismaster : False, secondary: True, hosts : [ host1, host2, host3 ] }
  17. 17. 17 What Does ismaster show? >>> pprint.pprint( db.command( "ismaster" )) {u'hosts': [u'JD10Gen-old.local:27017', u'JD10Gen-old.local:27018', u'JD10Gen-old.local:27019'], u'ismaster' : False, u'secondary': True, u'setName' : u'replset', …} >>>
  18. 18. 18 Topology Current Topology ismaster New Topology
  19. 19. 19 Client Side View Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2
  20. 20. 20 Client Side View Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3
  21. 21. 21 Client Side View Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code
  22. 22. 22 Next Is Insert c = MongoClient( "host1, host2", replicaSet="replset" ) client.db.col.insert_one( { "a" : "b" } )
  23. 23. 23 Insert Will Block Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code Insert
  24. 24. 24 ismaster response from Host 1 Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code Insert ismaster
  25. 25. 25 Now Write Can Proceed Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code Insert Insert
  26. 26. 26 Later Host 3 Responds Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code
  27. 27. 27 Steady State Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code
  28. 28. 28 Life Intervenes Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code ✖
  29. 29. 29 Monitor may not detect Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code ✖ Insert ConnectionFailure
  30. 30. 30 So Retry Secondary host2 Secondary host3 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code ✖ Insert
  31. 31. 31 Check for Primary Secondary host2 Secondary host3 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code ✖ Insert
  32. 32. 32 Host 2 Is Primary Primary host2 Secondary host3 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code ✖ Insert
  33. 33. 33 Steady State Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code
  34. 34. 34 What Does This Mean? - Connect import pymongo client = pymongo.MongoClient() try: client.admin.command( "ismaster" ) except pymongo.errors.ConnectionFailure, e : print( "Cannot connect: %s" % e )
  35. 35. 35 What Does This Mean? - Queries import pymongo def find_with_recovery( collection, query ) : try: return collection.find_one( query ) except pymongo.errors.ConnectionFailure, e : logging.info( "Connection failure : %s" e ) return collection.find_one( query )
  36. 36. 36 What Does This Mean? - Inserts def insert_with_recovery( collection, doc ) : doc[ "_id" ] = ObjectId() try: collection.insert_one( doc ) except pymongo.errors.ConnectionFailure, e: logging.info( "Connection error: %s" % e ) try: collection.insert_one( doc ) except DuplicateKeyError: pass
  37. 37. 37 What Does This Mean? - Updates collection.update( { "_id" : 1 }, { "$inc" : { "counter" : 1 }})
  38. 38. 38 Configuration connectTimeoutMS : 30s serverTimeoutMS : 30s
  39. 39. 39 connectTimeoutMS Secondary host2 Secondary host3 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code ✖ Insert connectTimeoutMS
  40. 40. 40 serverTimeoutMS Secondary host2 Secondary host3 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code ✖ Insert serverTimeoutMS
  41. 41. 41 More Reading • The spec author Jess Jiryu Davis has a collection of links and his better version of this talk https://emptysqua.re/blog/server-discovery-and-monitoring-in-mongodb- drivers/ • The full server discovery and monitoring spec is on GitHub https://github.com/mongodb/specifications/blob/master/source/server- discovery-and-monitoring/server-discovery-and-monitoring.rst
  42. 42. 43 insert_one • Stages – Parse the parameters – Get a socket to write data on – Add the object Id – Convert the whole insert command and parameters to a SON object – Apply the writeConcern to the command – Encode the message into a BSON object – Send the message to the server via the socket (TCP/IP) – Check for writeErrors (e.g. DuplicateKeyError) – Check for writeConcernErrors (e.g.writeTimeout) – Return Result object
  43. 43. 44 Bulk Insert bulker = collection.initialize_ordered_bulk_op() bulker.insert( { "a" : "b" } ) bulker.insert( { "c" : "d" } ) bulker.insert( { "e" : "f" } ) try: bulker.execute() except pymongo.errors.BulkWriteError as e : print( "Bulk write error : %s" % e.detail )
  44. 44. 45 Bulk Write • Create Bulker object • Accumulate operations • Each operation is created as a SON object • The operations are accumulated in a list • Once execute is called – For ordered execute in order added – For unordered execute INSERT, UPDATEs then DELETE • Errors will abort the whole batch unless no write concern specified

×