Fun Teaching MongoDBNew TricksRobert J. MoorePresidentAllanbank Consulting, Inc.Robert.J.Moore@allanbank.com
Agenda   Inspiration   A Little Fun: MongoDB as     ... a Coordination Service?     ... a Message Broker?   https://g...
Disclaimer Dont try these at home They are hacks in the best sense Professional developer on an air gap  laptop Enough...
Inspiration - Topics Publish / Subscribe     One writer, many readers   MongoDB Capped Collection   Tailable Cursor  ...
Inspiration - Topics Create a Capped Collection   Acts like a ring buffer Tailable Cursor returns documents as  they ar...
Asynchronous Java Driver Website   http://www.allanbank.com/mongodb-async-driver Focus on performance and usability Lo...
Asynchronous Java DriverPerformance  http://www.allanbank.com/mongodb-async-driver/performance/ycsb.html
Coordination Service ZooKeeper   Notification that something of interest    changed     Watches   Track entities in a ...
Watches Replica Set Oplog   All Insert, Update, Delete operations   Capped – Tailable cursors again Want to convert fr...
Oplog Documents Insert     { "ts" : { "t" : 1362958492000, "i" : 1 },       "h" : NumberLong("5915409566571821368"), "v" ...
Initialize a Watcher - Codefinal DocumentAssignable wantQuery =        or(            where("ns").equals(ns)             ....
Watcher - CodeWatcher watcher = new Watcher(client, collection,        Pattern.compile(".*"), new WatchListener() {       ...
Watcher Demo
Group Management Notification of:    Adding a Group Member    Removing a Group Member        Or Group Member Disappear...
Group Manager Client - CodeGroupManager manager = new GroupManager(executor, client, collection, rootContext);manager.addL...
Group Demo
Queues Divide and Conquer   One writer, one reader MongoDB Capped Collection Want to “share” a Cursor
Restartable Cursors  What if we had multiple applications passing the   same “getmore” requests?  “getmore” message refe...
Queues    Create a Capped Collection    Initialize a Tailable Cursor    Share with multiple processes                  ...
Restartable Cursors Added the ability to:    Persist an iterator or stream as a doument    Halt or stop the iterator st...
Queues – Initialization Code Initialize a Tailable Cursor Share with multiple processes   Find.Builder builder = new Fin...
Queues – Consumer Code   MongoCollection index = client.getDatabase(args[1]).getCollection(           "lookup");   Documen...
Queues Demo
Restartable Cursors - Gotcha Remember this: builder.setAwaitData(false); // Sigh. That masks a small issue...Pin( long l...
Final Thoughts Coordination Service   Watchers   Group Management Message Broker   Topics   Queues Remember the Dis...
Questions?    Contact Information:        Robert.J.Moore@allanbank.com
Fun Teaching MongoDB New Tricks
Upcoming SlideShare
Loading in …5
×

Fun Teaching MongoDB New Tricks

2,667 views

Published on

MongoDB is the trusted document store we turn to when we have tough data store problems to solve. For this talk we are going to go a little bit off the path and explore what other roles we can fit MongoDB into. Others have discussed how to turn MongoDB’s capped collections into a publish/subscribe server. We stretch that a little further and turn MongoDB into a full fledged broker with both publish/subscribe and queue semantics, and a the ability to mix them. We will provide code and a running demo of the queue producers and consumers. Next we will turn to coordination services: We will explore the fundamental features and show how to implement them using MongoDB as the storage engine. Again we will show the code and demo the coordination of multiple applications.

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,667
On SlideShare
0
From Embeds
0
Number of Embeds
428
Actions
Shares
0
Downloads
19
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Fun Teaching MongoDB New Tricks

  1. 1. Fun Teaching MongoDBNew TricksRobert J. MoorePresidentAllanbank Consulting, Inc.Robert.J.Moore@allanbank.com
  2. 2. Agenda Inspiration A Little Fun: MongoDB as  ... a Coordination Service?  ... a Message Broker? https://github.com/allanbank/mongodb-tricks
  3. 3. Disclaimer Dont try these at home They are hacks in the best sense Professional developer on an air gap laptop Enough! On with the fun
  4. 4. Inspiration - Topics Publish / Subscribe  One writer, many readers MongoDB Capped Collection Tailable Cursor Can also have selectors
  5. 5. Inspiration - Topics Create a Capped Collection  Acts like a ring buffer Tailable Cursor returns documents as they are added in the future ... Cursor Document
  6. 6. Asynchronous Java Driver Website  http://www.allanbank.com/mongodb-async-driver Focus on performance and usability Lower latency, higher throughput even under heavy thread contention Benefit when using the synchronous interface Greater benefit using the asynchronous interface
  7. 7. Asynchronous Java DriverPerformance http://www.allanbank.com/mongodb-async-driver/performance/ycsb.html
  8. 8. Coordination Service ZooKeeper  Notification that something of interest changed  Watches  Track entities in a group  Group Management
  9. 9. Watches Replica Set Oplog  All Insert, Update, Delete operations  Capped – Tailable cursors again Want to convert from the Oplog  Different fields contain the _id of the document
  10. 10. Oplog Documents Insert { "ts" : { "t" : 1362958492000, "i" : 1 }, "h" : NumberLong("5915409566571821368"), "v" : 2, "op" : "i", "ns" : "test.test", "o" : { "_id" : "513d189c8544eb2b5e000001" } } Delete { ... "op" : "d", ..., "b" : true, "o" : { "_id" : "513d189c8544eb2b5e000001" } } Update { ... "op" : "u", ..., "o2" : { "_id" : "513d189c8544eb2b5e000001" }, "o" : { "$set" : { "i" : 1 } } }
  11. 11. Initialize a Watcher - Codefinal DocumentAssignable wantQuery = or( where("ns").equals(ns) .and("op").in(constant("i"), constant("d")) .and("o._id").matches(myContext), where("ns").equals(ns) .and("op").equals("u") .and("o2._id").matches(myContext));final Find.Builder builder = new Find.Builder();if (myLastTs != null) { final DocumentBuilder tsQuery = BuilderFactory.start(); tsQuery.push(myLastTs.getName()) .add(myLastTs.withName(ComparisonOperator.GT.getToken())); builder.setQuery(and(tsQuery, wantQuery));}else { builder.setQuery(wantQuery);}builder.tailable();MongoDatabase localDb = myMongoClient.getDatabase("local");MongoCollection oplog = localDb.getCollection("oplog.rs");myControls = oplog.streamingFind(new OpLogNotification(), builder.build());
  12. 12. Watcher - CodeWatcher watcher = new Watcher(client, collection, Pattern.compile(".*"), new WatchListener() { @Override public void changed(Operation op, String context, Document document) { if (op == Operation.DELETE) { System.out.println(context + ": " + op); } else { System.out.println(context + ": " + op + ": " + document); } } });watcher.start();
  13. 13. Watcher Demo
  14. 14. Group Management Notification of:  Adding a Group Member  Removing a Group Member  Or Group Member Disappears Combination of:  Watcher  Scheduled Task for Heartbeat  Expire old members
  15. 15. Group Manager Client - CodeGroupManager manager = new GroupManager(executor, client, collection, rootContext);manager.addListener(new GroupListener() { @Override public void memberRemoved(String context) { System.out.println(context + " - Removed"); } @Override public void memberAdded(String context) { System.out.println(context + " - Added"); }});manager.start();final GroupMember member = manager.addMember();// Faster cleanup, if we can.Runtime.getRuntime().addShutdownHook(new Thread() { @Override public void run() { member.remove(); }});
  16. 16. Group Demo
  17. 17. Queues Divide and Conquer  One writer, one reader MongoDB Capped Collection Want to “share” a Cursor
  18. 18. Restartable Cursors  What if we had multiple applications passing the same “getmore” requests?  “getmore” message references a cursorstruct { MsgHeader header; // standard message header int32 ZERO; // 0 - reserved for future use cstring fullCollectionName; // "dbname.collectionname" int32 numberToReturn; // number of documents to return int64 cursorID; // cursorID from the OP_REPLY}http://docs.mongodb.org/meta-driver/latest/legacy/mongodb-wire-protocol/#op-get-more  We need a way to save and restore the cursor  Lets call them Restartable Cursors, but they dont exist, yet
  19. 19. Queues Create a Capped Collection Initialize a Tailable Cursor Share with multiple processes ... Cursor Application 1 Application 2 Application 3 Document
  20. 20. Restartable Cursors Added the ability to:  Persist an iterator or stream as a doument  Halt or stop the iterator stream (after exhausting the already requested documents). For restart add the ability to restart the iterator or stream Allows conversion from an iterator to a stream and back.
  21. 21. Queues – Initialization Code Initialize a Tailable Cursor Share with multiple processes Find.Builder builder = new Find.Builder(BuilderFactory.start()); builder.setTailable(true); builder.setAwaitData(false); // Sigh. MongoIterator<Document> cursor = collection.find(builder.build()); // Graceful shutdown of the iterator locally but not on the server. cursor.stop(); while (cursor.hasNext()) { System.out.println(cursor.next()); } collection = db.getCollection("lookup"); collection.delete(BuilderFactory.start().add("_id", collectionName)); collection.insert(BuilderFactory.start().add("_id", collectionName) .add("cursor", cursor.asDocument()));
  22. 22. Queues – Consumer Code MongoCollection index = client.getDatabase(args[1]).getCollection( "lookup"); Document queueLookupDoc = index.findOne(BuilderFactory.start().add( "_id", args[2])); DocumentElement cursorElement = queueLookupDoc.get( DocumentElement.class, "cursor"); MongoIterator<Document> iter = client.restart(cursorElement .getDocument()); long lastCount = 0; while (iter.hasNext()) { Document doc = iter.next(); // Do stuff }
  23. 23. Queues Demo
  24. 24. Restartable Cursors - Gotcha Remember this: builder.setAwaitData(false); // Sigh. That masks a small issue...Pin( long long cursorid ) : _cursorid( INVALID_CURSOR_ID ) { recursive_scoped_lock lock( ccmutex ); ClientCursor *cursor = ClientCursor::find_inlock( cursorid, true ); if ( cursor ) { uassert( 12051, "clientcursor already in use? driver problem?", cursor->_pinValue < 100 ); cursor->_pinValue += 100; _cursorid = cursorid; }} Closes the cursor on the client https://jira.mongodb.org/browse/SERVER-8602 https://github.com/allanbank/mongo/tree/concurrent_cursor_support
  25. 25. Final Thoughts Coordination Service  Watchers  Group Management Message Broker  Topics  Queues Remember the Disclaimer
  26. 26. Questions?  Contact Information: Robert.J.Moore@allanbank.com

×