• Like

Building Real Time Systems on MongoDB Using the Oplog at Stripe

  • 988 views
Published

 

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
988
On SlideShare
0
From Embeds
0
Number of Embeds
5

Actions

Shares
Downloads
30
Comments
0
Likes
3

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. MongoDB and the Oplog EVAN BRODER @ebroder
  • 2. MongoDB and the Oplog EVAN BRODER @ebroder
  • 3. AGENDA INTRO TO THE OPLOG EXAMPLE APPLICATIONS
  • 4. INTRO TO THE OPLOG
  • 5. PRIMARY SECONDARIES APPLICATION
  • 6. APPLICATION save {_id: 1, a: 2}
  • 7. THINGS I’VE DONE: - save {_id: 1, a: 2}
  • 8. APPLICATION update where {a: 2}, {$set: {a: 3}}
  • 9. THINGS I’VE DONE: - save {_id: 1, a: 2} - update {_id: 1}, {$set: {a: 3}}
  • 10. THINGS I’VE DONE: - save {_id: 1, a: 2} - update {_id: 1}, {$set: {a: 3}} - insert… - delete… - delete… - save… - update…
  • 11. THINGS I’VE DONE: save … THINGS I’VE DONE:
  • 12. THINGS I’VE DONE: save … THINGS I’VE DONE: save …
  • 13. TRIGGERS
  • 14. GOAL: EVENT PROCESSING
  • 15. GOAL: DETECT INSERTIONS
  • 16. oplog = mongo_connection['local']['oplog.rs'] ns = 'eventdb.events' oplog.find({'op' => 'i', 'ns' => ns}) do |cursor| cursor.each do |op| puts op['o']['_id'] end end
  • 17. oplog = mongo_connection['local']['oplog.rs'] ns = 'eventdb.events' oplog.find({'op' => 'i', 'ns' => ns}) do |cursor| cursor.each do |op| puts op['o']['_id'] end end
  • 18. oplog = mongo_connection['local']['oplog.rs'] ns = 'eventdb.events' oplog.find({'op' => 'i', 'ns' => ns}) do |cursor| cursor.each do |op| puts op['o']['_id'] end end
  • 19. oplog = mongo_connection['local']['oplog.rs'] ns = 'eventdb.events' oplog.find({'op' => 'i', 'ns' => ns}) do |cursor| cursor.each do |op| puts op['o']['_id'] end end
  • 20. oplog.find({'op' => 'i', 'ns' => ns}) do |cursor| cursor.add_option(Mongo::Constants::OP_QUERY_TAILABLE) cursor.add_option(Mongo::Constants::OP_QUERY_AWAIT_DATA) loop do cursor.each do |op| puts op['o']['_id'] end end end
  • 21. oplog.find({'op' => 'i', 'ns' => ns}) do |cursor| cursor.add_option(Mongo::Constants::OP_QUERY_TAILABLE) cursor.add_option(Mongo::Constants::OP_QUERY_AWAIT_DATA) loop do cursor.each do |op| puts op['o']['_id'] end end end
  • 22. oplog.find({'op' => 'i', 'ns' => ns}) do |cursor| cursor.add_option(Mongo::Constants::OP_QUERY_TAILABLE) cursor.add_option(Mongo::Constants::OP_QUERY_AWAIT_DATA) loop do cursor.each do |op| puts op['o']['_id'] end end end
  • 23. oplog.find({'op' => 'i', 'ns' => ns}) do |cursor| cursor.add_option(Mongo::Constants::OP_QUERY_TAILABLE) cursor.add_option(Mongo::Constants::OP_QUERY_AWAIT_DATA) loop do cursor.each do |op| puts op['o']['_id'] end end end
  • 24. start_entry = oplog.find_one({}, {:sort => {'$natural' => -1}}) start = start_entry['ts'] oplog.find({'ts' => {'$gt' => start}, 'op' => 'i', 'ns' => ns}) do |cursor| cursor.add_option(Mongo::Constants::OP_QUERY_TAILABLE) cursor.add_option(Mongo::Constants::OP_QUERY_AWAIT_DATA) loop do cursor.each do |op| puts op['o']['_id'] end
  • 25. start_entry = oplog.find_one({}, {:sort => {'$natural' => -1}}) start = start_entry['ts'] oplog.find({'ts' => {'$gt' => start}, 'op' => 'i', 'ns' => ns}) do |cursor| cursor.add_option(Mongo::Constants::OP_QUERY_OPLOG_REPLAY) cursor.add_option(Mongo::Constants::OP_QUERY_TAILABLE) cursor.add_option(Mongo::Constants::OP_QUERY_AWAIT_DATA) loop do cursor.each do |op| puts op['o']['_id']
  • 26. start_entry = oplog.find_one({}, {:sort => {'$natural' => -1}}) start = start_entry['ts'] oplog.find({'ts' => {'$gt' => start}, 'op' => 'i', 'ns' => ns}) do |cursor| cursor.add_option(Mongo::Constants::OP_QUERY_OPLOG_REPLAY) cursor.add_option(Mongo::Constants::OP_QUERY_TAILABLE) cursor.add_option(Mongo::Constants::OP_QUERY_AWAIT_DATA) loop do cursor.each do |op| puts op['o']['_id'] end end end
  • 27. start_entry = oplog.find_one({}, {:sort => {'$natural' => -1}}) start = start_entry['ts'] oplog.find({'ts' => {'$gt' => start}, 'op' => 'i', 'ns' => ns}) do |cursor| cursor.add_option(Mongo::Constants::OP_QUERY_OPLOG_REPLAY) cursor.add_option(Mongo::Constants::OP_QUERY_TAILABLE) cursor.add_option(Mongo::Constants::OP_QUERY_AWAIT_DATA) loop do cursor.each do |op| puts op['o']['_id'] end end end
  • 28. DATA TRANSFORMATIONS
  • 29. GOAL: MONGODB TO POSTGRESQL
  • 30. start_entry = oplog.find_one({}, {:sort => {'$natural' => -1}}) start = start_entry['ts'] oplog.find({'ts' => {'$gt' => start}, 'op' => 'i', 'ns' => ns}) do |cursor| cursor.add_option(Mongo::Constants::OP_QUERY_OPLOG_REPLAY) cursor.add_option(Mongo::Constants::OP_QUERY_TAILABLE) cursor.add_option(Mongo::Constants::OP_QUERY_AWAIT_DATA) loop do cursor.each do |op| puts op['o']['_id'] end end end
  • 31. start_entry = oplog.find_one({}, {:sort => {'$natural' => -1}}) start = start_entry['ts'] oplog.find({'ts' => {'$gt' => start}}) do |cursor| cursor.add_option(Mongo::Constants::OP_QUERY_OPLOG_REPLAY) cursor.add_option(Mongo::Constants::OP_QUERY_TAILABLE) cursor.add_option(Mongo::Constants::OP_QUERY_AWAIT_DATA) loop do cursor.each do |op| puts op['o']['_id'] end end end
  • 32. cursor.each do |op| puts op['o']['_id'] end
  • 33. cursor.each do |op| case op['op'] when 'i' puts op['o']['_id'] else # ¯_(ツ)_/¯ end end
  • 34. cursor.each do |op| case op['op'] when 'i' query = "INSERT INTO #{op['ns']} (" + op['o'].keys.join(', ') + ') VALUES (' + op['o'].values.map(&:inspect).join(', ') else # ¯_(ツ)_/¯ end end
  • 35. cursor.each do |op| case op['op'] when 'i' query = "INSERT INTO #{op['ns']} (" + op['o'].keys.join(', ') + ') VALUES (' + op['o'].values.map(&:inspect).join(', ') when 'd' query = "DELETE FROM #{op['ns']} WHERE _id=" + op['o']['_id'].inspect else # ¯_(ツ)_/¯ end end
  • 36. query = "DELETE FROM #{op['ns']} WHERE _id=" + op['o']['_id'].inspect when 'u' query = "UPDATE #{op['ns']} SET" updates = op['o']['$set'] ? op['o']['$set'] : op['o'] updates.each do |k, v| query += " #{k}=#{v.inspect}" end query += " WHERE _id=" query += op['o2']['_id'].inspect else # ¯_(ツ)_/¯ end end
  • 37. cursor.each do |op| case op['op'] when 'i' query = "INSERT INTO #{op['ns']} (" + op['o'].keys.join(', ') + ') VALUES (' + op['o'].values.map(&:inspect).join(', ') when 'd' query = "DELETE FROM #{op['ns']} WHERE _id=" + op['o']['_id'].inspect when 'u' query = "UPDATE #{op['ns']} SET" updates = op['o']['$set'] ? op['o']['$set'] : op['o'] updates.each do |k, v| query += " #{k}=#{v.inspect}" end query += " WHERE _id=" + op['o2']['_id'].inspect else # ¯_(ツ)_/¯ end end
  • 38. github.com/stripe/mosql
  • 39. github.com/stripe/zerowing
  • 40. cursor.each do |op| case op['op'] when 'i' query = "INSERT INTO #{op['ns']} (" + op['o'].keys.join(', ') + ') VALUES (' + op['o'].values.map(&:inspect).join(', ') when 'd' query = "DELETE FROM #{op['ns']} WHERE _id=" + op['o']['_id'].inspect when 'u' query = "UPDATE #{op['ns']} SET" updates = op['o']['$set'] ? op['o']['$set'] : op['o'] updates.each do |k, v| query += " #{k}=#{v.inspect}" end query += " WHERE _id=" + op['o2']['_id'].inspect else # ¯_(ツ)_/¯ end end
  • 41. DISASTER RECOVERY
  • 42. task = collection.find_one({'finished' => nil} # do something with task… collection.update({'_id' => task.id}, {'$set' => {'finished' => Time.now.to_i}})
  • 43. loop do collection.remove( {'finished' => {'$lt' => Time.now.to_i - 30}}) sleep(10) end
  • 44. evan@caron:~$ mongo MongoDB shell version: 2.4.10 connecting to: test normal:PRIMARY> null < (Date.now() / 1000) - 30 true
  • 45. > db.getReplicationInfo() { "logSizeMB" : 48964.3541015625, "usedMB" : 46116.4, "timeDiff" : 316550, "timeDiffHours" : 87.93, "tFirst" : "Thu Apr 11 2013 07:24:29 GMT+0000 (UTC)", "tLast" : "Sun Apr 14 2013 23:20:19 GMT+0000 (UTC)", "now" : "Sat May 24 2014 07:52:35 GMT+0000 (UTC)" }
  • 46. > db.getReplicationInfo() { "logSizeMB" : 48964.3541015625, "usedMB" : 46116.4, "timeDiff" : 316550, "timeDiffHours" : 87.93, "tFirst" : "Thu Apr 11 2013 07:24:29 GMT+0000 (UTC)", "tLast" : "Sun Apr 14 2013 23:20:19 GMT+0000 (UTC)", "now" : "Sat May 24 2014 07:52:35 GMT+0000 (UTC)" }
  • 47. THINGS I’VE DONE: insert delete … THINGS I’VE DONE:
  • 48. new_oplog.find({'ts' => {'$gt' => start}}) do |cursor| cursor.add_option(Mongo::Constants::OP_QUERY_OPLOG_REPLAY) cursor.each do |op| if op['op'] == 'd' && op['ns'] == 'monsterdb.tasks' old_task = old_tasks.find_one({'_id' => op['o']['_id']}) if old_task['finished'] == nil # found one! # save old_task to a file, and we'll re-queue it later end end old_connection['admin'].command({'applyOps' => [op]}) end end
  • 49. new_oplog.find({'ts' => {'$gt' => start}}) do |cursor| cursor.add_option(Mongo::Constants::OP_QUERY_OPLOG_REPLAY) cursor.each do |op| if op['op'] == 'd' && op['ns'] == 'monsterdb.tasks' old_task = old_tasks.find_one({'_id' => op['o']['_id']}) if old_task['finished'] == false # found one! # save old_task to a file, and we'll re-queue it later end end old_connection['admin'].command({'applyOps' => [op]}) end end
  • 50. new_oplog.find({'ts' => {'$gt' => start}}) do |cursor| cursor.add_option(Mongo::Constants::OP_QUERY_OPLOG_REPLAY) cursor.each do |op| if op['op'] == 'd' && op['ns'] == 'monsterdb.tasks' old_task = old_tasks.find_one({'_id' => op['o']['_id']}) if old_task['finished'] == false # found one! # save old_task to a file, and we'll re-queue it later end end old_connection['admin'].command({'applyOps' => [op]}) end end
  • 51. THINGS I’VE DONE: save … THINGS I’VE DONE: save …
  • 52. QUESTIONS?