• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
MongoDB - visualisation of slow operations
 

MongoDB - visualisation of slow operations

on

  • 1,978 views

MongoDB allows to profile slow operations. However, it's difficult to get a quick overview of a sharded system or to have a historical view since MongoDB stores slow operations on every profiled node ...

MongoDB allows to profile slow operations. However, it's difficult to get a quick overview of a sharded system or to have a historical view since MongoDB stores slow operations on every profiled node in a capped collection. This talk, held during the MongoDB User Group Berlin on 4th of June 2013, gives a deeper insight how idealo solved these shortcomings.

Statistics

Views

Total Views
1,978
Views on SlideShare
1,931
Embed Views
47

Actions

Likes
0
Downloads
8
Comments
2

4 Embeds 47

https://twitter.com 39
https://www.rebelmouse.com 4
http://gazeta.yandex.ru 2
https://www.optimizelyedit.com 2

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

12 of 2 previous next

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • project has been open sourced https://github.com/idealo/mongodb-slow-operations-profiler
    Are you sure you want to
    Your message goes here
    Processing…
  • I'm happy to announce you that we have opensourced the slow-operations-profiler project:
    https://github.com/idealo/mongodb-slow-operations-profiler
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    MongoDB - visualisation of slow operations MongoDB - visualisation of slow operations Presentation Transcript

    • 1MongoDBvisualisation of slow operationsKay Agahd4 June 2013
    • 2idealo and MongoDB●idealo = Europes leading price comparison web site●Germany, Austria, United Kingdom, France, Italy, Poland and Spain●250 millions offers online (May 2013)●fast growing●different types of databases (MySQL, Oracle, MongoDB)●MongoDB in production since v1.6●sharding in production since MongoDB v1.8●MongoDB stores offers for back-end usage●30 mongoDB servers for offerStore + 3 servers for offerHistory●15 mongoDB servers for other purposes●nearly 15 TB of data all together
    • 3Review profiling●MongoDB supports profiling of “slow” operations●“slow” is a threshold to be set when turning profiling on (default 100 ms)●profiling per-database or per-instance on a running mongod●profiler writes collected data to a capped collection “system.profile”
    • 4Example of slow op entry (1/2){"ts" : ISODate("2013­04­05T01:41:31.710Z"),"op" : "getmore","ns" : "offerStore.offer","query" : {“query” : {"shopId" : 123,"onlineProductIds" : {"$ne" : null},"smallPicture" : {"$ne" : null},"_id" : {"$gt" : 1555008076},"lastChange" : {"$gt" : ISODate("2013­04­02T22:00:00Z")}},“orderby” : {“_id” : 1}},"cursorid" : NumberLong("5773493375904448215"),"ntoreturn" : 500, ...
    • 5Example of slow op entry (2/2)..."keyUpdates" : 0,"numYield" : 2350,"ts" : "lockStats" : {"timeLockedMicros" : {"r" : NumberLong(8724165),"w" : NumberLong(0)},"timeAcquiringMicros" : {"r" : NumberLong(5321722),"w" : NumberLong(7)}},"nreturned" : 500,"responseLength" : 94656,"millis" : 5322,"client" : "172.16.65.202","user" : "pl_parser"}
    • 6Inconveniences●each mongod needs to be handled separately●replSet: connect to master and every slave●sharding: incomplete view through router, thus replSet * n shards●gives only a view on a limited time span due to capped collection●different formats of “query” field makes querying more difficult●bug: ops through mongos omit the user (JIRA: SERVER-7538)
    • 7Example of different formats/schemata- “query” as flat document:{ "query" : { "shopId" : 123,     "onlineProductIds" : { "$ne" : null } },    "user" : "pl_parser"}- “query” embedded:{"query" : { "query" : { "shopId" : 123,       "onlineProductIds" : { "$ne" : null } },    "orderby" : { "_id" : NumberLong(1) } },  "user" : "pl_parser"}- “query” embedded as $query:{ "query" : { "$query" : { "shopId" : 123,           "onlineProductIds" : { "$ne" : null } },               "$orderby" : { "_id" : NumberLong(1) },              "$comment" : "profiling comment" },  "user" : "pl_parser" }
    • 8idealo requirements●quick overview of types of slow-ops and their quantity within a time period(“types” means op type, user, server, queried and sorted fields)●historical view to see how slow-ops evolve to extrapolate them●discovering spikes in time or in slow-op types●filtering by slow-op types and/or time range to drill down
    • 9Goals●faster queries●better adapted indexes●better adapted data schema●higher throughput by smarter workflow
    • 10Steps to go●two global steps:●1) collect and aggregate slow ops from all mongods into one globalcollection●2) GUI to query and show results
    • 11Step 1 of 2●global collection:●allows easy and fast querying of the whole mongoDB (shard) system●keeps historical data (no capped collection)●located on another replSet to avoid interfering with profiled mongods●collector:●guarantee that only 1 instance is running at once (or add logic to avoiddoubled entries)●use tailable cursors to collect data from profiled mongods●in case of failure: reconnect before data gets overwritten but avoid DoS●monitor it (nagios etc.)●profiled entries:●reduce size by keeping only interesting fields●make them easier to query (i.e. only 1 schema)●aggregate fields inside “query” and “orderby” to values●choose short field names
    • 12slow-op example●slow-op example of above becomes:{"_id" : ObjectId("512e43099bbcf52b9aff3602"),"ts" : ISODate("2013­04­05T01:41:31.710Z"),"adr" : "s233.ipx","op" : "getmore","fields" : ["shopId","onlineProductIds","smallPicture","_id",“lastChange“],"sort" : ["_id”],"nret" : 500,"reslen" : 94656,"millis" : 5322,"user" : "pl_parser"}
    • 13Step 2 of 2●GUI:●x-axis = execution time●y-axis = duration of slow op●size of point = quantity of slow-op type●zoomable in x or y axis
    • 14How to query slow ops●group by time component allows resolution by year, month, week etc.●group by server address, user, operation, queried fields and sorted fieldsallows to define different slow-op types●filter allows to focus on time period and specific slow ops●use slavePreferred option●error handling, i.e. result exceeds max of 16 MB
    • 15Query example{$match:{ts : {$gt : #, $lt : # }}},   fields : {$all : ["_id","shopId","bokey"]}{$group:{_id : {op : "$op",   user : "$user",   fields : "$fields",   year : { $year : "$ts" },   month : { $month : "$ts" },   dayOfMonth : { $dayOfMonth : "$ts" },   hour : { $hour : "$ts" }},   count : { $sum : 1 },millis : { $sum : "$millis" },avgMs : { $avg : "$millis" },minMs : { $min : "$millis" },maxMs : { $max : "$millis" },firstts : { $first : "$ts" }}},{ $sort:{ firstts : 1 }}FilterSlow-opResolutionData
    • 16GUI
    • 17Resolution by minute
    • 18Resolution by minute & filter
    • 19dygraph.js●general syntax:<script type="text/javascript">   g = new Dygraph(document.getElementById("graph"),    "x­name,   graph1­name,   graph2­name,   ..., graphN­namen" +     "x­value1, graph1­value1, graph2­value1, ..., graphN­value1n" +    "x­value2, graph1­value2, graph2­value2, ..., graphN­value2n" +    ...    "x­valueN, graph1­valueN, graph2­valueN, ..., graphN­valueNn"   ); </script>●example for 2 slow-op types:<script type="text/javascript"> g = new Dygraph(document.getElementById("graph"), "Date,op=query;fields=[_id;shopId],n,min,max,op=query;fields=[_id],n,min,maxn" +  "2013/03/17,  5.4, 10, 3.2,  7.8,            10.4, 123, 3.1, 20.2n" +  "2013/03/18, 12.4, 23, 3.4, 55.8,               0,   0,   0,    0n" + "2013/03/19,    0,  0,   0,    0,            33.5,  66, 3.1, 89.3n"    );</script>
    • 20dygraph.js Options 1/3●hide legend values from being drawn as graph:<script type="text/javascript"> g = new Dygraph(document.getElementById("graph"),  "Date,op=query;fields=[_id;shopId],n,min,max,op=query;fields=[_id],n,min,maxn" +   "2013/03/17, 5.4, 10, 3.2, 7.8, 10.4, 123, 3.1, 20.2n" +   "2013/03/18, 12.4, 23, 3.4, 55.8, 0, 0, 0, 0n" +    "2013/03/19, 0, 0, 0, 0, 33.5, 66, 3.1, 89.3n",  {//options:    visibility:[true, false, false, false, true, false, false, false],    showLabelsOnHighlight:false,    hideOverlayOnMouseOut:false,    labelsSeparateLines: true,    drawPoints: true,    legend: "always",    xlabel: "Date",    ylabel: "seconds",    ... more options ...  }); 
    • 21dygraph.js Options 2/3●show custom legend on mouse over: highlightCallback: function(e, x, pts, row) {  var text = "";  var legend = new Array();  for (var i = 0; i < pts.length; i++) {    var rangeY = g.yAxisRange();      if(pts[i].yval >= rangeY[0] && pts[i].yval <= rangeY[1]){//hide outside series        var seriesProps = g.getPropertiesForSeries(pts[i].name);        var count = g.getValue(row, seriesProps.column+1);        var minSec = g.getValue(row, seriesProps.column+2);        var maxSec = g.getValue(row, seriesProps.column+3);        if(pts[i].yval != 0 && count != 0){          legend.push([seriesProps.color, pts[i], count, minSec, maxSec]);        }}}//end for  legend.sort(function(a,b){return b[1].yval­a[1].yval});//sort by y­values  for (var i = 0; i < legend.length; i++) {    text += "<span style=color: " + legend[i][0] + ";> " + legend[i][1].name +        "</span><br/><span>" + Dygraph.dateString_(legend[i][1].xval) + " count:" +         legend[i][2] + " minSec:" + legend[i][3] + " maxSec:" + legend[i][4] + "avgSec:" +  legend[i][1].yval + " </span><br/>";  }  document.getElementById("status").innerHTML = text; }, ... more options ...
    • 22dygraph.js Options 3/3●draw circles with surface of count:  drawPointCallback : function(g, seriesName, ctx, cx, cy, color, pSize){    if(lastSeries != seriesName || isNaN(currentRow) ){lastSeries = seriesName;currentRow = g.getLeftBoundary_() ­ 1;    }    currentRow++;    var col = g.indexFromSetName(seriesName);    var count = g.getValue(currentRow, col+1);    ctx.strokeStyle = color;    ctx.lineWidth = 0.8;    ctx.beginPath();    ctx.arc(cx, cy, Math.sqrt(count/Math.PI), 0, 2 * Math.PI, false);     ctx.closePath();    ctx.stroke();  } }//end options);//end dygraph
    • 23Profiling status
    • 24Collector read/write status
    • 25Questions?
    • 26Thank you!