Advanced Analytics & Statistics with MongoDB

mongoDB advanced analytics and statistics with mongodb John A. De Goes @jdegoes http://precog.io 04/30/2012

mongoDB what do you want from your data?

mongoDB I want to get and I want aggregates I want deep insight put data MongoDB MongoDB Query Aggregation ??? Language Framework SQL data storage data intelligence

mongoDB I want to get and I want aggregates I want deep insight put data MongoDB MongoDB Map Query Aggregation Reduce Language Framework SQL data storage data intelligence

mongoDB function map() { emit(1, // Or put a GROUP BY key here {sum: this.value, // the ﬁeld you want stats for min: this.value, max: this.value, count:1, diff: 0, // M2,n: sum((val-mean)^2) }); } function reduce(key, values) { var a = values[0]; // will reduce into here for (var i=1/*!*/; i < values.length; i++){ var b = values[i]; // will merge b into a // temp helpers var delta = a.sum/a.count - b.sum/b.count; // a.mean - b.mean var weight = (a.count * b.count)/(a.count + b.count); // do the reducing a.diff += b.diff + delta*delta*weight; a.sum += b.sum; a.count += b.count; a.min = Math.min(a.min, b.min); a.max = Math.max(a.max, b.max); } return a; } function ﬁnalize(key, value){ value.avg = value.sum / value.count; value.variance = value.diff / value.count; value.stddev = Math.sqrt(value.variance); return value; }

mongoDB what if there were another way?

mongoDB introducing • Statistical query language for JSON data • Purely declarative • Implicitly parallel • Inherently composable

mongoDB a taste of quirrel pageViews := //pageViews bound := 1.5 * stdDev(pageViews.duration) avg := mean(pageViews.duration) lengthyPageViews := pageViews where pageViews.duration > (avg + bound) lengthyPageViews.userId

mongoDB a taste of quirrel pageViews := //pageViews bound := 1.5 * stdDev(pageViews.duration) Users who spend an unusually avg := mean(pageViews.duration) long time looking at a page! lengthyPageViews := pageViews where pageViews.duration > (avg + bound) lengthyPageViews.userId

mongoDB quirrel in 10 minutes

mongoDB set-oriented in Quirrel everything is a set of events

mongoDB event an event is a JSON value paired with an identity

mongoDB (really) basic queries quirrel> 1 [1] quirrel> true [true] quirrel> {userId: 1239823, name: “John Doe”} [{userId: 1239823, name: “John Doe”}] quirrel>1 + 2 [3] quirrel> sqrt(16) * 4 - 1 / 3 [5]

mongoDB loading data quirrel> //payments [{"amount":5,"date":1329741127233,"recipients": ["research","marketing"]}, ...] quirrel> load(“/payments”) [{"amount":5,"date":1329741127233,"recipients": ["research","marketing"]}, ...]

mongoDB variables quirrel> payments := //payments | payments [{"amount":5,"date":1329741127233,"recipients": ["research","marketing"]}, ...] quirrel> five := 5 | five * 2 [10]

mongoDB filtered descent quirrel> //users.userId [9823461231, 916727123, 23987183, ...] quirrel> //payments.recipients[0] ["engineering","operations","research", ...]

mongoDB reductions quirrel> count(//users) 24185132 quirrel> mean(//payments.amount) 87.39 quirrel> sum(//payments.amount) 921541.29 quirrel> stdDev(//payments.amount) 31.84

mongoDB identity matching a*b a e1 ? b e2 e8 e3 e9 e4 * e10 e5 e11 e6 e12 ? e7

mongoDB identity matching quirrel> orders := //orders | orders.subTotal + | orders.subTotal * | orders.taxRate + | orders.shipping + orders.handling [153.54805, 152.7618, 80.38365, ...]

mongoDB values quirrel> payments.amount * 0.10 [6.1, 27.842, 29.084, 50, 0.5, 16.955, ...]

mongoDB filtering quirrel> users := //users | segment := users.age > 19 & | users.age < 53 & users.income > 60000 | count(users where segment) [15]

mongoDB chaining pageViews := //pageViews bound := 1.5 * stdDev(pageViews.duration) avg := mean(pageViews.duration) lengthyPageViews := pageViews where pageViews.duration > (avg + bound) lengthyPageViews.userId

mongoDB user functions quirrel> pageViews := //pageViews | | statsForUser(userId) := | {userId: userId, | meanPageView: mean(pageViews.duration | where pageViews.userId = userId)} | | statsForUser [{"userId":12353,"meanPageView":100.66666666666667},{"userId": 12359,"meanPageView":83}, ...]

mongoDB lots more! • Cross-joins • Self-joins • Augmentation • Power-packed standard library

mongoDB quirrel -> mongodb • Quirrel is extremely expressive • Aggregation framework insufﬁcient • Working with 10gen on new primitives • Backup plan: AF + MapReduce

mongoDB quirrel -> mongodb pageViews := //pageViews bound := 1.5 * stdDev(pageViews.duration) one-pass avg := mean(pageViews.duration) map/reduce lengthyPageViews := pageViews where pageViews.duration > (avg + bound) lengthyPageViews.userId one-pass mongo ﬁlter

mongoDB qa John A. De Goes @jdegoes http://precog.io 04/30/2012

http://lanyrd.com	83
https://twitter.com	3
https://twimg0-a.akamaihd.net	1
https://si0.twimg.com	1

Advanced Analytics & Statistics with MongoDB

by John De Goes

Accessibility

Categories

Upload Details

Usage Rights

4 Embeds 88

Statistics

Advanced Analytics & Statistics with MongoDB Presentation Transcript