Big data guru John A. De Goes, CTO of Precog, presents an overview of Quirrel, a high-level, statistically-oriented, open source query language designed for advanced analytics and statistics on large-scale JSON data sets. John discusses how the language can be used to solve a variety of common problems encountered by modern application developers, and then overviews ongoing efforts to port the language to MongoDB as part of a pure open source distribution.

1. mongoDB
advanced analytics and
statistics with mongodb
John A. De Goes @jdegoes
http://precog.io 04/30/2012

2. mongoDB
what do you want
from your data?

3. mongoDB
I want to get and
I want aggregates I want deep insight
put data
MongoDB MongoDB
Query Aggregation ???
Language Framework
SQL
data storage data intelligence

4. mongoDB
I want to get and
I want aggregates I want deep insight
put data
MongoDB MongoDB
Map
Query Aggregation
Reduce
Language Framework
SQL
data storage data intelligence

5. mongoDB
function map() {
emit(1, // Or put a GROUP BY key here
{sum: this.value, // the field you want stats for
min: this.value,
max: this.value,
count:1,
diff: 0, // M2,n: sum((val-mean)^2)
});
}
function reduce(key, values) {
var a = values[0]; // will reduce into here
for (var i=1/*!*/; i < values.length; i++){
var b = values[i]; // will merge 'b' into 'a'
// temp helpers
var delta = a.sum/a.count - b.sum/b.count; // a.mean - b.mean
var weight = (a.count * b.count)/(a.count + b.count);
// do the reducing
a.diff += b.diff + delta*delta*weight;
a.sum += b.sum;
a.count += b.count;
a.min = Math.min(a.min, b.min);
a.max = Math.max(a.max, b.max);
}
return a;
}
function finalize(key, value){
value.avg = value.sum / value.count;
value.variance = value.diff / value.count;
value.stddev = Math.sqrt(value.variance);
return value;
}

6. mongoDB
what if there were
another way?

7. mongoDB
introducing
• Statistical query language for JSON data
• Purely declarative
• Implicitly parallel
• Inherently composable

8. mongoDB
a taste of quirrel
pageViews := //pageViews
bound := 1.5 * stdDev(pageViews.duration)
avg := mean(pageViews.duration)
lengthyPageViews :=
pageViews where pageViews.duration > (avg + bound)
lengthyPageViews.userId

9. mongoDB
a taste of quirrel
pageViews := //pageViews
bound := 1.5 * stdDev(pageViews.duration)
Users who spend an unusually
avg := mean(pageViews.duration) long
time looking at a page!
lengthyPageViews :=
pageViews where pageViews.duration > (avg + bound)
lengthyPageViews.userId

10. mongoDB
quirrel in 10 minutes

11. mongoDB
set-oriented
in Quirrel everything is
a set of events

12. mongoDB
event
an event is a JSON value
paired with an identity

13. mongoDB
(really) basic queries
quirrel> 1
[1]
quirrel> true
[true]
quirrel> {userId: 1239823, name: “John Doe”}
[{userId: 1239823, name: “John Doe”}]
quirrel>1 + 2
[3]
quirrel> sqrt(16) * 4 - 1 / 3
[5]

14. mongoDB
loading data
quirrel> //payments
[{"amount":5,"date":1329741127233,"recipients":
["research","marketing"]}, ...]
quirrel> load(“/payments”)
[{"amount":5,"date":1329741127233,"recipients":
["research","marketing"]}, ...]

15. mongoDB
variables
quirrel> payments := //payments
| payments
[{"amount":5,"date":1329741127233,"recipients":
["research","marketing"]}, ...]
quirrel> five := 5
| five * 2
[10]

16. mongoDB
filtered descent
quirrel> //users.userId
[9823461231, 916727123, 23987183, ...]
quirrel> //payments.recipients[0]
["engineering","operations","research", ...]

17. mongoDB
reductions
quirrel> count(//users)
24185132
quirrel> mean(//payments.amount)
87.39
quirrel> sum(//payments.amount)
921541.29
quirrel> stdDev(//payments.amount)
31.84

18. mongoDB
identity matching
a*b
a
e1
? b
e2 e8
e3 e9
e4 * e10
e5 e11
e6 e12
?
e7

19. mongoDB
identity matching
quirrel> orders := //orders
| orders.subTotal +
| orders.subTotal *
| orders.taxRate +
| orders.shipping + orders.handling
[153.54805, 152.7618, 80.38365, ...]

20. mongoDB
values
quirrel> payments.amount * 0.10
[6.1, 27.842, 29.084, 50, 0.5, 16.955, ...]

21. mongoDB
filtering
quirrel> users := //users
| segment := users.age > 19 &
| users.age < 53 & users.income > 60000
| count(users where segment)
[15]

22. mongoDB
chaining
pageViews := //pageViews
bound := 1.5 * stdDev(pageViews.duration)
avg := mean(pageViews.duration)
lengthyPageViews :=
pageViews where pageViews.duration > (avg + bound)
lengthyPageViews.userId

23. mongoDB
user functions
quirrel> pageViews := //pageViews
|
| statsForUser('userId) :=
| {userId: 'userId,
| meanPageView: mean(pageViews.duration
| where pageViews.userId = 'userId)}
|
| statsForUser
[{"userId":12353,"meanPageView":100.66666666666667},{"userId":
12359,"meanPageView":83}, ...]

24. mongoDB
lots more!
• Cross-joins
• Self-joins
• Augmentation
• Power-packed standard library

25. mongoDB
quirrel -> mongodb
• Quirrel is extremely expressive
• Aggregation framework insufficient
• Working with 10gen on new primitives
• Backup plan: AF + MapReduce

26. mongoDB
quirrel -> mongodb
pageViews := //pageViews
bound := 1.5 * stdDev(pageViews.duration)
one-pass
avg := mean(pageViews.duration) map/reduce
lengthyPageViews :=
pageViews where pageViews.duration > (avg + bound)
lengthyPageViews.userId
one-pass
mongo filter

27. mongoDB
qa
John A. De Goes @jdegoes
http://precog.io 04/30/2012