mongodb-aggregation-may-2012

What problem are we solving?• Map/Reduce can be used for aggregation… • Currently being used for totaling, averaging, etc• Map/Reduce is a big hammer • Simpler tasks should be easier • Shouldn’t need to write JavaScript • Avoid the overhead of JavaScript engine• We’re seeing requests for help in handling complex documents • Select only matching subdocuments or arrays

How will we solve the problem?• Our new aggregation framework • Declarative framework • No JavaScript required • Describe a chain of operations to apply • Expression evaluation • Return computed values • Framework: we can add new operations easily • C++ implementation • Higher performance than JavaScript

Aggregation - Pipelines• Aggregation requests specify a pipeline• A pipeline is a series of operations• Conceptually, the members of a collection are passed through a pipeline to produce a result • Similar to a command-line pipe

Pipeline Operations• $match • Uses a query predicate (like .find({…})) as a filter• $project • Uses a sample document to determine the shape of the result (similar to .find()’s optional argument) • This can include computed values• $unwind • Hands out array elements one at a time• $group • Aggregates items into buckets defined by a key

Pipeline Operations (continued)• $sort • Sort documents• $limit • Only allow the specified number of documents to pass• $skip • Skip over the specified number of documents

Projections• $project can reshape results • Include or exclude fields • Computed fields • Arithmetic expressions, including built-in functions • Pull fields from nested documents to the top • Push fields from the top down into new virtual documents

Unwinding• $unwind can “stream” arrays • Array values are doled out one at time in the context of their surrounding documents • Makes it possible to filter out elements before returning

Grouping• $group aggregation expressions • Define a grouping key as the _id of the result • Total grouped column values: $sum • Average grouped column values: $avg • Collect grouped column values in an array or set: $push, $addToSet • Other functions • $min, $max, $first, $last

Sorting• $sort can sort documents • Sort specifications are the same as today, e.g., $sort:{ key1: 1, key2: -1, …}

Computed Expressions• Available in $project operations• Prefix expression language • Add two fields: $add:[“$field1”, “$field2”] • Provide a value for a missing field: $ifNull:[“$field1”, “$field2”] • Nesting: $add:[“$field1”, $ifNull:[“$field2”, “$field3”]] • Other functions…. • And we can easily add more as required

Computed Expressions (continued)• String functions • toUpper, toLower, substr• Date field extraction • Get year, month, day, hour, etc, from ISODate• Date arithmetic• Null value substitution (like MySQL ifnull(), Oracle nvl())• Ternary conditional • Return one of two values based on a predicate

DemoDemo files are at https://gist.github.com/1401585

Usage Tips• Use $match in a pipeline as early as possible • The query optimizer can then choose to scan an index and avoid scanning the entire collection• Use $sort in a pipeline as early as possible • The query optimizer can then be used to choose an index to scan instead of sorting the result

Driver Support• Initial version is a command • For any language, build a JSON database object, and execute the command • In the shell: db.runCommand({ aggregate : <collection-name>, pipeline : {…} }); • Beware of command result size limit • Document size limit is 16MB

Sharding support• Initial release will support sharding• Mongos analyzes pipeline, and forwards operations up to $group or $sort to shards; combines shard server results and returns them

When is this being released?• In final development now • Adding an explain facility• Expect to see this in the near future

Future Plans• More optimizations• $out pipeline operation • Saves the document stream to a collection • Similar to M/R $out, but with sharded output • Functions like a tee, so that intermediate results can be saved

http://www.10gen.com	902
http://drupal1.10gen.cc	12
http://lanyrd.com	11
http://10gen.localhost	9
http://w.mongodb.org	1
http://aicbsep.mongodb.org	1
http://translate.googleusercontent.com	1
http://ww.mongodb.org	1
http://archive.10gen.com	1

mongodb-aggregation-may-2012

by Chris Westin

Accessibility

Categories

Upload Details

Usage Rights

9 Embeds 939

Statistics

mongodb-aggregation-may-2012 Presentation Transcript