Consulting Engineer, MongoDB
Bryan Reinero
#ConferenceHashTag
Time Series Data- Part 2
Aggregations in Action
Real Time Traffic Data Project
Our network of 16,000 speed sensors report
data every minute.
What we want from our data
Charting and Trending
What we want from our data
Historical & Predictive Analysis
What we want from our data
Real Time Traffic Dashboard
Document Structure
{ _id: ObjectId("5382ccdd58db8b81730344e2"),
linkId: 900006,
date: ISODate("2014-03-12T17:00:00Z"),
dat...
Sample Document Structure
Compound, unique
Index identifies the
Individual document
{ _id: ObjectId("5382ccdd58db8b8173034...
Sample Document Structure
Saves an extra index
{ _id: “900006:14031217”,
data: [
{ speed: NaN, time: NaN },
{ speed: NaN, ...
{ _id: “900006:14031217”,
data: [
{ speed: NaN, time: NaN },
{ speed: NaN, time: NaN },
{ speed: NaN, time: NaN },
...
],
...
{ _id: “900006:140312”,
data: [
{ speed: NaN, time: NaN },
{ speed: NaN, time: NaN },
{ speed: NaN, time: NaN },
...
],
co...
Charts
0
10
20
30
40
50
60
70
MonMar10201404:57:00…
MonMar10201405:31:00…
MonMar10201406:05:00…
MonMar10201406:39:00…
MonM...
Rollups
{ _id: "20484097:20140204",
hours: [
{ speed: { sum: 1889, count: 60 }
time: { sum: 20562, count: 60 },
conditions...
Document retention
Doc per hour
Doc per day
2 days
2 months
1year
Doc per Month
Analysis with The Aggregation
Framework
Pipelining operations
grep | sort | uniq
Piping command line operations
Pipelining operations
$match $group | $sort|
Piping aggregation operations
Stream of documents Result documents
What is the average speed for a
given road segment?
> db.linkData.aggregate(
{ $match: { ”_id" : /^20484097:/ } },
{ $proj...
What is the average speed for a
given road segment?
Select documents on the target segment
> db.linkData.aggregate(
{ $mat...
What is the average speed for a
given road segment?
Keep only the fields we really need
> db.linkData.aggregate(
{ $match:...
What is the average speed for a
given road segment?
Loop over the array of data points
> db.linkData.aggregate(
{ $match: ...
What is the average speed for a
given road segment?
Use the handy $avg operator
> db.linkData.aggregate(
{ $match: { ”_id"...
More Sophisticated Pipelines:
average speed with variance
{ "$project" : {
mean: "$meanSpd",
spdDiffSqrd : {
"$map" : {
"i...
Historic Analysis
How does weather and road conditions affect
traffic?
The Ask: what are the average speeds per
weather, s...
MapReduce
function map() {
for( var i = 0; i < this.data.length; i++ ) {
emit (
this.conditions.weather,
{ speed : this.da...
MapReduce
function map() {
for( var i = 0; i < this.data.length; i++ ) {
emit (
this.conditions.weather,
{ speed : this.da...
MapReduce
function map() {
for( var i = 0; i < this.data.length; i++ ) {
emit (
this.conditions.weather,
{ speed : this.da...
MapReduce
function map() {
for( var i = 0; i < this.data.length; i++ ) {
emit (
this.conditions.weather,
{ speed : this.da...
MapReduce
MapReduce
Weather: “Rain”, speed: 44
MapReduce
Weather: “Rain”, speed: 39
MapReduce
Weather: “Rain”, speed: 46
MapReduce
function reduce ( key, values ) {
var result = { count : 1, speedSum : 0 };
values.forEach( function( v ){
resul...
MapReduce
function reduce ( key, values ) {
var result = { count : 1, speedSum : 0 };
values.forEach( function( v ){
resul...
Results
results: [
{
"_id" : "Generally Clear and Dry Conditions",
"value" : {
"count" : 902,
"speedSum" : 45100
}
},
{
"_...
Processing Large Data Sets
• Need to break data into smaller pieces
• Process data across multiple nodes
Hadoop Hadoop Had...
Benefits of the Hadoop Connector
• Increased parallelism
• Access to analytics libraries
• Separation of concerns
• Integr...
• Drivers will be accessing the data via web, mobile
devices, and navigation systems
• We need to provide current average ...
Current Real-Time Conditions
Last ten minutes of speeds and
times
{ _id : “I-87:10656”,
description : "NYS Thruway Harrima...
{ _id : “I-87:10656”,
description : "NYS Thruway Harriman Section Exits 14A - 16",
update : ISODate(“2013-10-10T23:06:37.0...
{ _id : “I-87:10656”,
description : "NYS Thruway Harriman Section Exits 14A - 16",
update : ISODate(“2013-10-10T23:06:37.0...
db.linksAvg.update(
{"_id" : linkId},
{ "$set" : {"update " : date},
"$push" : {
"times" : { "$each" : [ time ], "$slice" ...
Putting it all together
Patterns common to time series
data:
• You need to store and manage an incoming
stream of data samples
• You need to compu...
Patterns common to time series
data:
• You need to store and manage an incoming
stream of data samples
• You need to compu...
HVDF: Reference Implementation
Screech -- High Volume Data Feed engine
REST
Service API
Processor
Plugins
Inline
Batch
Str...
HVDF:
https://github.com/10gen-labs/hvdf
Hadoop Connector:
https://github.com/mongodb/mongo-hadoop
Consulting Engineer, MongoDB Inc.
Bryan Reinero
#MongoDBWorld
Thank You
Upcoming SlideShare
Loading in...5
×

MongoDB for Time Series Data Part 2: Analyzing Time Series Data Using the Aggregation Framework and Hadoop

9,929

Published on

The United States will be deploying 16,000 traffic speed monitoring sensors - 1 on every mile of US interstate in urban centers. These sensors update the speed, weather, and pavement conditions once per minute. MongoDB will collect and aggregate live sensor data feeds from roadways around the country, support real-time queries from cars on traffic conditions on their route as well as be the platform for real-time dashboards displaying traffic conditions and more complex analytical queries used to identify traffic trends. In this session, we’ll implement a few different data aggregation techniques to query and dashboard the metrics gathered from the US interstate.

Published in: Technology
0 Comments
14 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
9,929
On Slideshare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
185
Comments
0
Likes
14
Embeds 0
No embeds

No notes for slide
  • Reports (group, summing, averaging)
    Analytics(incremental reporting, rollups)
    Analysis (trends, segmentation, anomalies)
    Analytics (regression, forecasting, filtering)
    Warehousing (long term storage and simplified querying)
  • Compound unique index on linkId & Interval
    update field used to identify new documents for aggregation
  • Compound unique index on linkId & Interval
    update field used to identify new documents for aggregation
  • Compound unique index on linkId & Interval
    update field used to identify new documents for aggregation
  • Compound unique index on linkId & Interval
    update field used to identify new documents for aggregation
  • Compound unique index on linkId & Interval
    update field used to identify new documents for aggregation
  • Priority
    Floating point number between 0..1000
    Highest member that is up to date wins
    Up to date == within 10 seconds of primary
    If a higher priority member catches up, it will force election and win

    Slave Delay
    Lags behind master by configurable time delay
    Automatically hidden from clients
    Protects against operator errors
    Fat fingering
    Application corrupts data
  • Compound unique index on linkId & Interval
    update field used to identify new documents for aggregation
  • Priority
    Floating point number between 0..1000
    Highest member that is up to date wins
    Up to date == within 10 seconds of primary
    If a higher priority member catches up, it will force election and win

    Slave Delay
    Lags behind master by configurable time delay
    Automatically hidden from clients
    Protects against operator errors
    Fat fingering
    Application corrupts data

  • Priority
    Floating point number between 0..1000
    Highest member that is up to date wins
    Up to date == within 10 seconds of primary
    If a higher priority member catches up, it will force election and win

    Slave Delay
    Lags behind master by configurable time delay
    Automatically hidden from clients
    Protects against operator errors
    Fat fingering
    Application corrupts data

  • Priority
    Floating point number between 0..1000
    Highest member that is up to date wins
    Up to date == within 10 seconds of primary
    If a higher priority member catches up, it will force election and win

    Slave Delay
    Lags behind master by configurable time delay
    Automatically hidden from clients
    Protects against operator errors
    Fat fingering
    Application corrupts data

  • Priority
    Floating point number between 0..1000
    Highest member that is up to date wins
    Up to date == within 10 seconds of primary
    If a higher priority member catches up, it will force election and win

    Slave Delay
    Lags behind master by configurable time delay
    Automatically hidden from clients
    Protects against operator errors
    Fat fingering
    Application corrupts data

  • Priority
    Floating point number between 0..1000
    Highest member that is up to date wins
    Up to date == within 10 seconds of primary
    If a higher priority member catches up, it will force election and win

    Slave Delay
    Lags behind master by configurable time delay
    Automatically hidden from clients
    Protects against operator errors
    Fat fingering
    Application corrupts data

  • Priority
    Floating point number between 0..1000
    Highest member that is up to date wins
    Up to date == within 10 seconds of primary
    If a higher priority member catches up, it will force election and win

    Slave Delay
    Lags behind master by configurable time delay
    Automatically hidden from clients
    Protects against operator errors
    Fat fingering
    Application corrupts data

  • Priority
    Floating point number between 0..1000
    Highest member that is up to date wins
    Up to date == within 10 seconds of primary
    If a higher priority member catches up, it will force election and win

    Slave Delay
    Lags behind master by configurable time delay
    Automatically hidden from clients
    Protects against operator errors
    Fat fingering
    Application corrupts data
  • Priority
    Floating point number between 0..1000
    Highest member that is up to date wins
    Up to date == within 10 seconds of primary
    If a higher priority member catches up, it will force election and win

    Slave Delay
    Lags behind master by configurable time delay
    Automatically hidden from clients
    Protects against operator errors
    Fat fingering
    Application corrupts data
  • Priority
    Floating point number between 0..1000
    Highest member that is up to date wins
    Up to date == within 10 seconds of primary
    If a higher priority member catches up, it will force election and win

    Slave Delay
    Lags behind master by configurable time delay
    Automatically hidden from clients
    Protects against operator errors
    Fat fingering
    Application corrupts data
  • Priority
    Floating point number between 0..1000
    Highest member that is up to date wins
    Up to date == within 10 seconds of primary
    If a higher priority member catches up, it will force election and win

    Slave Delay
    Lags behind master by configurable time delay
    Automatically hidden from clients
    Protects against operator errors
    Fat fingering
    Application corrupts data
  • Reports (group, summing, averaging)
    Analytics(incremental reporting, rollups)
    Analysis (trends, segmentation, anomalies)
    Analytics (regression, forecasting, filtering)
    Warehousing (long term storage and simplified querying)
  • Reports (group, summing, averaging)
    Analytics(incremental reporting, rollups)
    Analysis (trends, segmentation, anomalies)
    Analytics (regression, forecasting, filtering)
    Warehousing (long term storage and simplified querying)
  • Reports (group, summing, averaging)
    Analytics(incremental reporting, rollups)
    Analysis (trends, segmentation, anomalies)
    Analytics (regression, forecasting, filtering)
    Warehousing (long term storage and simplified querying)
  • MongoDB for Time Series Data Part 2: Analyzing Time Series Data Using the Aggregation Framework and Hadoop

    1. 1. Consulting Engineer, MongoDB Bryan Reinero #ConferenceHashTag Time Series Data- Part 2 Aggregations in Action
    2. 2. Real Time Traffic Data Project Our network of 16,000 speed sensors report data every minute.
    3. 3. What we want from our data Charting and Trending
    4. 4. What we want from our data Historical & Predictive Analysis
    5. 5. What we want from our data Real Time Traffic Dashboard
    6. 6. Document Structure { _id: ObjectId("5382ccdd58db8b81730344e2"), linkId: 900006, date: ISODate("2014-03-12T17:00:00Z"), data: [ { speed: NaN, time: NaN }, { speed: NaN, time: NaN }, { speed: NaN, time: NaN }, ... ], conditions: { status: "Snow / Ice Conditions", pavement: "Icy Spots", weather: "Light Snow" } }
    7. 7. Sample Document Structure Compound, unique Index identifies the Individual document { _id: ObjectId("5382ccdd58db8b81730344e2"), linkId: 900006, date: ISODate("2014-03-12T17:00:00Z"), data: [ { speed: NaN, time: NaN }, { speed: NaN, time: NaN }, { speed: NaN, time: NaN }, ... ], conditions: { status: "Snow / Ice Conditions", pavement: "Icy Spots", weather: "Light Snow" } }
    8. 8. Sample Document Structure Saves an extra index { _id: “900006:14031217”, data: [ { speed: NaN, time: NaN }, { speed: NaN, time: NaN }, { speed: NaN, time: NaN }, ... ], conditions: { status: "Snow / Ice Conditions", pavement: "Icy Spots", weather: "Light Snow" } }
    9. 9. { _id: “900006:14031217”, data: [ { speed: NaN, time: NaN }, { speed: NaN, time: NaN }, { speed: NaN, time: NaN }, ... ], conditions: { status: "Snow / Ice Conditions", pavement: "Icy Spots", weather: "Light Snow" } } Sample Document Structure Range queries: /^900006:1403/ Regex must be left-anchored & case-sensitive
    10. 10. { _id: “900006:140312”, data: [ { speed: NaN, time: NaN }, { speed: NaN, time: NaN }, { speed: NaN, time: NaN }, ... ], conditions: { status: "Snow / Ice Conditions", pavement: "Icy Spots", weather: "Light Snow" } } Sample Document Structure Pre-allocated, 60 element array of per-minute data
    11. 11. Charts 0 10 20 30 40 50 60 70 MonMar10201404:57:00… MonMar10201405:31:00… MonMar10201406:05:00… MonMar10201406:39:00… MonMar10201407:13:00… MonMar10201407:47:00… MonMar10201408:21:00… MonMar10201408:55:00… MonMar10201409:29:00… MonMar10201410:04:00… MonMar10201410:38:00… MonMar10201411:55:00… TueMar11201402:41:00… TueMar11201403:15:00… TueMar11201403:49:00… TueMar11201404:39:00… TueMar11201405:13:00… TueMar11201405:47:00… TueMar11201406:21:00… TueMar11201406:55:00… TueMar11201407:29:00… TueMar11201408:03:00… TueMar11201408:37:00… TueMar11201409:18:00… TueMar11201410:44:00… TueMar11201411:18:00… TueMar11201411:53:00… TueMar11201412:27:00… TueMar11201413:04:00… TueMar11201413:38:00… TueMar11201414:15:00… TueMar11201416:56:00… WedMar12201401:45:00… WedMar12201402:19:00… WedMar12201402:53:00… WedMar12201403:27:00… WedMar12201406:46:00… WedMar12201408:26:00… WedMar12201409:00:00… WedMar12201410:12:00… WedMar12201410:46:00… db.linkData.find( { _id : /^20484097:2014031/ } )
    12. 12. Rollups { _id: "20484097:20140204", hours: [ { speed: { sum: 1889, count: 60 } time: { sum: 20562, count: 60 }, conditions: { status: "Snow / Ice Conditions", pavement: "Icy Spots", weather: "Light Snow" } }, { speed: {m: 1892, count: 60 }, time: {sum: 20442, count: 60 }, conditions: { status: "Snow / Ice Conditions", pavement: "Slush", weather: "Light Snow" } } ]}
    13. 13. Document retention Doc per hour Doc per day 2 days 2 months 1year Doc per Month
    14. 14. Analysis with The Aggregation Framework
    15. 15. Pipelining operations grep | sort | uniq Piping command line operations
    16. 16. Pipelining operations $match $group | $sort| Piping aggregation operations Stream of documents Result documents
    17. 17. What is the average speed for a given road segment? > db.linkData.aggregate( { $match: { ”_id" : /^20484097:/ } }, { $project: { "data.speed": 1 } } , { $unwind: "$data"}, { $group: { _id: “”, ave: { $avg: "$data.speed"} } } ); { "_id" : 20484097, "ave" : 47.067650676506766 }
    18. 18. What is the average speed for a given road segment? Select documents on the target segment > db.linkData.aggregate( { $match: { ”_id" : /^20484097:/ } }, { $project: { "data.speed": 1, linkId: 1 } } , { $unwind: "$data"}, { $group: { _id: "$linkId", ave: { $avg: "$data.speed"} } } ); { "_id" : 20484097, "ave" : 47.067650676506766 }
    19. 19. What is the average speed for a given road segment? Keep only the fields we really need > db.linkData.aggregate( { $match: { ”_id" : /^20484097:/ } }, { $project: { "data.speed": 1, linkId: 1 } } , { $unwind: "$data"}, { $group: { _id: "$linkId", ave: { $avg: "$data.speed"} } } ); { "_id" : 20484097, "ave" : 47.067650676506766 }
    20. 20. What is the average speed for a given road segment? Loop over the array of data points > db.linkData.aggregate( { $match: { ”_id" : /^20484097:/ } }, { $project: { "data.speed": 1, linkId: 1 } } , { $unwind: "$data"}, { $group: { _id: "$linkId", ave: { $avg: "$data.speed"} } } ); { "_id" : 20484097, "ave" : 47.067650676506766 }
    21. 21. What is the average speed for a given road segment? Use the handy $avg operator > db.linkData.aggregate( { $match: { ”_id" : /^20484097:/ } }, { $project: { "data.speed": 1, linkId: 1 } } , { $unwind: "$data"}, { $group: { _id: "$linkId", ave: { $avg: "$data.speed"} } } ); { "_id" : 20484097, "ave" : 47.067650676506766 }
    22. 22. More Sophisticated Pipelines: average speed with variance { "$project" : { mean: "$meanSpd", spdDiffSqrd : { "$map" : { "input": { "$map" : { "input" : "$speeds", "as" : "samp", "in" : { "$subtract" : [ "$$samp", "$meanSpd" ] } } }, as: "df", in: { $multiply: [ "$$df", "$$df" ] } } } } }, { $unwind: "$spdDiffSqrd" }, { $group: { _id: mean: "$mean", variance: { $avg: "$spdDiffSqrd" } } }
    23. 23. Historic Analysis How does weather and road conditions affect traffic? The Ask: what are the average speeds per weather, status and pavement
    24. 24. MapReduce function map() { for( var i = 0; i < this.data.length; i++ ) { emit ( this.conditions.weather, { speed : this.data[i].speed } ); emit ( this.conditions.status, { speed : this.data[i].speed } ); emit ( this.conditions.pavement, { speed : this.data[i].speed } ); } }
    25. 25. MapReduce function map() { for( var i = 0; i < this.data.length; i++ ) { emit ( this.conditions.weather, { speed : this.data[i].speed } ); emit ( this.conditions.status, { speed : this.data[i].speed } ); emit ( this.conditions.pavement, { speed : this.data[i].speed } ); } } “Snow”, 34
    26. 26. MapReduce function map() { for( var i = 0; i < this.data.length; i++ ) { emit ( this.conditions.weather, { speed : this.data[i].speed } ); emit ( this.conditions.status, { speed : this.data[i].speed } ); emit ( this.conditions.pavement, { speed : this.data[i].speed } ); } } “Icy spots”, 34
    27. 27. MapReduce function map() { for( var i = 0; i < this.data.length; i++ ) { emit ( this.conditions.weather, { speed : this.data[i].speed } ); emit ( this.conditions.status, { speed : this.data[i].speed } ); emit ( this.conditions.pavement, { speed : this.data[i].speed } ); } } “Delays”, 34
    28. 28. MapReduce
    29. 29. MapReduce Weather: “Rain”, speed: 44
    30. 30. MapReduce Weather: “Rain”, speed: 39
    31. 31. MapReduce Weather: “Rain”, speed: 46
    32. 32. MapReduce function reduce ( key, values ) { var result = { count : 1, speedSum : 0 }; values.forEach( function( v ){ result.speedSum += v.speed; result.count++; }); return result; }
    33. 33. MapReduce function reduce ( key, values ) { var result = { count : 1, speedSum : 0 }; values.forEach( function( v ){ result.speedSum += v.speed; result.count++; }); return result; }
    34. 34. Results results: [ { "_id" : "Generally Clear and Dry Conditions", "value" : { "count" : 902, "speedSum" : 45100 } }, { "_id" : "Icy Spots", "value" : { "count" : 242, "speedSum" : 9438 } }, { "_id" : "Light Snow", "value" : { "count" : 122, "speedSum" : 7686 } }, { "_id" : "No Report", "value" : { "count" : 782, "speedSum" : NaN } }
    35. 35. Processing Large Data Sets • Need to break data into smaller pieces • Process data across multiple nodes Hadoop Hadoop Hadoop Hadoop Hadoop Hadoop Hadoop HadoopHadoop Hadoop
    36. 36. Benefits of the Hadoop Connector • Increased parallelism • Access to analytics libraries • Separation of concerns • Integrates with existing tool chains
    37. 37. • Drivers will be accessing the data via web, mobile devices, and navigation systems • We need to provide current average speed, travel time and weather per road segment Real-time Dashboard
    38. 38. Current Real-Time Conditions Last ten minutes of speeds and times { _id : “I-87:10656”, description : "NYS Thruway Harriman Section Exits 14A - 16", update : ISODate(“2013-10-10T23:06:37.000Z”), speeds : [ 52, 49, 45, 51, ... ], times : [ 237, 224, 246, 233,... ], pavement: "Wet Spots", status: "Wet Conditions", weather: "Light Rain”, averageSpeed: 50.23, averageTime: 234, maxSafeSpeed: 53.1, location" : { "type" : "LineString", "coordinates" : [ [ -74.056, 41.098 ], [ -74.077, 41.104 ] } }
    39. 39. { _id : “I-87:10656”, description : "NYS Thruway Harriman Section Exits 14A - 16", update : ISODate(“2013-10-10T23:06:37.000Z”), speeds : [ 52, 49, 45, 51, ... ], times : [ 237, 224, 246, 233,... ], pavement: "Wet Spots", status: "Wet Conditions", weather: "Light Rain”, averageSpeed: 50.23, averageTime: 234, maxSafeSpeed: 53.1, location" : { "type" : "LineString", "coordinates" : [ [ -74.056, 41.098 ], [ -74.077, 41.104 ] } } Current Real-Time Conditions Pre-aggregated metrics
    40. 40. { _id : “I-87:10656”, description : "NYS Thruway Harriman Section Exits 14A - 16", update : ISODate(“2013-10-10T23:06:37.000Z”), speeds : [ 52, 49, 45, 51, ... ], times : [ 237, 224, 246, 233,... ], pavement: "Wet Spots", status: "Wet Conditions", weather: "Light Rain”, averageSpeed: 50.23, averageTime: 234, maxSafeSpeed: 53.1, location" : { "type" : "LineString", "coordinates" : [ [ -74.056, 41.098 ], [ -74.077, 41.104 ] } } Current Real-Time Conditions Geo-spatially indexed road segment
    41. 41. db.linksAvg.update( {"_id" : linkId}, { "$set" : {"update " : date}, "$push" : { "times" : { "$each" : [ time ], "$slice" : -10 }, "speeds" : {"$each" : [ speed ], "$slice" : -10} } }) Maintaining the current conditions Each update pops the last element off the array and pushes the new value
    42. 42. Putting it all together
    43. 43. Patterns common to time series data: • You need to store and manage an incoming stream of data samples • You need to compute derivative data sets based on these samples • You need low latency access to up-to-date data
    44. 44. Patterns common to time series data: • You need to store and manage an incoming stream of data samples • You need to compute derivative data sets based on these samples • You need low latency access to up-to-date data Introducing The High Volume Data Feed
    45. 45. HVDF: Reference Implementation Screech -- High Volume Data Feed engine REST Service API Processor Plugins Inline Batch Stream Channel Data Storage Raw Channel Data Aggregated Rollup T1 Aggregated Rollup T2 Query Processor Streaming spout Custom Stream Processing Logic Incoming Sample Stream POST /feed/channel/data GET /feed/channeldata?time=XX X&range=YYY Real-time Queries
    46. 46. HVDF: https://github.com/10gen-labs/hvdf Hadoop Connector: https://github.com/mongodb/mongo-hadoop
    47. 47. Consulting Engineer, MongoDB Inc. Bryan Reinero #MongoDBWorld Thank You
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×