Your SlideShare is downloading. ×
0
#mongodb

MongoDB Usage within
Oil, Gas, and Energy
Kevin Hanson
Senior Account Executive / Solutions Architect, MongoDB I...
Agenda
• Common Themes in MongoDB Usage
• What is MongoDB?
• Use-Cases and Examples
• Thinking Ahead

• Questions
Common Themes in MongoDB
Usage
Machine Generated Data
Fast Moving Data
• Hundreds of thousands of records per second
• Fast response required
• Sometimes all data kept, sometim...
Massive Amounts of Data
• Widely applicable data model
• Applies to several different “data use cases”
• Various schema an...
Data is Structured, but Varied…
• A machine generates a specific kind of data
• The data model is unlikely to change
• But...
Time Series Data
• Event data written multiple times per second,

minute, or hour
• Tracking progression of metrics over t...
What is MongoDB?
MongoDB is a ___________
database
• Open source
• High performance
• Full featured

• Document-oriented
• Horizontally sca...
Full Featured
• Dynamic (ad-hoc) queries
• Built-in online aggregation
• Rich query capabilities

• Traditionally consiste...
Document-Oriented Database
• A document is a nestable associative array

• Document schemas are flexible

• Documents can ...
Horizontally Scalable (Add Shards)
Replication Within a Shard
Enabling Global Deployments
Use-Case: Oil Rig Data Analysis
3 Points of Data Creation /
Collection
Hour Level Data

Rig Site
(Middle of the
Ocean)

Day Level Data

Regional Center
(N...
MongoDB on all 3 Sites

Hour Level Data

Rig Site
(Middle of the
Ocean)

Day Level Data

Regional Center
(Nearby Continent...
MongoDB on the Rig
{

machine-id: “derrick-72”,
utilization-rate: 92,
depth: 172,
ts: ISODate("2013-10-16T22:07:38.000-050...
MongoDB at the Regional Center
{

rig-id: “gulf-1a23v”,
machine-failures: 0,
efficiency: 82,
ts: ISODate("2014-07-13T22:12...
MongoDB at the Regional Center
MongoDB at Headquarters
{

{

region: “Atlantic”,
total-rigs: 102,
producing-rigs: 95,
barrels: 97000,
ts: ISODate("2014-0...
Powered by MongoDB Replication & the
Oplog
> db.replsettest.insert({_id:1,value:1})
{ "ts" : Timestamp(1350539727000, 1), ...
Use-Case: Predictive Energy Network
Analysis
Maintaining a Power Grid

Expensive Last Minute Resource Allocation
Use Data to Help Predict the Future
• Weather Radar Data
• Climate Models
• Syslog Data from Power Generating Entities
• G...
Sensor Data
• Straightforward to store in MongoDB documents

• With strategic document design, a single server can

save h...
Data Updates
db.sf-meter.update(
{
timestamp_minute: ISODate("2013-10-10T23:06:00.000Z"),
type: “richmond-district”
},
{
{...
Data Management
• Data stored at different granularity levels for read

performance
• Collections are organized into speci...
Aggregation Framework
• MongoDB has a built-in Aggregation Framework

that supports ad-hoc analysis tasks over data sets
•...
Pre-Aggregated Log Data
{
timestamp_minute: ISODate("2000-10-10T20:55:00Z"),
resource: ”sensor-5a3524s",
usage-values: {
0...
MongoDB Makes Sense
Massive Amounts of Data
• Commodity Storage
• Add Nodes for Scale
• No SAN Needed

• MongoDB Replication for HA
High Performance
• Massive Write Scale
• Massive Read Scale
• Real-Time Response
Flexible Data Model
• A single sensor isn’t likely to change its data

model…
• But what about the other sensors?
• Dynami...
Lower Total Cost of Ownership
• Open Source vs. Proprietary
• Commodity Hardware
• Reduced Development Time
Questions?
Resources
• Schema Design for Time Series Data in MongoDB

http://blog.mongodb.org/post/65517193370/schema-design-for-time...
Upcoming SlideShare
Loading in...5
×

Webinar: MongoDB Use Cases within the Oil, Gas, and Energy Industries

4,479

Published on

In this session we will dive into some of the use-cases companies are currently deploying MongoDB for in the energy space. It is becoming more important for companies to make data driven decisions, and MongoDB can often be the right tool for analyzing the massive amounts of data coming in. Whether tracking oil well site statistics, power meter data, or feeds from sensors, MongoDB can be a great fit for tracking and analyzing that data, using it to make smart, informed business decisions.

Published in: Technology
1 Comment
4 Likes
Statistics
Notes
No Downloads
Views
Total Views
4,479
On Slideshare
0
From Embeds
0
Number of Embeds
11
Actions
Shares
0
Downloads
93
Comments
1
Likes
4
Embeds 0
No embeds

No notes for slide
  • We have all these fantastic machines… they give the same metrics they used to, but now they transmit the data. We have metrics about metrics, and we need a place to store the data. We need a place to understand what the data means.
  • Transcript of "Webinar: MongoDB Use Cases within the Oil, Gas, and Energy Industries"

    1. 1. #mongodb MongoDB Usage within Oil, Gas, and Energy Kevin Hanson Senior Account Executive / Solutions Architect, MongoDB Inc. @hungarianhc ~ kevin@mongodb.com
    2. 2. Agenda • Common Themes in MongoDB Usage • What is MongoDB? • Use-Cases and Examples • Thinking Ahead • Questions
    3. 3. Common Themes in MongoDB Usage
    4. 4. Machine Generated Data
    5. 5. Fast Moving Data • Hundreds of thousands of records per second • Fast response required • Sometimes all data kept, sometimes just summary • Horizontal scalability required
    6. 6. Massive Amounts of Data • Widely applicable data model • Applies to several different “data use cases” • Various schema and modeling options • Application requirements drive schema design
    7. 7. Data is Structured, but Varied… • A machine generates a specific kind of data • The data model is unlikely to change • But there are so many different machines… • Queryability across all types
    8. 8. Time Series Data • Event data written multiple times per second, minute, or hour • Tracking progression of metrics over time
    9. 9. What is MongoDB?
    10. 10. MongoDB is a ___________ database • Open source • High performance • Full featured • Document-oriented • Horizontally scalable
    11. 11. Full Featured • Dynamic (ad-hoc) queries • Built-in online aggregation • Rich query capabilities • Traditionally consistent • Many advanced features • Support for many programming languages
    12. 12. Document-Oriented Database • A document is a nestable associative array • Document schemas are flexible • Documents can contain various data types (numbers, text, timestamps, blobs, etc)
    13. 13. Horizontally Scalable (Add Shards)
    14. 14. Replication Within a Shard Enabling Global Deployments
    15. 15. Use-Case: Oil Rig Data Analysis
    16. 16. 3 Points of Data Creation / Collection Hour Level Data Rig Site (Middle of the Ocean) Day Level Data Regional Center (Nearby Continent) Headquarters (Texas? )
    17. 17. MongoDB on all 3 Sites Hour Level Data Rig Site (Middle of the Ocean) Day Level Data Regional Center (Nearby Continent) Headquarters (Texas? )
    18. 18. MongoDB on the Rig { machine-id: “derrick-72”, utilization-rate: 92, depth: 172, ts: ISODate("2013-10-16T22:07:38.000-0500") } • Queried and analyzed by on-site rig personnel • High volume data with real-time response • Aggregations compute high level statistics • Statistics are transmitted to regional center
    19. 19. MongoDB at the Regional Center { rig-id: “gulf-1a23v”, machine-failures: 0, efficiency: 82, ts: ISODate("2014-07-13T22:12:21.000-0800") } • Monitoring important statistics from multiple rigs • Aggregating rig data to report regional data to headquarters
    20. 20. MongoDB at the Regional Center
    21. 21. MongoDB at Headquarters { { region: “Atlantic”, total-rigs: 102, producing-rigs: 95, barrels: 97000, ts: ISODate("2014-07-13") region: “Pacific”, total-rigs: 82, producing-rigs: 77, barrels: 44000, ts: ISODate("2014-07-13") } } • Regional views of the data • Real-time stats • Integration with hadoop for large batch processing jobs
    22. 22. Powered by MongoDB Replication & the Oplog > db.replsettest.insert({_id:1,value:1}) { "ts" : Timestamp(1350539727000, 1), "h" : NumberLong("6375186941486301201"), "op" : "i", "ns" : "test.replsettest", "o" : { "_id" : 1, "value" : 1 } } > db.replsettest.update({_id:1},{$inc:{value:10}}) { "ts" : Timestamp(1350539786000, 1), "h" : NumberLong("5484673652472424968"), "op" : "u", "ns" : "test.replsettest", "o2" : { "_id" : 1 }, "o" : { "$set" : { "value" : 11 } } }
    23. 23. Use-Case: Predictive Energy Network Analysis
    24. 24. Maintaining a Power Grid Expensive Last Minute Resource Allocation
    25. 25. Use Data to Help Predict the Future • Weather Radar Data • Climate Models • Syslog Data from Power Generating Entities • Geotagged Meter Usage
    26. 26. Sensor Data • Straightforward to store in MongoDB documents • With strategic document design, a single server can save hundreds of thousands of sensor reads per second
    27. 27. Data Updates db.sf-meter.update( { timestamp_minute: ISODate("2013-10-10T23:06:00.000Z"), type: “richmond-district” }, { {$set: {“values.59”: 2000000 }}, {$inc: {num_samples: 1, total_samples: 2000000 }} } ) • Single update required to add new data and increment associated counts
    28. 28. Data Management • Data stored at different granularity levels for read performance • Collections are organized into specific intervals • Retention is managed by simply dropping collections as they age out • Document structure is pre-created to maximize write performance
    29. 29. Aggregation Framework • MongoDB has a built-in Aggregation Framework that supports ad-hoc analysis tasks over data sets • “What counties had the highest average power utilization bracketed daily?” • “Which meters have the most surge problems per week?”
    30. 30. Pre-Aggregated Log Data { timestamp_minute: ISODate("2000-10-10T20:55:00Z"), resource: ”sensor-5a3524s", usage-values: { 0: 50, … 59: 250 } } • Leverage time-series style bucketing • Track individual metrics • Improve performance for reads/writes • Minimal processing overhead
    31. 31. MongoDB Makes Sense
    32. 32. Massive Amounts of Data • Commodity Storage • Add Nodes for Scale • No SAN Needed • MongoDB Replication for HA
    33. 33. High Performance • Massive Write Scale • Massive Read Scale • Real-Time Response
    34. 34. Flexible Data Model • A single sensor isn’t likely to change its data model… • But what about the other sensors? • Dynamic schema is a necessity • Easily drop collections for data management
    35. 35. Lower Total Cost of Ownership • Open Source vs. Proprietary • Commodity Hardware • Reduced Development Time
    36. 36. Questions?
    37. 37. Resources • Schema Design for Time Series Data in MongoDB http://blog.mongodb.org/post/65517193370/schema-design-for-time-seriesdata-in-mongodb • Operational Intelligence Use Case http://docs.mongodb.org/ecosystem/use-cases/#operational-intelligence • Data Modeling in MongoDB http://docs.mongodb.org/manual/data-modeling/ • Schema Design (webinar) http://www.mongodb.com/events/webinar/schema-design-oct2013
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×