• Like

Webinar: MongoDB Use Cases within the Oil, Gas, and Energy Industries

  • 4,149 views
Uploaded on

In this session we will dive into some of the use-cases companies are currently deploying MongoDB for in the energy space. It is becoming more important for companies to make data driven decisions, …

In this session we will dive into some of the use-cases companies are currently deploying MongoDB for in the energy space. It is becoming more important for companies to make data driven decisions, and MongoDB can often be the right tool for analyzing the massive amounts of data coming in. Whether tracking oil well site statistics, power meter data, or feeds from sensors, MongoDB can be a great fit for tracking and analyzing that data, using it to make smart, informed business decisions.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
4,149
On Slideshare
0
From Embeds
0
Number of Embeds
10

Actions

Shares
Downloads
89
Comments
1
Likes
4

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • We have all these fantastic machines… they give the same metrics they used to, but now they transmit the data. We have metrics about metrics, and we need a place to store the data. We need a place to understand what the data means.

Transcript

  • 1. #mongodb MongoDB Usage within Oil, Gas, and Energy Kevin Hanson Senior Account Executive / Solutions Architect, MongoDB Inc. @hungarianhc ~ kevin@mongodb.com
  • 2. Agenda • Common Themes in MongoDB Usage • What is MongoDB? • Use-Cases and Examples • Thinking Ahead • Questions
  • 3. Common Themes in MongoDB Usage
  • 4. Machine Generated Data
  • 5. Fast Moving Data • Hundreds of thousands of records per second • Fast response required • Sometimes all data kept, sometimes just summary • Horizontal scalability required
  • 6. Massive Amounts of Data • Widely applicable data model • Applies to several different “data use cases” • Various schema and modeling options • Application requirements drive schema design
  • 7. Data is Structured, but Varied… • A machine generates a specific kind of data • The data model is unlikely to change • But there are so many different machines… • Queryability across all types
  • 8. Time Series Data • Event data written multiple times per second, minute, or hour • Tracking progression of metrics over time
  • 9. What is MongoDB?
  • 10. MongoDB is a ___________ database • Open source • High performance • Full featured • Document-oriented • Horizontally scalable
  • 11. Full Featured • Dynamic (ad-hoc) queries • Built-in online aggregation • Rich query capabilities • Traditionally consistent • Many advanced features • Support for many programming languages
  • 12. Document-Oriented Database • A document is a nestable associative array • Document schemas are flexible • Documents can contain various data types (numbers, text, timestamps, blobs, etc)
  • 13. Horizontally Scalable (Add Shards)
  • 14. Replication Within a Shard Enabling Global Deployments
  • 15. Use-Case: Oil Rig Data Analysis
  • 16. 3 Points of Data Creation / Collection Hour Level Data Rig Site (Middle of the Ocean) Day Level Data Regional Center (Nearby Continent) Headquarters (Texas? )
  • 17. MongoDB on all 3 Sites Hour Level Data Rig Site (Middle of the Ocean) Day Level Data Regional Center (Nearby Continent) Headquarters (Texas? )
  • 18. MongoDB on the Rig { machine-id: “derrick-72”, utilization-rate: 92, depth: 172, ts: ISODate("2013-10-16T22:07:38.000-0500") } • Queried and analyzed by on-site rig personnel • High volume data with real-time response • Aggregations compute high level statistics • Statistics are transmitted to regional center
  • 19. MongoDB at the Regional Center { rig-id: “gulf-1a23v”, machine-failures: 0, efficiency: 82, ts: ISODate("2014-07-13T22:12:21.000-0800") } • Monitoring important statistics from multiple rigs • Aggregating rig data to report regional data to headquarters
  • 20. MongoDB at the Regional Center
  • 21. MongoDB at Headquarters { { region: “Atlantic”, total-rigs: 102, producing-rigs: 95, barrels: 97000, ts: ISODate("2014-07-13") region: “Pacific”, total-rigs: 82, producing-rigs: 77, barrels: 44000, ts: ISODate("2014-07-13") } } • Regional views of the data • Real-time stats • Integration with hadoop for large batch processing jobs
  • 22. Powered by MongoDB Replication & the Oplog > db.replsettest.insert({_id:1,value:1}) { "ts" : Timestamp(1350539727000, 1), "h" : NumberLong("6375186941486301201"), "op" : "i", "ns" : "test.replsettest", "o" : { "_id" : 1, "value" : 1 } } > db.replsettest.update({_id:1},{$inc:{value:10}}) { "ts" : Timestamp(1350539786000, 1), "h" : NumberLong("5484673652472424968"), "op" : "u", "ns" : "test.replsettest", "o2" : { "_id" : 1 }, "o" : { "$set" : { "value" : 11 } } }
  • 23. Use-Case: Predictive Energy Network Analysis
  • 24. Maintaining a Power Grid Expensive Last Minute Resource Allocation
  • 25. Use Data to Help Predict the Future • Weather Radar Data • Climate Models • Syslog Data from Power Generating Entities • Geotagged Meter Usage
  • 26. Sensor Data • Straightforward to store in MongoDB documents • With strategic document design, a single server can save hundreds of thousands of sensor reads per second
  • 27. Data Updates db.sf-meter.update( { timestamp_minute: ISODate("2013-10-10T23:06:00.000Z"), type: “richmond-district” }, { {$set: {“values.59”: 2000000 }}, {$inc: {num_samples: 1, total_samples: 2000000 }} } ) • Single update required to add new data and increment associated counts
  • 28. Data Management • Data stored at different granularity levels for read performance • Collections are organized into specific intervals • Retention is managed by simply dropping collections as they age out • Document structure is pre-created to maximize write performance
  • 29. Aggregation Framework • MongoDB has a built-in Aggregation Framework that supports ad-hoc analysis tasks over data sets • “What counties had the highest average power utilization bracketed daily?” • “Which meters have the most surge problems per week?”
  • 30. Pre-Aggregated Log Data { timestamp_minute: ISODate("2000-10-10T20:55:00Z"), resource: ”sensor-5a3524s", usage-values: { 0: 50, … 59: 250 } } • Leverage time-series style bucketing • Track individual metrics • Improve performance for reads/writes • Minimal processing overhead
  • 31. MongoDB Makes Sense
  • 32. Massive Amounts of Data • Commodity Storage • Add Nodes for Scale • No SAN Needed • MongoDB Replication for HA
  • 33. High Performance • Massive Write Scale • Massive Read Scale • Real-Time Response
  • 34. Flexible Data Model • A single sensor isn’t likely to change its data model… • But what about the other sensors? • Dynamic schema is a necessity • Easily drop collections for data management
  • 35. Lower Total Cost of Ownership • Open Source vs. Proprietary • Commodity Hardware • Reduced Development Time
  • 36. Questions?
  • 37. Resources • Schema Design for Time Series Data in MongoDB http://blog.mongodb.org/post/65517193370/schema-design-for-time-seriesdata-in-mongodb • Operational Intelligence Use Case http://docs.mongodb.org/ecosystem/use-cases/#operational-intelligence • Data Modeling in MongoDB http://docs.mongodb.org/manual/data-modeling/ • Schema Design (webinar) http://www.mongodb.com/events/webinar/schema-design-oct2013