MongoDB as a Universal Data Store for Process Data

1,634 views

Published on

‘Dynamic schema’, what’s in it for manufacturers? This session will give you a brief overview about the process industries’ IT landscape and how the sector dealt with Big Data long before the term was even invented. We shed a quick light on the evolution of data interface standards such as OPC UA. Taking a quick look into the motivation (and ongoing struggle) for seamless real-time system integration between Control and ERP, this talk will conclude with hard facts on why MongoDB qualifies so well in the industrial solution context. Deprecating the closed-shop data silo, inmation will share first-hand experience from MongoDB-centric system development.

Published in: Technology, Travel
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,634
On SlideShare
0
From Embeds
0
Number of Embeds
333
Actions
Shares
0
Downloads
54
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

MongoDB as a Universal Data Store for Process Data

  1. 1. MongoDB as a universal data store for Process Data Munich 2013
  2. 2. Munich 2013 The talk in brief It is all about two technology standards (coming together)…
  3. 3. Munich 2013 Timo Klingenmeier Co-founder, GM and Technical Lead inmation.com timo.klingenmeier@inmation.com 89-93 EMP (Engineering Company) Software Developer 93-03 GM, Founder iDV/best solutions AG Industrial System Integration, Custom SW Development 03-10 GM, Matrikon Deutschland AG Industrial Software Specialist, Real-Time Connectivity, Alarm Management, Performance Monitoring 11/12 Consultant Offshore Wind Projects 13 inmation.com NextGen Information Management Software
  4. 4. Munich 2013
  5. 5. Munich 2013
  6. 6. Munich 2013 Realtime 1ms – 1sec
  7. 7. Munich 2013 ERP / B2B / B2C MES and PIMS Control & Automation
  8. 8. Munich 2013 MES and PIMS ACRONYM ALERT! MES = Manufacturing Execution System PIMS = Process Information Management System
  9. 9. Munich 2013 1 Refining Company 15 Sites 2000 Applications CAPEX OPEX (Maintenance Contracts, etc.)
  10. 10. Munich 2013 Control Typical Production Process Applications Advanced Process Control Mass Balancing Laboratory Information Management Production Scheduling Alarm Management Quality Management Operational Excellence Performance, Reliability, Safety Management ERP Continuous Improvement (Generic Data-Mining)
  11. 11. Munich 2013 The evolution of industrial system integration (a brief history of time)
  12. 12. Munich 2013 Industry 3.0 Emptying the parking lots Source: flickr.com/acarlos1000
  13. 13. Munich 2013
  14. 14. Munich 2013 Today Distributed Control Systems Entire Sites Countless Items and Actors
  15. 15. Munich 2013 The Spinal Cord in Production: Contextualized Time-Series historization of numerical data and process events.
  16. 16. Munich 2013 The easiest way to improve a prediction is to add data. You can’t infer without data. So, store the data now and analyze it later,…[] Dwight Merriman, MongoDB evangelist
  17. 17. Munich 2013 The Babylonian Era
  18. 18. Munich 2013 Protocol Differences Information Contextualization Bandwidth and Throughput Implementation Complexity Robustness
  19. 19. Munich 2013
  20. 20. Munich 2013 Birth of a standard
  21. 21. Munich 2013
  22. 22. Munich 2013 You can talk DCOM to me. I will present you a qualified and more or less structured namespace, consisting of symbolically named items (tags). Each item may have additional properties. I can give you the actual value, the milisecond accuracy and the value quality.
  23. 23. Munich 2013 You can talk DCOM to me. On request, I will constantly deliver new Alarms & Events of any kind. Depending on the subordinated control systems I am connected to, the detail content of a single event record may vary. I can either supply all or filtered events.
  24. 24. Munich 2013 You can talk DCOM to me. Similar to my DA colleague on the left, I maintain a structured namespace of tags. Unlike this guy, I know about the entire history of their values. On request I will return raw values and statistical aggregates for any period of time.
  25. 25. Munich 2013 You can talk SOAP / XML to me. I can do what the DA guy does, but a little simplified.
  26. 26. Munich 2013 You can talk either binary TCP or XML to me. I offer various options for secure communications. I have many different profiles and facets. (Some people are confused about me) My smallest incarnation can work in a single chip solution, while I’m still qualifying for an enterprise-wide service. Obviously – Unified Architecture – I can supply all services of those little guys in one.
  27. 27. Munich 2013
  28. 28. Munich 2013 You can talk to me. (Everybody knows that) Very similar to the SQL language, I’m not young of age but definitely not willing to retire!
  29. 29. Munich 2013 One standard – many faces
  30. 30. Munich 2013 Globally standardized Universal Real-time Data Access Unknown to broader audience
  31. 31. Munich 2013 Industrial IT Data Storage Strategies Today Proprietary High-Frequency Time-Series Data Formats SQL Databases
  32. 32. Munich 2013 Application Differences Supported Interfaces Data Scope Extraction Data Format Storage Interfaces to next level
  33. 33. Munich 2013 Too many data silos
  34. 34. Munich 2013 Data Reduction & Loss
  35. 35. Munich 2013 How can industries create affordable, maintainable, open data stores which allow for the “Merriman paradigm” in the specific context of industrial data mining requirements ?
  36. 36. Munich 2013 the approach
  37. 37. Munich 2013 The problem to solve
  38. 38. Munich 2013 OS Platform? Scalability? Make your choice! ...or simply grow as required
  39. 39. Munich 2013 How we use MongoDB in our product 12 months ago, there were SQL parts There was homegrown data serialization Today, we only use MongoDB for any kind of data storage All network transports use inner BSON chunks, extended for efficient real-time object communication
  40. 40. Munich 2013 BSON Command Processor MongoDB Core cmd cmd_errors inSys cmd_processed BSON Working Model Working Model Core Object Connector Object BSON BSON Connector Object Endpoint Object Endpoint Object Endpoint Connector
  41. 41. Munich 2013 Remote OPC UA Server Endpoint Remote XML-DA Server Endpoint Remote/Local OPC DA Server Endpoint Remote/Local OPC A&E Server Endpoint Remote /Local OPC HDA Server Endpoint Connector Service
  42. 42. Munich 2013 Remote OPC UA Server Endpoint Remote XML-DA Server SPROX Protocol (Single Port TCP) Secure Prioritized Realtime Object Xchange Endpoint Remote/Local OPC DA Server Endpoint Remote/Local OPC A&E Server Endpoint Remote /Local OPC HDA Server Endpoint Connector Service Core Service
  43. 43. Munich 2013 Remote OPC UA Server Endpoint Remote XML-DA Server Schema Design: • Multiple Databases • Multiple Collections Database Design: • Replication (Redundancy) • Port TCP) SPROX Protocol (Single Sharding (Horizontal Scaling) Secure Prioritized Realtime Object Xchange Endpoint Remote/Local OPC DA Server Endpoint Connector Service MongoDB Remote/Local OPC A&E Server Different OPC Servers Endpoint Remote /Local OPC HDA Server Endpoint Mongo BSON Object Bulk Inserts Core Service
  44. 44. Munich 2013 Third-Party Stack (API) Command Application • • • • • • Realtime Data Access Component Registration Historical Data Access ComponentEvents Alarms and Configuration System Health Monitoring Any MongoDB Driver C++, C#, Java, … Any MongoDB Driver C++, C#, Java, … Connector Service Core Service MongoDB
  45. 45. Munich 2013 DataStudio • • • • • Component Registration Component Configuration System Health Monitoring Item Monitoring Log Analysis BOX Protocol (Single Port TCP) BSON Object Xchange Connector Service Core Service MongoDB
  46. 46. Munich 2013 OPC UA Server DataProxy BOX Protocol (Single Port TCP) BSON Object Xchange Connector Service Core Service MongoDB BSON MongoDB
  47. 47. Munich 2013
  48. 48. Munich 2013
  49. 49. Munich 2013 Using MongoDB-based Real-time Data (and a little bit of Google, too)
  50. 50. Munich 2013
  51. 51. Munich 2013 Summary – MongoDB qualifies for process data storage It has the performance and the scalability options required to store hi-freq data in huge amounts Timestamp accuracy is sufficient Schema (-less) flexibility fits the variant data structures, originated in (OPC) source systems Unbeatable value offer (both software and hardware utilization) It exposes ‘natural’ data structures which make any kind of analysis fun It satisfies IT people and engineers
  52. 52. Munich 2013 Thank you very much for your attention

×