Your SlideShare is downloading. ×

Mongo db as a universal data store for process data

781
views

Published on

Published in: Technology, Travel

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
781
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
75
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. MongoDB as a universal data store for Process Data Munich 2013
  • 2. Munich 2013 The talk in brief It is all about two technology standards (coming together)…
  • 3. Munich 2013 Timo Klingenmeier Co-founder, GM and Technical Lead inmation.com timo.klingenmeier@inmation.com 89-93 EMP (Engineering Company) Software Developer 93-03 GM, Founder iDV/best solutions AG Industrial System Integration, Custom SW Development 03-10 GM, Matrikon Deutschland AG Industrial Software Specialist, Real-Time Connectivity, Alarm Management, Performance Monitoring 11/12 Consultant Offshore Wind Projects 13 inmation.com NextGen Information Management Software
  • 4. Munich 2013
  • 5. Munich 2013
  • 6. Munich 2013 Realtime 1ms – 1sec
  • 7. Munich 2013 ERP / B2B / B2C MES and PIMS Control & Automation
  • 8. Munich 2013 MES and PIMS ACRONYM ALERT! MES = Manufacturing Execution System PIMS = Process Information Management System
  • 9. Munich 2013 1 Refining Company 15 Sites 2000 Applications CAPEX OPEX (Maintenance Contracts, etc.)
  • 10. Munich 2013 Control Typical Production Process Applications Advanced Process Control Mass Balancing Laboratory Information Management Production Scheduling Alarm Management Quality Management Operational Excellence Performance, Reliability, Safety Management ERP Continuous Improvement (Generic Data-Mining)
  • 11. Munich 2013 The evolution of industrial system integration (a brief history of time)
  • 12. Munich 2013 Industry 3.0 Emptying the parking lots Source: flickr.com/acarlos1000
  • 13. Munich 2013
  • 14. Munich 2013 Today Distributed Control Systems Entire Sites Countless Items and Actors
  • 15. Munich 2013 The Spinal Cord in Production: Contextualized Time-Series historization of numerical data and process events.
  • 16. Munich 2013 The easiest way to improve a prediction is to add data. You can’t infer without data. So, store the data now and analyze it later,…[] Dwight Merriman, MongoDB evangelist
  • 17. Munich 2013 The Babylonian Era
  • 18. Munich 2013 Protocol Differences Information Contextualization Bandwidth and Throughput Implementation Complexity Robustness
  • 19. Munich 2013
  • 20. Munich 2013 Birth of a standard
  • 21. Munich 2013
  • 22. Munich 2013 You can talk DCOM to me. I will present you a qualified and more or less structured namespace, consisting of symbolically named items (tags). Each item may have additional properties. I can give you the actual value, the milisecond accuracy and the value quality.
  • 23. Munich 2013 You can talk DCOM to me. On request, I will constantly deliver new Alarms & Events of any kind. Depending on the subordinated control systems I am connected to, the detail content of a single event record may vary. I can either supply all or filtered events.
  • 24. Munich 2013 You can talk DCOM to me. Similar to my DA colleague on the left, I maintain a structured namespace of tags. Unlike this guy, I know about the entire history of their values. On request I will return raw values and statistical aggregates for any period of time.
  • 25. Munich 2013 You can talk SOAP / XML to me. I can do what the DA guy does, but a little simplified.
  • 26. Munich 2013 You can talk either binary TCP or XML to me. I offer various options for secure communications. I have many different profiles and facets. (Some people are confused about me) My smallest incarnation can work in a single chip solution, while I’m still qualifying for an enterprise-wide service. Obviously – Unified Architecture – I can supply all services of those little guys in one.
  • 27. Munich 2013
  • 28. Munich 2013 You can talk to me. (Everybody knows that) Very similar to the SQL language, I’m not young of age but definitely not willing to retire!
  • 29. Munich 2013 One standard – many faces
  • 30. Munich 2013 Globally standardized Universal Real-time Data Access Unknown to broader audience
  • 31. Munich 2013 Industrial IT Data Storage Strategies Today Proprietary High-Frequency Time-Series Data Formats SQL Databases
  • 32. Munich 2013 Application Differences Supported Interfaces Data Scope Extraction Data Format Storage Interfaces to next level
  • 33. Munich 2013 Too many data silos
  • 34. Munich 2013 Data Reduction & Loss
  • 35. Munich 2013 How can industries create affordable, maintainable, open data stores which allow for the “Merriman paradigm” in the specific context of industrial data mining requirements ?
  • 36. Munich 2013 the approach
  • 37. Munich 2013 The problem to solve
  • 38. Munich 2013 OS Platform? Scalability? Make your choice! ...or simply grow as required
  • 39. Munich 2013 How we use MongoDB in our product 12 months ago, there were SQL parts There was homegrown data serialization Today, we only use MongoDB for any kind of data storage All network transports use inner BSON chunks, extended for efficient real-time object communication
  • 40. Munich 2013 BSON Command Processor MongoDB Core cmd cmd_errors inSys cmd_processed BSON Working Model Working Model Core Object Connector Object BSON BSON Connector Object Endpoint Object Endpoint Object Endpoint Connector
  • 41. Munich 2013 Remote OPC UA Server Endpoint Remote XML-DA Server Endpoint Remote/Local OPC DA Server Endpoint Remote/Local OPC A&E Server Endpoint Remote /Local OPC HDA Server Endpoint Connector Service
  • 42. Munich 2013 Remote OPC UA Server Endpoint Remote XML-DA Server SPROX Protocol (Single Port TCP) Secure Prioritized Realtime Object Xchange Endpoint Remote/Local OPC DA Server Endpoint Remote/Local OPC A&E Server Endpoint Remote /Local OPC HDA Server Endpoint Connector Service Core Service
  • 43. Munich 2013 Remote OPC UA Server Endpoint Remote XML-DA Server Schema Design: • Multiple Databases • Multiple Collections Database Design: • Replication (Redundancy) • Port TCP) SPROX Protocol (Single Sharding (Horizontal Scaling) Secure Prioritized Realtime Object Xchange Endpoint Remote/Local OPC DA Server Endpoint Connector Service MongoDB Remote/Local OPC A&E Server Different OPC Servers Endpoint Remote /Local OPC HDA Server Endpoint Mongo BSON Object Bulk Inserts Core Service
  • 44. Munich 2013 Third-Party Stack (API) Command Application • • • • • • Realtime Data Access Component Registration Historical Data Access ComponentEvents Alarms and Configuration System Health Monitoring Any MongoDB Driver C++, C#, Java, … Any MongoDB Driver C++, C#, Java, … Connector Service Core Service MongoDB
  • 45. Munich 2013 DataStudio • • • • • Component Registration Component Configuration System Health Monitoring Item Monitoring Log Analysis BOX Protocol (Single Port TCP) BSON Object Xchange Connector Service Core Service MongoDB
  • 46. Munich 2013 OPC UA Server DataProxy BOX Protocol (Single Port TCP) BSON Object Xchange Connector Service Core Service MongoDB BSON MongoDB
  • 47. Munich 2013
  • 48. Munich 2013
  • 49. Munich 2013 Using MongoDB-based Real-time Data (and a little bit of Google, too)
  • 50. Munich 2013
  • 51. Munich 2013 Summary – MongoDB qualifies for process data storage It has the performance and the scalability options required to store hi-freq data in huge amounts Timestamp accuracy is sufficient Schema (-less) flexibility fits the variant data structures, originated in (OPC) source systems Unbeatable value offer (both software and hardware utilization) It exposes ‘natural’ data structures which make any kind of analysis fun It satisfies IT people and engineers
  • 52. Munich 2013 Thank you very much for your attention