Key Data Management Requirements for the IoT
•Download as PPTX, PDF•
17 likes•8,111 views
The document discusses key data management requirements for Internet of Things (IoT) applications. It notes that IoT will generate massive amounts of structured and unstructured data from a large number of connected devices and sensors. This data must be managed in a way that allows for rich applications, a unified view of data, real-time operational insights, business agility, and continuous innovation. It argues that traditional relational databases may not be well-suited for IoT data management and that NoSQL databases can provide scalability, flexibility, analytics and a unified view of data from multiple sources.
More Related Content
What's hot
What's hot (20)
Similar to Key Data Management Requirements for the IoT
Similar to Key Data Management Requirements for the IoT (20)
More from MongoDB
More from MongoDB (20)
Recently uploaded
Recently uploaded (20)
Key Data Management Requirements for the IoT
- 1. Key Data Management Requirements for the IoT
- 2. Register to Reserve your Seat Webinar 1: Vision & Use Cases Webinar 2: Data Management Requirements for the IoT Webinar 3: From Concept to Code Joint Webinar Series
- 3. Today’s Speakers Dirk Slama, Director of Business Development Bosch Software Innovations 20+ years of experience in large-scale distributed application projects including M2M and IoT projects Joe Drumgoole, Director Partner Technical Services MongoDB 20+years experience of commercial software delivery
- 4. • IoT Vision from Bosch: Recap of Part 1 • Data Management Requirements in IoT • Next Steps Agenda
- 5. 5 7,1tn IoT Solutions Revenue | IDC Some Big Numbers: 1,9tn IoT Economic Value Add | Gartner 309bn IoT Supplier Revenue | Gartner 50bn Connected Devices | Cisco 14bn Connected Devices | Bosch SI Some small numbers: http://postscapes.com/internet-of-things-market-size Peter Middleton, Gartner: “By 2020, component costs will have come down to the point that connectivity will become a standard feature, even for processors costing less than $1 “ IoT Predictions by 2020 - 22
- 6. TensHundredsThousandsMillionsBillionsConnections Internet of Things Machine-to-Machine Monitored Smart Systems (Intelligence in Subnets of Things ) Telemetry and Telematics Smart Homes Connected Cars Intelligent Buildings Intelligent Transport Systems Smart Meters and Grids Smart Retailing Smart Enterprise Management Remotely controlled and managed Building automation Manufacturing Security Utilities Internet of Things Sensors Devices Systems Things Processes People Industries Products Services Source: Machina Research 2014 Growth in connections generates an unparalleled scale of data
- 7. Data Big data Changing data models Real-time Processing Aggregation Internet of Things Large estates of devices Evolving applications All forms of data Data streaming and processing Pre-IoT (M2M) Limited estate of devices Single purpose applications Structured / Semi- structured Data transfers (sensors and actuators) Evolution from M2M to IoT and Big Data Source: Machina Research 2014
- 8. Data Big data Changing data models Real-time processing Aggregation Databases will need to address new requirements Scalability Flexibility Analytics Unified View Source: Machina Research 2014
- 9. IoT Foundation: Bosch Suite for IoT A D C B Scale Flexibility Analytics Unified View
- 10. IoT Foundation: Bosch Suite for IoT A D C B Scale Flexibility Analytics Unified View
- 11. Data has Changed • 90% of the world’s data was created in the last two years • 80% of enterprise data is unstructured • Unstructured data growing 2x faster than structured
- 12. Yesterday’s Tool for Today’s Data?
- 13. IoT Data Management Requirements Rich Applications Single View Operational Insight Real-Time Business Agility Continuous Innovation Enterprise-Ready Secure & Reliable Multiple Data Sources Process Convergence
- 14. Building Rich Applications Farm to Fork: Track production through supply chain Quality Assurance: Proactively reduce product wastage Fleet Management: Compare drivers & vehicles
- 15. • IoT apps generate multi- structured data • Modeled more efficiently as JSON documents • Exposed to powerful analytics • Developers more productive – Less time wrestling ORMs – More time creating apps Modeling Complex Data { vehicle_id: ‘123abc’, vehicle_driver: ‘Miller’, base: ‘London’, tracking: [ { timestamp: ‘2014-01-17- 12:00:00’, location: [51.123,-0.232], speed: 55, … }, { timestamp: ‘2014-01-17- 12:15:00’, location: [51.224,-0.238], speed: 5, … } } }
- 16. Unlocking Business Agility Customer Insight: Optimize in-store product placement Smart Factory: Flexible assembly lines, autonomous production modules Fleet Management: Extend to fuel efficiency, driver safety
- 17. Continuous Integration ID PSI Temp Loc New Column 3 months later… • Dynamic database schema • No need to define upfront • Enables agile methodologies – Evolves as the application changes – Eliminates teams co-ordinating ALTER TABLE operations
- 18. Creating a Single View Single View of the Customer: Across channels Single View of Production Across multiple lines Single View of the Fleet Across real-time and service history
- 19. Business Process Convergence New Table New Table New Column • Aggregate data from multiple source systems – Real time sensor data blended with enterprise data • Define single schema, update whenever the source systems change…or JOIN hundreds of tables! • Document model & dynamic schema makes single view a reality
- 20. Real-Time Operational Insight Inventory Management: Track stock levels, predict demand Optimize Production Lines Analyze robotic performance Preventative Maintenance Correlate baselines to diagnostics
- 21. Powerful Analytics on Live Data
- 22. Enterprise Ready Platform Secure Customer Data Privacy & compliance Continuous Availability Maximize production capacity Scale Data Volumes More sensors, more vehicles
- 23. Scalable, Reliable, Secure Automatic Sharing & Replica Sets Defense in Depth
- 24. Case Study
- 25. Field Data Capturing Project SCFD Structured Capturing of Field Data Components: Car brakes, power steering, etc. Usage patterns: temperature, voltage, etc. Predictive maintenance, product optimization Why MongoDB: Constantly evolving system, from a data capturing and a data analytics point of view Large amount of streaming data Asset Management Stream Processing Big Data Management Analytics BRM BRM
- 26. Next Steps
- 27. Services to Support IoT Apps TRAINING Training for developers and administrators – online and in-person CONSULTING Expert resources for all phases of IoT implementations
- 28. • Listen On-Demand Part 1: IoT Vision & Use Cases, Bosch & Machina Research • Register for Part 3: From Concept to Code Register Now • Download the Bosch SI & MongoDB Whitepaper IoT & MongoDB Learn More
- 30. Any Questions
Editor's Notes
- At the heart of the change is data generated by a myriad of new sensors and devices, and the role in plays in modern apps • 90% of the world’s data was created in the last two years • 80% of enterprise data is unstructured • Unstructured data growing 2x faster than structured In the IoT economy, data is the raw currency. How you stores, manages, analyzes and uses data has a direct impact on the your success.
- RDBMS was only real database option up until relatively recently – great for structured data, but no good for multi-structured, polymorphic data generated by todays IoT applications (polymorphic = similar but different, e.g. address fields, or user profiles). Even historically, the RDBMS only held 15-20% of an organisation’s information assets. We now have the tools and technologies that can harness the other 80%
- Based on both ours and Bosch’s own research and experience, we have identified the following 5 key capabilities for data management in IoT: Creating rich, functional applications: Data management must support the development of functionally rich applications with complex data and algorithms, with fast time to market and at low cost. Unlocking Business Agility: The ability to support many new and frequently changing business requirements, causing fast and continuous evolution of the underlying data model. Enabling a Single Point of Truth & Business Convergence: Aggregate multiple views of related data from multiple systems into one consistent version of the data. Real-Time Operational Insight: Support both transactional as well as analytical applications from the same data source Enterprise-Grade Platform: Provide highly scalable, cloud-based, robust and secure applications. We have all seen the impact of security breaches on a organisations. Reputational risk is a key challenge for a modern global brand. Explore each of these in turn
- Work through requirements, tie back to 3 industry use cases we used in the first webinar – Retail, Manu and Telematics Today’s applications now incorporate a wide variety of data, bringing structured, semi-structured and unstructured data together to yield deeper operational insight into all areas of the business: • Retail: “farm-to-fork” initiatives and increasing regulatory requirements to prove food lineage require the addi-tion of sensors to generate audit trails tracking food production and transportation through the supply chain. • Manufacturing: Capturing time-series, event based sensor data directly from the production line enables manu-facturers to detect when processes are exceeding predefined tolerances and quickly take corrective action to avoid product wastage. • Telematics and Mobility: The engine data bus is already established as a standard in luxury cars and trucks, con-solidating individual events from sensors for engine diagnostics. The addition of new sensors enables richer appli-cations and an increasing amount of data is being pushed back to central servers, enabling a fleet management company to start building new asset management applications that compare drivers and vehicles across their fleet, identifying best -- and worst -- practices.
- Semi-structured and unstructured data does not lend itself to be stored and processed in the rigid row and column format imposed by relational databases, and cannot be fully harnessed for analytics if stored in BLOBS or flat files. With sub-documents and arrays, JSON documents also align with the data structure of objects at the application level. This makes it easy for developers to map the information model of the device or asset to its associated document in the database. In contrast, trying to map the same object representation of the data to the tabular representation of an RDBMS slows down development. Adding Object Relational Mappers (ORMs) can create additional complexity by reducing the flexibility to evolve schemas and to optimize queries to meet new application requirements. Instead of spending a lot of time dealing with the impedance mismatch between the programming language and the database, the developers must be enabled to focus on creating rich, functional applications.
- IoT is in its infancy. Changes in customer requirements, emerging standards and new use-cases demand flexible and dynamic development methodologies and data storage architecture. Retail: Technologies such as NFC and Apple’s iBeacon enable retailers to derive as much insight from customer movement around their physical stores as they are used to getting from tracking customer movement around their eCommerce stores. By capturing and visualizing data from these location-based sensors, retailers can build heat maps to optimize the placement of high-margin products. Manufacturing: Smart Factory concepts are proposing more flexible assembly lines and support for smaller batch sizes by moving away from centrally controlled systems towards chains of intelligent and more autonomous production modules which interact with each other directly via the products, e.g. using RFID to create a “product memory”. Telematics and Mobility: Rapid developments in automation and fleet management see each new generation of vehicle bristling with more sensors! Established telematics applications such as geo-location and engine diagnostics are being complemented by new services extending to areas such as fuel efficiency, driver safety, theft prevention and more.
- Key here is the dynamic database schema The rapid evolution of IoT applications can be constrained by traditional software development methodologies -- for example, the waterfall approach places enormous dependency on the requirements defined upfront. In the IoT age organizations need flexible, iterative development practices to make it easy for teams to respond to new business and market demands, without being held back by rigid data models. We can’t predict what we will want to do with the data, how we will enrich it in the future and how it will be linked to other data. MongoDB’s dynamic schema means that application development and ongoing evolution are straightforward, enabling continuous integration as developers add new features. MongoDB enables developers to evolve the database schema through iterative and agile methodologies. Developers can start writing code and persist the objects without first pre-defining their structure. Each document (analogous to a row in a relational database) can have its own set of fields. Users can adapt the structure of a document’s schema just by adding new fields or deleting existing ones, making it very simple to handle to the rapidly changing data generated fast moving IoT applications. Contrast this with a traditional relational database -- the developer and DBA working on a new project must first start by specifying the database schema, before any code is written. At minimum this will take days; it often takes weeks or months. When in production need to schedule the necessary ALTER TABLE operations – can cause downtime, take weeks on large databases. As MongoDB allows schemas to evolve dynamically, such an upgrade requires modifying just the application, with typically no action required for MongoDB.
- Building a single view of a business entity -- whether a physical asset or a customer -- can deliver a range of benefits, from improved cross-sell and upsell to enhanced operational insight and reduced costs: Retail: Building on the NFC and Beacon example earlier, retailers can instantiate a single view of their customers in real time, converging actual location in the store with their profile, purchase history and loyalty card details in order to deliver timely and targeted promotions. Manufacturing: Production line machines contain many discrete components, each with their own sensors. Bringing these together, along with the relevant service history can ensure optimum asset utilization and production line efficiency. Telematics / Mobility: Fleet managers can blend views of a vehicle’s real time operational performance and diagnostics against asset registers that track service history to optimize preventative maintenance schedules.
- Creating this “single point of truth” requires aggregating multiple views of related data distributed across different source systems into one consistent view. Using a relational database, the development and DBA team would first have to undertake lengthy design reviews in order to pre-define a common schema. Subsequent changes to the any of the source schemas would then necessitate associated changes to the single view schema. MongoDB’s dynamic schema and flexible document model do not impose the same constraints, enabling source systems to continuously evolve without impacting the single view needed by the business.
- IoT applications enable new levels of operational insight and business discovery, but their value can only be fully realized when analysis is delivered in real time -- providing the ability to react and respond as processes are in-flight. Retail: Inventory is tracked as it moves from shelf to basket while the retailer concurrently performs analytics that attempt to match available supply to predicted demand, adjusting for any deviation automatically through warehouse operations and the supply chain. Manufacturing: Sensor data from robotic systems is persisted to the database while analytics work in the background to identify optimizations to the production line Telematics / Mobility: Engine diagnostics is enhanced by writing a continuous stream of sensor data to the database while simultaneously performing analytics comparing current status to historical baseline readings in order to proactively identify deviations and potential faults. For example, changes in oil or engine temperature may indicate the need to perform preventative maintenance.
- Many traditional databases support operational applications by capturing structured data as it is generated. They then rely on slow moving batch ETL (Extract Transform Load) processes to replicate the data to the Enterprise Data Warehouse (EDW) where it is blended with semi-and-unstructured data for OLAP (OnLine Analytical Processing). To eliminate the analytics latency that inhibits real time business insight, it is necessary for the database to support both operational and analytical processes across the same data source handling structured, semi-structured and unstructured data. You can do this on MongoDB – powerful query framework over multi-structured data. Could be run against secondaries so separate operational from analytical workloads
- As IoT applications become embedded within the operational fabric of the business, they must deliver the scalability, availability and security demanded by any enterprise application. Business continuity and security are typically governed by strict mandates in every industry vertical. Specific examples include: ● Retail: The recent security breach of 70 million customer accounts at Target contributed to a 46% drop in net profits. The 2011 Playstation network breach at Sony Corporation is estimated to have cost the business over $4.5bn. As IoT applications are integrated into retail operations, they must provide security against attack if other businesses are not to suffer the same costs. Standards such as PCI-DSS and HIPAA (for those retailers selling pharmaceutical products) are also top of mind. ● Manufacturing: If a production line were to stop due to an unplanned failure -- even for a short period of time -- the costs can be significant, including lost production capacity, idle workers and scrapped product. As IoT is at the very heart of many production line systems, continuous availability should be a prime concern. ● Telematics / Mobility: more sensors in each vehicle. Smart vehicle technology filters down product lines into economy models
- While databases such as MongoDB offer new capabilities for flexible data management and agile development methodologies, they cannot compromise on the enterprise-grade capabilities of traditional relational databases. Using MongoDB organizations can build fault tolerant and secure applications that scale-out on commodity hardware as data volumes generated by sensors continues to explode. Scale out on commodity hardware using application transparent automatic sharding Integrated replication enables us to create self healing clusters replicated within and across data centers Most robust security mechanisms of any leading NoSQL database, with authentication via LDAP, Kerberos or PKI certificates, authorisation via RBAC, field level redaction to restrict access to specific fields, in built auditing and encryption
- We can help you get started – Bosch and MongoDB collaboratively deliver consulting and training