Building a reliable and scalable IoT platform with MongoDB and HiveMQ

Building a reliable and scalable IoT platform with MongoDB and HiveMQ
Building a Scalable and Reliable IoT Platform with HiveMQ and MongoDB Dr. Christian Kurze Principal Solutions Architect MongoDB Dominik Obermaier CTO and Co-Founder HiveMQ
How to optimize your Business with IoT? Typical IoT Challenges HiveMQ and MongoDB Solution Fleet Management Demo What you will learn today Q&A
Or: Why should you care? How to optimize your Business with IoT?
Digital Services are the Key Differentiator in Global Competition Intensifying Competition ... … Eroding Margins ... … leading to the need to rethink current business models 8 5 41 24 18 28 53 24 Asia-Pacific Americas Europe Rest of World 1998 2019 General industrial machinery Already commoditized Future price premium at stake Special purpose machinery Risk of hardware commoditization McKinsey: IIoT platforms: The technology stack as value driver in industrial equipment and machinery Services Software Machinery 1998 2019 Production of machinery by region (%) Indicative Value Pools
Digital Services are the Key Differentiator in Global Competition Intensifying Competition ... … Eroding Margins ... … leading to the need to rethink current business models
Leaders gain >15% positive impact on cost and revenue via IoT McKinsey: Best Practices Separating IoT Leaders from Laggards
Develop new IoT Products and Services e.g. apps, fleet management Business Strategies Drive Sales and Service Efficiency e.g. monitoring, field services, maintenance, staffing Optimize Business Operations e.g. manufacturing, supply chain, R&D
Devices with Sensors and Actuators e.g. cars, buildings, equipment, wearables Device Enablement Platforms e.g. obtaining, importing, and processing data using standard protocols IT Initiatives Business Applications e.g. customer- and/or device-facing functionality, dashboards, mobile apps Cloud and Edge Computing e.g. for new workloads and cost optimizations
A Digital Twin can represent almost everything: Machines, Processes, Places, Buildings, Cities, even Humans like You and Me.
Global leaders leverage IoT powered by MongoDB Source of Industries: IHS Markit
What is the core of IoT & Digital Twins? R&D Production Operation Maintenance Decommissioning Describe Predict Recommend Combines all data across the whole lifecycle of a product Outlives the physical product in order to optimize business processes and unlock new revenue streams Product Life-Cycle Information Life-Cycle
Only ~30% of relevant IoT solutions are in company-wide roll-out. Delivering IoT at scale requires the ability to extract, interpret, and harmonize data from disparate systems that were not designed to work together. McKinsey: Best Practices Separating IoT Leaders from Laggards
Or: Why does not everyone benefit? Typical IoT Challenges
People on the Internet
Devices on the Internet
Web Technology used today is built for the Internet of Humans, NOT for the Internet of Things
Many Different Types of Data ● Device data arrives in different formats (JSON, AVRO, Protobuf, custom binary formats) ● Often in time series data ● Data agnostic message brokers used to distribute data into the backend ● Relational databases are sometimes not well suited for IoT data
Responsiveness of Systems ● Low latency is critical for many IoT use cases ● End users expect responsive IoT applications ● Unreliable cellular networks can have a significant impact on responsiveness BMW Case Study: bit.ly/bmw-casestudy
Scalability ● IoT solutions need to scale to accommodate growth (100s - 1,000,000s of devices) ● Scale-up and scale-down to accommodate spikes
MQTT Broker Cluster IoT Devices Extension Extension Extension Enterprise Integration ● IoT data needs to be integrated into enterprise systems ● Device to Cloud and Cloud to Device data integration
On-premise Self hosted Managed Service Deployment Agnostic Atlas
Many different types of data Top IoT Challenges Responsiveness of systems Enterprise integration Deployment agnostic Scalability to thousands & millions of devices
Or: How to tackle the typical challenges? HiveMQ & MongoDB Solution
IoT Simplified: Same Pattern for any Application Sensors & Actuators Wireless communication over industry standard protocols Edge Gateway Offline Storage, Local Processing Streaming & Routing Standard Protocols and Tools, e.g. MQTT, Kafka Data Storage Hot & Cold Data for Real-Time & Batch Access Dashboards Visual Insights Applications User-Facing Applications & Automations Advanced Analytics & Machine Learning Gaining Insights into Data, Predict & Act
HiveMQ’s MQTT Broker for Communication Sensors & Actuators Wireless communication over industry standard protocols Edge Gateway Offline Storage, Local Processing Streaming & Routing Standard Protocols and Tools, e.g. MQTT, Kafka Data Storage Hot & Cold Data for Real-Time & Batch Access Dashboards Visual Insights Applications User-Facing Applications & Automations Advanced Analytics & Machine Learning Gaining Insights into Data, Predict & Act
Publish/subscribe based architecture Easy (I)IoT Messaging Protocol Minimal Overhead Designed for reliable communication over unreliable channels Binary Data agnostic What is MQTT?
Publish / Subscribe Pattern
Logistics Tele- communication IoT Messaging Middleware MQTT Use Cases Connected Car IIoT / Industry 4.0
HiveMQ Enterprise MQTT Platform
Building new digital products Improving customer experience Creating more efficient operations and insights Avoiding data loss with efficient system Our Customers …and more ...and more
Challenge: Responsiveness of Systems & Deployment Agnostic Challenge: Scalability to thousands & millions of devicesChallenge: Scalability to thousands & millions of devices Challenge: Enterprise Integration Tackling IoT Challenges World-class scalable MQTT ● Masterless architecture, Auto healing, elastic scaling ● Private or Public deployments ● K8s, OpenShift, AWS, Azure, GCP ● Ideal for multi-cloud Reliable Cloud Native Architecture ● Visibility for operations team ● Live debug of individual clients ● Trace recording for playback on message sequences Real-time Monitoring Across Device Fleets ● Data integration with existing enterprise systems ● Integration with other MQTT clients and broker ● Off-the-shelf integrations and custom extensions Extension Framework and Marketplace ● Scales to 10 million connections and more ● MQTT 5 fully supported ● Full hybrid support of 3.1.1 and 3.1
HiveMQ & MongoDB on-premises IoT Devices Extension Extension Extension
HiveMQ Cloud & MongoDB Atlas Extension Extension Extension IoT Devices Atlas
MongoDB’s Data Platform for IoT Sensors & Actuators Wireless communication over industry standard protocols Edge Gateway Offline Storage, Local Processing Streaming & Routing Standard Protocols and Tools, e.g. MQTT, Kafka Data Storage Hot & Cold Data for Real-Time & Batch Access Dashboards Visual Insights Applications User-Facing Applications & Automations Advanced Analytics & Machine Learning Gaining Insights into Data, Predict & Act
MongoDB Atlas: End-2-End Data Platform for IoT Primary Secondary Secondary High Volume Real Time Operational Data Analytical Analytical Real Time Analytical Data Data Lake, offline, queryable archive Lucene based Text Search Sharding and Replica Sets Multi-Cloud Data Platform OneQueryLanguage,API,andSQL Sensors & Actuators Wireless communication over industry standard protocols Edge Gateway Offline Storage, Local Processing Streaming & Routing Standard Protocols and Tools, e.g. MQTT, Kafka Mobile Database Edge to Cloud Sync Native Visualizations Applications & Microservices Advanced Analytics Triggers & Events Reporting
Benefits by Using One Single Data Platform Fast Time-to-Market One database query language for all platforms, incl. the data lake One codebase independent of deployment strategy, Optimization done once Transfer of resources between different teams Lower TCO Training (dev & ops) to be done once Efficiency of scale for operational costs (one team vs. platform-dedicated teams) Time-to-Value for new features is the same on all platforms One agreement across all platforms, same support team High Security Secure by default on different cloud providers Reusable security concept on multiple platforms Encryption (At Rest, In Use, In Flight), Authentication/Authorization, Auditing Low Operational Effort Same operations tooling and APIs independent of platform Same scaling approach and reusable integrations across all platforms Technical Support by the same Service Engineers independent of platform
Modelling a Basic “Thing” Unique Identifier Title, Description Creation, Modification Date Base Thing Relational Schema
Enriching a “Thing” Requires more Tables Unique Identifier Title, Description Creation, Modification Date Base Thing Properties & Schema Actions & Input / Output Events & Schema / Subscription / Cancellation Relational Schema
Further Enrichment Creates Complex Schemas Unique Identifier Title, Description Creation, Modification Date Base Thing Properties & Schema Actions & Input / Output Events & Schema / Subscription / Cancellation Translations Security Schemes Form Representation User-Defined Data ? Relational Schema
Strict Schema vs. Flexible Data Model id title description 12345-WoTLamp-1234 My Lamp A lamp in the room 67890-WoTLamp-1234 My Other Lamp Another lamp in the room id thing_id title readOnly writeOnly 4711-p1 12345-WoTLamp-1234 status false false id thing_id title safe idempodent 4711-a1 12345-WoTLamp-1234 toggle false false id thing_id title readOnly writeOnly 4711-e1 12345-WoTLamp-1234 overheating false false Thing Property Action Event { "id": "123456-WoTLamp-1234", "title": "My Lamp", "description": "A lamp in the room", "properties": { "status": { "type": "string", "readOnly" : false, "writeOnly" : false } }, "actions": { "toggle": { "safe": false, "idempodent": false } }, "events": { "overheating": { "data": { "type": "string", "readOnly" : false, "writeOnly" : false } } } } Relational Example Flexible Document Model
Extensible at Runtime { "id": "123456-WoTLamp-1234", "title": "My Lamp", "description": "A lamp in the room", + "properties": { ... }, "actions": { "toggle": { "safe": false, "idempotent": false } }, + "events": { ... } } { "id": "123456-WoTLamp-1234", "title": "My Lamp", "description": "A lamp in the room", "securityDefinitions": { "basic_sc": { "scheme": "basic", "in": "header" } }, + "properties": { ... }, "actions": { "toggle": { "safe": false, "idempodent": false, "forms": [{ "op": "invokeaction", "href": "https://mylamp.example.com/toggle", "contentType": "application/json" }] } }, + "events": { ... } } Adding Security and Action Information Schema-Free based on well-defined standards like JSON-LD and W3C’s Web of Things Vocabulary
What about Timeseries? Schema Design Pattern: Bucketing https://www.mongodb.com/collateral/time-series-best-practices
Telediagnostics: The Future of Mercedes Benz Services Learn more about the project in Madalin Broscaru’s MongoDB.live presentation: Telediagnostics@Mercedes Benz powered by MongoDB … the Vehicle Data Conditioning (VDC) where these technical vehicle events are processed … CAC Retail Customer … and the follow-up processes are triggered with real-time recommendations for actions. Vehicles are transmitting regularly status and health data into ... Aggregated Quality Analysis
Telediagnostics: The Future of Mercedes Benz Services { ............. "schema": "3.1.0" "createdAt": {..}, ............. "vehicleIdentData": { "chassisNumber": "WDD24708A5432J63", "countryCode": "4f3490b14e238a5f", "modelSeries": "f16ad22d42064811", "modelType": "2196868af1c70d74", "modelYear": "5e01ac15d73c3e4a", "steering": "4a3424fe6411461c" }, "basicData": { "mileage": {..}, "batteries": [..], "tanks": [..], "tiresPressure": [..], }, "controlUnits": [..], "affectedFunctions": [..], "vehicleClusterMessagesData": [..], "maintenanceData": {...}, .............. } "controlUnits": [ { "ecuId": {..} "name": "a927e49b0549f00f71", "detailsHardware": {}, "detailsSoftware": {}, "dtc": [ { "code": "B214F73", ... "failureText": "81957650ea", "environmentalData": {...} }, ... ] }, ... ]
Or: How does a solution look like? Show me the architecture and the code! Fleet Management Demo
IoT is broad - let’s look into a specific example Source of Industries: IHS Markit Fleet Management Covers fleets of commercial vehicles, fork lifts, trains, goods, literally anything that forms a fleet and needs to be monitored and actively managed. Major Challenges High distribution of fleet, unstable network connections, multitude of device types and data structures, scaling from 100’s to 1,000’s to 1,000,000s Typical Benefits Reduction of shipping costs, goods arrive in time, less outages and damages, reaction times in minutes instead of hours, CO2 reporting and reduction, higher fleet utilization, compliance with legal requirements
Getting Started with MongoDB & HiveMQ MongoDB Atlas Free Tier Cluster Retention ½ Day HiveMQ vehicles/trucks/truck-XXXXX MQTT Subscriber Python Trucks on tour sending location, speed, break time MQTT Subscriber Python Truck Simulator: - >9000 Warehouses across Germany - Trucks travel between random warehouses - Every second, the current location, speed, speed limit and break information is sent Real-Time Position of Trucks: - Subscription to MQTT topics for live visualization Analysis of Trucks: - Visualization of historical data based on truck routes
Getting Started with MongoDB & HiveMQ MongoDB Atlas Free Tier Cluster Retention ½ Day HiveMQ vehicles/trucks/truck-XXXXX MQTT Subscriber Python Trucks on tour sending location, speed, break time MQTT Subscriber Python Truck Simulator: - >9000 Warehouses across Germany - Trucks travel between random warehouses - Every second, the current location, speed, speed limit and break information is sent Real-Time Position of Trucks: - Subscription to MQTT topics for live visualization Analysis of Trucks: - Visualization of historical data based on truck routes
Or: What about X, Y and Z? Q&A
Get Started Today! MongoDB Atlas https://cloud.mongodb.com HiveMQ https://hivemq.com/ We love to hear your feedback! Dr. Christian Kurze | Principal Solutions Architect | christian.kurze@mongodb.com Dominik Obermaier | CTO and Co-Founder | dominik.obermaier@hivemq.com

Check these out next

Building a reliable and scalable IoT platform with MongoDB and HiveMQ

Recommended

More Related Content

Slideshows for you(20)

Similar to Building a reliable and scalable IoT platform with MongoDB and HiveMQ(20)

More from Dominik Obermaier(16)

Recently uploaded(20)

Building a reliable and scalable IoT platform with MongoDB and HiveMQ