Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Enabling Smarter Cities and Connected Vehicles with an Event Streaming Platform / Apache Kafka

904 views

Published on

Many cities are investing in technologies to transform their cities into smart city- environments in which data collection and analysis is utilized to manage assets and resources efficiently. Modern technology can help connect the right data, at the right time, to the right people, processes and systems. Innovations around smart cities and the Internet of Things give cities the ability to improve motor safety, unify and manage transportation systems and traffic, save energy and provide a better experience for the residents.

By utilizing an event streaming platform, like Confluent, cities are able to process data in real-time from thousands of sources, such as sensors. By aggregating that data and analyzing real-time data streams, more informed decisions can be made and fine-tuned operations developed for a positive impact on everyday challenges faced by cities.

Learn how to:
-Overcome challenges for building a smarter city
-Build a real time infrastructure to correlate relevant events
-Connect thousands of devices, machines, and people
-Leverage open source and fully managed solutions from the Apache Kafka ecosystem

Published in: Software
  • Be the first to comment

Enabling Smarter Cities and Connected Vehicles with an Event Streaming Platform / Apache Kafka

  1. 1. 1 Kai Waehner | Technology Evangelist, Confluent contact@kai-waehner.de | LinkedIn | @KaiWaehner | www.confluent.io | www.kai-waehner.de Enabling Smarter Cities and Connected Vehicles with an Event Streaming Platform Slides created together with Robert Cowart
  2. 2. 2 Agenda 1. Goals for Creating a Better World 2. Challenges for Building a Smarter City 3. The Smarter City Nervous System 4. Architecture Patterns for Edge, Hybrid and Global Infrastructures 5. Streaming Connectivity to Devices, Machines and People 6. Data Correlation in Real Time 7. Integration and Correlation between 100000 Connected Cars www.kai-waehner.de | @KaiWaehner
  3. 3. 3 Agenda 1. Goals for Creating a Better World 2. Challenges for Building a Smarter City 3. The Smarter City Nervous System 4. Architecture Patterns for Edge, Hybrid and Global Infrastructures 5. Streaming Connectivity to Devices, Machines and People 6. Data Correlation in Real Time 7. Integration and Correlation between 100000 Connected Cars www.kai-waehner.de | @KaiWaehner
  4. 4. 4 Goals for The Smarter City Improve Pedestrian Safety Improve Vehicle Safety Proactively Engage First Responders Reduce Traffic Congestion Enable Connected/Autonomous Vehicles Improve Customer Experience Automate Business Processes www.kai-waehner.de | @KaiWaehner
  5. 5. 5 Innovative new business models emerging… https://www.wejo.com/ https://parknowgroup.com/on-street-cashless-mobile-parking-payments/ https://www.scheidt-bachmann.de/en/article/news/ticketless-parking-management-system-the- future-has-begun-motorists-can-now-park-and-pay-without/ www.kai-waehner.de | @KaiWaehner
  6. 6. 6 Virtual Singapore: A Digital Twin of the (Smart) City Possible Uses of Virtual Singapore • Urban Planning (e.g. Crowd Simulation) • Collaboration and Decision-Making • Communication and Visualisation • Improved Accessibility • Analysis on Potential for Solar Energy Production • … https://www.nrf.gov.sg/programmes/virtual-singapore www.kai-waehner.de | @KaiWaehner
  7. 7. 7 Goals for The Smarter City The right insights (enriched and analyzed) At the right time (increasingly “real-time”) To the right people, processes and systems www.kai-waehner.de | @KaiWaehner
  8. 8. 8 Agenda 1. Goals for Creating a Better World 2. Challenges for Building a Smarter City 3. The Smarter City Nervous System 4. Architecture Patterns for Edge, Hybrid and Global Infrastructures 5. Streaming Connectivity to Devices, Machines and People 6. Data Correlation in Real Time 7. Integration and Correlation between 100000 Connected Cars www.kai-waehner.de | @KaiWaehner
  9. 9. 9 Integration with different data sources and technologies… Traffic Cameras (video & metrics) LIDAR Real-time Traffic Services Traffic Signals Other sensors • MetroTech IntelliSection with RTT • Swarm Analytics Perception Box • Quanergy • Velodyne • HERE • Bing Maps • Tom Tom • Automated Traffic Signal Performance Measures (ATSPM) • SAE J2735 (DSRC/WAVE) via Roadside Unit (RSU) • Surface Temperature • Pressure • Induction www.kai-waehner.de | @KaiWaehner
  10. 10. 10 Integration with different data sources and technologies… Traffic Cameras (video & metrics) LIDAR Real-time Traffic Services Traffic Signals Other sensors • MetroTech IntelliSection with RTT • Swarm Analytics Perception Box • Quanergy • Velodyne • HERE • Bing Maps • Tom Tom • Automated Traffic Signal Performance Measures (ATSPM) • SAE J2735 (DSRC/WAVE) via Roadside Unit (RSU) • Surface Temperature • Pressure • Induction And that is just some of the traffic related data! www.kai-waehner.de | @KaiWaehner
  11. 11. 111111 The requirement for multiple perspectives … www.kai-waehner.de | @KaiWaehner
  12. 12. 12 The need for transformation and correlation… 2019-09-30 00:00:00.500,80,82,52 2019-09-30 00:00:00.600,80,43,4 2019-09-30 00:00:00.700,80,2,6 2019-09-30 00:00:00.700,80,2,2 2019-09-30 00:00:00.900,80,82,6 2019-09-30 00:00:01.000,80,43,6 2019-09-30 00:00:02.473,80,400,0 2019-09-30 00:00:03.000,80,82,9 2019-09-30 00:00:03.900,80,81,9 2019-09-30 00:00:04.200,80,82,9 2019-09-30 00:00:04.400,80,8,6 2019-09-30 00:00:04.400,80,81,6 2019-09-30 00:00:04.400,80,4,6 2019-09-30 00:00:04.400,80,7,6 2019-09-30 00:00:04.400,80,8,2 2019-09-30 00:00:04.400,80,4,2 2019-09-30 00:00:04.400,80,7,2 2019-09-30 00:00:04.500,80,81,9 2019-09-30 00:00:04.500,80,44,6 2019-09-30 00:00:08.500,80,9,2 GAP OUT GREEN TERMINATION BEGIN YELLOW CLEARANCE www.kai-waehner.de | @KaiWaehner Traffic light station sensor information
  13. 13. 13 Why the “right time” is “real time”… www.kai-waehner.de | @KaiWaehner
  14. 14. 14 Why the “right time” is “real time”… https://www.ntsb.gov/investigations/AccidentReports/Pages/HWY18MH010-prelim.aspx www.kai-waehner.de | @KaiWaehner
  15. 15. 15 Why data correlation is important… www.kai-waehner.de | @KaiWaehner
  16. 16. 16 Agenda 1. Goals for Creating a Better World 2. Challenges for Building a Smarter City 3. The Smarter City Nervous System 4. Architecture Patterns for Edge, Hybrid and Global Infrastructures 5. Streaming Connectivity to Devices, Machines and People 6. Data Correlation in Real Time 7. Integration and Correlation between 100000 Connected Cars www.kai-waehner.de | @KaiWaehner
  17. 17. 17 A Streaming Platform is the Underpinning of an Event-driven Architecture Sensors Cameras CRM Mobile Real-time routing Cross selling Data warehouse Producers Consumers Object detection Sensor event CRM data Customer experiences Streams of real time events Stream processing apps Connectors Connectors Stream processing apps
  18. 18. 18 Apache Kafka – The Commit Log Time P C1 C2 C3 www.kai-waehner.de | @KaiWaehner
  19. 19. 19 Apache Kafka – A Distributed System Broker 1 Topic1 partition1 Broker 2 Broker 3 Broker 4 Topic1 partition1 Topic1 partition1 Leader Follower Topic1 partition2 Topic1 partition2 Topic1 partition2 Topic1 partition3 Topic1 partition4 Topic1 partition3 Topic1 partition3 Topic1 partition4 Topic1 partition4 www.kai-waehner.de | @KaiWaehner
  20. 20. 20 Apache Kafka (kafka.apache.org) includes Kafka Connect and Kafka Streams Kafka Streams Your app sinksource Kafka ConnectKafka Connect www.kai-waehner.de | @KaiWaehner
  21. 21. 21 Building the Smarter City Nervous System with Confluent • Middleware • Streaming ETL (Transform, Enrichment, Multi-Stream) • Business Applicationswww.kai-waehner.de | @KaiWaehner
  22. 22. 22 Agenda 1. Goals for Creating a Better World 2. Challenges for Building a Smarter City 3. The Smarter City Nervous System 4. Architecture Patterns for Edge, Hybrid and Global Infrastructures 5. Streaming Connectivity to Devices, Machines and People 6. Data Correlation in Real Time 7. Integration and Correlation between 100000 Connected Cars www.kai-waehner.de | @KaiWaehner
  23. 23. 23 A Kafka Cluster Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Producer Consumer Kafka Connect Kafka Connect www.kai-waehner.de | @KaiWaehner
  24. 24. 24 Disaster Recovery with 2 Kafka Clusters Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Producer Consumer Kafka Connect Kafka Connect Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Producer Consumer Kafka Connect Kafka Connect Data Center City-North Data Center City-South Streaming Replication www.kai-waehner.de | @KaiWaehner
  25. 25. 25 Aggregation of Kafka Clusters Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Producer Consumer Kafka Connect Kafka Connect Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Producer Consumer Kafka Connect Kafka Connect Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Producer Consumer Kafka Connect Kafka Connect Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Producer Consumer Kafka Connect Kafka Connect Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Kafka Connect Kafka Connect Kafka Broker Kafka Broker Kafka Broker www.kai-waehner.de | @KaiWaehner Analytics Data Center / Cloud Data Collection Data Center City-North Data Collection Data Center City-East Data Collection Data Center City-South Data Collection Data Center City-West
  26. 26. 26 Regional Edge Processing with Kafka Clusters Zookeeper Kafka Broker Schema Registry OPC-UA MQTT PLC4X KSQL Grafana Postgres Kafka Connect Zookeeper Kafka Broker Schema Registry OPC-UA MQTT PLC4X KSQL Grafana Postgres Kafka Connect Zookeeper Kafka Broker Schema Registry OPC-UA MQTT PLC4X KSQL Grafana Postgres Kafka Connect Zookeeper Kafka Broker Schema Registry OPC-UA MQTT PLC4X KSQL Grafana Postgres Kafka Connect Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Kafka Connect Kafka Connect Kafka Broker Kafka Broker Kafka Broker Real Time Correlations DC North Real Time Correlations DC East Real Time Correlations DC South Real Time Correlations DC West www.kai-waehner.de | @KaiWaehner Synchronization Data Center / Cloud
  27. 27. 27 Architecture patterns for distributed, hybrid, edge and global Apache Kafka deployments https://www.kai-waehner.de/blog/2020/01/29/ deployment-patterns-distributed-hybrid-edge-global-multi-data-center-kafka-architecture/ www.kai-waehner.de | @KaiWaehner
  28. 28. 28 Agenda 1. Goals for Creating a Better World 2. Challenges for Building a Smarter City 3. The Smarter City Nervous System 4. Architecture Patterns for Edge, Hybrid and Global Infrastructures 5. Streaming Connectivity to Devices, Machines and People 6. Data Correlation in Real Time 7. Integration and Correlation between 100000 Connected Cars www.kai-waehner.de | @KaiWaehner
  29. 29. 29 Ingestion Edge Collection Edge to Cloud Cloud Ingress Kafka Ingress dslink-scala- kafka10 toketi-kafka- connect-iothub kafka-connect- gcp-pubsub www.kai-waehner.de | @KaiWaehner
  30. 30. 30 Data Processing Producers Consumers• Client Libraries • Kafka Connectors • Proxies • Client Libraries • Kafka Connectors • Proxies X Consumer Groups www.kai-waehner.de | @KaiWaehner
  31. 31. 31 Kafka Connect Elasticsearch Sink Connector Kafka Connect InfluxDB Sink Connector Kafka Clients JavaScript, Golang, C, C++, Python, REST, etc. Stream Services User Facing Dashboards Infrastructure Health Egress www.kai-waehner.de | @KaiWaehner
  32. 32. 32 Kafka Connect - Data Sources and Sinks Data Diode Pre-built Connectors Hundreds of open source and commercial connectors available
  33. 33. 33 Event Streaming and IoT Platforms are Complementary Kafka Cluster Siemens MindSphere Kafka Client (Java, .NET, Go, Python) Kafka Streams ksqlDB Sensors REST Proxy MQTT Broker MQTT Connector Kafka Connect Azure IoT Hub Mobile App
  34. 34. 34 Agenda 1. Goals for Creating a Better World 2. Challenges for Building a Smarter City 3. The Smarter City Nervous System 4. Architecture Patterns for Edge, Hybrid and Global Infrastructures 5. Streaming Connectivity to Devices, Machines and People 6. Data Correlation in Real Time 7. Integration and Correlation between 100000 Connected Cars www.kai-waehner.de | @KaiWaehner
  35. 35. 35 Data Processing and Correlation Topic (observation-raw) www.kai-waehner.de | @KaiWaehner
  36. 36. 36 Data Processing and Correlation Topic (observation-raw) Metadata and Geo (lat/long) Enrichment www.kai-waehner.de | @KaiWaehner
  37. 37. 37 Traditional Database Event Streaming Process SELECT * FROM DB_TABLE CREATE TABLE T AS SELECT * FROM EVENT_STREAM Active Query: Passive Data: DB Table Active Data: Passive Query: Event Stream www.kai-waehner.de | @KaiWaehner
  38. 38. 38 Data Processing and Correlation Topic (observation-raw) Metadata and Geo (lat/long) Enrichment Topic (observation-meta) www.kai-waehner.de | @KaiWaehner
  39. 39. 39 Data Processing and Correlation Topic (observation-raw) Metadata and Geo (lat/long) Enrichment Topic (observation-meta) Streams Health Score & Incident Detection www.kai-waehner.de | @KaiWaehner
  40. 40. 4040 STREAM PROCESSING Create and store materialized views Filter Analyze in-flight Time C CC www.kai-waehner.de | @KaiWaehner
  41. 41. 41 Data Processing and Correlation Topic (observation-raw) Metadata and Geo (lat/long) Enrichment Topic (observation-meta) Streams Health Score & Incident Detection Topic (observation-out) www.kai-waehner.de | @KaiWaehner
  42. 42. 42 Data Processing and Correlation Topic (observation-raw) Metadata and Geo (lat/long) Enrichment Topic (observation-meta) Streams Health Score & Incident Detection Topic (observation-out) Elasticsearch Kafka Connect Elasticsearch Sink Connector www.kai-waehner.de | @KaiWaehner
  43. 43. 43 Agenda 1. Goals for Creating a Better World 2. Challenges for Building a Smarter City 3. The Smarter City Nervous System 4. Architecture Patterns for Edge, Hybrid and Global Infrastructures 5. Streaming Connectivity to Devices, Machines and People 6. Data Correlation in Real Time 7. Integration and Correlation between 100000 Connected Cars www.kai-waehner.de | @KaiWaehner
  44. 44. 44 Streaming Analytics with Kafka and TensorFlow MQTT Proxy Elastic Search Grafana Kafka Cluster Kafka Connect Car Sensors Kafka Ecosystem TensorFlow Other Components Kafka Streams (Java) All Data Critical Data Ingest Data Potential Detect KSQL TensorFlow Train Analytic Model Consume Data Preprocess Data Analytic Model Deploy Analytic Model Python https://github.com/kaiwaehner/hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-inference www.kai-waehner.de | @KaiWaehner
  45. 45. 45 Architecture for 100000 Connected Cars Kafka + KSQL + MQTT + TensorFlow + Kubernetes https://www.youtube.com/watch?v=7oVSLt0AZ3M www.kai-waehner.de | @KaiWaehner
  46. 46. 46 Questions? Let’s connect... Kai Waehner Technology Evangelist kai.waehner@confluent.io @KaiWaehner www.confluent.io www.kai-waehner.de LinkedIn

×