Apache Kafka in Manufacturing and Industry 4.0

1. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Event Streaming Use Cases and Architectures in Manufacturing 4.0 Automotive, Aerospace, Semiconductors, Oil&Gas, Chemical, Food, and others… Kai Waehner Technology Evangelist contact@kai-waehner.de LinkedIn @KaiWaehner www.confluent.io www.kai-waehner.de

2. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.deApache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de 1. Industry 4.0 - Digital Transformation for the Industrial IoT (IIoT) 2. 10000 Foot View – Event Streaming for Industry 4.0 3. Track&Trace / Production Control / Plant Logistics 4. Quality Assurance / Yield Management and Predictive Maintenance 5. Supply Chain Management 6. Cybersecurity 7. Servitization with Digital Twins 8. Additive Manufacturing 9. Augmented Reality Agenda

4. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Industry 4.0 / Industrial IoT (IIoT)

5. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Digitalization changes the whole manufacturing process https://www.mckinsey.com/industries/advanced-electronics/our-insights/reimagining-industrial-supply-chains

6. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de History of Automation Industry vs. Big Data and Cloud Christofer Dutz (codecentric) https://foss-backstage.de/sites/foss-backstage.de/files/2018-07/Revolutionizing%20Industrial%20IoT%20with%20Apache%20PLC4X.pdf IIoT != IoT Legacy / Proprietary Monoliths Long lifecycles No Security

7. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Evolution of Convergence between IT and Industrial Automation https://iot-analytics.com/5-industrial-connectivity-trends-driving-the-it-ot-convergence

8. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Huge demand to build an open, flexible, scalable platform • Real time • Scalability • High availability • Decoupling • Cost reduction • Flexibility • Elasticity • Standards-based • Extendibility • Security • Infrastructure-independent • Multi-region / global

9. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.deApache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de 1. Industry 4.0 - Digital Transformation for the Industrial IoT (IIoT) 2. 10000 Foot View – Event Streaming for Industry 4.0 3. Track&Trace / Production Control / Plant Logistics 4. Quality Assurance / Yield Management and Predictive Maintenance 5. Supply Chain Management 6. Cybersecurity 7. Servitization 8. Additive Manufacturing 9. Augmented Reality Agenda

10. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.deApache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de STREAM PROCESSING Create and store materialized views Filter Analyze in-flight Time C CC Event Streaming

11. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de An Event Streaming Platform is the Underpinning of an Event-driven Architecture MES ERP Sensors Mobile Customer 360 Real-time Alerting System Data warehouse Producers Consumers Streams of real time events Stream processing apps Connectors Connectors Stream processing apps Supplier Alert ForecastInventory Customer Order

12. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Hybrid Architecture - On Premise vs. Public Cloud VM SELF MANAGED FULLY MANAGED Edge vs. regional vs. global deployments Cloud-first (greenfield) vs. Hybrid architecture vs. Strategic move from on premise to cloud

13. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Digital Service at the Edge Single Kafka Broker or Cluster Digital Service Self-managed or certified OEM Hardware Kafka Cluster in DC / Cloud Replicator Siemens S7, Modbus, Allen Bradley, Beckhoff ADS

14. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Centralized Digital Service Digital Service Single Kafka Broker (or Cluster) Self-managed or certified OEM Hardware Siemens S7, Modbus, Allen Bradley, Beckhoff ADS Single Kafka Broker (or Cluster) Self-managed or certified OEM Hardware Siemens S7, Modbus, Allen Bradley, Beckhoff ADS

15. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Global Event Streaming Aggregate Small Footprint Edge Deployments with Replication (Aggregation) Simplify Disaster Recovery Operations with Multi-Region Clusters with RPO=0 and RTO=0 Stream Data Globally with Replication and Cluster Linking

16. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Tesla Trillions of messages per day for IoT use cases https://www.confluent.io/kafka-summit-san-francisco-2019/0-60-teslas-streaming-data-platform/ https://www.confluent.io/blog/stream-processing-iot-data-best-practices-and-techniques/

18. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Traceability and individual product serialization Production control Plant logistics Minimization of waste Elimination of fraud Inventory transparency

19. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Kafka Connect Kafka Cluster MQTT Integration Domain-Driven Design (DDD) for your Event Steaming Platform SAP Integration Real Time Predictions IoT Platform Connector Java / Python / ”you-name-it” Schema Registry Event Streaming Platform IoT Domain OT Domain Analytics Domain è Independent and loosely coupled, but scalable, highly available and reliable! 22

20. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de More than just Messaging! Data Correlation in Real-Time… https://eng.lyft.com/a-new-real-time-map-matching-algorithm-at-lyft-da593ab7b006

21. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Construction site management Collaborative planning Inventory and asset management Track, manage, and locate tools and equipment anytime and anywhere https://www.confluent.io/customers/bosch/ https://events.confluent.io/online-talks/bosch-power-toolse-nables-real-time-analytics-on-iot-event-streams

22. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Food Value Chain IoT-Based and Data-Driven Single source of truth across the food value chain (in the factories, and across regions) Business critical operations (tracking, calculations, alerts, …) https://www.confluent.io/blog/creating-iot-based-data-driven-food-value-chain-with-confluent-cloud/

23. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Location-Analytics and Geofencing • Geofence: Virtual perimeter for a real-world geographic area • Dynamically generated or predefined set of boundaries (such as plant area) • Location-aware device of a location-based service (LBS) user entering or exiting a geofence • Trigger events / alerts (enter, exist, outside, inside) https://www.slideshare.net/gschmutz/location-analytics-realtime-geofencing-using-apache-kafka-203115651 https://developer.confluent.io/podcast/location-data-and-geofencing-with-apache-kafka-ft-guido-schmutz

25. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Overall Equipment Effectiveness (OEE) The gold standard for measuring manufacturing productivity https://www.oee.com/oee-six-big-losses.html OEE score 100% = 100% Quality (only good parts) + 100% Performance (as fast as possible) + 100% Availability (no stop time)

26. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Analytics Database Data Lake CRM Kafka as Data Historian Continuous real time data ingestion, processing and monitoring 24/7 at scale MQTT Modbus Firewall Siemens S7 Kafka Connect Integration | Storage | Processing | Access | Cloud | Security Storage Kafka Streams / ksqlDB OPC-UA Stateless + Stateful Kafka Connect Kafka Clients Java, C++. Python, Go, … REST Proxy SCADA DCS ERP MES SyslogFile X Y Z A B C Cloud Factory

27. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Reduction in Production Cost – Early Part Scrapping Station 1 Station 2 Cost Before 100$ Total Cost 450$ (or more) 200$ 150$ Scrap? Scrap?

28. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de BI Tool AI/ML Machine Vision for Quality Assurance and Yield Management Apache Kafka and Applied Machine Learning Filter, transform aggregate, orchestrate APP Real-time alerting Sensor Data SCADA MES PLCs OT Team Plant Manager Images from Products of Assembly Lines IT Team Live Ops Machine Vision for Quality Inspection Reporting Backup Data Science Team Data Lake

30. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Supply Chain Management (SCM) • Planning and coordination of all the people, processes, and technology involved in creating value for a company • Cross cutting processes, including purchasing / procurement, logistics, operations / manufacturing, and others • Automation, robustness, flexibility, hybrid deployment (edge + cloud) Six Sigma (6σ)

31. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de SCM = Zoo of technologies and products Options: No software Buy Rent Make Mix TMS Transport Management System WMS Warehouse Management System WES Warehouse Execution System DPS Demand Planning System DRP Distribution Requirements Planning LMS Labor Management System CRM Customer Relationship Management SRM Supplier Relationship Management ERP Enterprise Resources Planning BI Business Intelligence MES Manufacturing Execution System PLM Product Lifecycle Management

32. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Postmodern ERP (coined by Gartner) Replace legacy, monolithic and highly customized ERP suites by a mixture of loosely coupled, exchangeable cloud-based and on-premises applications. TMS Legacy Proprietary SOAP Web Services Supplier Alert ForecastInventory Customer Order Core ERP CRM SaaS Kafka Interface MES Proprietary HTTP Web Services LMS Legacy Homegrown Database + CDC SRM Kafka-native

33. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Supply Chain Planning Streaming and Batch Analytics Use Case • Prepare for the unknown and the unknowable • Act at the right time in the right context • Yossi Sheffi: “You need to have sensors in the ground”, then SCM becomes a process of sensing and responding What if…?

34. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de From a batch-oriented systems architecture to a streaming micro services platform Thoughtworks: https://www.confluent.io/kafka-summit-sf17/Fast-Data-in-Supply-Chain-Planning

35. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Strangler Design Pattern The Big Bang does not always work… “The most important reason to consider a strangler fig application over a cut-over rewrite is reduced risk.” Martin Fowler Thoughtworks: https://www.confluent.io/kafka-summit-sf17/Fast-Data-in-Supply-Chain-Planning

36. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Supply Chain Purchasing Real-time Natural Language Processing (NLP) for Digital Contract Intelligence

37. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de BMW Group Industry-ready NLP Service Framework Based on Kafka https://www.confluent.io/kafka-summit-lon19/industry-ready-nlp-service-framework-kafka/

38. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Supply-chain operations reference (SCOR) model

39. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Cross-Company Supply Chain Integration Streaming Replication and API Management MirrorMaker 2 Confluent Replicator Cluster Linking Tier 2 Supplier OEM Streaming integration between companies API Management (REST et al) is not appropriate for streaming data Infosec and politics are your biggest hurdle Tier 1 Supplier

40. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Kafka vs. Blockchain for Supply Chain Management 46 Use Tamper-proof Apache Kafka for • Enterprise infrastructure • Open, scalable, real-time requirements • Flexible architectures for many use cases Use Hyperledger, Ethereum, et al. for • Deployment over various independent organizations • Participants verify the distributed ledger contents themselves. • Specific use cases • Server-side managed and controlled by multiple organizations • Scenarios where the business value overturns the added complexity and project risk Use Kafka and Blockchain together to combine the benefits of both (where this makes sense!) https://events.confluent.io/blockchain-kafka https://www.kai-waehner.de/blog/2020/07/17/apache-kafka-blockchain-dlt-comparison-kafka-native-vs-hyperledger-ethereum-ripple-iota-libra/

42. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.deApache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de https://news.microsoft.com/transform/hackers-hit-norsk-hydro-ransomware-company-responded-transparency/ https://www.insurancejournal.com/news/international/2019/04/30/525093.htm

43. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Cybersecurity The threat is real! Challenges Stealing IP DDoS Ransomware / wiperware WannaCry, NotPetya, … Damage: Billions of dollars ”Supply chain attack” Industry 4.0 Networking Communication Connectivity Open standards ”Always-on”

44. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Legacy SIEM needs to evolve ForwarderNetwork traffic Firewall logs RDBMS Application logs Adaptors Beats Sensor Data Challenges: ● Proprietary forwarders that can only send data to single source ● Data locked from being shared ● Difficult to scale with growing data volumes ● Prohibitively high indexing costs ● Unable to filter out noisy data ● Slow batch processing HTTP proxy logs

45. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de AI/ML Modernized security information and event management (SIEM) Filter, transform, aggregate APP SIEM Index Search Curated streams Forensic Archive HDFS S3 Big Query Syslog CDC Network traffic Firewall logs RDBMS Application logs Sensor Data HTTP proxy logs QRadar Arcsight Splunk Elastic

46. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Hybrid Cyber Data Platform Secure intermediary (on Linux) between the existing (Windows) hardware and modern (Linux) infrastructure Customer/Site 1 Customer/Site 2 Customer/Site N Central Data Center / Public Cloud

48. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Software and Digital Services become the Key Differentiator 54 https://www.mckinsey.com/industries/advanced-electronics/our-insights/iiot-platforms-the-technology-stack-as-value-driver-in-industrial-equipment-and-machinery

49. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Digital Twin – Merging the Physical and the Digital World 56 • Downtime reduction • Inventory management • Fleet management • What-if simulations • Operational planning • Servitization • Product development • Healthcare • Customer experience “Virtual representation of something else (Physical thing, process, service)” “A living model that drives a business outcome” https://www.youtube.com/watch?v=Ri0TD7kYsIQ

50. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Digital Thread 57 Digital Twin vs. Digital Thread? I only use the term Digital Twin in the following slides. Both terms overlap, often meaning the same. Span the entire lifecycle

51. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Apache Kafka Kafka as IoT Platform for the Digital Twin 58 Siemens S7, Modbus, Allen Bradley, Beckhoff ADS Digital Twin Real Time App Batch App Request Response App Kafka Connect Connectivity Homogenization Reprogrammable and smart Digital traces Modularity Storage Processing

52. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Apache Kafka Kafka as Digital Twin 59 Siemens S7, Modbus, Allen Bradley, Beckhoff ADS Digital Twin Real Time App Batch App Request Response App Kafka Connect Storage Processing Connectivity Homogenization Reprogrammable and smart Digital traces Modularity

53. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Connected Car Infrastructure 61 https://www.youtube.com/watch?v=yGLKi3TMJv8 • Real Time Data Analysis • Swarm Intelligence • Collaboration with Partners • Predictive AI • …

55. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Additive Manufacturing (aka 3D Printing) https://www.researchgate.net/figure/Products-that-Prosumers-want-to-realize-with-3D-printing_tbl5_320243961 (A prosumer is a person who consumes and produces a product) Prosumer at Home Smart Factory

56. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Distributed Manufacturing • 3D printing everything from houses and human organs to parts for the Mars Rover and even pizza • Game changer for supply chains • Continuous streaming and process analytics • Design files are intellectual property (and can easily be replicated) • Streaming design files • transforms how we consume physical products • secures illegal use https://www.computerworld.com/article/2836714/how-3d-printing-and-the-netflix-model-are-changing-everything.html https://spaceflight101.com/iss/made-in-space-1st-iss-3d-printer

57. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Additive Manufacturing (aka 3D Printing) Use Cases for Event Streaming with Apache Kafka Production Continuous streaming and process analytics for cost reduction / quality improvement Customer 360 Customer care, sales, support for full product lifecycle Security Securing intellectual property through a streaming model instead of file sharing

59. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Augmented Reality in Manufacturing Hand holding mobile smartphone use AR application to check and control welding robotics automatic arms machine Virtual Prototyping Smart Assistance Logistics & Warehouse Management

60. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Augmented Reality (AR) Tools and SDKs AR SDKs Vuforia, Wikitude, ARKit, ARCore, MaxST, EasyAR, Kudan, ARToolKit, Armedia, DeepAR.ai Virtual Dimension Center (VDC) Fellbach

61. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Augmented Reality for Smart Assistence with Apache Kafka, Kafka Connect and ksqlDB Pre-Processing and Data Correlation (Kafka Streams / ksqlDB) Receive Command Operator (REST Proxy) MES (Java) Send Live Metrics Send Command Send Production StatusRobots (C++) Receive Correlated Information

62. Why Confluent

63. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.deApache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Fully Managed Cloud ServiceSelf Managed Software FREEDOM OF CHOICE COMMITTER-DRIVEN EXPERTISE PartnersTrainingProfessional Services Enterprise Support Apache Kafka EFFICIENT OPERATIONS AT SCALE PRODUCTION- STAGE PREREQUISITES UNRESTRICTED DEVELOPER PRODUCTIVITY SQL-based Stream Processing KSQL (ksqlDB) Rich Pre-built Ecosystem Connectors | Hub | Schema Registry Multi-language Development non-Java clients | REST Proxy GUI-driven Mgmt & Monitoring Control Center Flexible DevOps Automation Operator | Ansible Dynamic Performance & Elasticity Auto Data Balancer | Tiered Storage Enterprise-grade Security RBAC | Secrets | Audit logs Data Compatibility Schema Registry | Schema Validation Global Resilience Multi-Region Clusters | Replicator Developer Operator Architect Open Source | Community licensed PARTNERSHIP FOR BUSINESS SUCCESS Complete Engagement Model Revenue / Cost / Risk Impact TCO / ROI Executive Buyer

64. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.deApache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de I N V E S T M E N T & T I M E VALUE 3 4 5 1 2 Event Streaming Maturity Model Initial Awareness / Pilot (1 Kafka Cluster) Start to Build Pipeline / Deliver 1 New Outcome (1 Kafka Cluster) Mission-Critical Deployment (Stretched, Hybrid, Multi- Region) Build Contextual Event- Driven Apps (Stretched, Hybrid, Multi-Region) Central Nervous System (Global Kafka) Product, Support, Training, Partners, Technical Account Management...

65. Apache Kafka in Manufacturing and Industry 4.0 - @KaiWaehner - www.kai-waehner.de Kai Waehner Technology Evangelist contact@kai-waehner.de @KaiWaehner www.kai-waehner.de www.confluent.io LinkedIn Questions? Feedback? Let’s connect!

Apache Kafka in Manufacturing and Industry 4.0

You are reading a preview.

Check these out next

Apache Kafka in Manufacturing and Industry 4.0

More Related Content

More from Kai Wähner (20)

Recently uploaded (20)