Complete ML space and Data Mesh Architecture slides

1. Data Analytics Platform for
Machine Learning
IoT(IIoT) Solution with AWS IoT
Narendra Ganesana

2. Key Design
Considerations
IIRA(Industrial Internet
Reference Architecture) Design
Architechure must consider
below points:
• Control domain
• Operations domain
• Information domain
• Application domain
• Business domain
ViewPoint in IIRA

3. IIoT Reference
Architechure & usecases
• Industrial Data for Predictive Quality
• Industrial Data for Asset Condition
Monitoring
• Industrial data for Predictive Maintenance
• Industrial Data for Asset Tracking TPT
Monitoring
• Industrial Data for Pricing Model

4. Sources of Machine Data
Sensors Caluclations
Predictions
Automation
Metadata Decisions
Transactions

5. KeyTerminologies
• Industry 4.0
• IoT Communication Standards

6. How to Collect the Machine data ?
• KEPServerEX
• Particle
• Samsara
• AWS IoT SiteWise,
deviceWISE and Many
more
• MQTT
• OPC-UA
• OPC-DA
HMI

7. Industrial DataCenter
Plant
Edge Gateway Servers
connectors:
• AWS Sitewise connector
• AWS greengrass&sitewise
connector
• Losant
Servers:
• Moxa
• OnLogic
• Aws Sitewise edge

8. Sample Industrial DataCenter Architechure
**Source: Siemens Edge Solution for IIOT

9. Highlevel Flow of Machine data collection

10. AWS IoT Greengrass Architechure
OPC-DA machine
OPC UA machine
Other Assets
Gateway Server
Ethernet
AWS IoT Greengrass
Industrial Datacenter
Other protocols
OPC UA protocols
OPC DA protocols
OPC UA STREAM
OWN STREAMING
PROTOCOL
Machine to cloud
data publisher
Streaming Manager
Connection Layer
OPC DA STREAM
IOT Connectivity HUB

11. IIoT Data architechure in AWS
1
2
3
4
5
6
7
8
9
10

12. AWS Lambda
Function
AWS Kinesis
IIoT Streaming Layer
AWS Kafka AWS Lambda AWS Firehose AWS S3
AWS Glue
AWS Lake
Formation
AWS Athena
AWS Redshift
AWS EMR
AWS Sagemaker
AWS QuickSight
IOT Core Rule AWS Simple
Storage Service
Producers Consumers Event Source Downstream
workflows
Machine data
Sensors
Windmills
Etc..
Event1
Event2
Event3
Etc..
Enterprise DataLake
AWS API Gateway
AWS Cognito

13. IIoT ML Layer
1. Setting up storage
2. Moving data
3. Preparing and
cataloguing data
4. Configuring
security policies
5. Making data
available for
consumption

14. Core AWS Technologies for IIoT Platform

15. Three Tier Architechure to the Functional Domains
• Edge Tier
• Platform Tier
• Enterprise Tier

16. IIOT AWS ML Architecture birds eye view

17. Data Driven Architectures
• Data Mesh vs. Data Fabric
• Data Lake vs Data Hub
• How to Choose ?
Data Mesh: API-driven [solution] for developers, unlike [data] fabric
Data Fabric: low-code, no-code, which means that the API integration is
happening inside of the fabric without actually leveraging it directly
Data hub as an architectural concept is different from data hub as a database
Finally, they are architectural frameworks, not architectures. You don’t have
architecture until the frameworks are adapted and customized to your needs, your
data, your processes, and your terminology

18. Technologies Involved and Justification
Technologies Tools
Architechure Framework for IIOT Data Mesh
Streaming IOT layer rules AWS MSK and Kinesis
Serverless Lambda
ML Engine workspace Sagemaker, EMR, AWS ECS,
Database DynamoDB, Redshift
Dashboards Amazon Quick Sight
ETL & ELT Aws Glue, EMR
Binary Management Jfrog Artifactory
Monitor Cloudwatch & Prometheus
CI/CD Pipelines Jenkins
Graphdatabases Neptune
Serverless Query AWS Athena
Infrastructure as Code Terraform
Cataloging AWS LakeFormation
Why Kafka ?
1. Supports replay, analytics, and
machine learning based use cases.
2. Reduced Cost
3. Integration with Enterprise Data
Lake
4. Optimize Machine Learning
Workloads
5. Compliance Backloading data to
other system

19. Questions ?