Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Optimize Smart Factories Using Data Lakes and Machine Learning on AWS

372 views

Published on



近年來,物聯網(IoT)產業正以驚人的速度成長中,相關的應用與服務也陸續問市,而包含物聯網(IoT)、人工智慧(AI)與機器學習(machine learning)等科技的革新,致使從新創企業乃至傳統產業皆投入這一波新科技的發展,更添產業動能,尤其是在製造業、運輸及物流、公共安全、智慧城市等幾大面向。

據消費及商業企業服務供應商JT Group白皮書預估,到2020年全球物聯網累計裝置量(installed base)將達204.1億台。更多企業也將改變其商業模式,由產品導向轉為服務導向,客戶機構採購的不再只是硬體產品,而是尋求整套完整的解決方案。AWS 即將於3/7 (四) 舉辦『AWS AIoT未來智造高峰論壇』,持續以創新、全球視野帶領您與您的企業一起探索物聯網價值最大化的關鍵。

  • Be the first to comment

Optimize Smart Factories Using Data Lakes and Machine Learning on AWS

  1. 1. Optimize Smart Factories Using Data Lakes and Machine Learning on AWS Ivan Cheng (鄭志帆) AWS Solutions Architect
  2. 2. Smart factory vision A smart factory represents a leap forward from more traditional automation to a fully connected and flexible system—one that can use a constant stream of data from connected operations and production systems to learn and adapt to new demands 1st 2nd 3rd 4th
  3. 3. Why smart factory? Build to order Increase yield and throughput Improve quality with low overhead Minimize inventory at all stages Inventory manager Production manager Quality manager Plant manager
  4. 4. Smart factory solution components • Data lake • Data ingestion & analysis • Command & control center • Machine learning models
  5. 5. Data lake Store all structured and unstructured data, affecting manufacturing, at any scale Run different types of analytics— from visualizations to big data processing, real-time analytics, and machine learning to guide better decisions.
  6. 6. Challenges with building smart factory data lake • Disparate source systems talking different protocols • Security • Performance of closed loop systems
  7. 7. Data ingestion & analysis
  8. 8. Data sources Planning – Weekly feed from SAP Supplier – Daily Excel Manufacturing – Personalization of component. Real-time feed from machine Assembly – Adding component to print head. Real-time feed from machine Planning Supplier Manufacturing Assembly
  9. 9. IIoT Data IIoT Data ingestion & analysis • AWS Greengrass to get data out from manufacturing floor • AWS IoT Core to route the message • Real time analysis with Amazon Kinesis • Store metrics in Amazon DynamoDB AWS IoT Greengrass AWS IoT Core Kinesis Data Streams Kinesis Data Analytics Kinesis Data Streams AWS Lambda Amazon DynamoDB
  10. 10. IIoT Data IIoT Data ingestion & analysis AWS IoT Greengrass AWS IoT Core Kinesis Data Streams Kinesis Data Analytics Kinesis Data Streams AWS Lambda Amazon DynamoDB
  11. 11. AWS Greengrass: Communication with factory equipment • Use predefined protocol adapters or create new adapter • Each equipment after basic filtering writes to a specific MQTT topic • Use X.509 certificates and IAM policies for security
  12. 12. Internet Enterprise DMZ Industrial AWS Greengrass VPC VPN connection DX Machine Tool / PLC Manufacturing to AWS connectivity
  13. 13. IIoT Data IIoT Data ingestion & analysis AWS IoT Greengrass AWS IoT Core Kinesis Data Streams Kinesis Data Analytics Kinesis Data Streams AWS Lambda Amazon DynamoDB
  14. 14. Use Case Aggregate Multiple Data points to calculate in 5 minutes interval Source Time t0 t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12 t13 t14 t15
  15. 15. Kinesis Data Analytics: Aggregation CREATE OR replace pump ”output2_pump" AS INSERT INTO "destination_mfg_stream_002” SELECT machineid, sum( CASE WHEN programming_status = 'GOOD' THEN 1 ELSE 0 END) OVER sliding_window AS good_outs, sum( CASE WHEN programming_status <> 'GOOD' THEN 1 ELSE 0 END) OVER sliding_window AS bad_outs FROM "MFG_STREAM_001" window sliding_window AS ( partition BY machineid range interval ‘5’ minute PRECEDING);
  16. 16. Use Case Anomaly Detection / Find Outlier in Machine Data
  17. 17. Kinesis Data Analytics: Anomaly detection CREATE OR REPLACE PUMP "STREAM_PUMP" AS INSERT INTO "TEMP_STREAM" SELECT STREAM ”machineId", ”sensorId", ”sensorValue", "ANOMALY_SCORE" FROM TABLE(RANDOM_CUT_FOREST( CURSOR(SELECT STREAM * FROM ”MFG_STREAM_001") )); CREATE OR REPLACE PUMP "OUTPUT1_PUMP" AS INSERT INTO "DESTINATION_MFG_STREAM_001" SELECT STREAM * FROM "TEMP_STREAM" ORDER BY FLOOR("TEMP_STREAM".ROWTIME TO SECOND), ANOMALY_SCORE DESC;
  18. 18. IIoT Data IIoT Data ingestion & analysis AWS IoT Greengrass AWS IoT Core Kinesis Data Streams Kinesis Data Analytics Kinesis Data Streams AWS Lambda Amazon DynamoDB Command & control center Amazon EC2
  19. 19. Batch data ingestion & analysis • Planning & supplier data extracted from source systems and copied to Amazon S3. Manufacturing floor data via Amazon Kinesis Data Firehose. • Scheduled AWS Glue jobs to transform data • Transformed data available from Amazon Athena, Amazon Redshift, Amazon DynamoDB for dashboards and ad-hoc analysis Amazon DynamoDB Batch Data Amazon Athena S3-Transformed Data Amazon Redshift AWS Glue Amazon S3- Raw Data Manufacturing Floor Kinesis Firehose
  20. 20. Batch analysis sc = SparkContext() glueContext = GlueContext(sc) spark = glueContext.spark_session job = Job(glueContext) job.init(args['JOB_NAME'], args) # some job-specific variables compression_type = 'snappy’ source_path = 's3://smartfactory/rawdata/mfgfile01’ destination_path = 's3://smartfactory/parquet/mfgfile01’ # CSV to Parquet conversion df = spark.read.option('delimiter','|').option('header','true').csv(source_path) df.write.mode("overwrite").format('parquet').option('compression', compression_type).save(destination_path ) job.commit() CREATE EXTERNAL TABLE manufacturing_inventory ( requestBeginTime string, incomingMaterialId string, outgoingPartId string, checkinTimestamp bigint, checkoutTimestamp bigint) LOCATION 's3://smartfactory/parquet/mfgfile01’
  21. 21. Amazon QuickSight visualization Amazon QuickSight allows business users to create their own charts to embed Amazon QuickSight can read data from Amazon S3 via Athena and Amazon Redshift Amazon AthenaS3-Transformed Data Amazon Redshift Amazon QuickSight
  22. 22. Command & control center
  23. 23. Command & control center Inventory manager Production manager Plant manager End-to-end view of the smart factory. Role-based dashboards showing metrics & alerts for each role with the ability to take action. Overall production compared to plan Current & predicted Inventory at each stage. Highlight shortage. Yield at each stage. Highlight low yield.
  24. 24. Command & control center Command & control center architecture • Interactive (data driven documents) charts - JavaScript based web app • DynamoDB as the data store • Control manufacturing equipment by updating device shadow • Use Amazon SageMaker-hosted endpoints to consumer predictive data directly or from DynamoDB Amazon S3 Amazon EC2 Amazon SageMaker Amazon DynamoDB AWS IoT Core
  25. 25. Machine learning models
  26. 26. Machine learning • Developers/data scientist to explore Amazon S3 data in Amazon SageMaker • Built-in algorithms for classification, regression, deep learning to compare results Amazon S3 – transformed data Amazon SageMaker
  27. 27. Why Machine Learning at the Edge?
  28. 28. Law of physics Law of economics Law of the land
  29. 29. Inference Training Machine Learning at the Edge Local actions Edge Cloud
  30. 30. Predictive maintenance • Build models for each critical machine using its unique data and features • Use Amazon SageMaker Linear Learner for binary classification (machine is going to fail or not in a future period) or regression (compute the remaining useful life) • Performing local inference saves data transfer costs
  31. 31. Computer vision-based quality inspection • Train Amazon SageMaker image classification model in Transfer Learning mode to identify defective parts. Deploy on AWS Greengrass. • AWS Lambda running on AWS Greengrass generates an alert if defective part is identified
  32. 32. Demand forecasting and inventory management Time Series Data is traditionally used to predict future demand and plan inventory Amazon SageMaker DeepAR is a supervised learning algorithm for forecasting using recurrent neural networks (RNN)
  33. 33. Reference architecture & Implementation advice
  34. 34. Command & Control Center Smart factory data lake & ML reference architecture Amazon Athena Amazon S3- Raw Data S3-Transformed Data IIoT Data Batch Data Kinesis Data Streams AWS Glue Kinesis Data Analytics Amazon Kinesis Firehose AWS Lambda AWS IoT Core AWS Greengrass Amazon EC2 Amazon SageMaker Amazon DynamoDB Amazon Redshift Amazon QuickSight Kinesis Data Streams
  35. 35. Implementation advice • Start small – One critical product line, factory • Finalize visualization & analysis required based on roles. Work backwards to design data lake. • Decide business problem to be solved with machine learning, create models and set up training pipeline • Involve cyber security and review networking & data ingestion • Augment staff with SA/ProServ/Partner
  36. 36. Thank you.

×