This summarizes my work during my first year of PhD at Institute for Manufacturing, University of Cambridge where I investigate the feasibility of deploying machine learning under uncertainty for cyber-physical manufacturing systems.

1. Deploying Machine Learning under Uncertainty in
Cyber-Physical Manufacturing System (CPMS)
Bang Xiang Yong
Supervisor: Dr. Alexandra Brintrup
Advisor: Prof Duncan McFarlane

2. • In the recent Made Smarter Review [1], the total value of AI/ML technologies to the United Kingdom industry
was estimated to be ￡198.7bn between 2017-2027.
• Classical statistic methods such as linear regression and multivariate regression were designed for data with
few dozen input variables and sample sizes which would be considered small by today’s standard. By contrast,
many ML algorithms are capable of analyzing high dimensional data [6].
• Classical statistics and ML are not mutually exclusive : classical statistics can be used to pre-process the data
before feeding into ML model such as Artificial Neural Network (ANN).
• As manufacturers begin to capture large volumes of data describing their process through Cyber Physical
Manufacturing Systems (CPMS), there is a need to derive value from the sensor data streams which are
complex, high dimensional and non-linear.
Introduction

3. • Manufacturing systems are characterised by heterogeneous distributed systems, dynamic and unpredictable
environments.
• A missing ingredient from existing approaches is the measurement of uncertainty of ML – current approaches
focus on
• maximising the accuracy of predictions
• executing them in the fastest way via scalable algorithms
• A first motivating question would be, "Can we still trust the predictions of ML, given that faults and
environment changes have occurred?".
• Without reasoning on the uncertainty of data-driven systems in dynamic environments, manufacturers are
exposed to risk in events – sensor failures, overconfident and erroneous predictions
Problem Statement

4. • Definition of Machine learning
• Data-driven approaches which build mathematical models based on sample data to make predictions or
decisions. [21]
• In my definition : an automated process of building a mathematical model using example data,
characterised by training and inference phases which relies on minimal prior knowledge of the underlying
relationships between variables in data.
• Methodologies such as CRISP-DM [24] have been developed to provide industrial standard frameworks for
encapsulating the process of data analytics, which have been adapted to various domains [25].
Research Background

6. • Definition of Cyber-Physical Manufacturing Systems:
• Rajkumar et al. [30] defined CPS as systems which bridge the cyber-world of computing and communications with the
physical world.
• Sanislav and Miclea [32] emphasised CPS through the input and feedback to and fro the cyber-physical environments,
management of distributed control, real-time executions, large spatial distribution and large scale control systems
within systems.
• My definition: Distributed and highly interconnected computer systems in manufacturing which are capable of obtaining
real-time data from the physical world through sensors and acting upon them.
Research Background

7. Research Background

8. Systematic literature review
• Systematic literature reviews are characterised by explicit
and reproducible steps with clear guidelines to reproduce.
• The methods of systematic review advocated by Pickering
and Byrne [39] were adopted.
• Modern manufacturing has focused less on studying ML methods compared to
other technologies as shown in a review by Liao et al. [41].

9. Systematic literature review
RQ1: What are the types of learning algorithms and technologies used in ML-CPMS?
RQ2: What are the types and properties of data in ML-CPMS?
RQ3: What are the characteristics and communication protocols of ML-CPMS framework?
RQ4: What are the application areas and use cases of ML-CPMS?
RQ5: What are the challenges and issues in ML-CPMS?
"What is the state of machine learning research in Cyber-Physical Manufacturing Systems?"
Main review question
Sub review questions

10. Systematic literature review
Keyword search at Scopus and Web of Science:
("cyber physical system*" OR "industrial internet") AND
("machine learning“ OR "knowledge discover*" OR "data
mine*" OR "artificial intelligence") AND "manufacturing"

11. Customised web application for systematic literature review

12. RQ1: What are the types of learning algorithms and technologies used in ML-CPMS?

13. RQ1: What are the types of learning algorithms and technologies used in ML-CPMS?

14. RQ2: What are the types and properties of data in ML-CPMS?

15. RQ3: What are the characteristics and communication protocols of ML-CPMS framework?

16. RQ4: What are the application areas and use cases of ML-CPMS?

17. RQ5: What are the challenges and issues in ML-CPMS?
There is not a uniform definition for uncertainty in ML-CPMS:
• Uncertainty of raw measurements [82]
• Economics and systems perspective [109]
• Decision-making [110]
• Events [111]
• Part quality [58]
• Model uncertainty [70, 112].

18. RQ5: What are the challenges and issues in ML-CPMS?

19. Uncertainty of machine learning in CPMS
Relevant scenarios for measuring uncertainty of ML were inferred:
• Overconfident predictions
• Concept drift
• Sensor faults and degradation
Uncertainty of ML - The level of confidence of the output given by a model.

20. Research Gaps
1. Lack of study addressing uncertainty holistically
2. Methods to quantify uncertainty of predictions.
3. Isolated study of scenarios relevant to uncertainty.
4. Acting upon quantified uncertainty.

21. Towards developing a framework for deploying machine learning in Cyber-
Physical Manufacturing Systems which quantifies uncertainty of prediction
and acts on it.
Research aim

22. Research Questions and Tasks
1. What are the characteristics of ML-CPMS and the uncertainty of ML-CPMS?
• Task 1.1 : Carry out systematic literature review to understand the status of MLCPMS
research by enumerating the types of learning algorithms, technologies,
challenges and issues which contribute to uncertainty.
• Task 1.2 : Study extant literature to identify the sources of ML uncertainty,
relevant scenarios and existing methods of dealing with these scenarios.
• Task 1.3 : Define a conceptual model of uncertainty in ML-CPMS based on Task 1.2.
2. How can a framework of ML-CPMS manage the uncertainties?
• Task 2.1 : Define components of framework and algorithms for uncertainty quantification
of ML predictions, sensors fault detection and concept drift detection.
• Task 2.2 : Identify ML algorithms which are capable of expressing uncertainty in
their predictions.
• Task 2.3 : Identify methods and algorithms which are able to reduce the uncertainty
of prediction.
• Task 2.4 : Combine outputs of Task 2.1, Task 2.2, Task 2.3 into a framework with
the ability to quantify uncertainty, detect the sources and subsequently reduce it.

23. Research Questions and Tasks
3. How effective is the proposed framework in handling uncertainty?
• Task 3.1 : Identify quantitative or qualitative performance metrics of proposed
framework.
• Task 3.2 : Develop a testbed to implement algorithms and ML-CPMS framework
on real industrial data-sets.
• Task 3.3 : Define and conduct laboratory experiments to evaluate the performance
of proposed framework.
• Task 3.4 : Deploy proposed framework in industrial case studies.
• Task 3.5 : Evaluate the outcomes of the deployment and compare with the
laboratory studies.

24. Research Scope Research Methodology

25. Relevant journals
1. International Journal of Production Research
2. Future Generation Computing Systems
3. Sensors
4. Computers in Industry
5. Journal of Manufacturing and Materials Processing
6. Journal of Machine Learning Research.
Dissemination
• Webinar was recorded to disseminate the developed agent-based system for handling uncertainty,
agentMET4FOF software package and is publicly available for viewing
• SOHOMA19 paper titled "Multi Agent System for Machine Learning Under
Uncertainty in Cyber Physical Manufacturing System"
Current publication

26. Preliminary Conceptual Framework

27. Preliminary Conceptual Framework

28. Preliminary Conceptual Framework

29. Exploratory Studies : 1. Agent-based system
“How can a software architecture for machine
learning be implemented which is modular and
flexible to investigate various use cases,
configurations, scenarios, algorithms in the
context of Cyber-Physical Manufacturing
System?”

31. Screenshot of
agentMET4FOF
web application

32. Exploratory Studies : 2. Relationship between uncertainty thresholds and accuracy of ML

33. Exploratory Studies : 2. Relationship between uncertainty thresholds and accuracy of ML
Classification of Hydraulic System condition using Bayesian Neural Network

34. Exploratory Studies : 2. Relationship between uncertainty thresholds and accuracy of ML
Classification of Hydraulic System condition using Bayesian Neural Network

35. Exploratory Studies : 2. Relationship between uncertainty thresholds and accuracy of ML
Regression for prognosis of electromechanical cylinder using Bayesian Neural Network

36. Exploratory Studies : 2. Relationship between uncertainty thresholds and accuracy of ML
Regression for quality prediction of radial forge, AFRC

37. Exploratory Studies : 2. Relationship between uncertainty thresholds and accuracy of ML
Regression for quality prediction of radial forge, AFRC

38. Exploratory study: 3. Deployment of agent-based system on embedded systems

39. Future works
1. Further develop, implement and analyse proposed architecture based on outline
performance criteria
2. Simulate behaviour of multi-agent system under conditions of uncertain scenarios
3. Analyse relationship between predictive uncertainty and uncertainty and how agents
can make decisions to manage uncertainty of ML models
Contributions
• Theoretical contribution - theoretical framework of ML-CPMS which quantifies, detects and acts
on uncertainty of prediction.
• Practical contribution - software package based on agent-based system which readily implements
the methods for quantifying, detecting and acts on uncertainty in ML-CPMS