Digital transformation (DX) is spreading across all industries. Artificial Intelligence (AI), especially machine learning, is inevitable for effective use of the data collected and stored in DX, and systems that utilize machine learning have been developed in various industries and companies. The development of machine learning application systems (MLASs) presents different difficulties from traditional IT system development. Therefore, software engineering (especially project management) for MLAS has become one of the most important issues in these days. In this paper, we have classified the difficulties of MLAS development based on various documents and interviews, and created a difficulty map consisting of 12 categories. This difficulty map has some unique features that include the introduction of the relationship between the difficulties and the dual MLAS development process (implementation process and exploitation process). In this paper, we then propose a method of expressing and sharing these difficulties among stakeholders, based on MLAS Project FMEA (Failure Mode Effect Analysis). The proposed method was evaluated using two illustrative MLAS examples.

1. PICMET2022@Portland
Naoshi Uchihira
Japan Advanced Institute of Science and Technology
Project FMEA for Recognizing
Difficulties in Machine Learning
Application System Development
JAIST
TD-05 Artificial Intelligence-2
Tuesday, 8/9/2022, 14:00 - 15:30 Room: Morrison

2. 2
Agenda
1. Background: AI Project Management
2. Literature Review of Risk Management Methods
3. New Project Risk Management Method for AI Project
4. Evaluation

3. 3
Software Engineering for AI Applications is Required
Google Scholar "Software Engineering for Machine Learning"
0
50
100
150
200
250
2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
Number of Papers

4. 4
Software Engineering for MLAS Development
Efficient data collection,
maintenance, and preprocessing
methods
MLAS: Machine Learning Application System
Requirements analysis for
MLASs
Project management for MLAS development
Programming languages,
frameworks, and environments for
efficient MLAS development
Architectures used for
designing MLASs
Testing and verification,
debugging, and monitoring
methods for MLASs
Platforms, infrastructure, and
hardware to support MLASs

5. 5
Objectives of this Paper
Since Project management of ML application systems
(MLASs) development has different difficulties from
traditional IT system, objectives of this paper are
1. To collect and classify the difficulties of MLAS
development as a difficulty map.
2. To propose project risk management method for
MLAS (MLAS Project FMEA) for sharing these
difficulties among stakeholders.
3. To evaluate the proposed method by illustrative
examples.

6. 6
Agenda
1. Background: AI Project Management
2. Literature Review of Risk Management Methods
3. New Project Risk Management Method for AI Project
4. Evaluation

7. 7
Literature Review: (A) Risk Analysis
• Various methods have been proposed to analyze the risk of
products, services, and systems.
– FTA: Fault Tree Analysis
– FMEA: Failure Mode Effect Analysis
– STAMP/STPA: Systems-Theoretic Accident Model and
Processes/System-Theoretic Process Analysis
– FRAM: Functional Resonance Analysis Method
• FMEA is a method for extracting failure mode components and
analyzing their effects. FMEA has been developed and used
mainly for products and services.
• This paper focuses on risk analysis for project management.

8. 8
Literature Review: (B) Project Risk Management
•Project risk management is one of
the 10 knowledge areas in PMBOK.
•Royer (2001) described project risk
management and listed nine types of
project management risks.
•Kwan and Leung (2011) categorized
risk management practices into five categories.
•The risk management process has been well
investigated and standardized as PMBOK,
especially for IT system development.

9. 9
Literature Review: (C) Tools for Project Risk Management
• Carbone and Tippett (2004) proposed Project Risk FMEA (RFMEA).
• Bahrami et al. (2012) applied FMEA to a civil engineering project,
which is a customized version of FMEA risk assessment
• Santos et al. (2008) proposed a 10-step project risk management
methodology based on FMEA.
• Ahmadi et al. (2017) applied FMEA for highway projects.
• Taghipour et al. (2020) applied FMEA for IT projects.
• Chao and Ishii (2007) proposed FMEA as a product development
process.
• Although these tools and methods may be used in MLAS development
projects, additional consideration should be required due to
difficulties inherent in MLAS.

10. 10
Agenda
1. Background: AI Project Management
2. Literature Review of Risk Management Methods
3. New Project Risk Management Method for AI Project
4. Evaluation

11. 11
Proposed Method: MLAS Project FMEA
Category Difficulty Phase of Implementation Level of Exploitation
RA DP ME ST OM L1 L2 L3 L
D1:
Reliability
and Safety
D11: Gaps between training data and operational data
** * *
D12: Balancing model refinement and cost
** * *
D13: No stop condition for model verification
** * *
D14: Integration of induction (machine learning) and
deduction (logical rules) * ** * * *
D15: Dealing with malicious users
** * *
D2:
Efficiency
and
Productivity
D21: Appropriate selection of models and algorithms
** * *
D22: Data collection and tagging cost and time
reduction ** * * *
D23: Model reuse and versioning
** ** * *
D3: Process
Managemen
t
D31: Moving goals
** * * *
D32: Unpredictable model accuracy
** * * *
D33: Unpredictable implementation cost.
** * * * *
D34: Difficulties in collecting data from the field
** **
D35: Appropriate model maintenance during
operation **
*
*
D4:
Relationship
between
Humans and
AI
D41: Refusal of the field/customer to accept the
results of AI * ** *
D42: Diminished human ability due to relying too
much on AI. ** *
Category
Failure Mode
(Expected
Difficulty)
Implementation
Phase
Exploitation
Level
Effect Cause Possible
Countermeasure
MLAS Project FMEA
MLAS Difficulty Map
Forced Association
Dual Development
Process in MLAS
Difficulties of MLAS
Development

12. 12
Classification of Difficulties of MLAS Development
• D1: Reliability and safety
• D2: Efficiency and productivity
• D3: Process management
• D4: Relationship between humans and AI
• Business
• Standards and guidelines
• AI awareness
• AI human resource development
• Data and model distribution
• Policy and social system
• Security and privacy
• Legal systems and regulations
We developed a map
of challenges and
difficulties of MLAS
development based on:
 Japanese founding
agency reports:
NEDO, IPA.
 Interviews:
Toshiba, Panasonic,
Aisin, Denso, and
Subaru.

13. 13
Classification of Difficulties of MLAS Development
Category Explanation Typical Difficulties
Reliability
and Safety
Difficulties in quality assurance of
MLAS (reliability, safety, and
performance).
Gaps between training data and operational data; Balancing model refinement
and cost; No stop condition for model verification ; Integration of induction
(machine learning) and deduction (logical rules); Dealing with malicious users.
Efficiency
and
Productivity
Difficulties related to the
efficiency and productivity of
MLAS development (cost and
time)
Appropriate selection of models and algorithms; Tagging cost reduction; Model
reuse and versioning.
Process
Management
Difficulties in process
management of MLAS
implementation and operation.
Moving goals; Unpredictable model accuracy; Unpredictable implementation
cost, Difficulties in collecting data from the field; Appropriate model
maintenance during operation.
Relationship
between
Humans and
AI
Difficulties arising from the
immaturity of the human-AI
relationship
Refusal of the field/customer to accept the results of AI; Diminished human
ability due to relying too much on AI.
Business and
Monetizing
Difficulties related to investment
and return in MLAS
The effect of MLAS cannot be predicted in advance; Limitation of Proof of
Concept (PoC); Difficulty in monetizing.
Standards and
Guidelines
Difficulties caused by the lack of
standards and guidelines
recognized in society and industry
Necessity of safety standards and quality assurance guidelines; Liability and
indemnification in case of failure and accident.
AI Awareness Difficulties caused by
stakeholders' misperceptions of AI
Phenomenon tossed around in the AI boom (the introduction of AI has become
a goal); Lack of understanding of the limitations of AI.
AI Human
Resource
Development
Difficulties related to the talent
shortage capable of leveraging AI
Shortage of executives, users, managers and system developers who can
properly use AI; AI human resources education.
Data and
Model
Distribution
Difficulties related to the
distribution and protection of data
and models
Ownership of models and data; Distribution and protection of data and models;
Elimination of data monopoly.
Policy and
Social System
Difficulties due to the immaturity
of the policy and social system for
AI
National AI research support system; Tax system for AI industries.
Security and
Privacy,
Difficulties in ensuring security
and protecting privacy
Hacking; Anti-attack strategies; Personal information and confidential
information protection.
Legal
Systems and
Regulations
Difficulties due to the immaturity
of the legal and regulatory system
Protection of AI users' rights; Gaps between current law and latest AI
applications.
D1
D2
D3
D4

14. 14
Dual Development Process in MLAS
•There are two types of development processes:
–Implementation process
–Exploitation process.
RA DP ME ST OM
L1
L2
L3
L4

15. 15
MLAS Implementation Process
RA: Requirement definition and architecture design
DP: Data collection and pre-processing
ME: Model building and evaluation
ST: System integration and testing
OM: Operation and monitoring
RA DP ME ST OM

16. 16
MLAS Exploitation Process
•L1: Visualization system for human decisions
•L2: MLAS supporting human decisions: Human > MLAS
•L3: MLAS decisions checked by humans: Human < MLAS
•L4: Autonomous MLAS without human checking
L1
L2
L3
L4
Human > MLAS
Visualization
Autonomous
Human < MLAS

17. 17
MLAS Exploitation Process: Example
Level 0
No Automation
Level 1
Driver Assistance
Level 2
Partial Automation
Level 3
Conditional Automation
Level 4
High Automation
Level 5
Full Automation
•L1: Visualization system
for human decisions
•L2: MLAS supporting
human decisions
•L3: MLAS decisions
checked by humans
•L4: Autonomous MLAS
without human checking
Automated driving system

18. 18
Implementation Process
Exploitation
Process
R
A
D
P
M
E
S
T
O
M
L2
L3
L4
L1
R
A
D
P
M
E
S
T
O
M
R
A
D
P
M
E
S
T
O
M
R
A
D
P
M
E
S
T
O
M Goal
Sequential Implementation Processes where L4 is a Final Goal
Example 1: Dual Development Process

19. 19
Concurrent Implementation Processes where L3 is a Final Goal
Example 2: Dual Development Process
RA DP ME ST OM
L2
L3
L1
RA DP ME ST OM
RA DP ME ST OM Goal
Implementation Process
Exploitation
Process

20. 20
Relationship between Difficulties and Dual Development Process
•It is possible to logically identify the relationship between
the difficulties and the dual MLAS development.
•For example, the difficulty “D11: gaps between training
data and operational data” occurs only in OM (Operation
and Monitoring) of L2, L3, and L4. Because data is used
only for visualization in L1, D11 does not occur in L1.
Category Difficulty Phase of Implementation Level of Exploitation
RA DP ME ST OM L1 L2 L3 L4
D1: Reliability
and Safety
D11: Gaps between
training data and
operational data
** * * *
** Strongly Related, *Weekly Related

21. 21
Relationship between Difficulties and Dual Development Process
Category Difficulty Phase of Implementation Level of Exploitation
RA DP ME ST OM L1 L2 L3 L
D1:
Reliability
and Safety
D11: Gaps between training data and operational data
** * *
D12: Balancing model refinement and cost
** * *
D13: No stop condition for model verification
** * *
D14: Integration of induction (machine learning) and
deduction (logical rules) * ** * * *
D15: Dealing with malicious users
** * *
D2:
Efficiency
and
Productivity
D21: Appropriate selection of models and algorithms
** * *
D22: Data collection and tagging cost and time
reduction ** * * *
D23: Model reuse and versioning
** ** * *
D3: Process
Managemen
t
D31: Moving goals
** * * *
D32: Unpredictable model accuracy
** * * *
D33: Unpredictable implementation cost.
** * * * *
D34: Difficulties in collecting data from the field
** **
D35: Appropriate model maintenance during
operation **
*
*
D4:
Relationship
between
Humans and
AI
D41: Refusal of the field/customer to accept the
results of AI * ** *
D42: Diminished human ability due to relying too
much on AI. ** *
** Strongly Related, *Weekly Related

22. 22
Category
Failure Mode
(Expected
Difficulty)
Implementation
Phase
Exploitation
Level
Effect Cause Possible
Countermeasure
MLAS Project FMEA

23. 23
MLAS Project FMEA
This method supports managers in effectively recognizing
possible difficulties by forced association.
Example: Project X: Level 2, Phase DP
Category Difficulty Phase of Implementation Level of Exploitation
RA DP ME ST OM L1 L2 L3 L
D1:
Reliability
and Safety
D11: Gaps between training data and operational data
** * *
D12: Balancing model refinement and cost
** * *
D13: No stop condition for model verification
** * *
D14: Integration of induction (machine learning) and
deduction (logical rules) * ** * * *
D15: Dealing with malicious users
** * *
D2:
Efficiency
and
Productivity
D21: Appropriate selection of models and algorithms
** * *
D22: Data collection and tagging cost and time
reduction ** * * *
D23: Model reuse and versioning
** ** * *
D3: Process
Managemen
t
D31: Moving goals
** * * *
D32: Unpredictable model accuracy
** * * *
D33: Unpredictable implementation cost.
** * * * *
D34: Difficulties in collecting data from the field
** **
D35: Appropriate model maintenance during
operation **
*
*
D4:
Relationship
between
Humans and
AI
D41: Refusal of the field/customer to accept the
results of AI * ** *
D42: Diminished human ability due to relying too
much on AI. ** *
D22: Data collection
and Tagging cost
and time reduction
Expected
Difficulty Type
Failure Mode
Category
Failure Mode
(Expected
Difficulty)
Implementation
Phase
Exploitation
Level
Effect Cause Possible
Countermeasure
MLAS Project FMEA
MLAS Difficulty Map
Forced
Association

24. 24
Proposed Method: MLAS Project FMEA
MLAS Project FMEA
MLAS Difficulty Map
Forced Association
Dual Development
Process in MLAS
Difficulties of MLAS
Development
Category Difficulty Phase of Implementation Level of Exploitation
RA DP ME ST OM L1 L2 L3 L
D1:
Reliability
and Safety
D11: Gaps between training data and operational data
** * *
D12: Balancing model refinement and cost
** * *
D13: No stop condition for model verification
** * *
D14: Integration of induction (machine learning) and
deduction (logical rules) * ** * * *
D15: Dealing with malicious users
** * *
D2:
Efficiency
and
Productivity
D21: Appropriate selection of models and algorithms
** * *
D22: Data collection and tagging cost and time
reduction ** * * *
D23: Model reuse and versioning
** ** * *
D3: Process
Managemen
t
D31: Moving goals
** * * *
D32: Unpredictable model accuracy
** * * *
D33: Unpredictable implementation cost.
** * * * *
D34: Difficulties in collecting data from the field
** **
D35: Appropriate model maintenance during
operation **
*
*
D4:
Relationship
between
Humans and
AI
D41: Refusal of the field/customer to accept the
results of AI * ** *
D42: Diminished human ability due to relying too
much on AI. ** *
Category
Failure Mode
(Expected
Difficulty)
Implementation
Phase
Exploitation
Level
Effect Cause Possible
Countermeasure

25. 25
Agenda
1. Background: AI Project Management
2. Literature Review of Risk Management Methods
3. New Project Risk Management Method for AI Project
4. Evaluation

26. 26
Illustrative examples
•Credit Risk Management System
This system can predict the customer’s bankruptcy
risk that was calculated by machine learning model
based on the customer’s past financial data from
the financial database.
•Software Quality Management System
This system can support the project managers to
decide resources allocation by QCD prediction model
based on various data about software and developer
activities in the project.

27. 27
MLAS Project FMEA for Credit Risk Evaluation System
Category Failure Mode
(Expected Difficulty)
Implementation
Phase
Exploitation
Level
Effect Cause Possible Countermeasu
Reliability and
Safety
D11: Gaps between
training data and
operational data
OM L3 Misjudgment of credit
risk
A few specific companies
have specific financial
data．
Introduction of excepti
handling
Process
Management
D32: Unpredictable
model accuracy
when starting the
project.
RA,
&
ME
L3 Trouble between a
customer and MLAS
developer
Gaps between initial
expectation and final
accuracy.
Mutual understanding a
agreement with
stakeholders about mod
accuracy
Process
Management
D35: Appropriate
model maintenance
during operation
OM L3 The reconstructed
model degrades.
High level data scientists
moved to other projects.
Detail documentation o
model construction.
Relationship
between
Humans and AI
D41:Refusal of the
risk manager to
accept the results of
AI
OM L3 Overall performance
may be down.
Risk manager does not rely
on AI.
Increasing trust in AI w
small successful results
Relationship
between
Humans and AI
D41: Refusal of the
customer to accept
the results of AI
OM L3 Trouble between a
company and a
customer
Some customers cannot
understand and agree AI
results．
Introduction of
explainable AI technolo

28. 28
MLAS Project FMEA for Credit Risk Evaluation System
•D11 (gaps between training data and operational data)
A few specific customer companies (ex., power and
railroad companies) have specific financial data. This
system may show the wrong credit risk score for these
specific companies in the OM phase. Exception handling is
required as a countermeasure.
•D41 (refusal of the customer to accept the results of AI)
If the customer companies cannot understand the bad
score produced by AI, trouble may occur between the
customers and the company that refuse to trade due to a
bad credit risk score. Logical explaining the score to the
customer companies requires explainable AI technology.

29. 29
MLAS Project FMEA for Software Quality Management System
Category Failure Mode (Expected
Difficulty)
Implementation
Phase
Exploitation
Level
Effect Cause Possible Countermeas
Process
Management
D31: Moving goals (Users
don’t understand how to
use)
RA L1 The project cannot
start.
Final goals cannot be
shared with
stakeholders.
Starting from
visualization and
confirming the
availability of AI.
Efficiency and
Productivity
D22: Difficulties in
collecting data from the
field
DP L1 Data collection
requires time and
cost.
Definition of data is
different in
organizations
Fixing and standardizin
field data.
Reliability and
Safety
D11: Gaps between training
data and operational data
OM L2 The model cannot
be perfect.
A new project is
different from a past
project.
Categorization of the
project types.
Relationship
between
Humans and AI
D41: Refusal of the project
managers to accept the
results of AI
OM L2 The system is not
used regularly.
Gaps between their
intuition and AI
results.
AI results should be us
as means of awareness.
Process
Management
D35: Appropriate model
maintenance during
operation
OM L2 The reconstructed
model degrades.
The organization must
maintain the model by
itself.
Detail documentation o
model construction.

30. 30
MLAS Project FMEA for Software Quality Management System
•D31 (moving goals)
Managers in the field are skeptical about machine learning
and do not know how to use it in project management.
Therefore, the MLAS development project is hard to begin
because final goals cannot be shared with stakeholders. In
this situation, starting from visualization (L1) is acceptable.
•D22 (difficulties in collecting data from the field)
Data collection requires time and cost in the DP phase
because the definition of project data (ex., document
review style and cost) varies and differs by organization.
Standardization of project data is required in L1.

31. 31
Evaluation for Functional Adequacy
•The method based on forced association
functionally works in the examples.
•The two examples confirm that there are certain
patterns of risk in MLAS in different domains.
•Separating the implementation process and the
exploitation process makes it clear when
difficulties occur.
•This method may be especially effective for
companies with little experience in MLAS.

32. 32
Conclusion
•MLAS Project FMEA is proposed which can reduce the
possibility of failure of MLAS projects.
•Characteristic features
– Introduction of dual development processes
(implementation process and exploitation process)
– Classification of the difficulties in
MLAS development projects
– Linkage of the difficulties and
the dual development processes
(MLAS Difficulty Map).
•The proposed method is evaluated for functional
adequacy through two illustrative examples.
Category Difficulty Phase of Implementation Level of Exploitation
RA DP ME ST OM L1 L2 L3 L
D1:
Reliability
and Safety
D11: Gaps between training data and operational data
** * *
D12: Balancing model refinement and cost
** * *
D13: No stop condition for model verification
** * *
D14: Integration of induction (machine learning) and
deduction (logical rules) * ** * * *
D15: Dealing with malicious users
** * *
D2:
Efficiency
and
Productivity
D21: Appropriate selection of models and algorithms
** * *
D22: Data collection and tagging cost and time
reduction ** * * *
D23: Model reuse and versioning
** ** * *
D3: Process
Managemen
t
D31: Moving goals
** * * *
D32: Unpredictable model accuracy
** * * *
D33: Unpredictable implementation cost.
** * * * *
D34: Difficulties in collecting data from the field
** **
D35: Appropriate model maintenance during
operation **
*
*
D4:
Relationship
between
Humans and
AI
D41: Refusal of the field/customer to accept the
results of AI * ** *
D42: Diminished human ability due to relying too
much on AI. ** *
Category
Failure Mode
(Expected
Difficulty)
Implementation
Phase
Exploitation
Level
Effect Cause Possible
Countermeasure
MLAS Project FMEA
MLAS Difficulty Map
Forced Association
Dual Development
Process in MLAS
Difficulties of MLAS
Development

33. 33
Limitation and Future Works
•Evaluation of the proposed method was done by
the only two illustrative examples.
•In future, the proposed method should be
evaluated by actual MLAS developments and
refined based on the experiences.