The document discusses machine learning software engineering patterns and their significance in bridging abstract paradigms with practical solutions in AI/ML applications. It details various engineering patterns, like two-stage predictions and transfer learning, while highlighting the need for a common language among stakeholders. Future prospects include better pattern extraction, organization, and integration into development processes.

1. Machine Learning Software Engineering
Patterns and Their Engineering
Hironori Washizaki
Waseda University, Professor
National Institute of Informatics, Visiting Processor
IEEE Computer Society, 2025 President
ISO/IEC/JTC1/SC7/WG20 Convenor
IPSJ-SIGSE Chair
http://www.washi.cs.waseda.ac.jp/
RAIE 2024 Keynote
Lisbon, April 16th, 2024

2. Agenda
• Machine Learning Software Engineering Patterns
• AI/ML Pattern Engineering
• Future Prospectives
2

3. Street Cafe
Problem: Needs to have a place where
people can sit lazily, legitimately, be on
view, and watch the world go by…
Solution: Encourage local cafes to spring
up in each neighborhood. Make them
intimate places, with several rooms, open
to a busy path … Alexander, Christopher, et al. A Pattern Language. Oxford University Press, 1977.
3
https://foreignpolicy.com/2021/09/12/portugals-sardine-capitalism-is-a-model-for-the-world/ https://tokyocheapo.com/food-and-drink/drinking/tokyo-terrace-tippling/

4. Example case of ML-based system design
• We wish to identify the type of
instrument for the sound picked up
by the phone and achieve recording
and response according to the type.
• However, the memory and
performance of the phone is
limited, and a large deep learning
model is unlikely to be loaded.
How can we do this?
4
Pretrained
Model
• Let's use Two-stage predictions where a
small model on the phone determines if a
sound is a musical instrument, and a large
model on the cloud classifies the type of
sound only if it is a musical instrument.
• For the large model, we will adopt Transfer
Learning to achieve precise classification.
Machine Learning Design Patterns (V. Lakshmanan, et al. 2020)
Two-stage predictions
• Problem: There is a need to maintain the
performance of models that are large and
complex in nature, even when deployed
to edge or distributed devices.
• Solution: The utilization flow is divided
into two phases, with only the simple
phase performed on the edge.
Transfer Learning
• Problem: There is a lack of large data sets
needed to train complex machine learning
models.
• Solution: Some layers of the trained
model are taken out and the weights are
frozen and used in the new model to solve
similar problems without being trained.

5. AI/ML software engineering needs patterns!
• Bridge between abstract paradigms and concrete
cases/tools
– Documenting Know-Why, Know-What and Know-How
– Reusing solutions and problems
– Getting consistent architecture
• Common language among stakeholders
– Software engineers, data scientist, domain experts,
network engineers, …
5
Paradigm
Case Tool
FW
Instruction
？
？

6. AI/ML software engineering patterns
• Architecture and design patterns
– Software Engineering Patterns for ML
applications [SEP4MLA]
– Machine Learning Design Patterns
[MLDP]
• Assurance argument patterns
– Safety Case Pattern for ML systems
[Safety]
– Security Argument Patterns for DNN
[Security][Security2]
• Responsible AI patterns
– Responsible AI System Design Patterns
[Responsible]
• Development and management
practices
– Lifecycle phase practices [Practice1]
– Issues and development practices
[Practice2]
• Prompt engineering patterns
– Prompt Pattern Catalog and
Taxonomy [Prompt][Prompt2]
6
[MLDP] V. Lakshmanan, et al., “Machine Learning Design Patterns,” O’Reilly, 2020
[SEP4MLA] H. Washizaki, et al. “Software Engineering Design Patterns for Machine Learning Applications,” IEEE Computer 55(3) 2022
[Safety] E. Wozniak, et al., “A Safety Case Pattern for Systems with Machine Learning Components,” SAFECOMP 2020 Workshop
[Security] M. Zeroual, et al., “Security Argument patterns for Deep Neural Network Development,” PLoP 2023
[Security2] M. Mutsche, et al. “Robustness-based Security Case Verification for Deep Neural Networks,” AsianPLoP 2024
[Responsible] Q. Lu, et al., “Responsible-AI-by-Design: a Pattern Collection for Designing Responsible AI Systems,” IEEE Software, 2023
[Practice1] M. S. Rahman, et al., “Machine Learning Application Development: Practitioners’ Insights,” Software Quality Journal, 31, 2023
[Practice2] Y. Watanabe, et al., “Preliminary Literature Review of Machine Learning System Development Practices,” COMPSAC 2021 Fast Abstract
[Prompt] J, White, et al., “A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT,” arXiv 2302.11382, 2023
[Prompt2] Y. Sasaki, et al., “A Taxonomy and Review of Prompt Engineering Patterns in Software Engineering,” COMPSAC 2024

7. SE Patterns for ML applications [Computer’22]
• 15 patterns extracted from around 40 scholarly and gray documents
• Classified into three types: Topology, Programming, and Model Operation
7
Hironori Washizaki, Foutse Khomh, Yann-Gael Gueheneuc, Hironori Takeuchi, Naotake Natori, Takuo Doi, Satoshi Okuda,
“Software Engineering Design Patterns for Machine Learning Applications,” IEEE Computer, Vol. 55, No. 3, pp. 30-39, 2022. (Best Paper Award)
Encapsulate ML Models within Rule-based
Safeguards
• Problem: ML models are known to be
unstable and vulnerable to adversarial
attacks, noise, and data drift.
• Solution: Encapsulate functionality provided
by ML models and deal with the inherent
uncertainty in the containing system using
deterministic and verifiable rules.
Business
Logic API
Rule-based
Safeguard
Inference
(Prediction)
Encapsulated
ML model
Input
Output
Rule
Explainable Proxy Model
• Problem: A surrogate ML model
must be built to provide
explainability.
• Solution: Run the explainable
inference pipeline in parallel with
the primary inference pipeline to
monitor prediction differences.
Input
Decoy model Data lake
Proxy model
(E.g., Decision
tree) Monitoring
and
comparison
Reproduce
and
retraining
Production
model
(E.g., DNN)
Best Paper Award

8. Practitioners’ insights on patterns
[ICSME’20]
• 118 developers answered
• Developers were unfamiliar with
most patterns.
• Most respondents indicated
considering the use of patterns
in future designs.
• As respondents become more
organized in their approach to
design problems by reuse, the
pattern usage ratio increased.
8
Knew it Didn’t know it
0 20 40 60 80 100 120
Data Flows Up, Model Flows Down
Secure Aggregation
Deployable Canary Model
Kappa Architecture for ML
Parameter-Server Abstraction
Different Workloads in Different Computing…
Encapsulate ML models within rule-base…
ML Gateway Routing Architecture
Lambda Architecture for ML
Separation of Concerns and Modularization of…
Distinguish Business Logic from ML Models
Data Lake for ML
Discard PoC code
Microservice Architecture for ML
ML Versioning
Used it Never used it Consider using it Not consider
Hironori Washizaki, Hironori Takeuchi, Foutse Khomh, Naotake Natori, Takuo Doi, Satoshi Okuda, “Practitioners’ insights on machine-learning software
engineering design patterns: a preliminary study,” 36th IEEE International Conference on Software Maintenance and Evolution (ICSME 2020), Late Breaking Ideas

9. Argument-based assurance case
• Assurance case
– Structured claims, arguments, and
evidence which gives confidence that
a system will possess the particular
qualities or properties that need to be
assured
– Safety, security, resilience, …
• Goal Structuring Notation (GSN)
– Graphical notation for visualizing
arguments that assure critical
properties
• Adoption for AI application risk-
based assessment
– E.g., UK ICO AI Explainability Guide
9
Jordi Cabot, Goal Structuring Notation – a short introduction
https://modeling-languages.com/goal-structuring-notation-introduction/

10. ML a
10
ML assurance argument patterns
(e.g., “DNN Robustness Case Verification”)
[AsianPLoP’24]
Goal (Claim)
Solution
(Evidence)
Strategy
M. Mutsche, et al. “Robustness-based Security Case Verification for Deep Neural Networks,” AsianPLoP 2024

11. Prompt engineering patterns
[COMPSAC’24]
20+ patterns in around 30
papers Learning
approaches (5)
• Interaction-focused
methods (4)
• Task specific (7)
• Model optimization (2)
• Systematization (3)
11
Gradual Execution and Output
Problem: Enhancing LLMs’
complex task performance with
improved user control and
collaboration.
Solution: Decomposing tasks
using chained LLM steps for
enhanced quality, transparency,
and user satisfaction.
Y. Sasaki, H. Washizaki, J. Li, D. Sander, N. Yoshioka and Y. Fukazawa., “A Taxonomy
and Review of Prompt Engineering Patterns in Software Engineering,” COMPSAC 2024

12. IEEE-CS’ SWEBOK Guide V4: AI and SE incl. patterns
• Modern engineering, practice update, BOK grows and recently developed areas
Requirements
Design
Construction
Testing
Maintenance
Configuration Management
Engineering Management
Process
Models and Methods
Quality
Professional Practice
Economics
Computing Foundations
Mathematical Foundations
Engineering Foundations
Requirements
Architecture
Design
Construction
Testing
Operations
Maintenance
Configuration Management
Engineering Management
Process
Models and Methods
Quality
Security
Professional Practice
Economics
Computing Foundations
Mathematical Foundations
Engineering Foundations
V3 V4
Agile,
DevOps
AI for
SE, SE
for AI

13. Agenda
• Machine Learning Software Engineering Patterns
• AI/ML Pattern Engineering
• Future Prospectives
13

14. • Problem: …
(AI/ML) pattern engineering
• Extraction: Identifying and formulating recurring problems and solutions into
“new” patterns to have reusable patterns
• Detection: Detecting “known” patterns in software processes and products to
comprehend and identify further improvement opportunities
• Application: Selecting, concretizing and deploying patterns on software
processes and products to resolve particular problems
• Organization: Organizing patterns to build a system (i.e., language) of patterns
• Integration: Integrating into pattern-oriented development and management
14
• Problem: …
• Solution: ….
AI/ML pattern
Extraction Application
Similar
results
Detection
Pattern
instances
Organization
Process
Management
Integration

15. Pytorch vs. Keras
ML pattern detection: Example of pattern instances [APSEC’23]
15
Weitao Pan, Hironori Washizaki, Nobukazu Yoshioka, Yoshiaki Fukazawa, Foutse Khomh, Yann-Gaël Guéhéneuc,
“A Machine Learning Based Approach to Detect Machine Learning Design Patterns,” APSEC 2023 ERA
Embeddings
• Problem: High-cardinality features
where closeness relationships are
important to preserve.
• Solution: Learn to map high-
cardinality data into a lower
dimensional space in such a way
that the information relevant to
the learning problem is preserved.

16. ML pattern
detection by ML
16
Revised Text-CNN Model
Weitao Pan, Hironori Washizaki, Nobukazu Yoshioka, Yoshiaki Fukazawa, Foutse Khomh, Yann-Gaël Guéhéneuc,
“A Machine Learning Based Approach to Detect Machine Learning Design Patterns,” APSEC 2023 ERA

17. ML pattern application: Deployment of assurance argument patterns [AsianPLoP’24]
17
1. Identifying and
showing pattern
candidates
2. Showing
potential impact
and result before
deployment
3. Deploying a
selected pattern
T. Ayukawa, et al., “Machine Learning Design Pattern Application Support,” AsianPLoP 2024

18. Java実装
ML pattern application: Modeling (e.g., “Adversarial Example Defense”) [AsianPLoP’24]
Pattern model Application
Pattern description
Context
Model is not trained
well…
Problem:
Model is vulnerable
to adversarial
samples with
intentional noise…
Solution:
Employ adversarial
training …
OCL description
Java description
T. Ayukawa, et al., “Machine Learning Design Pattern Application Support,” AsianPLoP 2024

19. ML a
19
ML pattern application:
Detail specification
(e.g., “DNN Robustness Case Verification”)
[AsianPLoP’24]
M. Mutsche, et al. “Robustness-based Security Case Verification for Deep Neural Networks,” AsianPLoP 2024
Acceptable (L2 Norm = 2) Unacceptable (L2 Norm = 3)
Goal: Robustness >= 2

20. Agenda
• Machine Learning Software Engineering Patterns
• ML Pattern Engineering
• Future Prospectives
20

21. Future prospectives
• Etraction:
– More related patterns from relatively mature areas
– Maybe semi-automated pattern extraction: (NOTE) Patterns are NOT
just parameterized templates.
• Organization: Building pattern “languages”
– Communication among stakeholders
– Connecting dots to form larger languages over different areas
• Integration: AI software engineering integration
– Extraction, application, detection, …
21

22. IEEE CS Technology Predictions 2024
Chance of success
higher than impact on
humanity
Impact on humanity
higher than chance of
success (worth investing in)
IEEE CS Technology Prediction Team (Chair: Dejan Milojicic) https://www.computer.org/resources/2024-top-technology-predictions
• Patterns particularly from “Mature” topics
• Consideration of “Impact on Humanity”

23. AI architecture
patterns
IoT design patterns
Towards pattern languages across layers and properties
… OK, so, to attract
many people to our city,
Small Public Squares
should be located in the
center. At the SMALL
PUBLIC SQUARE, make
Street Cafes be
Opening to the Street
...
23
https://unsplash.com/photos/EdpbTj3Br-Y
https://unsplash.com/photos/GqurqYbj7aU
https://unsplash.com/photos/zFoRwZirFvY
AI management
practices
AI assurance
argument
patterns
Governance
Responsible
AI patterns
AI design patterns
Small Public
Square
Street
Cafe
Opening to
the Street

24. Metamodel for consistency
and traceability [ICEBE’23]
ML Canvas
AI Project Canvas
Assurance Case
KAOS Goal Model
STAMP/STPA
Architecture
(SysML)
ML workflow
pipeline
24
Hironori Takeuchi, Jati H. Husen, Hnin Thandar Tun, Hironori Washizaki and Nobukazu Yoshioka, “Enterprise Architecture-based Metamodel for a Holistic
Business – IT Alignment View on Machine Learning Projects,” IEEE International Conference on E-Business Engineering (ICEBE 2023), Best Paper Award
Best Paper Award

25. 25
AI/ML software engineering patterns
• Architecture and design patterns
– Software Engineering Patterns for ML
applications [SEP4MLA]
– Machine Learning Design Patterns
[MLDP]
• Assurance argument patterns
– Safety Case Pattern for ML systems
[Safety]
– Security Argument Patterns for DNN
[Security][Security2]
• Responsible AI patterns
– Responsible AI System Design Patterns
[Responsible]
• Development and management
practices
– Lifecycle phase practices [Practice1]
– Issues and development practices
[Practice2]
• Prompt engineering patterns
– Prompt Pattern Catalog and
Taxonomy [Prompt][Prompt2]
6
[MLDP] V. Lakshmanan, et al., “Machine Learning Design Patterns,” O’Reilly, 2020
[SEP4MLA] H. Washizaki, et al. “Software Engineering Design Patterns for Machine Learning Applications,” IEEE Computer 55(3) 2022
[Safety] E. Wozniak, et al., “A Safety Case Pattern for Systems with Machine Learning Components,” SAFECOMP 2020 Workshop
[Security] M. Zeroual, et al., “Security Argument patterns for Deep Neural Network Development,” PLoP 2023
[Security2] M. Mutsche, et al. “Robustness-based Security Case Verification for Deep Neural Networks,” AsianPLoP 2024
[Responsible] Q. Lu, et al., “Responsible-AI-by-Design: a Pattern Collection for Designing Responsible AI Systems,” IEEE Software, 2023
[Practice1] M. S. Rahman, et al., “Machine Learning Application Development: Practitioners’ Insights,” Software Quality Journal, 31, 2023
[Practice2] Y. Watanabe, et al., “Preliminary Literature Review of Machine Learning System Development Practices,” COMPSAC 2021 Fast Abstract
[Prompt] J, White, et al., “A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT,” arXiv 2302.11382, 2023
[Prompt2] Y. Sasaki, et al., “A Taxonomy and Review of Prompt Engineering Patterns in Software Engineering,” COMPSAC 2024
AI architecture
patterns
IoT design patterns
Towards pattern languages across layers and properties
… OK, so, to attract
many people to our city,
Small Public Squares
should be located in the
center. At the SMALL
PUBLIC SQUARE, make
Street Cafes be
Opening to the Street
...
23
https://unsplash.com/photos/EdpbTj3Br-Y
https://unsplash.com/photos/GqurqYbj7aU
https://unsplash.com/photos/zFoRwZirFvY
AI management
practices
AI assurance
argument
patterns
Governance
Responsible
AI patterns
AI design patterns
Small Public
Square
Street
Cafe
Opening to
the Street
ML software engineering needs patterns!
• Bridge between abstract paradigms and concrete
cases/tools
– Documenting Know-Why, Know-What and Know-How
– Reusing solutions and problems
– Getting consistent architecture
• Common language among stakeholders
– Software engineers, data scientist, domain experts,
network engineers, …
5
Paradigm
Case Tool
FW
Instruction
？
？
• Problem: …
(AI/ML) pattern engineering
• Extraction: Identifying and formulating recurring problems and solutions into
“new” patterns to have reusable patterns
• Detection: Detecting “known” patterns in software processes and products to
comprehend and identify further improvement opportunities
• Application: Selecting, concretizing and deploying patterns on software
processes and products to resolve particular problems
• Organization: Organizing patterns to build a system (i.e., language) of patterns
• Integration: Integrating into pattern-oriented development and management
14
• Problem: …
• Solution: ….
AI/ML pattern
Extraction Application
Similar
results
Detection
Pattern
instances
Organization
Process
Management
Integration