SlideShare a Scribd company logo
1 of 25
Download to read offline
Machine Learning Software Engineering
Patterns and Their Engineering
Hironori Washizaki
Waseda University, Professor
National Institute of Informatics, Visiting Processor
IEEE Computer Society, 2025 President
ISO/IEC/JTC1/SC7/WG20 Convenor
IPSJ-SIGSE Chair
http://www.washi.cs.waseda.ac.jp/
RAIE 2024 Keynote
Lisbon, April 16th, 2024
Agenda
• Machine Learning Software Engineering Patterns
• AI/ML Pattern Engineering
• Future Prospectives
2
Street Cafe
Problem: Needs to have a place where
people can sit lazily, legitimately, be on
view, and watch the world go by…
Solution: Encourage local cafes to spring
up in each neighborhood. Make them
intimate places, with several rooms, open
to a busy path … Alexander, Christopher, et al. A Pattern Language. Oxford University Press, 1977.
3
https://foreignpolicy.com/2021/09/12/portugals-sardine-capitalism-is-a-model-for-the-world/ https://tokyocheapo.com/food-and-drink/drinking/tokyo-terrace-tippling/
Example case of ML-based system design
• We wish to identify the type of
instrument for the sound picked up
by the phone and achieve recording
and response according to the type.
• However, the memory and
performance of the phone is
limited, and a large deep learning
model is unlikely to be loaded.
How can we do this?
4
Pretrained
Model
• Let's use Two-stage predictions where a
small model on the phone determines if a
sound is a musical instrument, and a large
model on the cloud classifies the type of
sound only if it is a musical instrument.
• For the large model, we will adopt Transfer
Learning to achieve precise classification.
Machine Learning Design Patterns (V. Lakshmanan, et al. 2020)
Two-stage predictions
• Problem: There is a need to maintain the
performance of models that are large and
complex in nature, even when deployed
to edge or distributed devices.
• Solution: The utilization flow is divided
into two phases, with only the simple
phase performed on the edge.
Transfer Learning
• Problem: There is a lack of large data sets
needed to train complex machine learning
models.
• Solution: Some layers of the trained
model are taken out and the weights are
frozen and used in the new model to solve
similar problems without being trained.
AI/ML software engineering needs patterns!
• Bridge between abstract paradigms and concrete
cases/tools
– Documenting Know-Why, Know-What and Know-How
– Reusing solutions and problems
– Getting consistent architecture
• Common language among stakeholders
– Software engineers, data scientist, domain experts,
network engineers, …
5
Paradigm
Case Tool
FW
Instruction
?
?
AI/ML software engineering patterns
• Architecture and design patterns
– Software Engineering Patterns for ML
applications [SEP4MLA]
– Machine Learning Design Patterns
[MLDP]
• Assurance argument patterns
– Safety Case Pattern for ML systems
[Safety]
– Security Argument Patterns for DNN
[Security][Security2]
• Responsible AI patterns
– Responsible AI System Design Patterns
[Responsible]
• Development and management
practices
– Lifecycle phase practices [Practice1]
– Issues and development practices
[Practice2]
• Prompt engineering patterns
– Prompt Pattern Catalog and
Taxonomy [Prompt][Prompt2]
6
[MLDP] V. Lakshmanan, et al., “Machine Learning Design Patterns,” O’Reilly, 2020
[SEP4MLA] H. Washizaki, et al. “Software Engineering Design Patterns for Machine Learning Applications,” IEEE Computer 55(3) 2022
[Safety] E. Wozniak, et al., “A Safety Case Pattern for Systems with Machine Learning Components,” SAFECOMP 2020 Workshop
[Security] M. Zeroual, et al., “Security Argument patterns for Deep Neural Network Development,” PLoP 2023
[Security2] M. Mutsche, et al. “Robustness-based Security Case Verification for Deep Neural Networks,” AsianPLoP 2024
[Responsible] Q. Lu, et al., “Responsible-AI-by-Design: a Pattern Collection for Designing Responsible AI Systems,” IEEE Software, 2023
[Practice1] M. S. Rahman, et al., “Machine Learning Application Development: Practitioners’ Insights,” Software Quality Journal, 31, 2023
[Practice2] Y. Watanabe, et al., “Preliminary Literature Review of Machine Learning System Development Practices,” COMPSAC 2021 Fast Abstract
[Prompt] J, White, et al., “A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT,” arXiv 2302.11382, 2023
[Prompt2] Y. Sasaki, et al., “A Taxonomy and Review of Prompt Engineering Patterns in Software Engineering,” COMPSAC 2024
SE Patterns for ML applications [Computer’22]
• 15 patterns extracted from around 40 scholarly and gray documents
• Classified into three types: Topology, Programming, and Model Operation
7
Hironori Washizaki, Foutse Khomh, Yann-Gael Gueheneuc, Hironori Takeuchi, Naotake Natori, Takuo Doi, Satoshi Okuda,
“Software Engineering Design Patterns for Machine Learning Applications,” IEEE Computer, Vol. 55, No. 3, pp. 30-39, 2022. (Best Paper Award)
Encapsulate ML Models within Rule-based
Safeguards
• Problem: ML models are known to be
unstable and vulnerable to adversarial
attacks, noise, and data drift.
• Solution: Encapsulate functionality provided
by ML models and deal with the inherent
uncertainty in the containing system using
deterministic and verifiable rules.
Business
Logic API
Rule-based
Safeguard
Inference
(Prediction)
Encapsulated
ML model
Input
Output
Rule
Explainable Proxy Model
• Problem: A surrogate ML model
must be built to provide
explainability.
• Solution: Run the explainable
inference pipeline in parallel with
the primary inference pipeline to
monitor prediction differences.
Input
Decoy model Data lake
Proxy model
(E.g., Decision
tree) Monitoring
and
comparison
Reproduce
and
retraining
Production
model
(E.g., DNN)
Best Paper Award
Practitioners’ insights on patterns
[ICSME’20]
• 118 developers answered
• Developers were unfamiliar with
most patterns.
• Most respondents indicated
considering the use of patterns
in future designs.
• As respondents become more
organized in their approach to
design problems by reuse, the
pattern usage ratio increased.
8
Knew it Didn’t know it
0 20 40 60 80 100 120
Data Flows Up, Model Flows Down
Secure Aggregation
Deployable Canary Model
Kappa Architecture for ML
Parameter-Server Abstraction
Different Workloads in Different Computing…
Encapsulate ML models within rule-base…
ML Gateway Routing Architecture
Lambda Architecture for ML
Separation of Concerns and Modularization of…
Distinguish Business Logic from ML Models
Data Lake for ML
Discard PoC code
Microservice Architecture for ML
ML Versioning
Used it Never used it Consider using it Not consider
Hironori Washizaki, Hironori Takeuchi, Foutse Khomh, Naotake Natori, Takuo Doi, Satoshi Okuda, “Practitioners’ insights on machine-learning software
engineering design patterns: a preliminary study,” 36th IEEE International Conference on Software Maintenance and Evolution (ICSME 2020), Late Breaking Ideas
Argument-based assurance case
• Assurance case
– Structured claims, arguments, and
evidence which gives confidence that
a system will possess the particular
qualities or properties that need to be
assured
– Safety, security, resilience, …
• Goal Structuring Notation (GSN)
– Graphical notation for visualizing
arguments that assure critical
properties
• Adoption for AI application risk-
based assessment
– E.g., UK ICO AI Explainability Guide
9
Jordi Cabot, Goal Structuring Notation – a short introduction
https://modeling-languages.com/goal-structuring-notation-introduction/
ML a
10
ML assurance argument patterns
(e.g., “DNN Robustness Case Verification”)
[AsianPLoP’24]
Goal (Claim)
Solution
(Evidence)
Strategy
M. Mutsche, et al. “Robustness-based Security Case Verification for Deep Neural Networks,” AsianPLoP 2024
Prompt engineering patterns
[COMPSAC’24]
20+ patterns in around 30
papers Learning
approaches (5)
• Interaction-focused
methods (4)
• Task specific (7)
• Model optimization (2)
• Systematization (3)
11
Gradual Execution and Output
Problem: Enhancing LLMs’
complex task performance with
improved user control and
collaboration.
Solution: Decomposing tasks
using chained LLM steps for
enhanced quality, transparency,
and user satisfaction.
Y. Sasaki, H. Washizaki, J. Li, D. Sander, N. Yoshioka and Y. Fukazawa., “A Taxonomy
and Review of Prompt Engineering Patterns in Software Engineering,” COMPSAC 2024
IEEE-CS’ SWEBOK Guide V4: AI and SE incl. patterns
• Modern engineering, practice update, BOK grows and recently developed areas
Requirements
Design
Construction
Testing
Maintenance
Configuration Management
Engineering Management
Process
Models and Methods
Quality
Professional Practice
Economics
Computing Foundations
Mathematical Foundations
Engineering Foundations
Requirements
Architecture
Design
Construction
Testing
Operations
Maintenance
Configuration Management
Engineering Management
Process
Models and Methods
Quality
Security
Professional Practice
Economics
Computing Foundations
Mathematical Foundations
Engineering Foundations
V3 V4
Agile,
DevOps
AI for
SE, SE
for AI
Agenda
• Machine Learning Software Engineering Patterns
• AI/ML Pattern Engineering
• Future Prospectives
13
• Problem: …
(AI/ML) pattern engineering
• Extraction: Identifying and formulating recurring problems and solutions into
“new” patterns to have reusable patterns
• Detection: Detecting “known” patterns in software processes and products to
comprehend and identify further improvement opportunities
• Application: Selecting, concretizing and deploying patterns on software
processes and products to resolve particular problems
• Organization: Organizing patterns to build a system (i.e., language) of patterns
• Integration: Integrating into pattern-oriented development and management
14
• Problem: …
• Solution: ….
AI/ML pattern
Extraction Application
Similar
results
Detection
Pattern
instances
Organization
Process
Management
Integration
Pytorch vs. Keras
ML pattern detection: Example of pattern instances [APSEC’23]
15
Weitao Pan, Hironori Washizaki, Nobukazu Yoshioka, Yoshiaki Fukazawa, Foutse Khomh, Yann-Gaël Guéhéneuc,
“A Machine Learning Based Approach to Detect Machine Learning Design Patterns,” APSEC 2023 ERA
Embeddings
• Problem: High-cardinality features
where closeness relationships are
important to preserve.
• Solution: Learn to map high-
cardinality data into a lower
dimensional space in such a way
that the information relevant to
the learning problem is preserved.
ML pattern
detection by ML
16
Revised Text-CNN Model
Weitao Pan, Hironori Washizaki, Nobukazu Yoshioka, Yoshiaki Fukazawa, Foutse Khomh, Yann-Gaël Guéhéneuc,
“A Machine Learning Based Approach to Detect Machine Learning Design Patterns,” APSEC 2023 ERA
ML pattern application: Deployment of assurance argument patterns [AsianPLoP’24]
17
1. Identifying and
showing pattern
candidates
2. Showing
potential impact
and result before
deployment
3. Deploying a
selected pattern
T. Ayukawa, et al., “Machine Learning Design Pattern Application Support,” AsianPLoP 2024
Java実装
ML pattern application: Modeling (e.g., “Adversarial Example Defense”) [AsianPLoP’24]
Pattern model Application
Pattern description
Context
Model is not trained
well…
Problem:
Model is vulnerable
to adversarial
samples with
intentional noise…
Solution:
Employ adversarial
training …
OCL description
Java description
T. Ayukawa, et al., “Machine Learning Design Pattern Application Support,” AsianPLoP 2024
ML a
19
ML pattern application:
Detail specification
(e.g., “DNN Robustness Case Verification”)
[AsianPLoP’24]
M. Mutsche, et al. “Robustness-based Security Case Verification for Deep Neural Networks,” AsianPLoP 2024
Acceptable (L2 Norm = 2) Unacceptable (L2 Norm = 3)
Goal: Robustness >= 2
Agenda
• Machine Learning Software Engineering Patterns
• ML Pattern Engineering
• Future Prospectives
20
Future prospectives
• Etraction:
– More related patterns from relatively mature areas
– Maybe semi-automated pattern extraction: (NOTE) Patterns are NOT
just parameterized templates.
• Organization: Building pattern “languages”
– Communication among stakeholders
– Connecting dots to form larger languages over different areas
• Integration: AI software engineering integration
– Extraction, application, detection, …
21
IEEE CS Technology Predictions 2024
Chance of success
higher than impact on
humanity
Impact on humanity
higher than chance of
success (worth investing in)
IEEE CS Technology Prediction Team (Chair: Dejan Milojicic) https://www.computer.org/resources/2024-top-technology-predictions
• Patterns particularly from “Mature” topics
• Consideration of “Impact on Humanity”
AI architecture
patterns
IoT design patterns
Towards pattern languages across layers and properties
… OK, so, to attract
many people to our city,
Small Public Squares
should be located in the
center. At the SMALL
PUBLIC SQUARE, make
Street Cafes be
Opening to the Street
...
23
https://unsplash.com/photos/EdpbTj3Br-Y
https://unsplash.com/photos/GqurqYbj7aU
https://unsplash.com/photos/zFoRwZirFvY
AI management
practices
AI assurance
argument
patterns
Governance
Responsible
AI patterns
AI design patterns
Small Public
Square
Street
Cafe
Opening to
the Street
Metamodel for consistency
and traceability [ICEBE’23]
ML Canvas
AI Project Canvas
Assurance Case
KAOS Goal Model
STAMP/STPA
Architecture
(SysML)
ML workflow
pipeline
24
Hironori Takeuchi, Jati H. Husen, Hnin Thandar Tun, Hironori Washizaki and Nobukazu Yoshioka, “Enterprise Architecture-based Metamodel for a Holistic
Business – IT Alignment View on Machine Learning Projects,” IEEE International Conference on E-Business Engineering (ICEBE 2023), Best Paper Award
Best Paper Award
25
AI/ML software engineering patterns
• Architecture and design patterns
– Software Engineering Patterns for ML
applications [SEP4MLA]
– Machine Learning Design Patterns
[MLDP]
• Assurance argument patterns
– Safety Case Pattern for ML systems
[Safety]
– Security Argument Patterns for DNN
[Security][Security2]
• Responsible AI patterns
– Responsible AI System Design Patterns
[Responsible]
• Development and management
practices
– Lifecycle phase practices [Practice1]
– Issues and development practices
[Practice2]
• Prompt engineering patterns
– Prompt Pattern Catalog and
Taxonomy [Prompt][Prompt2]
6
[MLDP] V. Lakshmanan, et al., “Machine Learning Design Patterns,” O’Reilly, 2020
[SEP4MLA] H. Washizaki, et al. “Software Engineering Design Patterns for Machine Learning Applications,” IEEE Computer 55(3) 2022
[Safety] E. Wozniak, et al., “A Safety Case Pattern for Systems with Machine Learning Components,” SAFECOMP 2020 Workshop
[Security] M. Zeroual, et al., “Security Argument patterns for Deep Neural Network Development,” PLoP 2023
[Security2] M. Mutsche, et al. “Robustness-based Security Case Verification for Deep Neural Networks,” AsianPLoP 2024
[Responsible] Q. Lu, et al., “Responsible-AI-by-Design: a Pattern Collection for Designing Responsible AI Systems,” IEEE Software, 2023
[Practice1] M. S. Rahman, et al., “Machine Learning Application Development: Practitioners’ Insights,” Software Quality Journal, 31, 2023
[Practice2] Y. Watanabe, et al., “Preliminary Literature Review of Machine Learning System Development Practices,” COMPSAC 2021 Fast Abstract
[Prompt] J, White, et al., “A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT,” arXiv 2302.11382, 2023
[Prompt2] Y. Sasaki, et al., “A Taxonomy and Review of Prompt Engineering Patterns in Software Engineering,” COMPSAC 2024
AI architecture
patterns
IoT design patterns
Towards pattern languages across layers and properties
… OK, so, to attract
many people to our city,
Small Public Squares
should be located in the
center. At the SMALL
PUBLIC SQUARE, make
Street Cafes be
Opening to the Street
...
23
https://unsplash.com/photos/EdpbTj3Br-Y
https://unsplash.com/photos/GqurqYbj7aU
https://unsplash.com/photos/zFoRwZirFvY
AI management
practices
AI assurance
argument
patterns
Governance
Responsible
AI patterns
AI design patterns
Small Public
Square
Street
Cafe
Opening to
the Street
ML software engineering needs patterns!
• Bridge between abstract paradigms and concrete
cases/tools
– Documenting Know-Why, Know-What and Know-How
– Reusing solutions and problems
– Getting consistent architecture
• Common language among stakeholders
– Software engineers, data scientist, domain experts,
network engineers, …
5
Paradigm
Case Tool
FW
Instruction
?
?
• Problem: …
(AI/ML) pattern engineering
• Extraction: Identifying and formulating recurring problems and solutions into
“new” patterns to have reusable patterns
• Detection: Detecting “known” patterns in software processes and products to
comprehend and identify further improvement opportunities
• Application: Selecting, concretizing and deploying patterns on software
processes and products to resolve particular problems
• Organization: Organizing patterns to build a system (i.e., language) of patterns
• Integration: Integrating into pattern-oriented development and management
14
• Problem: …
• Solution: ….
AI/ML pattern
Extraction Application
Similar
results
Detection
Pattern
instances
Organization
Process
Management
Integration

More Related Content

Similar to Machine Learning Software Engineering Patterns and Their Engineering

Taming Complexity: On Studying the Application of Model-Driven Engineering to...
Taming Complexity: On Studying the Application of Model-Driven Engineering to...Taming Complexity: On Studying the Application of Model-Driven Engineering to...
Taming Complexity: On Studying the Application of Model-Driven Engineering to...
Florian Rademacher
 
SADP PPTs of all modules - Shanthi D.L.pdf
SADP PPTs of all modules - Shanthi D.L.pdfSADP PPTs of all modules - Shanthi D.L.pdf
SADP PPTs of all modules - Shanthi D.L.pdf
B.T.L.I.T
 
Systematic Architecture Design
Systematic Architecture DesignSystematic Architecture Design
Systematic Architecture Design
GESSI UPC
 

Similar to Machine Learning Software Engineering Patterns and Their Engineering (20)

Design Patterns - General Introduction
Design Patterns - General IntroductionDesign Patterns - General Introduction
Design Patterns - General Introduction
 
Taming Complexity: On Studying the Application of Model-Driven Engineering to...
Taming Complexity: On Studying the Application of Model-Driven Engineering to...Taming Complexity: On Studying the Application of Model-Driven Engineering to...
Taming Complexity: On Studying the Application of Model-Driven Engineering to...
 
Oose unit 4 ppt
Oose unit 4 pptOose unit 4 ppt
Oose unit 4 ppt
 
OOSE Unit 4 PPT.ppt
OOSE Unit 4 PPT.pptOOSE Unit 4 PPT.ppt
OOSE Unit 4 PPT.ppt
 
2016-03-02 research seminar
2016-03-02 research seminar2016-03-02 research seminar
2016-03-02 research seminar
 
Introduction to MDE
Introduction to MDEIntroduction to MDE
Introduction to MDE
 
Se research update
Se research updateSe research update
Se research update
 
Model-Driven Architecture for Cloud Applications Development, A survey
Model-Driven Architecture for Cloud Applications Development, A surveyModel-Driven Architecture for Cloud Applications Development, A survey
Model-Driven Architecture for Cloud Applications Development, A survey
 
Model-Driven Architecture for Cloud Applications Development, A survey
Model-Driven Architecture for Cloud Applications Development, A surveyModel-Driven Architecture for Cloud Applications Development, A survey
Model-Driven Architecture for Cloud Applications Development, A survey
 
Model-Driven Architecture for Cloud Applications Development, A survey
Model-Driven Architecture for Cloud Applications Development, A survey Model-Driven Architecture for Cloud Applications Development, A survey
Model-Driven Architecture for Cloud Applications Development, A survey
 
[2015/2016] Software systems engineering PRINCIPLES
[2015/2016] Software systems engineering PRINCIPLES[2015/2016] Software systems engineering PRINCIPLES
[2015/2016] Software systems engineering PRINCIPLES
 
Chapter 1(1) system development life .ppt
Chapter 1(1) system development life .pptChapter 1(1) system development life .ppt
Chapter 1(1) system development life .ppt
 
SADP PPTs of all modules - Shanthi D.L.pdf
SADP PPTs of all modules - Shanthi D.L.pdfSADP PPTs of all modules - Shanthi D.L.pdf
SADP PPTs of all modules - Shanthi D.L.pdf
 
Systematic Architecture Design
Systematic Architecture DesignSystematic Architecture Design
Systematic Architecture Design
 
Developing Digital Twins
Developing Digital TwinsDeveloping Digital Twins
Developing Digital Twins
 
Towards Design-space Exploration of Component Chains in Vehicle Software
Towards Design-space Exploration of Component Chains in Vehicle SoftwareTowards Design-space Exploration of Component Chains in Vehicle Software
Towards Design-space Exploration of Component Chains in Vehicle Software
 
BCS302- Digital Design and computer organization -VTU-2022 scheme-Expectation...
BCS302- Digital Design and computer organization -VTU-2022 scheme-Expectation...BCS302- Digital Design and computer organization -VTU-2022 scheme-Expectation...
BCS302- Digital Design and computer organization -VTU-2022 scheme-Expectation...
 
H1803044651
H1803044651H1803044651
H1803044651
 
UNIT 01 SMD.pptx
UNIT 01 SMD.pptxUNIT 01 SMD.pptx
UNIT 01 SMD.pptx
 
V5 i3201613
V5 i3201613V5 i3201613
V5 i3201613
 

More from Hironori Washizaki

More from Hironori Washizaki (20)

IEEE Computer Society 2024 Technology Predictions Update
IEEE Computer Society 2024 Technology Predictions UpdateIEEE Computer Society 2024 Technology Predictions Update
IEEE Computer Society 2024 Technology Predictions Update
 
鷲崎弘宜, "国際規格ISO/IEC 24773とその意義", 情報処理学会 第86回全国大会
鷲崎弘宜, "国際規格ISO/IEC 24773とその意義", 情報処理学会 第86回全国大会鷲崎弘宜, "国際規格ISO/IEC 24773とその意義", 情報処理学会 第86回全国大会
鷲崎弘宜, "国際規格ISO/IEC 24773とその意義", 情報処理学会 第86回全国大会
 
TISO/IEC JTC1におけるソフトウェア工学知識体系、技術者認証および品質の標準化と研究・教育他への活用
TISO/IEC JTC1におけるソフトウェア工学知識体系、技術者認証および品質の標準化と研究・教育他への活用TISO/IEC JTC1におけるソフトウェア工学知識体系、技術者認証および品質の標準化と研究・教育他への活用
TISO/IEC JTC1におけるソフトウェア工学知識体系、技術者認証および品質の標準化と研究・教育他への活用
 
アジャイル品質のパターンとメトリクス Agile Quality Patterns and Metrics (QA2AQ) 20240225
アジャイル品質のパターンとメトリクス Agile Quality Patterns and Metrics (QA2AQ) 20240225アジャイル品質のパターンとメトリクス Agile Quality Patterns and Metrics (QA2AQ) 20240225
アジャイル品質のパターンとメトリクス Agile Quality Patterns and Metrics (QA2AQ) 20240225
 
Joseph Yoder : Being Agile about Architecture
Joseph Yoder : Being Agile about ArchitectureJoseph Yoder : Being Agile about Architecture
Joseph Yoder : Being Agile about Architecture
 
世界標準のソフトウェア工学知識体系SWEBOK Guide最新第4版を通じた開発アップデート
世界標準のソフトウェア工学知識体系SWEBOK Guide最新第4版を通じた開発アップデート世界標準のソフトウェア工学知識体系SWEBOK Guide最新第4版を通じた開発アップデート
世界標準のソフトウェア工学知識体系SWEBOK Guide最新第4版を通じた開発アップデート
 
デジタルトランスフォーメーション(DX)におけるソフトウェアの側面とダイバーシティ・インクルーシブに関する研究実践動向
デジタルトランスフォーメーション(DX)におけるソフトウェアの側面とダイバーシティ・インクルーシブに関する研究実践動向デジタルトランスフォーメーション(DX)におけるソフトウェアの側面とダイバーシティ・インクルーシブに関する研究実践動向
デジタルトランスフォーメーション(DX)におけるソフトウェアの側面とダイバーシティ・インクルーシブに関する研究実践動向
 
SQuBOKガイドV3概説 ~IoT・AI・DX時代のソフトウェア品質とシステム監査~
SQuBOKガイドV3概説 ~IoT・AI・DX時代のソフトウェア品質とシステム監査~SQuBOKガイドV3概説 ~IoT・AI・DX時代のソフトウェア品質とシステム監査~
SQuBOKガイドV3概説 ~IoT・AI・DX時代のソフトウェア品質とシステム監査~
 
人生100年・60年カリキュラム時代のDX人材育成: スマートエスイー 2021年度成果および2022年度募集
人生100年・60年カリキュラム時代のDX人材育成: スマートエスイー 2021年度成果および2022年度募集人生100年・60年カリキュラム時代のDX人材育成: スマートエスイー 2021年度成果および2022年度募集
人生100年・60年カリキュラム時代のDX人材育成: スマートエスイー 2021年度成果および2022年度募集
 
スマートエスイーコンソーシアムの概要と2021年度成果紹介
スマートエスイーコンソーシアムの概要と2021年度成果紹介スマートエスイーコンソーシアムの概要と2021年度成果紹介
スマートエスイーコンソーシアムの概要と2021年度成果紹介
 
DXの推進において企業内に求められる人材やデジタル人材の育て方
DXの推進において企業内に求められる人材やデジタル人材の育て方DXの推進において企業内に求められる人材やデジタル人材の育て方
DXの推進において企業内に求められる人材やデジタル人材の育て方
 
対応性のある運用のパターン
対応性のある運用のパターン対応性のある運用のパターン
対応性のある運用のパターン
 
モデル訓練のパターン
モデル訓練のパターンモデル訓練のパターン
モデル訓練のパターン
 
パターンのつながりとAI活用成熟度
パターンのつながりとAI活用成熟度パターンのつながりとAI活用成熟度
パターンのつながりとAI活用成熟度
 
データ表現のパターン
データ表現のパターンデータ表現のパターン
データ表現のパターン
 
機械学習デザインパターンの必要性と機械学習ライフサイクル
機械学習デザインパターンの必要性と機械学習ライフサイクル機械学習デザインパターンの必要性と機械学習ライフサイクル
機械学習デザインパターンの必要性と機械学習ライフサイクル
 
青山幹雄先生を偲んで(開拓、理論、実践、コミュニティ&国際)
青山幹雄先生を偲んで(開拓、理論、実践、コミュニティ&国際)青山幹雄先生を偲んで(開拓、理論、実践、コミュニティ&国際)
青山幹雄先生を偲んで(開拓、理論、実践、コミュニティ&国際)
 
機械学習デザインパターンおよび機械学習システムの品質保証の取り組み
機械学習デザインパターンおよび機械学習システムの品質保証の取り組み機械学習デザインパターンおよび機械学習システムの品質保証の取り組み
機械学習デザインパターンおよび機械学習システムの品質保証の取り組み
 
Rubric-based Assessment of Programming Thinking Skills and Comparative Evalua...
Rubric-based Assessment of Programming Thinking Skills and Comparative Evalua...Rubric-based Assessment of Programming Thinking Skills and Comparative Evalua...
Rubric-based Assessment of Programming Thinking Skills and Comparative Evalua...
 
機械学習デザインパターン Machine Learning Design Patterns
機械学習デザインパターン Machine Learning Design Patterns機械学習デザインパターン Machine Learning Design Patterns
機械学習デザインパターン Machine Learning Design Patterns
 

Recently uploaded

Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
drm1699
 

Recently uploaded (20)

Auto Affiliate AI Earns First Commission in 3 Hours..pdf
Auto Affiliate  AI Earns First Commission in 3 Hours..pdfAuto Affiliate  AI Earns First Commission in 3 Hours..pdf
Auto Affiliate AI Earns First Commission in 3 Hours..pdf
 
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
 
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
 
Software Engineering - Introduction + Process Models + Requirements Engineering
Software Engineering - Introduction + Process Models + Requirements EngineeringSoftware Engineering - Introduction + Process Models + Requirements Engineering
Software Engineering - Introduction + Process Models + Requirements Engineering
 
Community is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea GouletCommunity is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea Goulet
 
Automate your OpenSIPS config tests - OpenSIPS Summit 2024
Automate your OpenSIPS config tests - OpenSIPS Summit 2024Automate your OpenSIPS config tests - OpenSIPS Summit 2024
Automate your OpenSIPS config tests - OpenSIPS Summit 2024
 
Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024
 
Weeding your micro service landscape.pdf
Weeding your micro service landscape.pdfWeeding your micro service landscape.pdf
Weeding your micro service landscape.pdf
 
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024
 
Navigation in flutter – how to add stack, tab, and drawer navigators to your ...
Navigation in flutter – how to add stack, tab, and drawer navigators to your ...Navigation in flutter – how to add stack, tab, and drawer navigators to your ...
Navigation in flutter – how to add stack, tab, and drawer navigators to your ...
 
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
 
The mythical technical debt. (Brooke, please, forgive me)
The mythical technical debt. (Brooke, please, forgive me)The mythical technical debt. (Brooke, please, forgive me)
The mythical technical debt. (Brooke, please, forgive me)
 
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
 
Your Ultimate Web Studio for Streaming Anywhere | Evmux
Your Ultimate Web Studio for Streaming Anywhere | EvmuxYour Ultimate Web Studio for Streaming Anywhere | Evmux
Your Ultimate Web Studio for Streaming Anywhere | Evmux
 
Transformer Neural Network Use Cases with Links
Transformer Neural Network Use Cases with LinksTransformer Neural Network Use Cases with Links
Transformer Neural Network Use Cases with Links
 
From Theory to Practice: Utilizing SpiraPlan's REST API
From Theory to Practice: Utilizing SpiraPlan's REST APIFrom Theory to Practice: Utilizing SpiraPlan's REST API
From Theory to Practice: Utilizing SpiraPlan's REST API
 
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale IbridaUNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
 
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
 
Abortion Pill Prices Mthatha (@](+27832195400*)[ 🏥 Women's Abortion Clinic In...
Abortion Pill Prices Mthatha (@](+27832195400*)[ 🏥 Women's Abortion Clinic In...Abortion Pill Prices Mthatha (@](+27832195400*)[ 🏥 Women's Abortion Clinic In...
Abortion Pill Prices Mthatha (@](+27832195400*)[ 🏥 Women's Abortion Clinic In...
 
From Knowledge Graphs via Lego Bricks to scientific conversations.pptx
From Knowledge Graphs via Lego Bricks to scientific conversations.pptxFrom Knowledge Graphs via Lego Bricks to scientific conversations.pptx
From Knowledge Graphs via Lego Bricks to scientific conversations.pptx
 

Machine Learning Software Engineering Patterns and Their Engineering

  • 1. Machine Learning Software Engineering Patterns and Their Engineering Hironori Washizaki Waseda University, Professor National Institute of Informatics, Visiting Processor IEEE Computer Society, 2025 President ISO/IEC/JTC1/SC7/WG20 Convenor IPSJ-SIGSE Chair http://www.washi.cs.waseda.ac.jp/ RAIE 2024 Keynote Lisbon, April 16th, 2024
  • 2. Agenda • Machine Learning Software Engineering Patterns • AI/ML Pattern Engineering • Future Prospectives 2
  • 3. Street Cafe Problem: Needs to have a place where people can sit lazily, legitimately, be on view, and watch the world go by… Solution: Encourage local cafes to spring up in each neighborhood. Make them intimate places, with several rooms, open to a busy path … Alexander, Christopher, et al. A Pattern Language. Oxford University Press, 1977. 3 https://foreignpolicy.com/2021/09/12/portugals-sardine-capitalism-is-a-model-for-the-world/ https://tokyocheapo.com/food-and-drink/drinking/tokyo-terrace-tippling/
  • 4. Example case of ML-based system design • We wish to identify the type of instrument for the sound picked up by the phone and achieve recording and response according to the type. • However, the memory and performance of the phone is limited, and a large deep learning model is unlikely to be loaded. How can we do this? 4 Pretrained Model • Let's use Two-stage predictions where a small model on the phone determines if a sound is a musical instrument, and a large model on the cloud classifies the type of sound only if it is a musical instrument. • For the large model, we will adopt Transfer Learning to achieve precise classification. Machine Learning Design Patterns (V. Lakshmanan, et al. 2020) Two-stage predictions • Problem: There is a need to maintain the performance of models that are large and complex in nature, even when deployed to edge or distributed devices. • Solution: The utilization flow is divided into two phases, with only the simple phase performed on the edge. Transfer Learning • Problem: There is a lack of large data sets needed to train complex machine learning models. • Solution: Some layers of the trained model are taken out and the weights are frozen and used in the new model to solve similar problems without being trained.
  • 5. AI/ML software engineering needs patterns! • Bridge between abstract paradigms and concrete cases/tools – Documenting Know-Why, Know-What and Know-How – Reusing solutions and problems – Getting consistent architecture • Common language among stakeholders – Software engineers, data scientist, domain experts, network engineers, … 5 Paradigm Case Tool FW Instruction ? ?
  • 6. AI/ML software engineering patterns • Architecture and design patterns – Software Engineering Patterns for ML applications [SEP4MLA] – Machine Learning Design Patterns [MLDP] • Assurance argument patterns – Safety Case Pattern for ML systems [Safety] – Security Argument Patterns for DNN [Security][Security2] • Responsible AI patterns – Responsible AI System Design Patterns [Responsible] • Development and management practices – Lifecycle phase practices [Practice1] – Issues and development practices [Practice2] • Prompt engineering patterns – Prompt Pattern Catalog and Taxonomy [Prompt][Prompt2] 6 [MLDP] V. Lakshmanan, et al., “Machine Learning Design Patterns,” O’Reilly, 2020 [SEP4MLA] H. Washizaki, et al. “Software Engineering Design Patterns for Machine Learning Applications,” IEEE Computer 55(3) 2022 [Safety] E. Wozniak, et al., “A Safety Case Pattern for Systems with Machine Learning Components,” SAFECOMP 2020 Workshop [Security] M. Zeroual, et al., “Security Argument patterns for Deep Neural Network Development,” PLoP 2023 [Security2] M. Mutsche, et al. “Robustness-based Security Case Verification for Deep Neural Networks,” AsianPLoP 2024 [Responsible] Q. Lu, et al., “Responsible-AI-by-Design: a Pattern Collection for Designing Responsible AI Systems,” IEEE Software, 2023 [Practice1] M. S. Rahman, et al., “Machine Learning Application Development: Practitioners’ Insights,” Software Quality Journal, 31, 2023 [Practice2] Y. Watanabe, et al., “Preliminary Literature Review of Machine Learning System Development Practices,” COMPSAC 2021 Fast Abstract [Prompt] J, White, et al., “A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT,” arXiv 2302.11382, 2023 [Prompt2] Y. Sasaki, et al., “A Taxonomy and Review of Prompt Engineering Patterns in Software Engineering,” COMPSAC 2024
  • 7. SE Patterns for ML applications [Computer’22] • 15 patterns extracted from around 40 scholarly and gray documents • Classified into three types: Topology, Programming, and Model Operation 7 Hironori Washizaki, Foutse Khomh, Yann-Gael Gueheneuc, Hironori Takeuchi, Naotake Natori, Takuo Doi, Satoshi Okuda, “Software Engineering Design Patterns for Machine Learning Applications,” IEEE Computer, Vol. 55, No. 3, pp. 30-39, 2022. (Best Paper Award) Encapsulate ML Models within Rule-based Safeguards • Problem: ML models are known to be unstable and vulnerable to adversarial attacks, noise, and data drift. • Solution: Encapsulate functionality provided by ML models and deal with the inherent uncertainty in the containing system using deterministic and verifiable rules. Business Logic API Rule-based Safeguard Inference (Prediction) Encapsulated ML model Input Output Rule Explainable Proxy Model • Problem: A surrogate ML model must be built to provide explainability. • Solution: Run the explainable inference pipeline in parallel with the primary inference pipeline to monitor prediction differences. Input Decoy model Data lake Proxy model (E.g., Decision tree) Monitoring and comparison Reproduce and retraining Production model (E.g., DNN) Best Paper Award
  • 8. Practitioners’ insights on patterns [ICSME’20] • 118 developers answered • Developers were unfamiliar with most patterns. • Most respondents indicated considering the use of patterns in future designs. • As respondents become more organized in their approach to design problems by reuse, the pattern usage ratio increased. 8 Knew it Didn’t know it 0 20 40 60 80 100 120 Data Flows Up, Model Flows Down Secure Aggregation Deployable Canary Model Kappa Architecture for ML Parameter-Server Abstraction Different Workloads in Different Computing… Encapsulate ML models within rule-base… ML Gateway Routing Architecture Lambda Architecture for ML Separation of Concerns and Modularization of… Distinguish Business Logic from ML Models Data Lake for ML Discard PoC code Microservice Architecture for ML ML Versioning Used it Never used it Consider using it Not consider Hironori Washizaki, Hironori Takeuchi, Foutse Khomh, Naotake Natori, Takuo Doi, Satoshi Okuda, “Practitioners’ insights on machine-learning software engineering design patterns: a preliminary study,” 36th IEEE International Conference on Software Maintenance and Evolution (ICSME 2020), Late Breaking Ideas
  • 9. Argument-based assurance case • Assurance case – Structured claims, arguments, and evidence which gives confidence that a system will possess the particular qualities or properties that need to be assured – Safety, security, resilience, … • Goal Structuring Notation (GSN) – Graphical notation for visualizing arguments that assure critical properties • Adoption for AI application risk- based assessment – E.g., UK ICO AI Explainability Guide 9 Jordi Cabot, Goal Structuring Notation – a short introduction https://modeling-languages.com/goal-structuring-notation-introduction/
  • 10. ML a 10 ML assurance argument patterns (e.g., “DNN Robustness Case Verification”) [AsianPLoP’24] Goal (Claim) Solution (Evidence) Strategy M. Mutsche, et al. “Robustness-based Security Case Verification for Deep Neural Networks,” AsianPLoP 2024
  • 11. Prompt engineering patterns [COMPSAC’24] 20+ patterns in around 30 papers Learning approaches (5) • Interaction-focused methods (4) • Task specific (7) • Model optimization (2) • Systematization (3) 11 Gradual Execution and Output Problem: Enhancing LLMs’ complex task performance with improved user control and collaboration. Solution: Decomposing tasks using chained LLM steps for enhanced quality, transparency, and user satisfaction. Y. Sasaki, H. Washizaki, J. Li, D. Sander, N. Yoshioka and Y. Fukazawa., “A Taxonomy and Review of Prompt Engineering Patterns in Software Engineering,” COMPSAC 2024
  • 12. IEEE-CS’ SWEBOK Guide V4: AI and SE incl. patterns • Modern engineering, practice update, BOK grows and recently developed areas Requirements Design Construction Testing Maintenance Configuration Management Engineering Management Process Models and Methods Quality Professional Practice Economics Computing Foundations Mathematical Foundations Engineering Foundations Requirements Architecture Design Construction Testing Operations Maintenance Configuration Management Engineering Management Process Models and Methods Quality Security Professional Practice Economics Computing Foundations Mathematical Foundations Engineering Foundations V3 V4 Agile, DevOps AI for SE, SE for AI
  • 13. Agenda • Machine Learning Software Engineering Patterns • AI/ML Pattern Engineering • Future Prospectives 13
  • 14. • Problem: … (AI/ML) pattern engineering • Extraction: Identifying and formulating recurring problems and solutions into “new” patterns to have reusable patterns • Detection: Detecting “known” patterns in software processes and products to comprehend and identify further improvement opportunities • Application: Selecting, concretizing and deploying patterns on software processes and products to resolve particular problems • Organization: Organizing patterns to build a system (i.e., language) of patterns • Integration: Integrating into pattern-oriented development and management 14 • Problem: … • Solution: …. AI/ML pattern Extraction Application Similar results Detection Pattern instances Organization Process Management Integration
  • 15. Pytorch vs. Keras ML pattern detection: Example of pattern instances [APSEC’23] 15 Weitao Pan, Hironori Washizaki, Nobukazu Yoshioka, Yoshiaki Fukazawa, Foutse Khomh, Yann-Gaël Guéhéneuc, “A Machine Learning Based Approach to Detect Machine Learning Design Patterns,” APSEC 2023 ERA Embeddings • Problem: High-cardinality features where closeness relationships are important to preserve. • Solution: Learn to map high- cardinality data into a lower dimensional space in such a way that the information relevant to the learning problem is preserved.
  • 16. ML pattern detection by ML 16 Revised Text-CNN Model Weitao Pan, Hironori Washizaki, Nobukazu Yoshioka, Yoshiaki Fukazawa, Foutse Khomh, Yann-Gaël Guéhéneuc, “A Machine Learning Based Approach to Detect Machine Learning Design Patterns,” APSEC 2023 ERA
  • 17. ML pattern application: Deployment of assurance argument patterns [AsianPLoP’24] 17 1. Identifying and showing pattern candidates 2. Showing potential impact and result before deployment 3. Deploying a selected pattern T. Ayukawa, et al., “Machine Learning Design Pattern Application Support,” AsianPLoP 2024
  • 18. Java実装 ML pattern application: Modeling (e.g., “Adversarial Example Defense”) [AsianPLoP’24] Pattern model Application Pattern description Context Model is not trained well… Problem: Model is vulnerable to adversarial samples with intentional noise… Solution: Employ adversarial training … OCL description Java description T. Ayukawa, et al., “Machine Learning Design Pattern Application Support,” AsianPLoP 2024
  • 19. ML a 19 ML pattern application: Detail specification (e.g., “DNN Robustness Case Verification”) [AsianPLoP’24] M. Mutsche, et al. “Robustness-based Security Case Verification for Deep Neural Networks,” AsianPLoP 2024 Acceptable (L2 Norm = 2) Unacceptable (L2 Norm = 3) Goal: Robustness >= 2
  • 20. Agenda • Machine Learning Software Engineering Patterns • ML Pattern Engineering • Future Prospectives 20
  • 21. Future prospectives • Etraction: – More related patterns from relatively mature areas – Maybe semi-automated pattern extraction: (NOTE) Patterns are NOT just parameterized templates. • Organization: Building pattern “languages” – Communication among stakeholders – Connecting dots to form larger languages over different areas • Integration: AI software engineering integration – Extraction, application, detection, … 21
  • 22. IEEE CS Technology Predictions 2024 Chance of success higher than impact on humanity Impact on humanity higher than chance of success (worth investing in) IEEE CS Technology Prediction Team (Chair: Dejan Milojicic) https://www.computer.org/resources/2024-top-technology-predictions • Patterns particularly from “Mature” topics • Consideration of “Impact on Humanity”
  • 23. AI architecture patterns IoT design patterns Towards pattern languages across layers and properties … OK, so, to attract many people to our city, Small Public Squares should be located in the center. At the SMALL PUBLIC SQUARE, make Street Cafes be Opening to the Street ... 23 https://unsplash.com/photos/EdpbTj3Br-Y https://unsplash.com/photos/GqurqYbj7aU https://unsplash.com/photos/zFoRwZirFvY AI management practices AI assurance argument patterns Governance Responsible AI patterns AI design patterns Small Public Square Street Cafe Opening to the Street
  • 24. Metamodel for consistency and traceability [ICEBE’23] ML Canvas AI Project Canvas Assurance Case KAOS Goal Model STAMP/STPA Architecture (SysML) ML workflow pipeline 24 Hironori Takeuchi, Jati H. Husen, Hnin Thandar Tun, Hironori Washizaki and Nobukazu Yoshioka, “Enterprise Architecture-based Metamodel for a Holistic Business – IT Alignment View on Machine Learning Projects,” IEEE International Conference on E-Business Engineering (ICEBE 2023), Best Paper Award Best Paper Award
  • 25. 25 AI/ML software engineering patterns • Architecture and design patterns – Software Engineering Patterns for ML applications [SEP4MLA] – Machine Learning Design Patterns [MLDP] • Assurance argument patterns – Safety Case Pattern for ML systems [Safety] – Security Argument Patterns for DNN [Security][Security2] • Responsible AI patterns – Responsible AI System Design Patterns [Responsible] • Development and management practices – Lifecycle phase practices [Practice1] – Issues and development practices [Practice2] • Prompt engineering patterns – Prompt Pattern Catalog and Taxonomy [Prompt][Prompt2] 6 [MLDP] V. Lakshmanan, et al., “Machine Learning Design Patterns,” O’Reilly, 2020 [SEP4MLA] H. Washizaki, et al. “Software Engineering Design Patterns for Machine Learning Applications,” IEEE Computer 55(3) 2022 [Safety] E. Wozniak, et al., “A Safety Case Pattern for Systems with Machine Learning Components,” SAFECOMP 2020 Workshop [Security] M. Zeroual, et al., “Security Argument patterns for Deep Neural Network Development,” PLoP 2023 [Security2] M. Mutsche, et al. “Robustness-based Security Case Verification for Deep Neural Networks,” AsianPLoP 2024 [Responsible] Q. Lu, et al., “Responsible-AI-by-Design: a Pattern Collection for Designing Responsible AI Systems,” IEEE Software, 2023 [Practice1] M. S. Rahman, et al., “Machine Learning Application Development: Practitioners’ Insights,” Software Quality Journal, 31, 2023 [Practice2] Y. Watanabe, et al., “Preliminary Literature Review of Machine Learning System Development Practices,” COMPSAC 2021 Fast Abstract [Prompt] J, White, et al., “A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT,” arXiv 2302.11382, 2023 [Prompt2] Y. Sasaki, et al., “A Taxonomy and Review of Prompt Engineering Patterns in Software Engineering,” COMPSAC 2024 AI architecture patterns IoT design patterns Towards pattern languages across layers and properties … OK, so, to attract many people to our city, Small Public Squares should be located in the center. At the SMALL PUBLIC SQUARE, make Street Cafes be Opening to the Street ... 23 https://unsplash.com/photos/EdpbTj3Br-Y https://unsplash.com/photos/GqurqYbj7aU https://unsplash.com/photos/zFoRwZirFvY AI management practices AI assurance argument patterns Governance Responsible AI patterns AI design patterns Small Public Square Street Cafe Opening to the Street ML software engineering needs patterns! • Bridge between abstract paradigms and concrete cases/tools – Documenting Know-Why, Know-What and Know-How – Reusing solutions and problems – Getting consistent architecture • Common language among stakeholders – Software engineers, data scientist, domain experts, network engineers, … 5 Paradigm Case Tool FW Instruction ? ? • Problem: … (AI/ML) pattern engineering • Extraction: Identifying and formulating recurring problems and solutions into “new” patterns to have reusable patterns • Detection: Detecting “known” patterns in software processes and products to comprehend and identify further improvement opportunities • Application: Selecting, concretizing and deploying patterns on software processes and products to resolve particular problems • Organization: Organizing patterns to build a system (i.e., language) of patterns • Integration: Integrating into pattern-oriented development and management 14 • Problem: … • Solution: …. AI/ML pattern Extraction Application Similar results Detection Pattern instances Organization Process Management Integration