This document discusses the landscape of patterns for Internet of Things (IoT) and machine learning (ML). It analyzes 33 papers on IoT patterns to classify them by abstraction level, domain specificity, and quality characteristics addressed. It also identifies common issues in ML system development by analyzing 9 papers and categorizes ML practices. Finally, it summarizes the publication trends of ML architecture and design patterns based on 10 papers and 28 gray documents.

1. Landscape of IoT and
Machine Learning Patterns
Hironori Washizaki
IoT Pattern Survey with Nobukazu Yoshioka, Atsuo
Hazeyama, Takehisa Kato, Haruhiko Kaiya, Shinpei Ogata,
Takao Okubo and Eduardo B. Fernandez
ML Pattern Survey with Yasuhiro Watanabe,
Foutse Khomh and Yann-Gael Gueheneuc
Smart SE Seminar, Tokyo, Sep 9th, 2019

2. Agenda
• Landscape of IoT patterns
• (Partial) Landscape of ML patterns
2

3. Landscape of IoT Patterns
• IoT architecture and design patterns at different
abstraction levels are not well classified and
studied.
• RQ1. What are the publication trends of IoT
patterns?
– We identified the publication years and venues of the
33 papers surveyed.
• RQ2. Are all existing IoT patterns really IoT
patterns?
– We confirmed whether or not each proposed or used
pattern the IoT context is actually a pattern
addressing specific problems and solutions in IoT.
3
H. Washizaki, et al. “Landscape of IoT Patterns,” SERP4IoT’19 https://arxiv.org/abs/1902.09718

4. IoT Pattern Classification
• Abstraction level
– High: Reference architectures
– Middle: Architecture patterns
– Low: Design patterns
• Domain specificity
– Any: General architecture/design patterns that can be
adopted to design IoT systems and software
– General IoT: Applicable to any IoT systems and software.
– Specific IoT: Addressing specific problem/technical
domains
• Quality characteristic
– All quality characteristics except for functional suitability in
ISO/IEC 25010
– + Emerging characteristics common in IoT such as
scalability and privacy
4
H. Washizaki, et al. “Landscape of IoT Patterns,” SERP4IoT’19 https://arxiv.org/abs/1902.09718

5. Systematic literature review (SLR)
• Initial Search: 63 papers 2014–2018 in Scopus
– “IoT” AND ( “design pattern” OR “architecture pattern” )
• Impurity Removal: 56
• Inclusion and Exclusion Criteria: 33
– Inclusion: Addressing patterns for designing IoT systems
and software, and papers written in English
• Data Extraction
– Publication title, publication year, publication venue
– Types of patterns proposed or used, pattern names
– Domain names in the case of Specific IoT patterns
– Quality characteristics addressed
5
Scopus
Initial
Search
63
Impurity
Removal
56
Inclusion and
Exclusion Criteria
33
TotalH. Washizaki, et al. “Landscape of IoT Patterns,” SERP4IoT’19 https://arxiv.org/abs/1902.09718

6. Publication (RQ1)
• The high numbers of conference papers and
journal papers suggest IoT architecture and
design patterns are maturing.
• Since 2016, IoT patterns have become an
important and eye-catching aspect of research,
and interest has been expanding each year.
6
Workshop Symposium Conference Book chapter Journal Total
2014 1 1
2015 1 1
2016 1 3 2 6
2017 1 1 7 3 12
2018 4 5 1 3 13
Total 5 2 17 1 8 33
H. Washizaki, et al. “Landscape of IoT Patterns,” SERP4IoT’19 https://arxiv.org/abs/1902.09718

7. IoT Patterns (RQ2)
• 136 patterns mentioned in 33 papers
• 75 general (non-IoT) patterns
– Incl. 8 non-IoT patterns appeared in multiple papers:
Publish-Subscribe, Client-Server, Peer-to-Peer, REST,
SOA, RBAC, MVC and Reflection
– 14 papers used such non-IoT patterns only. IoT
systems and software are often designed via
conventional architecture and design patterns.
• 61 IoT patterns in 19 papers
7
Any General IoT Specific IoT Total
High 20 3 2 25
Middle 14 14 15 43
Low 41 24 3 68
Total 75 41 20 136
Domain specificityAbstraction
level
H. Washizaki, et al. “Landscape of IoT Patterns,” SERP4IoT’19 https://arxiv.org/abs/1902.09718

8. E.g. Reference Architecture
8
鄭顕志,”アーキテクチャ・品質エンジニアリング”, スマートエスイー, 2018
E.g. Lambda-style architecture
H. Washizaki, et al. “Landscape of IoT Patterns,” SERP4IoT’19 https://arxiv.org/abs/1902.09718

9. IoT Pattern Classification (RQ3)
• IoT patterns are not recognized by different research groups
– Only Operator-Controller-Module mentioned in multiple papers
– Pattern authors are encouraged to carefully check existing IoT
patterns
• Combinations of abstraction level and domain specificity
– Most of IoT design patterns are applicable to any domain.
– Many IoT architecture patterns exist for specific domains. Unique
nature of IoT adoption in specific domains appears at the
architecture level.
• Quality characteristics: Performance efficiency, usability,
reliability, and maintainability are commonly addressed.
9
Any General IoT Specific IoT Total
High 20 3 2 25
Middle 14 14 15 43
Low 41 24 3 68
Total 75 41 20 136
Domain specificityAbstraction
level
H. Washizaki, et al. “Landscape of IoT Patterns,” SERP4IoT’19 https://arxiv.org/abs/1902.09718

10. Agenda
• Landscape of IoT patterns
• (Partial) Landscape of ML patterns
• Attribute-Driven Design integrating IoT/ML
patterns
10

11. Landscape of ML Patterns
• ML system architecture and design patterns at
different abstraction levels are not well classified
and studied.
• RQ1. What are common issues and practices in
ML system development?
– We identified issues and practices aligned with ML
development process.
• RQ2. What are the publication trends of ML
architecture and design patterns?
– We identified 10 papers and 28 gray documents
surveyed.
• RQ3. What kind of ML architecture and design
patterns are identified?
– (TBD!) 11

12. ML development & operation process
12
Data Cleaning
Data collection
Data labeling
Feature Engineering
Model Requirement
Model Deployment
Model Evaluation
Model monitoring
Cross cutting
Model Training
Trials and tribulations of developers of intelligent systems: A field study, VL/HCC 2016
The emerging role of data scientists on software development teams, ICSE 2016
Ux design innovation: Challenges for working with machine learning as a design material, CHI 2017
Human model evaluation in interactive supervised learning, CHI 2011
Professional Identity and Information Use: On Becoming a Machine Learning Developer, iConference 2019
A methodology to involve domain experts and machine learning techniques in the design of human-centered algorithms, HWID 2019
Silly lamp: Study of a relationship with engaging machine learning artefacts, CHI 2018
Software engineering for machine learning: A case study, ICSE 2019.
Rules of machine learning: Best practices for ml engineering, http://martin.zinkevich.org/rules of ml/rules of ml.pdf 2017

13. RQ1. Issues (excerpt)
• We identified around 25 issues in 9 papers.
• Model Requirement
– Difficult to understand ML algorithm and its potential
– Lack of Ethical Considerations
• Data Collection
– Difficult to understand data from third party
– Difficult to understand data format
• Data Cleaning
– The method to preprocess data is not mentioned
• Data Labeling
– Strain on resources for labeling
• Feature Engineering
– Difficult to measure the effect of feature
– Important features affect the result 13
JST未来社会 QAMLプロジェクト Y. Watanabe, H. Washizakiほか調査

14. RQ1. Issues (excerpt)(cont.)
• Model Training
– Ad-hoc algorithm selection based on past experience
– Difficult to reuse the model for other domains or data
formats
• Model Evaluation
– Difficult to understand the results
– Frequent revisions initiated by model changes,
parameter tuning, and data updates
• Model Deployment
– Copy pipeline and drop necessary data
– Concerned with other modules
• Cross-Cutting
– Highly dependent on individuals
– Lack of collaboration between developers and other
roles 14
JST未来社会 QAMLプロジェクト Y. Watanabe, H. Washizakiほか調査

15. RQ1. Practice categories
• We identified around 25 practices in 9 papers.
– Some of them can be categorized into the followings:
• Start with small: Practices to start with simplified issues.
• Goal-Oriented: Practices to focus on a goal of the project
• Data Concerns: Practices to deal with issues related to data
• Separation of Concerns: Practices to identify sub issues and
deal with them step-by-step
• Measure: Practices to measure uncertainty in developments
• Heuristic: Practices which rely on developers' experiences
• Traditional Practice: Practices employed even in traditional
software developments
– Requirement elicitation from users, traceability management,
refactoring, and testing
15
JST未来社会 QAMLプロジェクト Y. Watanabe, H. Washizakiほか調査

16. Practice categories aligned with phases
16
Practice
category
Model
requir
ement
Data
collec
tion
Data
cleani
ng
Data
labeling
Feat
ure
engi
neeri
ng
Model
trainin
g
Mode
l
evalu
ation
Mode
l
deplo
ymen
t
Model
monitori
ng
Start small ○ ○ ○ ○
Goal-
oriented
○ ○
Data
concern
○ ○ ○ ○ ○
Separation
of concerns
○ ○ ○
Measureme
nt
○ ○ ○
Heuristic ○ ○
Traditional
practices
○ ○ ○
JST未来社会 QAMLプロジェクト Y. Watanabe, H. Washizakiほか調査

17. RQ2. Publication trends of ML architecture and
design patterns
• 10 papers by using Engineering Village
– Developing Machine Learning Products Better and Faster at Startups
– Hidden technical debt in machine learning systems
– Applied Machine Learning at Facebook: A Datacenter Infrastructure
Perspective
– Continuous Integration of Machine Learning Models with ease.ml/ci:
Towards a Rigorous Yet Practical Treatment
– Uncertainty in machine learning applications: A practice-driven
classification of uncertainty
– Software Architecture of a Learning Apprentice System in Medical
Billing
– ClearTK 2.0: Design patterns for machine learning in UIMA
– Solution patterns for machine learning
– A Survey on Security Threats and Defensive Techniques of Machine
Learning: A Data Driven View Learning
– Machine Learning System Architectural Pattern for Improving
Operational Stability 17

18. RQ2. Publication trends of ML architecture and
design patterns (cont.)
• 28 gray documents by using Google
• Collection of patterns
– Design Patterns for Deep Learning
– Solution Patterns for Machine Learning
– Design Patterns for Machine Learning in Production
– Patterns (and Anti-Patterns) for Developing Machine Learning
Systems
– Design Patterns for Deep Learning Architectures
– Exploring Development Patterns in Data Science
• Individual pattern
– The MVC for Machine Learning: Data-Model-Learner (DML)
– Closed-Loop Intelligence
– Daisy Architecture
– Event-Driven Architecture
– Data Lake Architecture 18

19. E.g. Multi-Layer architecture pattern
19Haruki Yokoyama, Machine Learning System Architectural Pattern for Improving Operational Stability, ICSA-C, 2019
https://ieeexplore.ieee.org/document/8712157

20. Summary
• Landscape of IoT patterns
– IoT systems and software are often designed via
conventional architecture and design patterns.
– IoT patterns may not be recognized by different
research groups.
– Unique nature of IoT adoption in specific domains
appears at the architecture level.
• (Partial) Landscape of ML patterns
– We identified issues and practices aligned with ML
development process.
– We identified 10 papers and 28 gray documents
surveyed.
20