Hironori Washizaki, Software Engineering Patterns for Machine Learning Applications, 2021 IEEE International Conference on Electronic Technology, Communication and Information (ICETCI 2021), Keynote, August 28, Online, 2021.
This is a power-point presentation prepared for the students who are studying SYSTEM ENGINEERING in Fourth Semester (CBCS) of the branches of colleges affiliated to RGPV, Bhopal (M.P.). In this presentation, topics of the first unit in the syllabus are covered. I hope it will be helpful to the students.
This presentation is about a lecture I gave within the "Software systems and services" immigration course at the Gran Sasso Science Institute, L'Aquila (Italy): http://cs.gssi.infn.it/.
http://www.ivanomalavolta.com
Software engineering 18 user interface designVaibhav Khanna
System users often judge a system by its
interface rather than its functionality
λ A poorly designed interface can cause a user to
make catastrophic errors
λ Poor user interface design is the reason why so
many software systems are never used
Unified Modeling Language (UML) is a modeling language, used for design. Designed based on OMG Standard, Object this helps to express and design documents, software. This is particularly useful for OO design. Here is a brief tutorial that talks about UML usage.
Human Computer Interaction Chapter 3 HCI in the Software Process and Design ...VijiPriya Jeyamani
HCI in the Software Process:
Introduction
The software lifecycle
Usability engineering
Interactive design and prototyping
Design rationale
3.2 Design Rules
Introduction
Principles to support usability
Standards
Guidelines
Golden rules and heuristics
HCI patterns
Machine Learning Software Engineering Patterns and Their EngineeringHironori Washizaki
Hironori Washizaki, "Machine Learning Software Engineering Patterns and Their Engineering," 2nd International Workshop on Responsible AI Engineering (RAIE’24), Keynote, Lisbon, April 16th, 2024.
This is a power-point presentation prepared for the students who are studying SYSTEM ENGINEERING in Fourth Semester (CBCS) of the branches of colleges affiliated to RGPV, Bhopal (M.P.). In this presentation, topics of the first unit in the syllabus are covered. I hope it will be helpful to the students.
This presentation is about a lecture I gave within the "Software systems and services" immigration course at the Gran Sasso Science Institute, L'Aquila (Italy): http://cs.gssi.infn.it/.
http://www.ivanomalavolta.com
Software engineering 18 user interface designVaibhav Khanna
System users often judge a system by its
interface rather than its functionality
λ A poorly designed interface can cause a user to
make catastrophic errors
λ Poor user interface design is the reason why so
many software systems are never used
Unified Modeling Language (UML) is a modeling language, used for design. Designed based on OMG Standard, Object this helps to express and design documents, software. This is particularly useful for OO design. Here is a brief tutorial that talks about UML usage.
Human Computer Interaction Chapter 3 HCI in the Software Process and Design ...VijiPriya Jeyamani
HCI in the Software Process:
Introduction
The software lifecycle
Usability engineering
Interactive design and prototyping
Design rationale
3.2 Design Rules
Introduction
Principles to support usability
Standards
Guidelines
Golden rules and heuristics
HCI patterns
Machine Learning Software Engineering Patterns and Their EngineeringHironori Washizaki
Hironori Washizaki, "Machine Learning Software Engineering Patterns and Their Engineering," 2nd International Workshop on Responsible AI Engineering (RAIE’24), Keynote, Lisbon, April 16th, 2024.
Studying Software Engineering Patterns for Designing Machine Learning SystemsHironori Washizaki
Hironori Washizaki, Hiromu Uchida, Foutse Khomh and Yann-Gaël Guéhéneuc, “Studying Software Engineering Patterns for Designing Machine Learning Systems,” The 10th International Workshop on Empirical Software Engineering in Practice (IWESEP 2019), Tokyo, Japan, on December 13-14, 2019.
Patterns for New Software Engineering: Machine Learning and IoT Engineering P...Hironori Washizaki
Hironori Washizaki, "Patterns for New Software Engineering: Machine Learning and IoT Engineering Patterns", Keynote, AsianPLoP 2020: 9th Asian Conference on Pattern Languages of Programs, Sep 3rd, 2020.
2 September - 4 September, 2020
What do Practitioners Expect from the Meta-modeling Tools? A SurveyObeo
Modeling languages are defined with a meta-model, which are specified using the meta-modeling tools that produce the editors for specifying models in accordance with the meta-models. While many different meta-modeling tools have been available today, it is not yet clear what the expectations of practitioners are from the meta-modeling tools and what sort of challenges that practitioners face with. So, we designed and conducted a survey, which was responded by 103 practitioners from 24 different countries. The survey participants represent the different profiles of the population who differ in terms of the work industries, the problem domains, job positions, and years of experiences. Our survey investigates three important research questions, which essentially focus on the usage frequencies of the existing meta-modeling tools, practitioners’ expectations from the meta-modeling tools, and any challenges that practitioners face with. The survey questionnaire considers the notation, semantics, editor services, model-transformation, validation, testing, and composability requirements for meta-modeling tools.
The survey results lead to many interesting findings regarding the practical use of meta-modeling tools from different viewpoints. The survey also reveals many important challenges in each type of requirements. We strongly believe that the survey results are expected to be useful for anyone who consider developing their own DSMLs (domain-specific modeling languages) in understanding the top-used meta-modeling tools for different domains. Also, the tool vendors could use the survey results in learning the expectations of practitioners from the meta-modeling tools and any challenges encountered.
Assoc.Prof.Dr. Mert Ozkaya, Yeditepe University
Very preliminar intro to MDE for software developer communities and other kind of software practitioners. Contains material from several recognized sources.
Critiquing CS Assessment from a CS for All lens: Dagstuhl Seminar PosterMark Guzdial
Poster presented at the Dagstuhl Seminar "Assessing Learning in Introductory Computer Science" (http://www.dagstuhl.de/en/program/calendar/semhp/?semnr=16072). I argue that we have to consider what the learner wants to do and wants to be (i.e., their desired Community of Practice) when assessing learning. Different CoP, different outcomes, different assessments.
NL based Object Oriented modeling - EJSR 35(1)IT Industry
Imran Sarwar Bajwa, Shahzad Mumtaz, Ali Samad [2009], "Object Oriented Software Modeling using NLP Based Knowledge Extraction", European Journal of Scientific Research, Aug 2009, Vol. 35 No. 01, pp:22-33
Takashi Kobayashi and Hironori Washizaki, "SWEBOK Guide and Future of SE Education," First International Symposium on the Future of Software Engineering (FUSE), June 3-6, 2024, Okinawa, Japan
Rubric-based Assessment of Programming Thinking Skills and Comparative Evalua...Hironori Washizaki
Hironori Washizaki, "Rubric-based Assessment of Programming Thinking Skills and Comparative Evaluation of Introductory Programming Environments," 4th International Annual Meeting on STEM Education (IAMSTEM 2021), Keynote, August 12-14, 2021, Keelung, Taiwan and Online
Cyaniclab : Software Development Agency Portfolio.pdfCyanic lab
CyanicLab, an offshore custom software development company based in Sweden,India, Finland, is your go-to partner for startup development and innovative web design solutions. Our expert team specializes in crafting cutting-edge software tailored to meet the unique needs of startups and established enterprises alike. From conceptualization to execution, we offer comprehensive services including web and mobile app development, UI/UX design, and ongoing software maintenance. Ready to elevate your business? Contact CyanicLab today and let us propel your vision to success with our top-notch IT solutions.
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...Globus
Large Language Models (LLMs) are currently the center of attention in the tech world, particularly for their potential to advance research. In this presentation, we'll explore a straightforward and effective method for quickly initiating inference runs on supercomputers using the vLLM tool with Globus Compute, specifically on the Polaris system at ALCF. We'll begin by briefly discussing the popularity and applications of LLMs in various fields. Following this, we will introduce the vLLM tool, and explain how it integrates with Globus Compute to efficiently manage LLM operations on Polaris. Attendees will learn the practical aspects of setting up and remotely triggering LLMs from local machines, focusing on ease of use and efficiency. This talk is ideal for researchers and practitioners looking to leverage the power of LLMs in their work, offering a clear guide to harnessing supercomputing resources for quick and effective LLM inference.
top nidhi software solution freedownloadvrstrong314
This presentation emphasizes the importance of data security and legal compliance for Nidhi companies in India. It highlights how online Nidhi software solutions, like Vector Nidhi Software, offer advanced features tailored to these needs. Key aspects include encryption, access controls, and audit trails to ensure data security. The software complies with regulatory guidelines from the MCA and RBI and adheres to Nidhi Rules, 2014. With customizable, user-friendly interfaces and real-time features, these Nidhi software solutions enhance efficiency, support growth, and provide exceptional member services. The presentation concludes with contact information for further inquiries.
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...Juraj Vysvader
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I didn't get rich from it but it did have 63K downloads (powered possible tens of thousands of websites).
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...informapgpstrackings
Keep tabs on your field staff effortlessly with Informap Technology Centre LLC. Real-time tracking, task assignment, and smart features for efficient management. Request a live demo today!
For more details, visit us : https://informapuae.com/field-staff-tracking/
Large Language Models and the End of ProgrammingMatt Welsh
Talk by Matt Welsh at Craft Conference 2024 on the impact that Large Language Models will have on the future of software development. In this talk, I discuss the ways in which LLMs will impact the software industry, from replacing human software developers with AI, to replacing conventional software with models that perform reasoning, computation, and problem-solving.
Advanced Flow Concepts Every Developer Should KnowPeter Caitens
Tim Combridge from Sensible Giraffe and Salesforce Ben presents some important tips that all developers should know when dealing with Flows in Salesforce.
Understanding Globus Data Transfers with NetSageGlobus
NetSage is an open privacy-aware network measurement, analysis, and visualization service designed to help end-users visualize and reason about large data transfers. NetSage traditionally has used a combination of passive measurements, including SNMP and flow data, as well as active measurements, mainly perfSONAR, to provide longitudinal network performance data visualization. It has been deployed by dozens of networks world wide, and is supported domestically by the Engagement and Performance Operations Center (EPOC), NSF #2328479. We have recently expanded the NetSage data sources to include logs for Globus data transfers, following the same privacy-preserving approach as for Flow data. Using the logs for the Texas Advanced Computing Center (TACC) as an example, this talk will walk through several different example use cases that NetSage can answer, including: Who is using Globus to share data with my institution, and what kind of performance are they able to achieve? How many transfers has Globus supported for us? Which sites are we sharing the most data with, and how is that changing over time? How is my site using Globus to move data internally, and what kind of performance do we see for those transfers? What percentage of data transfers at my institution used Globus, and how did the overall data transfer performance compare to the Globus users?
Gamify Your Mind; The Secret Sauce to Delivering Success, Continuously Improv...Shahin Sheidaei
Games are powerful teaching tools, fostering hands-on engagement and fun. But they require careful consideration to succeed. Join me to explore factors in running and selecting games, ensuring they serve as effective teaching tools. Learn to maintain focus on learning objectives while playing, and how to measure the ROI of gaming in education. Discover strategies for pitching gaming to leadership. This session offers insights, tips, and examples for coaches, team leads, and enterprise leaders seeking to teach from simple to complex concepts.
How Recreation Management Software Can Streamline Your Operations.pptxwottaspaceseo
Recreation management software streamlines operations by automating key tasks such as scheduling, registration, and payment processing, reducing manual workload and errors. It provides centralized management of facilities, classes, and events, ensuring efficient resource allocation and facility usage. The software offers user-friendly online portals for easy access to bookings and program information, enhancing customer experience. Real-time reporting and data analytics deliver insights into attendance and preferences, aiding in strategic decision-making. Additionally, effective communication tools keep participants and staff informed with timely updates. Overall, recreation management software enhances efficiency, improves service delivery, and boosts customer satisfaction.
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...Anthony Dahanne
Les Buildpacks existent depuis plus de 10 ans ! D’abord, ils étaient utilisés pour détecter et construire une application avant de la déployer sur certains PaaS. Ensuite, nous avons pu créer des images Docker (OCI) avec leur dernière génération, les Cloud Native Buildpacks (CNCF en incubation). Sont-ils une bonne alternative au Dockerfile ? Que sont les buildpacks Paketo ? Quelles communautés les soutiennent et comment ?
Venez le découvrir lors de cette session ignite
Strategies for Successful Data Migration Tools.pptxvarshanayak241
Data migration is a complex but essential task for organizations aiming to modernize their IT infrastructure and leverage new technologies. By understanding common challenges and implementing these strategies, businesses can achieve a successful migration with minimal disruption. Data Migration Tool like Ask On Data play a pivotal role in this journey, offering features that streamline the process, ensure data integrity, and maintain security. With the right approach and tools, organizations can turn the challenge of data migration into an opportunity for growth and innovation.
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisGlobus
JASMIN is the UK’s high-performance data analysis platform for environmental science, operated by STFC on behalf of the UK Natural Environment Research Council (NERC). In addition to its role in hosting the CEDA Archive (NERC’s long-term repository for climate, atmospheric science & Earth observation data in the UK), JASMIN provides a collaborative platform to a community of around 2,000 scientists in the UK and beyond, providing nearly 400 environmental science projects with working space, compute resources and tools to facilitate their work. High-performance data transfer into and out of JASMIN has always been a key feature, with many scientists bringing model outputs from supercomputers elsewhere in the UK, to analyse against observational or other model data in the CEDA Archive. A growing number of JASMIN users are now realising the benefits of using the Globus service to provide reliable and efficient data movement and other tasks in this and other contexts. Further use cases involve long-distance (intercontinental) transfers to and from JASMIN, and collecting results from a mobile atmospheric radar system, pushing data to JASMIN via a lightweight Globus deployment. We provide details of how Globus fits into our current infrastructure, our experience of the recent migration to GCSv5.4, and of our interest in developing use of the wider ecosystem of Globus services for the benefit of our user community.
Into the Box Keynote Day 2: Unveiling amazing updates and announcements for modern CFML developers! Get ready for exciting releases and updates on Ortus tools and products. Stay tuned for cutting-edge innovations designed to boost your productivity.
Check out the webinar slides to learn more about how XfilesPro transforms Salesforce document management by leveraging its world-class applications. For more details, please connect with sales@xfilespro.com
If you want to watch the on-demand webinar, please click here: https://www.xfilespro.com/webinars/salesforce-document-management-2-0-smarter-faster-better/
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTier1 app
Even though at surface level ‘java.lang.OutOfMemoryError’ appears as one single error; underlyingly there are 9 types of OutOfMemoryError. Each type of OutOfMemoryError has different causes, diagnosis approaches and solutions. This session equips you with the knowledge, tools, and techniques needed to troubleshoot and conquer OutOfMemoryError in all its forms, ensuring smoother, more efficient Java applications.
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...Hivelance Technology
Cryptocurrency trading bots are computer programs designed to automate buying, selling, and managing cryptocurrency transactions. These bots utilize advanced algorithms and machine learning techniques to analyze market data, identify trading opportunities, and execute trades on behalf of their users. By automating the decision-making process, crypto trading bots can react to market changes faster than human traders
Hivelance, a leading provider of cryptocurrency trading bot development services, stands out as the premier choice for crypto traders and developers. Hivelance boasts a team of seasoned cryptocurrency experts and software engineers who deeply understand the crypto market and the latest trends in automated trading, Hivelance leverages the latest technologies and tools in the industry, including advanced AI and machine learning algorithms, to create highly efficient and adaptable crypto trading bots
Prosigns: Transforming Business with Tailored Technology SolutionsProsigns
Unlocking Business Potential: Tailored Technology Solutions by Prosigns
Discover how Prosigns, a leading technology solutions provider, partners with businesses to drive innovation and success. Our presentation showcases our comprehensive range of services, including custom software development, web and mobile app development, AI & ML solutions, blockchain integration, DevOps services, and Microsoft Dynamics 365 support.
Custom Software Development: Prosigns specializes in creating bespoke software solutions that cater to your unique business needs. Our team of experts works closely with you to understand your requirements and deliver tailor-made software that enhances efficiency and drives growth.
Web and Mobile App Development: From responsive websites to intuitive mobile applications, Prosigns develops cutting-edge solutions that engage users and deliver seamless experiences across devices.
AI & ML Solutions: Harnessing the power of Artificial Intelligence and Machine Learning, Prosigns provides smart solutions that automate processes, provide valuable insights, and drive informed decision-making.
Blockchain Integration: Prosigns offers comprehensive blockchain solutions, including development, integration, and consulting services, enabling businesses to leverage blockchain technology for enhanced security, transparency, and efficiency.
DevOps Services: Prosigns' DevOps services streamline development and operations processes, ensuring faster and more reliable software delivery through automation and continuous integration.
Microsoft Dynamics 365 Support: Prosigns provides comprehensive support and maintenance services for Microsoft Dynamics 365, ensuring your system is always up-to-date, secure, and running smoothly.
Learn how our collaborative approach and dedication to excellence help businesses achieve their goals and stay ahead in today's digital landscape. From concept to deployment, Prosigns is your trusted partner for transforming ideas into reality and unlocking the full potential of your business.
Join us on a journey of innovation and growth. Let's partner for success with Prosigns.
Prosigns: Transforming Business with Tailored Technology Solutions
Software Engineering Patterns for Machine Learning Applications
1. 2021 IEEE International Conference on Electronic Technology, Communication
and Information (ICETCI 2021)
Software Engineering Patterns for Machine
Learning Applications
Hironori Washizaki
Professor at Waseda University, Tokyo, Japan
washizaki@waseda.jp http://www.washi.cs.waseda.ac.jp/
Supported by JST MIRAI eAI JPMJMI20B8 https://www.jst.go.jp/mirai/jp/program/super-smart/JPMJMI20B8.html
• Hironori Washizaki, Hiromu Uchida, Foutse Khomh and Yann-Gael Gueheneuc, “Studying Software Engineering Patterns for
Designing Machine Learning Systems,” The 10th International Workshop on Empirical Software Engineering in Practice
(IWESEP 2019)
• Hironori Washizaki, Foutse Khomh, Yann-Gael Gueheneuc, “Software Engineering Patterns for Machine Learning Applications
(SEP4MLA),” 9th Asian Conference on Pattern Languages of Programs (AsianPLoP 2020)
• Hironori Washizaki, Foutse Khomh, Yann-Gaël Guéhéneuc, Hironori Takeuchi, Satoshi Okuda, Naotake Natori, Naohisa Shioura,
“Software Engineering Patterns for Machine Learning Applications (SEP4MLA) – Part 2”, the 27th Conference on Pattern
Languages of Programs in 2020 (PLoP’20)
• Hironori Washizaki, Hironori Takeuchi, Foutse Khomh, Naotake Natori, Takuo Doi, Satoshi Okuda, “Practitioners’ insights on
machine-learning software engineering design patterns: a preliminary study,” 36th IEEE International Conference on Software
Maintenance and Evolution (ICSME 2020), Late Breaking Ideas track
• Yasuhiro Watanabe, Hironori Washizaki, Kazunori Sakamoto, Daisuke Saito, Kiyoshi Honda, Naohiko Tsuda, Yoshiaki Fukazawa,
Nobukazu Yoshioka, “Preliminary Literature Review of Machine Learning System Development Practices,” 45th IEEE Computer
Society Signature Conference on Computers, Software and Applications (COMPSAC 2021), Fast Abstract
• Jati H. Husen, Hnin Thandar Tun, Nobukazu Yoshioka, Hironori Washizaki and Yoshiaki Fukazawa, “Goal-Oriented Machine
Learning-Based Component Development Process,” ACM/IEEE 24th International Conference on Model Driven Engineering
Languages and Systems (MODELS 2021), Poster
2. Prof. Dr. Hironori Washizaki
• Professor and the Associate Dean of the Research
Promotion Division at Waseda University in Tokyo
• Visiting Professor at the National Institute of Informatics
• Outside Directors of SYSTEM INFORMATION and
eXmotion
• Research and education projects
• Leading a large-scale grant at MEXT enPiT-Pro Smart
SE
• Leading framework team of JST MIRAI eAI project
• Professional contributions
• IEEE Computer Society Vice President for Professional
and Educational Activities
• Editorial Board Member of MDPI Education Sciences
• Steering Committee Member of the IEEE Conference on
Software Engineering Education and Training (CSEE&T)
• Associate Editor of IEEE Transactions on Emerging
Topics in Computing
• Advisory Committee Member of the IEEE-CS COMPSAC
• Steering Committee Member of Asia-Pacific Software
Engineering Conference (APSEC)
• Convener of ISO/IEC/JTC1 SC7/WG20
3. Agenda
• ML software engineering and patterns
• Literature review of software engineering
patterns for ML applications
• Classification of ML patterns
• Practitioners’ insights on ML patterns
3
4. 4
Street Cafe
Problem: Needs to have a place where
people can sit lazily, legitimately, be on
view, and watch the world go by…
Solution: Encourage local cafes to spring
up in each neighborhood. Make them
intimate places, with several rooms, open
to a busy path … Alexander, Christopher, et al. A Pattern Language. Oxford University Press, 1977.
https://unsplash.com/photos/8IKf54pc3qk https://unsplash.com/photos/zACLEreWKXE
5. Towards a pattern language
… OK, so, to attract many
people to our city, Small
Public Squares should be
located in the center. At the
SMALL PUBLIC SQUARE,
make Street Cafes be
Opening to the Street ...
5
Small Public
Square
Street
Cafe
Opening to the
Street
https://unsplash.com/photos/EdpbTj3Br-Y
https://unsplash.com/photos/GqurqYbj7aU
https://unsplash.com/photos/zFoRwZirFvY
6. ML software engineering:
Induction (and abduction)
6
Goal Data
Model Behavior
Goal Model Behavior Data
Conventional software engineering: Deduction
ML software engineering: Induction (and abduction)
H. Maruyama, “Machine Learning Engineering and Reuse of AI Work Products,” The First International Workshop on Sharing
and Reuse of AI Work Products, 2017
Hironori Washizaki, “Towards Software Value Co-Creation with AI”, The 44th IEEE Computer Society Signature Conference
on Computers, Software, and Applications (COMPSAC 2020), Fast Abstract
7. ML software engineering needs patterns!
• Bridge between abstract paradigms and
concrete cases/tools
– Documenting Know-Why, Know-What and
Know-How
– Reusing solutions and problems
– Getting consistent architecture
• Common language among stakeholders
– Software engineers, data scientist, domain
experts, network engineers, …
7
Paradigm
Case
Tool
FW
Instruction
?
?
8. Practices and patterns in ML-SE
• Researchers and practitioners studying
best practices strive to design Machine
Learning (ML) systems and software.
• Some practices are formalized as
patterns.
(NOTE: NOT handle ML model patterns.)
8
Data Lake for ML
K. M. Hazelwood, et al., Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective, HPCA 2018
Different Workloads in Different Computing Environments (e.g., Facebook)
https://unsplash.com/photos/_HPlEmsKgP0
FBLearner
Feature Store
FBLearner
Flow
FBLearner
Predictor
Trained
model
Deployed
model
Features
Data
FBLearner
Feature Store
FBLearner
Flow
FBLearner
Predictor
Storage CPU CPU + GPU CPU
H. Washizaki, et al. Software Engineering Patterns for Machine Learning Applications (SEP4MLA) – Part 2, PLoP’20
Structured
data
Unstructured
data
9. Problem and goal
• ML system architecture and design patterns at different
abstraction levels are not well classified and studied.
• Thus, we conducted a survey of software developers and
an Systematic Literature Review.
9
Well-documented patterns
Different Workloads in Different
Computing Environments
Practices with less
information
Data Lake
for ML
Scholarly papers
Gray documents
10. Agenda
• ML software engineering and patterns
• Literature review of software engineering
patterns for ML applications
• Classification of ML patterns
• Practitioners’ insights on ML patterns
10
11. Research questions
• RQ1. Does academic and gray literature address the
design of ML systems and software?
– 19 scholarly and 19 gray documents identified
– 15 SE patterns for ML applications extracted
• RQ2. Can ML patterns be classified?
– Categories of scopes: Topology, programming and model
– Quality attributes: ISO/IEC 25010:2011 System and
software product quality attributes, ML model and
prediction quality attributes
• RQ3. How do practitioners perceive ML patterns?
– Questionnaire-based survey for 600+ developers
– Developers were unfamiliar with most ML patterns,
although there were several major patterns used by 20%
11
12. RQ1. Does academic and gray literature address
the design of ML systems and software?
• Systematic Literature Review (SLR)
– Scholar papers: Engineering Village
– Gray documents: Google
• 19 scholarly papers and 19 gray documents identified
• 15 patterns extracted
12
((((system) OR (software)) AND (machine learning) AND (implementation pattern) OR
(pattern) OR (architecture pattern) OR (design pattern) OR (anti-pattern) OR (recipe) OR
(workflow) OR (practice) OR (issue) OR (template))) WN ALL) + ((cpx OR ins OR kna) WN
DB) AND (({ca} OR {ja} OR {ip} OR {ch}) WN DT)
(system OR software) "Machine learning" (pattern OR "implementation pattern" OR
"architecture pattern" OR "design pattern“ OR anti-pattern OR recipe OR workflow OR
practice OR issue OR template)
"machine implementation pattern" OR "architecture pattern" OR "design pattern“ OR anti-
pattern OR recipe OR workflow OR practice OR issue OR template
Engineering Village
Google
13. Numbers of documents per year
13
0
1
2
3
4
5
6
7
8
9
10
2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019
Scholarly
Gray
• ML application systems have recently become popular due to
the promotion of artificial intelligence.
• Since 2008, academic and gray documents have discussed
good (bad) practices of ML application systems design.
H. Washizaki, et al., Studying Software Engineering Patterns for Designing Machine Learning Systems, IWESEP 2019
14. Agenda
• ML software engineering and patterns
• Literature review of software engineering
patterns for ML applications
• Classification of ML patterns
• Practitioners’ insights on ML patterns
14
15. RQ2. Can ML patterns be classified?
• Model operation patterns that focus on ML models
• Programming patterns that define the design of a
particular component
• Topology patterns that define the entire system
architecture.
15
Training data
Trained model Prediction
Training
Infrastructure
Input data
Programming
patterns
Serving
Infrastructure
Model
operation
patterns
Topology patterns
H. Washizaki, et al., Practitioners’ insights on machine-learning software engineering design patterns: a preliminary study, ICSME 2020
16. Topology patterns
16
Pattern Problem Solution
Different Workloads in
Different Computing
Environments
It is necessary to separate and
quickly change the ML data
workload …
Physically isolate different workloads to
separate machines…
Distinguish Business
Logic from ML Models
The overall business logic should be
isolated from the ML models …
Separate the business logic and the
inference engine, loosely coupling the
business logic and ML-specific dataflows.
ML Gateway Routing
Architecture
Difficult to set up and manage
individual endpoints for each
service…
Install a gateway before a set of
applications …
Microservice
Architecture for ML
ML applications may be confined to
some
“known” ML frameworks …
Provide well-defined services to use for
ML frameworks….
Lambda Architecture
for ML
Real-time data processing requires
scalability,
fault tolerance, predictability …
The batch layer keeps producing views
while the speed layer creates the
relevant real-time views …
Kappa Architecture for
ML
It is necessary to deal with huge
amount of
data with less code resource …
Support both real-time data processing
and continuous reprocessing with a
single stream processing engine …
H. Washizaki, et al. Software Engineering Patterns for Machine Learning Applications (SEP4MLA) – Part 2, PLoP’20
17. Distinguish Business Logic from ML Models
• Problem: Business logic should be isolated from ML models so that they can
be changed without impacting rest of business logic.
• Solution: Separate the business logic and the inference engine, loosely
coupling the business logic and ML-specific dataflows.
17
H. Yokoyama, Machine Learning System Architectural Pattern for Improving Operational Stability, ICSA-C, 2019
H. Washizaki, et al., Software Engineering Patterns for Machine Learning Applications (SEP4MLA), AsianPLoP 2020
Data Layer
Logic Layer
Presentation Layer
User
Interface
Database
Data
Collection
Data Lake
Business
Logic
Data
Processing
Inference
Engine
Real
World
Business
Logic
Specific
ML
Specific
Architectural Layers
Deployed as ML System
Business Logic Data Flow
ML Runtime Data Flow
ML Development Data Flow
Legend
18. Usage of Distinguish Business Logic from ML Models
18
Data Layer
Logic Layer
Presentation Layer
User Interface
(Chatbot UI)
Web App
Front-end
Slack
Business Logic
(Chatbot Logic)
Web App
Back-end
Slack
Data Collection
(Dataset)
Datasets
Nagoya Univ.
Conversation
Corpus
Data Processing
(Text to Vector Transformer)
NN Model
pre- and post-
processing
TensorFlow
Inference Engine
(Language Model)
NN Model TensorFlow
Database
(Previous Q&A Store)
DB Server (None)
Data Lake
(Vectorized Corpus)
Word
Vector
TensorFlow
(Text)
Users
Data
Source
Input
Output
Datasets
ML
Input
ML Output
Architectural Elements
(Example Role as Chatbot)
What How
Business Logic Data Flow
ML Development Data Flow
ML Runtime Data Flow
Input Data
Output Data
Input Data
Datasets
Word
Vector
Legend
19. Programming patterns
19
Pattern Problem Solution
Data Lake for ML We cannot foresee the kind of
analyses that will be performed on
the data …
Store data, which range from structured
to unstructured, as “raw” as possible
into a data storage …
Separation of
Concerns and
Modularization of ML
Components
ML applications must accommodate
regular and frequent changes to
their ML components …
Decouple at different levels of
complexity from the simplest to the
most complex …
Encapsulate ML
Models within Rule-
based Safeguards
ML models are known to be unstable
and vulnerable to adversarial attacks,
drifts, …
Encapsulate functionality in the
containing system using deterministic
and verifiable rules …
Discard PoC Code The code created for Proof of
Concept (PoC) often includes code
that sacrifices maintainability …
Discard the code created for the PoC
and rebuild maintainable code …
H. Washizaki, et al. Software Engineering Patterns for Machine Learning Applications (SEP4MLA) – Part 2, PLoP’20
20. Encapsulate ML Models within
Rule-based Safeguards
• Problem: ML models are known to be unstable and vulnerable to
adversarial attacks, noise, and data drift.
• Solution: Encapsulate functionality provided by ML models and deal
with the inherent uncertainty in the containing system using
deterministic and verifiable rules.
• Know usage: E.g. Apollos’s object detection [Peng20]
20
Business
Logic API
Rule-based
Safeguard
Inference
(Prediction)
Encapsulated
ML model
Input
Output
Rule
H. Washizaki, et al. Software Engineering Patterns for Machine Learning Applications (SEP4MLA) – Part 2, PLoP’20
Z. Peng, et al., A First Look at the Integration of Machine Learning Models in Complex Autonomous Driving Systems, ESEC/FSE’20
21. Model operation patterns
21
Pattern Problem Solution
Parameter-Server
Abstraction
For distributed learning, widely
accepted abstractions are lacking …
Distribute both data and workloads over
worker nodes, while the server nodes
maintain globally shared parameters …
Data Flows Up, Model
Flows Down
Standard ML approaches require
centralizing the training data on one
machine …
Enable mobile devices to collaboratively
learn while keeping all the training data
on the device as federated learning …
Secure Aggregation The system needs to communicate
and aggregate model updates in a
secure and scalable way …
Encrypt data from each device and
calculate totals and averages without
individual examination …
Deployable Canary
Model
A surrogate ML that approximates
the behavior of best model must be
built to provide explainability …
Run the explainable inference pipeline
in parallel to monitor prediction
differences …
ML Versioning ML models and their different
versions may change the behavior of
the overall ML applications …
Record the ML model, dataset, and code
to ensure a reproducible training and
inference processes …
H. Washizaki, et al. Software Engineering Patterns for Machine Learning Applications (SEP4MLA) – Part 2, PLoP’20
22. Deployable Canary Model
• Problem: A surrogate ML that approximates the behavior of the
best ML model must be built to provide explainability.
• Solution: Run the explainable inference pipeline in parallel with
the primary inference pipeline to monitor prediction differences.
• Known usage: Image-based anomaly detection at factory
22
S. Ghanta et al., Interpretability and reproducibility in production machine learning applications, ICMLA 2018
Input
Decoy model Data lake
Canary model
(E.g., Decision
tree)
Production
model
(E.g., DNN)
Monitoring
and
comparison
Output
Output
Reproduce
and
retraining
23. 23
Pattern Perfor
mance
Compa
tibility
Relia
bility
Securi
ty
Maintai
nability
Porta
bility
Robus
tness
Explaina
bility
Accur
acy
Different Workloads in Different
Computing Environments
X X
Distinguish Business Logic from
ML Models
X
ML Gateway Routing
Architecture
X X
Microservice Architecture for ML X X X
Lambda Architecture for ML X X
Kappa Architecture for ML X X
Data Lake for ML X X X
Separation of Concerns and
Modularization of ML
Components
X
Encapsulate ML Models within
Rule-based Safeguards
X
Discard PoC Code X
Parameter-Server Abstraction X X
Data Flows Up, Model Flows
Down
X X X
Secure Aggregation X X X
Deployable Canary Model X X
ML Versioning X X X
24. Agenda
• ML software engineering and patterns
• Literature review of software engineering
patterns for ML applications
• Classification of ML patterns
• Practitioners’ insights on ML patterns
24
25. RQ3. Practitioners’ insights on quality
• Surveyed 300+ developers, 46 answered in ML development
• What product quality attributes considered?
– Maintainability, reliability, security, and usability
• What model and prediction quality attributes?
– Robustness, accuracy, and explainability
• Maintainability, reliability, robustness and accuracy are well
handled by ML patterns. There are demands for having ML
patterns addressing security, usability, and explainability,
which are not handled well now.
25
3
3
4
13
19
20
20
21
28
0 20 40
(not considered)
Compatibility
Portability
Performance efficiency
Usability
Reliability
Security
Maintainability
Functional suitability
10
4
20
21
26
0 10 20 30
(not considered)
Fairness
Accuracy
Explainability
Robustness
H. Washizaki, et al., Practitioners’ insights on machine-learning software engineering design patterns: a preliminary study, ICSME 2020
26. Practitioners’ insights on ML design patterns
• Surveyed 600+
developers, 118
answered
• Have you ever
referred to ML
patterns?
– Major:
ML Versioning,
Microservice
Architecture for
ML
– None:
Secure
Aggregation,
Data Flows Up
(aka. Federated
Learning)
26
Knew it Didn’t know it
0 20 40 60 80 100 120
Data Flows Up, Model Flows Down
Secure Aggregation
Deployable Canary Model
Kappa Architecture for ML
Parameter-Server Abstraction
Different Workloads in Different Computing…
Encapsulate ML models within rule-base…
ML Gateway Routing Architecture
Lambda Architecture for ML
Separation of Concerns and Modularization of…
Distinguish Business Logic from ML Models
Data Lake for ML
Discard PoC code
Microservice Architecture for ML
ML Versioning
Used it Never used it Consider using it Not consider
27. Practitioners’ insights on ML design patterns
• Have you ever referred to ML patterns?
– Developers were unfamiliar with most ML patterns, although there
were several major patterns used by 20+% of the respondents.
– For all patterns, most respondents indicated that they would
consider using them in future designs.
– Promoting existing ML patterns will increase their utilization
• How do you solve and share design challenges of ML
application systems?
– 37 (i.e., 31%) organized design patterns and past design results.
– As respondents become more organized in their approach to design
problems by reuse, the pattern usage ratio increased.
– Development teams and organizations will reuse more ML patterns
as they become more consistent in their reuse approach.
27
Design solution and reuse practice #Respondents #Patterns
used
Pattern
usage ratio
Lv3. Organizing, reusing patterns (and past results) 37 64 11.5%
Lv2. Reusing externally documented patterns 31 50 10.8%
Lv1. Resolving problems in an ad-hoc way 37 35 6.3%
Others 13 3 1.5%
28. Conclusion
• ML software engineering needs patterns!
– As bride and common language
• Literature review of academic and gray literature
– 19 scholarly and 19 gray documents identified.
– 15 SE patterns for ML applications extracted.
– Patterns at https://eai-transfer.github.io/ml-design-pattern/en/
– ML patterns can be classified by scopes and quality attributes
• Survey of practitioners’ insights
– Developers were unfamiliar with most ML patterns, although
there were several major patterns used by 20% (such as ML
Versioning and Microservice Architecture for ML)
– 31% organized design patterns and past design results.
– As respondents become more organized in their approach to
design problems by reuse, the pattern usage ratio increased. 28
29. Future research direction
• Identify ML patterns addressing specific quality attributes
that are not handled well now
– Security, usability, and explainability
• Investigate the impact of patterns on quality attributes of
systems
• Analyze relationships among patterns including related
ones towards a pattern language
– V. Lakshmanan et al., “Machine Learning Design Patterns,”
O’Reilly, 2020.
– Y. Shibui, “Machine Learning System Design Patterns”,
https://github.com/mercari/ml-system-design-pattern, 2020.
• Integration into framework to handle from requirements
ton implementations and testing/debugging
29
Jati H. Husen, Hnin Thandar Tun, Nobukazu Yoshioka, Hironori Washizaki and Yoshiaki Fukazawa, “Goal-Oriented Machine
Learning-Based Component Development Process,” ACM/IEEE 24th International Conference on Model Driven Engineering
Languages and Systems (MODELS), Poster