Software Testing: Issues and Challenges of Artificial Intelligence & Machine ...gerogepatton
The history of Artificial Intelligence and Machine Learning dates back to 1950’s. In recent years, there has been an increase in popularity for applications that implement AI and ML technology. As with traditional development, software testing is a critical component of an efficient AI/ML application. However, the approach to development methodology used in AI/ML varies significantly from traditional development. Owing to these variations, numerous software testing challenges occur. This paper aims to recognize and to explain some of the biggest challenges that software testers face in dealing with AI/ML applications. For future research, this study has key implications. Each of the challenges outlined in this paper is ideal for further investigation and has great potential to shed light on the way to more productive software testing strategies and methodologies that can be applied to AI/ML applications.
Software Testing: Issues and Challenges of Artificial Intelligence & Machine ...gerogepatton
The history of Artificial Intelligence and Machine Learning dates back to 1950’s. In recent years, there has been an increase in popularity for applications that implement AI and ML technology. As with traditional development, software testing is a critical component of an efficient AI/ML application. However, the approach to development methodology used in AI/ML varies significantly from traditional development. Owing to these variations, numerous software testing challenges occur. This paper aims to recognize and to explain some of the biggest challenges that software testers face in dealing with AI/ML applications. For
future research, this study has key implications. Each of the challenges outlined in this paper is ideal for further investigation and has great potential to shed light on the way to more productive software testing strategies and methodologies that can be applied to AI/ML applications.
SOFTWARE TESTING: ISSUES AND CHALLENGES OF ARTIFICIAL INTELLIGENCE & MACHINE ...ijaia
The history of Artificial Intelligence and Machine Learning dates back to 1950’s. In recent years, there has
been an increase in popularity for applications that implement AI and ML technology. As with traditional
development, software testing is a critical component of an efficient AI/ML application. However, the
approach to development methodology used in AI/ML varies significantly from traditional development.
Owing to these variations, numerous software testing challenges occur. This paper aims to recognize and
to explain some of the biggest challenges that software testers face in dealing with AI/ML applications. For
future research, this study has key implications. Each of the challenges outlined in this paper is ideal for
further investigation and has great potential to shed light on the way to more productive software testing
strategies and methodologies that can be applied to AI/ML applications.
Loan Default Prediction Using Machine Learning TechniquesIRJET Journal
This document discusses using machine learning techniques to predict loan defaults. It begins with an abstract that outlines using data collection, cleaning, and performance assessment to predict loan defaulters. It then discusses implementing models like decision trees and KNN (K-nearest neighbors) for classification and regression. The document evaluates the performance of these models on loan default prediction and concludes the KNN model performs better. It proposes using both models and comparing their accuracy to improve prediction performance and minimize risk.
A survey of predicting software reliability using machine learning methodsIAESIJAI
In light of technical and technological progress, software has become an urgent need in every aspect of human life, including the medicine sector and industrial control. Therefore, it is imperative that the software always works flawlessly. The information technology sector has witnessed a rapid expansion in recent years, as software companies can no longer rely only on cost advantages to stay competitive in the market, but programmers must provide reliable and high-quality software, and in order to estimate and predict software reliability using machine learning and deep learning, it was introduced A brief overview of the important scientific contributions to the subject of software reliability, and the researchers' findings of highly efficient methods and techniques for predicting software reliability.
Decision Making Framework in e-Business Cloud Environment Using Software Metr...ijitjournal
Cloud computing technology is most important one in IT industry by enabling them to offer access to their
system and application services on payment type. As a result, more than a few enterprises with Facebook,
Microsoft, Google, and amazon have started offer to their clients. Quality software is most important one in
market competition in this paper presents a hybrid framework based on the goal/question/metric paradigm
to evaluate the quality and effectiveness of previous software goods in project, product and organizations
in a cloud computing environment. In our approach it support decision making in the area of project,
product and organization levels using Neural networks and three angular metrics i.e., project metrics,
product metrics, and organization metrics
A Compendium of Various Applications of Machine LearningIRJET Journal
This document provides a review of various applications of machine learning. It begins with an introduction to machine learning and discusses its applications in fields such as energy efficiency, intrusion detection, anomaly detection, quantitative finance, and cancer prediction and prognosis. Specific machine learning algorithms and techniques discussed include decision trees, naive Bayes, k-nearest neighbors, artificial neural networks, support vector machines, and more. The document also provides examples of machine learning applications in each field and references various research papers to support the discussed applications.
A Novel Approach for Forecasting Disease Using Machine LearningIRJET Journal
This document discusses using machine learning models to predict diseases. It analyzes several supervised machine learning algorithms, including Naive Bayes, Decision Trees, K-Nearest Neighbors, Logistic Regression, and Convolutional Neural Networks. The key findings are:
1) K-Nearest Neighbors performed best at predicting kidney disease, Parkinson's disease, and heart disease based on the analyses.
2) Logistic Regression and Convolutional Neural Networks predicted breast cancer and common diseases accurately, respectively.
3) Supervised machine learning algorithms show potential for early disease detection when applied to electronic health data, which can help clinicians and improve patient outcomes.
Software Testing: Issues and Challenges of Artificial Intelligence & Machine ...gerogepatton
The history of Artificial Intelligence and Machine Learning dates back to 1950’s. In recent years, there has been an increase in popularity for applications that implement AI and ML technology. As with traditional development, software testing is a critical component of an efficient AI/ML application. However, the approach to development methodology used in AI/ML varies significantly from traditional development. Owing to these variations, numerous software testing challenges occur. This paper aims to recognize and to explain some of the biggest challenges that software testers face in dealing with AI/ML applications. For future research, this study has key implications. Each of the challenges outlined in this paper is ideal for further investigation and has great potential to shed light on the way to more productive software testing strategies and methodologies that can be applied to AI/ML applications.
Software Testing: Issues and Challenges of Artificial Intelligence & Machine ...gerogepatton
The history of Artificial Intelligence and Machine Learning dates back to 1950’s. In recent years, there has been an increase in popularity for applications that implement AI and ML technology. As with traditional development, software testing is a critical component of an efficient AI/ML application. However, the approach to development methodology used in AI/ML varies significantly from traditional development. Owing to these variations, numerous software testing challenges occur. This paper aims to recognize and to explain some of the biggest challenges that software testers face in dealing with AI/ML applications. For
future research, this study has key implications. Each of the challenges outlined in this paper is ideal for further investigation and has great potential to shed light on the way to more productive software testing strategies and methodologies that can be applied to AI/ML applications.
SOFTWARE TESTING: ISSUES AND CHALLENGES OF ARTIFICIAL INTELLIGENCE & MACHINE ...ijaia
The history of Artificial Intelligence and Machine Learning dates back to 1950’s. In recent years, there has
been an increase in popularity for applications that implement AI and ML technology. As with traditional
development, software testing is a critical component of an efficient AI/ML application. However, the
approach to development methodology used in AI/ML varies significantly from traditional development.
Owing to these variations, numerous software testing challenges occur. This paper aims to recognize and
to explain some of the biggest challenges that software testers face in dealing with AI/ML applications. For
future research, this study has key implications. Each of the challenges outlined in this paper is ideal for
further investigation and has great potential to shed light on the way to more productive software testing
strategies and methodologies that can be applied to AI/ML applications.
Loan Default Prediction Using Machine Learning TechniquesIRJET Journal
This document discusses using machine learning techniques to predict loan defaults. It begins with an abstract that outlines using data collection, cleaning, and performance assessment to predict loan defaulters. It then discusses implementing models like decision trees and KNN (K-nearest neighbors) for classification and regression. The document evaluates the performance of these models on loan default prediction and concludes the KNN model performs better. It proposes using both models and comparing their accuracy to improve prediction performance and minimize risk.
A survey of predicting software reliability using machine learning methodsIAESIJAI
In light of technical and technological progress, software has become an urgent need in every aspect of human life, including the medicine sector and industrial control. Therefore, it is imperative that the software always works flawlessly. The information technology sector has witnessed a rapid expansion in recent years, as software companies can no longer rely only on cost advantages to stay competitive in the market, but programmers must provide reliable and high-quality software, and in order to estimate and predict software reliability using machine learning and deep learning, it was introduced A brief overview of the important scientific contributions to the subject of software reliability, and the researchers' findings of highly efficient methods and techniques for predicting software reliability.
Decision Making Framework in e-Business Cloud Environment Using Software Metr...ijitjournal
Cloud computing technology is most important one in IT industry by enabling them to offer access to their
system and application services on payment type. As a result, more than a few enterprises with Facebook,
Microsoft, Google, and amazon have started offer to their clients. Quality software is most important one in
market competition in this paper presents a hybrid framework based on the goal/question/metric paradigm
to evaluate the quality and effectiveness of previous software goods in project, product and organizations
in a cloud computing environment. In our approach it support decision making in the area of project,
product and organization levels using Neural networks and three angular metrics i.e., project metrics,
product metrics, and organization metrics
A Compendium of Various Applications of Machine LearningIRJET Journal
This document provides a review of various applications of machine learning. It begins with an introduction to machine learning and discusses its applications in fields such as energy efficiency, intrusion detection, anomaly detection, quantitative finance, and cancer prediction and prognosis. Specific machine learning algorithms and techniques discussed include decision trees, naive Bayes, k-nearest neighbors, artificial neural networks, support vector machines, and more. The document also provides examples of machine learning applications in each field and references various research papers to support the discussed applications.
A Novel Approach for Forecasting Disease Using Machine LearningIRJET Journal
This document discusses using machine learning models to predict diseases. It analyzes several supervised machine learning algorithms, including Naive Bayes, Decision Trees, K-Nearest Neighbors, Logistic Regression, and Convolutional Neural Networks. The key findings are:
1) K-Nearest Neighbors performed best at predicting kidney disease, Parkinson's disease, and heart disease based on the analyses.
2) Logistic Regression and Convolutional Neural Networks predicted breast cancer and common diseases accurately, respectively.
3) Supervised machine learning algorithms show potential for early disease detection when applied to electronic health data, which can help clinicians and improve patient outcomes.
Review on Algorithmic and Non Algorithmic Software Cost Estimation Techniquesijtsrd
Effective software cost estimation is the most challenging and important activities in software development. Developers want a simple and accurate method of efforts estimation. Estimation of the cost before starting of work is a prediction and prediction always not accurate. Software effort estimation is a very critical task in the software engineering and to control quality and efficiency a suitable estimation technique is crucial. This paper gives a review of various available software effort estimation methods, mainly focus on the algorithmic model and non algorithmic model. These existing methods for software cost estimation are illustrated and their aspect will be discussed. No single technique is best for all situations, and thus a careful comparison of the results of several approaches is most likely to produce realistic estimation. This paper provides a detailed overview of existing software cost estimation models and techniques. This paper presents the strength and weakness of various cost estimation methods. This paper focuses on some of the relevant reasons that cause inaccurate estimation. Pa Pa Win | War War Myint | Hlaing Phyu Phyu Mon | Seint Wint Thu "Review on Algorithmic and Non-Algorithmic Software Cost Estimation Techniques" Published in International Journal of Trend in Scientific Research and Development (ijtsrd), ISSN: 2456-6470, Volume-3 | Issue-5 , August 2019, URL: https://www.ijtsrd.com/papers/ijtsrd26511.pdfPaper URL: https://www.ijtsrd.com/engineering/-/26511/review-on-algorithmic-and-non-algorithmic-software-cost-estimation-techniques/pa-pa-win
This document provides information about the Engineering Minor in Data Science offered by the School of Computer Science and Engineering. It describes what engineering minors are, lists the courses offered in the Data Science minor, and provides brief descriptions and outcomes of each course. The minor consists of six courses spanning four semesters that cover topics like data management, visualization, programming in R, predictive analytics, big data fundamentals, and cluster computing. The document also discusses career opportunities, industrial applications, special requirements, and contacts for additional information about the minor.
IRJET- Comparison of Classification Algorithms using Machine LearningIRJET Journal
This document compares several machine learning classification algorithms. It first provides background on machine learning and describes common algorithms like linear regression, support vector machines, and decision trees. It then outlines an experimental framework in Python using libraries like Pandas, Scikit-Learn, and Matplotlib. Various classification algorithms are applied to a dataset and their test and train errors are calculated and compared to determine the most accurate algorithm. The proposed algorithm is found to have the lowest test and train errors compared to other algorithms like ridge regression, KNN, Bayesian regression, decision trees, and SVM.
Proceedings of the 2015 Industrial and Systems Engineering Res.docxwkyra78
Proceedings of the 2015 Industrial and Systems Engineering Research Conference
S. Cetinkaya and J. K. Ryan, eds.
Use of Symbolic Regression for Lean Six Sigma Projects
Daniel Moreno-Sanchez, MSc.
Jacobo Tijerina-Aguilera, MSc.
Universidad de Monterrey
San Pedro Garza Garcia, NL 66238, Mexico
Arlethe Yari Aguilar-Villarreal, MEng.
Universidad Autonoma de Nuevo Leon
San Nicolas de los Garza, NL 66451, Mexico
Abstract
Lean Six Sigma projects and the quality engineering profession have to deal with an extensive selection of tools
most of them requiring specialized training. The increased availability of standard statistical software motivates the
use of advanced data science techniques to identify relationships between potential causes and project metrics. In
these circumstances, Symbolic Regression has received increased attention from researchers and practitioners to
uncover the intrinsic relationships hidden within complex data without requiring specialized training for its
implementation. The objective of this paper is to evaluate the advantages and drawbacks of using computer assisted
Symbolic Regression within the Analyze phase of a Lean Six Sigma project. An application of this approach in a
service industry project is also presented.
Keywords
Symbolic Regression, Data Science, Lean Six Sigma
1. Introduction
Lean Six Sigma (LSS) has become a well-known hybrid methodology for quality and productivity improvement in
organizations. Its wide adoption in several industries has shaped Process Innovation and Operational Excellence
initiatives, enabling LSS to become a main topic in quality practitioner sites of interest [1], recognized Six Sigma
(SS) certification body of knowledge contents [2], and professional society conferences [3].
However LSS projects and the quality engineering profession have to deal with an extensive selection of tools most
of them requiring specialized training. To assist LSS practitioners it is common to categorize tools based on the
traditional DMAIC model which stands for Define, Measure, Analyze, Improve, and Control phases. Table 1
presents an overview of the main tools that are commonly used in each phase of a LSS project, allowing team
members to progressively develop an understanding between realizing each phase’s intent and how the selected
tools can contribute to that purpose.
This paper focuses on the Analyze phase where tools for statistical model building are most likely to be selected.
The increased availability of standard statistical software motivates the use of advanced data science techniques to
identify relationships between potential causes and project metrics. In these circumstances Symbolic Regression
(SR) has received increased attention from researchers and practitioners even though SR is still in an early stage of
commercial availability.
The objective of this paper is to evaluate the advantages and drawbacks o ...
The adoption of machine learning techniques for software defect prediction: A...RAKESH RANA
The adoption of machine learning techniques for software defect prediction: An initial industrial validation
Presented at:
11th Joint Conference On Knowledge-Based Software Engineering, JCKBSE, Volgograd, Russia, 2014
Get full text of publication at:
http://rakeshrana.website/index.php/work/publications/
The document discusses project quality management for IT projects. It defines quality management and describes processes like quality planning, assurance, and control. It discusses tools for quality control like Pareto analysis, statistical sampling, Six Sigma, and testing. It summarizes contributions of quality experts like Deming and Juran. It describes how leadership, costs, standards, and maturity models relate to quality improvement for IT projects.
In Banking Loan Approval Prediction Using Machine LearningIRJET Journal
This document discusses using machine learning algorithms to predict loan approvals. It analyzes loan data using decision trees, logistic regression, and random forest algorithms. The random forest algorithm achieved the highest accuracy rate of 88.53% compared to 85% for decision trees and 83.04% for logistic regression. Therefore, the random forest algorithm is concluded to be best for loan approval prediction. Future work could involve applying these algorithms to other loan data sets and exploring additional machine learning methods.
Here is a guide for Successful testing of AI/ML, as the advancement in AI/ML have been increased by many folds in recent years be it the industry of healthcare, finance, or automobile manufacturing so the testing should be standardized and guided towards success.
Machine Learning Approaches to Predict Customer Churn in Telecommunications I...IRJET Journal
This project aimed to develop machine learning models to predict customer churn in the telecommunications industry. Four algorithms were evaluated - logistic regression, support vector machine, decision tree, and random forest. Logistic regression performed best with an accuracy of 79.25% and AUC score of 84.08%. The models analyzed customer attribute data to identify patterns and predict churn, helping telecom companies understand churn reasons and develop retention strategies. The results provide insights to improve customer experience and reduce costly customer churn.
The document discusses using data mining techniques to analyze crime data and predict crime trends. It describes collecting crime reports from various sources to create a database. Machine learning algorithms would then be applied to the crime data to discover patterns and relationships between different crimes. This analysis could help police identify crime hotspots and determine if a crime was committed in a known location. The proposed system aims to forecast crimes and trends based on past crime data, date and location to help prevent crimes. It discusses implementing the system using Python and testing it with sample input data.
The document summarizes an advisory board meeting for the CarLab research center. It discusses continuous assurance and how audit is evolving to be a continuous process using automation, sensors, and data monitoring. Research projects are looking at areas like control monitoring, process mining, remote auditing, and using techniques like continuity equations and clustering for anomaly detection. The document also discusses creating a rapid prototyping environment for testing advanced analytics and building dashboards to visualize risks and transactions for continuous monitoring.
Automated Feature Selection and Churn Prediction using Deep Learning ModelsIRJET Journal
This document discusses using deep learning models for churn prediction in the telecommunications industry. It begins with an introduction to churn prediction and feature selection challenges. It then provides an overview of deep learning techniques, including artificial neural networks, convolutional neural networks, and their applications. The document proposes three deep learning architectures for churn prediction and experiments with them on two telecom datasets. The results show deep learning models can achieve performance comparable to traditional models without manual feature engineering.
How Does Data Create Economic Value_ Foundations For Valuation Models.pdfDr. Anke Nestler
In order to clarify how data creates economic value, the measurement of data value and the process of valuation of data contribution to economic benefits is being discussed in this article leading to the identification of ways to operationalize data valuation methodologies. The four independent authors André Gorius, Véronique Blum, Andreas Liebl, and Anke Nestler are leading European IP and licensing specialists of the International Licensing Executives Society (LESI).
Um zu klären, wie Daten einen wirtschaftlichen Wert schaffen, werden in diesem Artikel die Messung von Datenwerten und der Prozess der Bewertung des Beitrags von Daten zum wirtschaftlichen Nutzen erörtert, um Wege zur Operationalisierung von Datenbewertungsmethoden zu finden. Die vier unabhängigen Autoren André Gorius, Véronique Blum, Andreas Liebl und Anke Nestler sind führende europäische IP- und Lizenzierungsspezialisten der International Licensing Executives Society (LESI).
The document describes a Driverless ML API that was created to automate machine learning workflows including feature engineering, model validation, tuning, selection, and deployment. The API uses machine learning interpretability techniques to provide visualizations and explanations of models. It aims to help scale data science efforts and enable both expert and junior data scientists to more quickly develop accurate, production-ready models. Key capabilities of the API include automated exploratory data analysis, feature selection and engineering, model selection and hyperparameter tuning using GPUs for faster training, and model interpretability visualizations.
Identification, Analysis & Empirical Validation (IAV) of Object Oriented Desi...rahulmonikasharma
Metrics and Measure are closely inter-related to each other. Measure is defined as way of defining amount, dimension, capacity or size of some attribute of a product in quantitative manner while Metric is unit used for measuring attribute. Software quality is one of the major concerns that need to be addressed and measured. Object oriented (OO) systems require effective metrics to assess quality of software. The paper is designed to identify attributes and measures that can help in determining and affecting quality attributes. The paper conducts empirical study by taking public dataset KC1 from NASA project database. It is validated by applying statistical techniques like correlation analysis and regression analysis. After analysis of data, it is found that metrics SLOC, RFC, WMC and CBO are significant and treated as quality indicators while metrics DIT and NOC are not significant. The results produced from them throws significant impact on improving software quality.
AN IMPROVED REPOSITORY STRUCTURE TO IDENTIFY, SELECT AND INTEGRATE COMPONENTS...ijseajournal
An ultimate goal of software development is to build high quality products. The customers of software
industry always demand for high-quality products quickly and cost effectively. The component-based
development (CBD) is the most suitable methodology for the software companies to meet the demands of
target market. To opt CBD, the software development teams have to customize generic components that are
available in the market and it is very difficult for the development teams to choose the suitable components
from the millions of third party and commercial off the shelf (COTS) components. On the other hand, the
development of in-house repository is tedious and time consuming. In this paper, we propose an easy and
understandable repository structure to provide helpful information about stored components like how to
identify, select, retrieve and integrate components. The proposed repository will also provide previous
assessments of developers and end-users about the selected component. The proposed repository will help
the software companies by reducing the customization effort, improving the quality of developed software
and preventing integrating unfamiliar components.
Insights on Research Techniques towards Cost Estimation in Software Design IJECEIAES
This document summarizes research on techniques for cost estimation in software design. It begins by describing common cost estimation techniques like Constructive Cost Modeling (COCOMO) and Function Point Analysis. It then analyzes research trends in cost estimation, effort estimation, and fault prediction based on literature from 2010 to present. Fewer than 50 papers were found related to overall cost estimation, less than 25 for effort estimation, and only 9 for fault prediction. The document then reviews existing research addressing general cost estimation, enhancement of Function Point Analysis, statistical modeling approaches, cost estimation for embedded systems, and estimation for fourth generation languages and NASA projects. Most techniques use COCOMO or extend existing models with techniques like fuzzy logic, neural networks, or statistical
AI TESTING: ENSURING A GOOD DATA SPLIT BETWEEN DATA SETS (TRAINING AND TEST) ...ijsc
This document discusses challenges in effectively splitting a dataset into training and test sets for machine learning models. It proposes using k-means clustering followed by decision tree analysis to improve the split. K-means clustering groups the data points into clusters to ensure each cluster is well-represented in both the training and test sets. Then a decision tree is used to split the clustered data, aiming to maximize purity within each subset and minimize overlap between training and test data. This approach aims to capture the full domain of the dataset and avoid underrepresenting any parts of the data in either the training or test sets.
Machine Learning: The First Salvo of the AI Business RevolutionCognizant
Machine learning (ML), a branch of artificial intelligence (AI), is coming into its own as a force in the business landscape, performing a variety of innovative and highly skilled activities that enhance customer experience and offer market advantages. This is a brief guide to getting started with ML, the thinking, tools and frameworks to make it a powerful business tool.
Robustness in Machine Learning Explanations Does It Matter-1.pdfDaniel983829
This document discusses the desirability of robustness in machine learning explanations. It argues that robustness is desirable to the extent we want explanations to reflect real patterns in the data and the world. The importance of explanations capturing real patterns depends on the problem context. In some contexts, non-robust explanations can pose moral hazards. Clarifying how much we want explanations to capture real patterns can help determine whether the "Rashomon effect" is beneficial or problematic.
Robustness and Regularization of Support Vector Machines.pdfDaniel983829
This document summarizes a research paper that establishes an equivalence between robust optimization and regularization as applied to support vector machines (SVMs). It shows that regularized SVMs can be formulated as a robust optimization problem that minimizes error under potential adversarial noise or disturbances in the training data. This provides an alternative explanation for why regularization improves generalization performance of SVMs. The document also discusses implications for algorithm development and analysis, including a new proof of consistency for SVMs based on their robustness interpretation.
More Related Content
Similar to machine-learning-development-audit-framework-assessment-and-inspection-of-risk-and-quality-of-data-model-and-development-process.pdf
Review on Algorithmic and Non Algorithmic Software Cost Estimation Techniquesijtsrd
Effective software cost estimation is the most challenging and important activities in software development. Developers want a simple and accurate method of efforts estimation. Estimation of the cost before starting of work is a prediction and prediction always not accurate. Software effort estimation is a very critical task in the software engineering and to control quality and efficiency a suitable estimation technique is crucial. This paper gives a review of various available software effort estimation methods, mainly focus on the algorithmic model and non algorithmic model. These existing methods for software cost estimation are illustrated and their aspect will be discussed. No single technique is best for all situations, and thus a careful comparison of the results of several approaches is most likely to produce realistic estimation. This paper provides a detailed overview of existing software cost estimation models and techniques. This paper presents the strength and weakness of various cost estimation methods. This paper focuses on some of the relevant reasons that cause inaccurate estimation. Pa Pa Win | War War Myint | Hlaing Phyu Phyu Mon | Seint Wint Thu "Review on Algorithmic and Non-Algorithmic Software Cost Estimation Techniques" Published in International Journal of Trend in Scientific Research and Development (ijtsrd), ISSN: 2456-6470, Volume-3 | Issue-5 , August 2019, URL: https://www.ijtsrd.com/papers/ijtsrd26511.pdfPaper URL: https://www.ijtsrd.com/engineering/-/26511/review-on-algorithmic-and-non-algorithmic-software-cost-estimation-techniques/pa-pa-win
This document provides information about the Engineering Minor in Data Science offered by the School of Computer Science and Engineering. It describes what engineering minors are, lists the courses offered in the Data Science minor, and provides brief descriptions and outcomes of each course. The minor consists of six courses spanning four semesters that cover topics like data management, visualization, programming in R, predictive analytics, big data fundamentals, and cluster computing. The document also discusses career opportunities, industrial applications, special requirements, and contacts for additional information about the minor.
IRJET- Comparison of Classification Algorithms using Machine LearningIRJET Journal
This document compares several machine learning classification algorithms. It first provides background on machine learning and describes common algorithms like linear regression, support vector machines, and decision trees. It then outlines an experimental framework in Python using libraries like Pandas, Scikit-Learn, and Matplotlib. Various classification algorithms are applied to a dataset and their test and train errors are calculated and compared to determine the most accurate algorithm. The proposed algorithm is found to have the lowest test and train errors compared to other algorithms like ridge regression, KNN, Bayesian regression, decision trees, and SVM.
Proceedings of the 2015 Industrial and Systems Engineering Res.docxwkyra78
Proceedings of the 2015 Industrial and Systems Engineering Research Conference
S. Cetinkaya and J. K. Ryan, eds.
Use of Symbolic Regression for Lean Six Sigma Projects
Daniel Moreno-Sanchez, MSc.
Jacobo Tijerina-Aguilera, MSc.
Universidad de Monterrey
San Pedro Garza Garcia, NL 66238, Mexico
Arlethe Yari Aguilar-Villarreal, MEng.
Universidad Autonoma de Nuevo Leon
San Nicolas de los Garza, NL 66451, Mexico
Abstract
Lean Six Sigma projects and the quality engineering profession have to deal with an extensive selection of tools
most of them requiring specialized training. The increased availability of standard statistical software motivates the
use of advanced data science techniques to identify relationships between potential causes and project metrics. In
these circumstances, Symbolic Regression has received increased attention from researchers and practitioners to
uncover the intrinsic relationships hidden within complex data without requiring specialized training for its
implementation. The objective of this paper is to evaluate the advantages and drawbacks of using computer assisted
Symbolic Regression within the Analyze phase of a Lean Six Sigma project. An application of this approach in a
service industry project is also presented.
Keywords
Symbolic Regression, Data Science, Lean Six Sigma
1. Introduction
Lean Six Sigma (LSS) has become a well-known hybrid methodology for quality and productivity improvement in
organizations. Its wide adoption in several industries has shaped Process Innovation and Operational Excellence
initiatives, enabling LSS to become a main topic in quality practitioner sites of interest [1], recognized Six Sigma
(SS) certification body of knowledge contents [2], and professional society conferences [3].
However LSS projects and the quality engineering profession have to deal with an extensive selection of tools most
of them requiring specialized training. To assist LSS practitioners it is common to categorize tools based on the
traditional DMAIC model which stands for Define, Measure, Analyze, Improve, and Control phases. Table 1
presents an overview of the main tools that are commonly used in each phase of a LSS project, allowing team
members to progressively develop an understanding between realizing each phase’s intent and how the selected
tools can contribute to that purpose.
This paper focuses on the Analyze phase where tools for statistical model building are most likely to be selected.
The increased availability of standard statistical software motivates the use of advanced data science techniques to
identify relationships between potential causes and project metrics. In these circumstances Symbolic Regression
(SR) has received increased attention from researchers and practitioners even though SR is still in an early stage of
commercial availability.
The objective of this paper is to evaluate the advantages and drawbacks o ...
The adoption of machine learning techniques for software defect prediction: A...RAKESH RANA
The adoption of machine learning techniques for software defect prediction: An initial industrial validation
Presented at:
11th Joint Conference On Knowledge-Based Software Engineering, JCKBSE, Volgograd, Russia, 2014
Get full text of publication at:
http://rakeshrana.website/index.php/work/publications/
The document discusses project quality management for IT projects. It defines quality management and describes processes like quality planning, assurance, and control. It discusses tools for quality control like Pareto analysis, statistical sampling, Six Sigma, and testing. It summarizes contributions of quality experts like Deming and Juran. It describes how leadership, costs, standards, and maturity models relate to quality improvement for IT projects.
In Banking Loan Approval Prediction Using Machine LearningIRJET Journal
This document discusses using machine learning algorithms to predict loan approvals. It analyzes loan data using decision trees, logistic regression, and random forest algorithms. The random forest algorithm achieved the highest accuracy rate of 88.53% compared to 85% for decision trees and 83.04% for logistic regression. Therefore, the random forest algorithm is concluded to be best for loan approval prediction. Future work could involve applying these algorithms to other loan data sets and exploring additional machine learning methods.
Here is a guide for Successful testing of AI/ML, as the advancement in AI/ML have been increased by many folds in recent years be it the industry of healthcare, finance, or automobile manufacturing so the testing should be standardized and guided towards success.
Machine Learning Approaches to Predict Customer Churn in Telecommunications I...IRJET Journal
This project aimed to develop machine learning models to predict customer churn in the telecommunications industry. Four algorithms were evaluated - logistic regression, support vector machine, decision tree, and random forest. Logistic regression performed best with an accuracy of 79.25% and AUC score of 84.08%. The models analyzed customer attribute data to identify patterns and predict churn, helping telecom companies understand churn reasons and develop retention strategies. The results provide insights to improve customer experience and reduce costly customer churn.
The document discusses using data mining techniques to analyze crime data and predict crime trends. It describes collecting crime reports from various sources to create a database. Machine learning algorithms would then be applied to the crime data to discover patterns and relationships between different crimes. This analysis could help police identify crime hotspots and determine if a crime was committed in a known location. The proposed system aims to forecast crimes and trends based on past crime data, date and location to help prevent crimes. It discusses implementing the system using Python and testing it with sample input data.
The document summarizes an advisory board meeting for the CarLab research center. It discusses continuous assurance and how audit is evolving to be a continuous process using automation, sensors, and data monitoring. Research projects are looking at areas like control monitoring, process mining, remote auditing, and using techniques like continuity equations and clustering for anomaly detection. The document also discusses creating a rapid prototyping environment for testing advanced analytics and building dashboards to visualize risks and transactions for continuous monitoring.
Automated Feature Selection and Churn Prediction using Deep Learning ModelsIRJET Journal
This document discusses using deep learning models for churn prediction in the telecommunications industry. It begins with an introduction to churn prediction and feature selection challenges. It then provides an overview of deep learning techniques, including artificial neural networks, convolutional neural networks, and their applications. The document proposes three deep learning architectures for churn prediction and experiments with them on two telecom datasets. The results show deep learning models can achieve performance comparable to traditional models without manual feature engineering.
How Does Data Create Economic Value_ Foundations For Valuation Models.pdfDr. Anke Nestler
In order to clarify how data creates economic value, the measurement of data value and the process of valuation of data contribution to economic benefits is being discussed in this article leading to the identification of ways to operationalize data valuation methodologies. The four independent authors André Gorius, Véronique Blum, Andreas Liebl, and Anke Nestler are leading European IP and licensing specialists of the International Licensing Executives Society (LESI).
Um zu klären, wie Daten einen wirtschaftlichen Wert schaffen, werden in diesem Artikel die Messung von Datenwerten und der Prozess der Bewertung des Beitrags von Daten zum wirtschaftlichen Nutzen erörtert, um Wege zur Operationalisierung von Datenbewertungsmethoden zu finden. Die vier unabhängigen Autoren André Gorius, Véronique Blum, Andreas Liebl und Anke Nestler sind führende europäische IP- und Lizenzierungsspezialisten der International Licensing Executives Society (LESI).
The document describes a Driverless ML API that was created to automate machine learning workflows including feature engineering, model validation, tuning, selection, and deployment. The API uses machine learning interpretability techniques to provide visualizations and explanations of models. It aims to help scale data science efforts and enable both expert and junior data scientists to more quickly develop accurate, production-ready models. Key capabilities of the API include automated exploratory data analysis, feature selection and engineering, model selection and hyperparameter tuning using GPUs for faster training, and model interpretability visualizations.
Identification, Analysis & Empirical Validation (IAV) of Object Oriented Desi...rahulmonikasharma
Metrics and Measure are closely inter-related to each other. Measure is defined as way of defining amount, dimension, capacity or size of some attribute of a product in quantitative manner while Metric is unit used for measuring attribute. Software quality is one of the major concerns that need to be addressed and measured. Object oriented (OO) systems require effective metrics to assess quality of software. The paper is designed to identify attributes and measures that can help in determining and affecting quality attributes. The paper conducts empirical study by taking public dataset KC1 from NASA project database. It is validated by applying statistical techniques like correlation analysis and regression analysis. After analysis of data, it is found that metrics SLOC, RFC, WMC and CBO are significant and treated as quality indicators while metrics DIT and NOC are not significant. The results produced from them throws significant impact on improving software quality.
AN IMPROVED REPOSITORY STRUCTURE TO IDENTIFY, SELECT AND INTEGRATE COMPONENTS...ijseajournal
An ultimate goal of software development is to build high quality products. The customers of software
industry always demand for high-quality products quickly and cost effectively. The component-based
development (CBD) is the most suitable methodology for the software companies to meet the demands of
target market. To opt CBD, the software development teams have to customize generic components that are
available in the market and it is very difficult for the development teams to choose the suitable components
from the millions of third party and commercial off the shelf (COTS) components. On the other hand, the
development of in-house repository is tedious and time consuming. In this paper, we propose an easy and
understandable repository structure to provide helpful information about stored components like how to
identify, select, retrieve and integrate components. The proposed repository will also provide previous
assessments of developers and end-users about the selected component. The proposed repository will help
the software companies by reducing the customization effort, improving the quality of developed software
and preventing integrating unfamiliar components.
Insights on Research Techniques towards Cost Estimation in Software Design IJECEIAES
This document summarizes research on techniques for cost estimation in software design. It begins by describing common cost estimation techniques like Constructive Cost Modeling (COCOMO) and Function Point Analysis. It then analyzes research trends in cost estimation, effort estimation, and fault prediction based on literature from 2010 to present. Fewer than 50 papers were found related to overall cost estimation, less than 25 for effort estimation, and only 9 for fault prediction. The document then reviews existing research addressing general cost estimation, enhancement of Function Point Analysis, statistical modeling approaches, cost estimation for embedded systems, and estimation for fourth generation languages and NASA projects. Most techniques use COCOMO or extend existing models with techniques like fuzzy logic, neural networks, or statistical
AI TESTING: ENSURING A GOOD DATA SPLIT BETWEEN DATA SETS (TRAINING AND TEST) ...ijsc
This document discusses challenges in effectively splitting a dataset into training and test sets for machine learning models. It proposes using k-means clustering followed by decision tree analysis to improve the split. K-means clustering groups the data points into clusters to ensure each cluster is well-represented in both the training and test sets. Then a decision tree is used to split the clustered data, aiming to maximize purity within each subset and minimize overlap between training and test data. This approach aims to capture the full domain of the dataset and avoid underrepresenting any parts of the data in either the training or test sets.
Machine Learning: The First Salvo of the AI Business RevolutionCognizant
Machine learning (ML), a branch of artificial intelligence (AI), is coming into its own as a force in the business landscape, performing a variety of innovative and highly skilled activities that enhance customer experience and offer market advantages. This is a brief guide to getting started with ML, the thinking, tools and frameworks to make it a powerful business tool.
Similar to machine-learning-development-audit-framework-assessment-and-inspection-of-risk-and-quality-of-data-model-and-development-process.pdf (20)
Robustness in Machine Learning Explanations Does It Matter-1.pdfDaniel983829
This document discusses the desirability of robustness in machine learning explanations. It argues that robustness is desirable to the extent we want explanations to reflect real patterns in the data and the world. The importance of explanations capturing real patterns depends on the problem context. In some contexts, non-robust explanations can pose moral hazards. Clarifying how much we want explanations to capture real patterns can help determine whether the "Rashomon effect" is beneficial or problematic.
Robustness and Regularization of Support Vector Machines.pdfDaniel983829
This document summarizes a research paper that establishes an equivalence between robust optimization and regularization as applied to support vector machines (SVMs). It shows that regularized SVMs can be formulated as a robust optimization problem that minimizes error under potential adversarial noise or disturbances in the training data. This provides an alternative explanation for why regularization improves generalization performance of SVMs. The document also discusses implications for algorithm development and analysis, including a new proof of consistency for SVMs based on their robustness interpretation.
Robustness of Machine Learning Models Beyond Adversarial Attacks.pdfDaniel983829
This document discusses assessing the robustness of machine learning models to "real-world perturbations" that occur naturally rather than being crafted by an adversary. It argues that adversarial robustness is not a valid measure of robustness to such natural perturbations. The document proposes a probabilistic approach to compute the likelihood that a real-world perturbation would change a model's prediction, quantifying its real-world robustness. This approach models possible input perturbations individually for each application and works for any "black-box" model. It illustrates the approach on two datasets and analytically solvable cases, and discusses estimating real-world robustness in high-dimensional spaces.
Robustness in Deep Learning - Single Image Denoising using Untrained Networks...Daniel983829
This document is a thesis submitted by Esha Singh to the University of Minnesota for the degree of Master of Science in May 2021. The thesis explores single image denoising using untrained neural networks. It first provides background on deep learning, inverse problems, image denoising and neural networks. It then reviews existing image denoising algorithms including spatial/transform domain and neural network methods. The thesis also discusses recent work on deep image priors and rethinking single image denoising using over-parameterization and low-rank matrix recovery. Preliminary experiments on denoising images with salt and pepper noise are presented to demonstrate the proposed methodology.
Data preparation for artificial intelligence in medical imaging - A comprehen...Daniel983829
This document provides a comprehensive guide to tools and platforms for preparing medical image data for artificial intelligence applications. It discusses key steps in a medical image preparation pipeline including image acquisition, de-identification, data curation, storage, and annotation. A variety of open-access tools are reviewed that can perform image de-identification, data curation, storage, and annotation. Examples of medical imaging datasets covering different organs and diseases are also provided. The guide aims to enable standardized and large-scale data preparation and AI development in medical imaging.
Generative Adversarial Networks for Robust Medical Image Analysis.pdfDaniel983829
This document presents two approaches for improving robustness in medical image segmentation using generative adversarial networks (GANs). The first approach, UltraGAN, uses a GAN to enhance the quality of ultrasound images and improve robustness to low image quality. The second approach, MedRobGAN, generates adversarial medical image examples to improve robustness against adversarial attacks. Both methods are evaluated on medical segmentation tasks to validate their effectiveness in improving robustness.
Application of generative adversarial networks (GAN) for ophthalmology image ...Daniel983829
This document reviews literature on the application of generative adversarial networks (GANs) for ophthalmology image domains. GANs have been used for tasks like segmentation, data augmentation, denoising, domain transfer, and super-resolution of ophthalmic images. The review identified 48 relevant papers and found that GANs have benefited analysis of fundus photographs, optical coherence tomography scans, and other eye images. However, GANs also have limitations such as mode collapse and need clinical validation before practical use in ophthalmology. Further development could improve GAN performance for ocular image analysis and disease diagnosis.
Auditing AI models for verified deployment under semantic specifications.pdfDaniel983829
The document proposes a framework called AuditAI for auditing deep learning models prior to deployment. It does this by creating a bridge between the deep learning model and a generative model, allowing them to share the same semantically-aligned latent space. This enables the use of unit tests to verify that predefined specifications are satisfied for controlled variations in the latent space, like accuracy over 95% for different angles of faces. The framework applies interval bound propagation to provide certifications for specifications involving latent space perturbations, providing interpretable and scalable auditing of deep learning models.
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slackshyamraj55
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
“An Outlook of the Ongoing and Future Relationship between Blockchain Technologies and Process-aware Information Systems.” Invited talk at the joint workshop on Blockchain for Information Systems (BC4IS) and Blockchain for Trusted Data Sharing (B4TDS), co-located with with the 36th International Conference on Advanced Information Systems Engineering (CAiSE), 3 June 2024, Limassol, Cyprus.
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Building RAG with self-deployed Milvus vector database and Snowpark Container...Zilliz
This talk will give hands-on advice on building RAG applications with an open-source Milvus database deployed as a docker container. We will also introduce the integration of Milvus with Snowpark Container Services.
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!SOFTTECHHUB
As the digital landscape continually evolves, operating systems play a critical role in shaping user experiences and productivity. The launch of Nitrux Linux 3.5.0 marks a significant milestone, offering a robust alternative to traditional systems such as Windows 11. This article delves into the essence of Nitrux Linux 3.5.0, exploring its unique features, advantages, and how it stands as a compelling choice for both casual users and tech enthusiasts.
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc
How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024?
In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores.
See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar will review:
- The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey
- The top challenges for privacy leaders, practitioners, and organizations in 2024
- Key themes to consider in developing and maintaining your privacy program
20 Comprehensive Checklist of Designing and Developing a WebsitePixlogix Infotech
Dive into the world of Website Designing and Developing with Pixlogix! Looking to create a stunning online presence? Look no further! Our comprehensive checklist covers everything you need to know to craft a website that stands out. From user-friendly design to seamless functionality, we've got you covered. Don't miss out on this invaluable resource! Check out our checklist now at Pixlogix and start your journey towards a captivating online presence today.
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIVladimir Iglovikov, Ph.D.
Presented by Vladimir Iglovikov:
- https://www.linkedin.com/in/iglovikov/
- https://x.com/viglovikov
- https://www.instagram.com/ternaus/
This presentation delves into the journey of Albumentations.ai, a highly successful open-source library for data augmentation.
Created out of a necessity for superior performance in Kaggle competitions, Albumentations has grown to become a widely used tool among data scientists and machine learning practitioners.
This case study covers various aspects, including:
People: The contributors and community that have supported Albumentations.
Metrics: The success indicators such as downloads, daily active users, GitHub stars, and financial contributions.
Challenges: The hurdles in monetizing open-source projects and measuring user engagement.
Development Practices: Best practices for creating, maintaining, and scaling open-source libraries, including code hygiene, CI/CD, and fast iteration.
Community Building: Strategies for making adoption easy, iterating quickly, and fostering a vibrant, engaged community.
Marketing: Both online and offline marketing tactics, focusing on real, impactful interactions and collaborations.
Mental Health: Maintaining balance and not feeling pressured by user demands.
Key insights include the importance of automation, making the adoption process seamless, and leveraging offline interactions for marketing. The presentation also emphasizes the need for continuous small improvements and building a friendly, inclusive community that contributes to the project's growth.
Vladimir Iglovikov brings his extensive experience as a Kaggle Grandmaster, ex-Staff ML Engineer at Lyft, sharing valuable lessons and practical advice for anyone looking to enhance the adoption of their open-source projects.
Explore more about Albumentations and join the community at:
GitHub: https://github.com/albumentations-team/albumentations
Website: https://albumentations.ai/
LinkedIn: https://www.linkedin.com/company/100504475
Twitter: https://x.com/albumentations
1. Machine Learning Development Audit Framework:
Assessment and Inspection of Risk and Quality of
Data, Model and Development Process
Jan Stodt, Christoph Reich
Abstract—The usage of machine learning models for prediction
is growing rapidly and proof that the intended requirements
are met is essential. Audits are a proven method to determine
whether requirements or guidelines are met. However, machine
learning models have intrinsic characteristics, such as the quality
of training data, that make it difficult to demonstrate the required
behavior and make audits more challenging. This paper describes
an ML audit framework that evaluates and reviews the risks of
machine learning applications, the quality of the training data,
and the machine learning model. We evaluate and demonstrate
the functionality of the proposed framework by auditing an steel
plate fault prediction model.
Keywords—Audit, machine learning, assessment, metrics.
I. INTRODUCTION
MACHINE LEARNING (ML) and Artificial Intelligence
(AI) are increasingly used in many sectors of the
economy. Especially in areas such as automation, sensor
technology, assistance systems, predictive maintenance and
resource management. AI-based applications have been
implemented particularly successfully in these areas [1]. The
technology thus makes a decisive contribution to securing or
improving the respective market position. However, this is
accompanied by various risks based on ”black box” modeling
as a basic principle of AI. Conventional computer programs
process data by explicit instructions or commands defined
by software to solve a problem. In contrast, AI is based
on independent learning processes, creating system autonomy,
which leads to completely new approaches to problem solving.
The complexity of machine learning algorithms used makes it
difficult or currently impossible for data scientists to follow
the decisions made by the machine learning algorithm.
Therefore, there is no guarantee that an AI application
will always reliably deliver good results. However, this is a
requirement that must be met by an autonomous vehicle, for
example. A common question that arises in the area of critical
Jan Stodt is with the Institute of Data Science, Cloud Computing and IT
Security, Hochschule Furtwangen of Applied Science, 78120 Furtwangen,
Germany (phone: +497723920-2379; e-mail: jan.stodt@hs-furtwangen.de).
Christoph Reich is with the Institute of Data Science, Cloud
Computing and IT Security, Hochschule Furtwangen of Applied
Science, 78120 Furtwangen, Germany (e-mail: christoph.reich@hs-
furtwangen.de).
This work has received funding from INTERREG Upper Rhine
(European Regional Development Fund) and the Ministries for Research of
Baden-Wuerttemberg, RheinlandPfalz and from the Region Grand Est in
the framework of the Science Offensive Upper Rhine for the project
HALFBACK. The authors would like to thank the Master student Janik
Baur for doing programming and evaluation support.
AI applications is how to prove safety while at the same time
not knowing how the software will behave.
First of all, you need to know the risk, secondly you need to
know what the requirements are for your AI-based application,
and thirdly you need to have a development process that
guarantees that the requirements are met. In order to achieve
high quality of your AI application, so that the customer keeps
a high level of trust in the product and legal problems are
avoided, an audit framework is required to ensure that the AI
application meets the requirements.
This paper is structured as follows: Section II gives an
overview of the related work. Then the paper refers to the
ML-specific audit III and refines the ML assessment of data,
ML model and development process in IV. In Section V an
audit framework is introduced and evaluated in Section VI.
Finally, a conclusion is drawn in Section VII.
II. RELATED WORK
This section has been divided in a) ML testing frameworks
for predictive models, b) metrics for data quality, c) metrics
for ML models d) data and ML model fairness:
a) There is little work on testing predictive models. Most
of them focus on the quality assessment of predictive models,
which is an important part of an audit, but not all of it.
An overview on benchmarking machine learning devices and
software frameworks can be found in Wei et. al [2]. Nishi et
al. [3] developed a quality assurance framework for machine
learning products and their underlining model. The framework
consists of a set of metrics that define model quality, a
model evaluation strategy, and development lifecycle testing
to maintain model quality. Bhatt et al. [4] point out that an
audit framework should include evaluation of the conceptual
soundness of the model, monitoring, and benchmarking of the
model, and should provide result analysis. Zhang et al. [5] give
a comprehensive overview of the state of the art in the field
of machine learning testing. Workflows to be tested, metrics
and characteristics of machine learning are presented.
b) Auditing machine learning models requires a known
quality of the test data. Stewart et al. [6] show in detail the
impact of poor data quality on different machine learning
algorithms and their accuracy and performance. Schelter
et al. [7] provide an overview of process, models and
metrics for validating the quality of labeled data against
application-specific requirements. Barrett et al. [8] propose
World Academy of Science, Engineering and Technology
International Journal of Computer and Information Engineering
Vol:15, No:3, 2021
187
International Scholarly and Scientific Research & Innovation 15(3) 2021 ISNI:0000000091950263
Open
Science
Index,
Computer
and
Information
Engineering
Vol:15,
No:3,
2021
publications.waset.org/10011898/pdf
2. methods and metrics to evaluate and improve the quality of
hand labeled data using statistical approaches.
c) Regarding the metrics for evaluating the quality of
machine learning, Handelman et al. [9] provide an overview
and evaluation of metrics of machine learning models for
gaining insight into the machine learning model. Noise
sensitivity [10] or robustness [11], [12] are important metrics
in an audit.
d) Another recent research area related to audits is the
fairness of machine learning algorithms. Rick et. al [13]
investigated how well different forms of audit are suitable for
this purpose. Not only predictive models were considered, but
potentially all algorithms for decision making. The object of
an audit would be understood as a black box. In order to
achieve transparency, works of Waltl et. al [14] or Arrieta et.
al [15] where used, which try to explain the mechanisms of
the neural network within the black box.
III. MACHINE LEARNING SPECIFIC AUDIT
The audit process includes the following steps: a) Planning,
b) Definition of audit objectives and scope, c) Collection
and evaluation of evidence, d) Documentation and reporting.
Is necessary to understand that there is a trade-off between
costs and risks that management must accept. The nature
of ML-based applications differs from traditional software
in several features. Before a risk assessment for the audit
objectives and scope can be performed, some general
objectives must be described, which are determined by the
nature of ML-based applications.
A. General Objectives of Machine Learning Audit
Like any algorithm- and data-driven process, ML gives the
internal audit a clear role in ensuring accuracy and reliability.
ML can only function properly if it analyzes good data and
evaluates it against valid criteria - areas where internal audit
can have a positive impact. An audit is a formal review
of an item or process with the objective of examining the
enforcement of policies and guidelines to mitigate risk (see
next section III-B). The nature of ML-based applications
imposes some additional objectives:
• ML applications should be classified into different risk
classes. Depending on the risk, they can then be approved,
reviewed or even continuously monitored.
• The testing of ML systems with high or very high
risk should be carried out by independent testing
organizations. The risk-based approach is an established
procedure of the European Single Market to combine
security and innovation.
• A prerequisite for the manufacturer-independent testing
of algorithmic systems is access to the safety-relevant
data required for this purpose (e.g. data for driver
assistance systems in cars).
• Continuous verification is necessary for the learning of
ML systems, since variations of the data (newly collected
data) lead to new models.
• Besides the verification of ML models, verification data
is essential.
B. Risk Assessment for Audit Objectives and Scope
for Machine Learning based Application
There are ML opportunities and risks [16], which can
be divided into economic risks, such as the acceptance of
AI-based applications by the client, etc., and technical risks.
Technical risks are of utmost interest to be considered in an
audit process.
• Logical Error: Like any software, ML is subject to the
risk of logic and implementation errors. This can affect
the effectiveness of algorithms, which can lead to a
reduced quality of results and thus to massive impacts
on the application context.
• Human Factor and Biases: There is a risk that
unintentional human bias may be introduced into the
design of the AI. Due to the lack of knowledge of a
domain, data for the training of neural networks might
be missing, which reduces the quality of the result. The
results of neural networks are interpreted by humans and
should not be taken for granted (e.g. in cancer diagnosis).
• Data Quality: The quality of ML results depends on
the input data, therefore the input data quality is
crucial. Achieving data quality involves checking for
consistency, balance, robustness, accuracy, compatibility,
completeness, timeliness, and duplicate or corrupted data
sets. The training data must be representative of the real
data, in particular it must be ensured that the sampling
rate is appropriate. It must also be considered that the
data sets are noisy, have outliers, missing values and
duplicates.
For example, robustness means safe behavior despite
unexpected or abnormal input data [17]. It should be
ensured that the intelligent system containing an ML
model is safe to operate with acceptable risk. If the model,
in the example in Fig. 1, receives an unexpected image
(e.g., darker image) instead of the trained image of a
traffic situation, it must not attempt to recognize it as a
new traffic situation. The operating limits defined by the
data set used for the training must be taken into account.
Fig. 1 Error of Autonomous Vehicle, because of Brightness Changes [18]
• Model Quality: Modeling quality describes how well an
ML model reflects reality and thus fulfills its purpose. It
World Academy of Science, Engineering and Technology
International Journal of Computer and Information Engineering
Vol:15, No:3, 2021
188
International Scholarly and Scientific Research & Innovation 15(3) 2021 ISNI:0000000091950263
Open
Science
Index,
Computer
and
Information
Engineering
Vol:15,
No:3,
2021
publications.waset.org/10011898/pdf
3. considers how accurate the predictions of a predictive
model are. When testing the modeling quality, the
predictions of the predictive model for several test data
are compared with the correct values, the ground truth
[14]. As Fig. 2 shows, the comparison of the predictions
ŷ and the ground truth is y*. Ideally the result of the
black box is identical to the ground truth. ML model
metrics try to quantify the expected difference to the
ground truth. The model must be correct (correct ML
type and architecture) - otherwise it will never fit the
data well, no matter how long the training phase will be
or how good the data might be.
Fig. 2 Black Box Model Predicting the Ground Truth
• Cyber Security: Digital attackers get an additional attack
vector by using AI. The integrity of ML models could
be compromised, e.g., noise is added to an X-ray image,
thereby turning predictions from normal scan to abnormal
[19], or confidentiality could be compromised, e.g.,
private training data used to train the algorithm has
been restored and faces have been reconstructed [20].
Furthermore, the type of information processed (personal
/ sensitive company data) can be an additional motivation
for attacks.
• Compliance: National and international legislation is
not (yet) fully adapted to the use of AI. This leads
to partially unclear legal situations and legal risks for
the involved actors. At the same time, there are strict
regulations for parts of the AI, e.g. in connection with the
mass processing of personal data by the European Data
Protection Basic Regulation (DSGVO). Non-compliance
with the requirements in this sensitive area is sanctioned
with heavy fines.
IV. MACHINE LEARNING ASSESSMENT: DATA, MODEL,
DEVELOPMENT PROCESS
Data, ML models and the ML development process are most
important to be reviewed and therefore investigated in detail.
A. Data and ML-Model Inspection With Metrics
Fig. 3 shows a selection of metrics for regression and
classifier quality assessment metrics. This choice of metric has
an impact on the test result, since the different metrics focus
on different quality criteria. For example, both the MSE and
MMRE metrics calculate the deviation from the prediction to
the ground truth, but use different formulas. MSE calculates
the mean squared error, while MMRE calculates the mean
Fig. 3 Selection of Quality Metrics
relative error with respect to the correct value. There are also
compound metrics that try to combine several basic metrics. It
is possible that predictive models perform well with one metric
but worse with another [21]. Which metric is most appropriate
can only be determined in relation to the specific application.
B. ML Development Process Inspection
ML-engineering and -development is today carried out and
used in almost every organization to a different extent -
quite analogous to the general software development over the
years. However, ML software has unique features that clearly
distinguish it from traditional enterprise software development.
The ML development process to be investigated can be
described as shown in Fig. 4, the ML development process
under investigation can be divided into 3 tasks: the validation
of data, the validation of ML models, and the validation of
the ML application itself.
Fig. 4 Assessment Overview
The details to be considered are:
• Continuous Data Quality Check: The models are based
on historical data, while the data itself is constantly
changing. Data quality is often taken for granted without
having been properly tested and validated, and its quality
must be continuously reviewed. Therefore, data quality
must be continuously checked, data must be versioned,
continuously collected to represent the ground truth and
World Academy of Science, Engineering and Technology
International Journal of Computer and Information Engineering
Vol:15, No:3, 2021
189
International Scholarly and Scientific Research & Innovation 15(3) 2021 ISNI:0000000091950263
Open
Science
Index,
Computer
and
Information
Engineering
Vol:15,
No:3,
2021
publications.waset.org/10011898/pdf
4. correspond as closely as possible to the real world, and
data quality must be continuously monitored.
• Continuous Model Quality Check: ML models must be
subject to continuous quality checks. Versioning, model
provisioning and quality monitoring of the processes must
also be ensured. And regular model updates through
training and ensuring the same quality are important.
• ML DevOps - Continuous Delivery: The quality of
traditional software development must be ensured. In
addition, however, the integration effort of ML models
into the product must also be regulated. How quickly
changes to the ML models developed by data scientists
can be integrated into the product lifecycle is crucial for
the timeliness of an ML application. ML-DevOps must
therefore be supported.
Therefore a special ML software development lifecycle
(ML-SDLC) with its quality indicators should be reviewed by
the auditors.
V. MACHINE LEARNING AUDIT FRAMEWORK
First the specification of audits is discussed, followed by the
ML audit framework, its workflow description, and possible
implementation.
A. Machine Learning Audit Specification
To the best of our knowledge, there are currently no
machine-readable specification languages for machine learning
(see [22]). For continuous auditing and automation an
xml-based audit language for ML is required. An overview
and comparison of audit specification languages in general
and specifically for cloud computing was presented by
Doelitzscher et al. [23]. Based on the overview in relation
to the domain of machine learning, there are the data and
ML models that need to be quality checked by a number of
metrics, such as accuracy, completeness, timeliness, etc. The
ML development process is a manual task by the internal or
external auditor.
B. Machine Learning Audit Framework Architecture
The framework architecture consists of four modules, which
in turn consist of several sub-modules. Fig. 5 visualizes
the degree of abstraction of the modules by the vertical
arrangement. The topmost module, Audit, has the highest
degree of abstraction. This is where the administrative and
organizational processes take place, such as the audit process,
the definition of the goals of an audit. The Inspection module
covers the targeted activities of an auditor to demonstrate
compliance with guidelines or specifications. Resources are
entities that are required to perform an audit. The Toolbox
module does not include entities, but rather the tools and
actions from which an audit can be assembled.
In summary, the Audit module specifies the audit objectives
according to the assessment requirements of the objective, the
ML application. The more detailed circumstances of the audit
that affect the modules Audit and Resources are defined and
provide evidence to the auditor. The specifications determine
Audit
Inspection
Toolbox Resources
Define
Define
Utilizes Utilizes
Provides Evidence
Fig. 5 Audit Framework Architecture Overview
the available resources and the necessary activities during the
tests, so that the tools from the toolbox can be used to generate
evidence.
C. ML Audit Workflow
The workflow shown in Fig. 6 begins with specification, in
which the audit objectives, environment and constraints are
defined, and the workflow is described as follows:
Fig. 6 Audit Workflow
1) Criteria: Defines the criteria of the data and model
to be evaluated in the audit workflow. The criteria are
evaluated to ensure adequacy, Inaccuracies are clarified.
2) Resources: Orchestrates the resources required to
perform the audit tasks, taking into account the defined
specifications. Examples of resources are the ML model
and the data set.
3) Examination: If the specifications imply resource
requirements, the audit task analyzes the specifications,
criteria, and resources to ensure the validity of the
audit results. This task is essential because faulty
specifications, criteria, or resources would nullify and
invalidate the audit results. The audit task also analyzes
the structure of the model and resources in order to gain
World Academy of Science, Engineering and Technology
International Journal of Computer and Information Engineering
Vol:15, No:3, 2021
190
International Scholarly and Scientific Research & Innovation 15(3) 2021 ISNI:0000000091950263
Open
Science
Index,
Computer
and
Information
Engineering
Vol:15,
No:3,
2021
publications.waset.org/10011898/pdf
5. insights that are necessary to determine the procedure
defined in the next task. If the examination determines
that the resources are insufficient to determine the audit
procedure, the resource task is repeated, taking into
account the results of the examination.
4) Procedure: Defines the procedure of the audit itself
using the toolbox to provide the elements necessary
to create the audit procedure. The structure of the
procedure depends on the defined criteria.
5) Planing: Creates the order in which the audit is to
be carried out, taking into account dependencies and
resource coordination. A well defined plan ensures a
smooth and efficient process. If no plan can be created
due to conditions that cannot be met, the procedure must
be changed.
6) Preparation: Consists of a preparatory action step for
the final results and evidence. Examples would be the
change of resources for the verification, the correction
of defects. Depending on the test to be performed,
preparation may not be necessary.
7) Review: At this point of the audit workflow all elements
of the audit are defined. The results of the previous task
are combined to create an executable audit.
8) Validation: The created executable audit is validated to
ensure its validity, since errors in the audit would falsify
the audit results. Defects discovered are corrected at
their point of origin (criteria, resources or procedures).
9) Execution: The validated executable check is processed
and the result is the result of the check. In case of
non-compliance with a audit objective, there are several
possible procedures: a) in case of a negative verdict for
the model, the test could be aborted, b) jump directly to
the report, or c) continue the audit as planned. In case
of a positive verdict, the next task is continued.
10) Proof: The audit results are recorded and compared with
the requirements, and detailed evidence is generated.
11) Monitoring: All generated evidence is collected,
summarized and made available as a report. The
granularity of the evidence report depends on the level
of detail required.
D. Framework Automation and Implementation
The audit framework for ML models can automate many
audit steps described in the audit workflow to minimize the
audit effort and possible human error in the audit process. In
this work, executable audits were created manually in Python,
but evidence was generated, the audit and the audit report were
processed automatically.
The framework uses the well known and proven machine
learning frameworks keras [24] and tensorflow [25] to create
its extensive functionality for the audit process.
The implementation of the audit framework (modules,
structure and processes) was inspired by various existing
audit frameworks. The framework developed by Holland et
al. [26] to assign appropriate functions for providing data and
generating results for each data set label has been adopted for
use with the prediction model. The concept of augmenting
datasets with coverage-guided fuzzing by Xie et al. [27]
has been adopted for transforming a data set to generate an
additional data set for test purposes. Burton et al. [28] argued
that a predictive model fulfills a certain requirement if it
provides a certain performance in an environment specified for
a benchmark. The concept known as the security case pattern
has been adapted for use in the audit framework. The audit
framework was also developed by the architecture of Nishi
et al. [3], which describes potential structures, tests and test
types that can be tested within a machine learning product.
VI. EVALUATION
For the evaluation of the functionality of the described audit
ML framework the data set ”Steel Plates Faults”, which is
part of the machine learning repository of the University of
California, Irvine [29] was used. This data set contains 27
independent variables describing 7 types of steel plate failures.
In our application example, we create a machine learning
model to determine steel plate failure classes based on different
steel plate properties. The audit objective of this use case is to
determine the accuracy, robustness, and spurious relationship
of our machine learning model and the occurrence of each
defect class in the test data set. We define a set of rules for
each metric to prove the conformity or violation of the model
during the audit process.
Rule 1 Fault class proportion in test dataset
1: for each Class in fault classes do
2: if Class proportion in dataset ≤ 10% then
3: return False
4: end if
5: end for
6: return True
Rule 1 defines that the test data set meets our requirements
if each failure class has a share of more than 10% in the
data set. Execution of the audit task shows that our machine
learning model violates rule 1, as shown in Fig. 7.
To obtain meaningful and correct evidence in subsequent
audit steps, the proportion of defect classes in the test data set
is rebalanced by removing disproportionate test data from the
data set.
Rule 2 Model accuracy
if Accuracy ≥ 72% then
2: return True
else
4: return False
end if
Rule 2 defines that the model meets our requirements if the
accuracy is greater than or equal to 72%.
In the next step, the prediction of our model is compared
to the prediction with the ground truth, and the results
are quantified by the above mentioned metrics accuracy,
World Academy of Science, Engineering and Technology
International Journal of Computer and Information Engineering
Vol:15, No:3, 2021
191
International Scholarly and Scientific Research & Innovation 15(3) 2021 ISNI:0000000091950263
Open
Science
Index,
Computer
and
Information
Engineering
Vol:15,
No:3,
2021
publications.waset.org/10011898/pdf
6. TABLE I
AUDIT RESULTS OF THE STEEL PLATES FAULT CLASSIFICATION
Inspection Criteria Metric Value required Value demonstrated Assessment
Accuracy C1 Accuracy 0.72 0.7201 Compliant
Accuracy C1 F1-Score 0.7 0.732 Compliant
Robustness R1 F1-Score 0.67 0.711 Compliant
Stability R2 Neuron Coverage 0.85 0.85 Compliant
Stability R3 Top-2 NC 0.7 0.625 Not compliant
Spurious relationship S1 Plausibility rate 0.8 1 Compliant
P
a
s
t
r
y
Z
S
c
r
a
t
c
h
K
S
c
r
a
t
c
h
S
t
a
i
n
s
D
i
r
t
y
n
e
s
s
B
u
m
p
s
O
t
h
e
r
F
a
u
l
t
s
0.1
0.15
0.2
0.25
0.3
Fig. 7 Proportion of Fault Classes in the Test Data Set
robustness and spurious relationship. Auditing the robustness
of the model requires creating an additional data set by
transforming each record in the test dataset by replacing a
random feature value with the average value for the feature
over all records. Auditing the robustness also encompasses
evaluate the relevance of the neurons within the model by
determining the structural coverage metrics In the last step of
the audit, the inference of the model is evaluated to determine
whether a false relationship exists. The procedure consists of
creating a local and linear approximation of the real inference.
The final step is the audit report, which consists of a
summary of the audit task performed, as shown in Fig. 8,
and the results for the audit itself for each criterion, as shown
in Table I.
Fig. 8 Audit Report Showing the Audit Workflow
VII. CONCLUSION
An ML test frame with a corresponding workflow was
designed and described. The framework facilitates the
execution of audits for ML data, ML models and the ML
development process. It defines the relevant elements, puts
them into context and regulates the audit process. Due to the
heterogeneity of ML models, the audit framework is abstract
and not limited to specific model architectures or platforms.
The applicability of this audit concept was evaluated on the
basis of the use case: steel sheet defects.
REFERENCES
[1] “Künstliche intelligenz im mittelstand - relevanz, anwendungen,
transfer,” 2019, wissenschaftliches Institut für Infrastruktur und
Kommunikationsdienste.
[2] W. Dai and D. Berleant, “Benchmarking contemporary deep learning
hardware and frameworks: A survey of qualitative metrics,” in 2019
IEEE First International Conference on Cognitive Machine Intelligence
(CogMI), Dec 2019, pp. 148–155.
[3] Y. Nishi, S. Masuda, H. Ogawa, and K. Uetsuki, “A Test Architecture
for Machine Learning Product,” in 2018 IEEE International Conference
on Software Testing, Verification and Validation Workshops (ICSTW),
Apr. 2018, pp. 273–278.
[4] U. Bhatt, A. Xiang, S. Sharma, A. Weller, A. Taly, Y. Jia, J. Ghosh,
R. Puri, J. M. F. Moura, and P. Eckersley, “Explainable Machine
Learning in Deployment,” p. 10, 2020.
[5] J. M. Zhang, M. Harman, L. Ma, and Y. Liu, “Machine Learning Testing:
Survey, Landscapes and Horizons,” IEEE Transactions on Software
Engineering, pp. 1–1, 2020, conference Name: IEEE Transactions on
Software Engineering.
[6] E. Stewart, K. Chellappan, S. Backhaus, D. Deka, M. Reno, S. Peisert,
D. Arnold, C. Chen, A. Florita, and M. Buckner, “Integrated Multi
Scale Data Analytics and Machine Learning for the Grid; Benchmarking
Algorithms and Data Quality Analysis,” Lawrence Livermore National
Lab.(LLNL), Livermore, CA (United States), Tech. Rep., 2018.
[7] S. Schelter, D. Lange, P. Schmidt, M. Celikel, F. Biessmann,
and A. Grafberger, “Automating large-scale data quality
verification,” Proceedings of the VLDB Endowment, vol. 11,
no. 12, pp. 1781–1794, Aug. 2018. (Online). Available:
http://dl.acm.org/citation.cfm?doid=3229863.3275547
[8] L. Barrett and M. W. Sherman, “Improving ML Training Data with
Gold-Standard Quality Metrics,” p. 4, 2019.
[9] G. S. Handelman, H. K. Kok, R. V. Chandra, A. H. Razavi,
S. Huang, M. Brooks, M. J. Lee, and H. Asadi, “Peering Into
the Black Box of Artificial I ntelligence: E valuation M etrics of
Machine Learning Methods,” American Journal of Roentgenology,
vol. 212, no. 1, pp. 38–43, Jan. 2019. (Online). Available:
https://www.ajronline.org/doi/10.2214/AJR.18.20224
[10] D. Rolnick, A. Veit, S. Belongie, and N. Shavit, “Deep learning
is robust to massive label noise,” 2018. (Online). Available:
https://openreview.net/forum?id=B1p461b0W
[11] S. Moosavi-Dezfooli, A. Fawzi, and P. Frossard, “Deepfool:
a simple and accurate method to fool deep neural
networks,” CoRR, vol. abs/1511.04599, 2015. (Online).
Available: http://arxiv.org/abs/1511.04599
World Academy of Science, Engineering and Technology
International Journal of Computer and Information Engineering
Vol:15, No:3, 2021
192
International Scholarly and Scientific Research & Innovation 15(3) 2021 ISNI:0000000091950263
Open
Science
Index,
Computer
and
Information
Engineering
Vol:15,
No:3,
2021
publications.waset.org/10011898/pdf
7. [12] N. Carlini and D. Wagner, “Towards evaluating the robustness of neural
networks,” in 2017 IEEE Symposium on Security and Privacy (SP), May
2017, pp. 39–57.
[13] R. Salay and K. Czarnecki, “Using machine learning safely in
automotive software: An assessment and adaption of software process
requirements in ISO 26262,” CoRR, vol. abs/1808.01614, 2018.
[Online]. Available: http://arxiv.org/abs/1808.01614
[14] B. Waltl and R. Vogl, “Increasing transparency in algorithmic-
decision-making with explainable ai,” Datenschutz Datensicherheit -
DuD, vol. 42, pp. 613 –617, Sep. 2018.
[15] A. B. Arrieta, N. Dı́az-Rodrı́guez, J. D. Ser, A. Bennetot, S. Tabik,
A. Barbado, S. Garcı́a, S. Gil-López, D. Molina, R. Benjamins,
R. Chatila, and F. Herrera, “Explainable artificial intelligence (xai):
Concepts, taxonomies, opportunities and challenges toward responsible
ai,” 2019.
[16] “Global perspectivesand insightsartificial intelligence – considerations
for the profession of int ernal auditing,” The Institute of Internal
Auditors. [Online]. Available: https://na.theiia.org/periodicals/Public
Documents/GPI-Artificial-Intelligence.pdf
[17] C. Hutchison, M. Zizyte, P. E. Lanigan, D. Guttendorf, M. Wagner, C. Le
Goues, and P. Koopman, “Robustness testing of autonomy software,”
in 2018 IEEE/ACM 40th International Conference on Software
Engineering: Software Engineering in Practice Track (ICSE-SEIP),
2018, pp. 276–285.
[18] K. Pei, Y. Cao, J. Yang, and S. Jana, “Deepxplore,” Proceedings of the
26th Symposium on Operating Systems Principles - SOSP ’17, 2017.
[Online]. Available: http://dx.doi.org/10.1145/3132747.3132785
[19] M. Paschali, S. Conjeti, F. Navarro, and N. Navab, “Generalizability vs.
robustness: Adversarial examples for medical imaging,” 2018.
[20] M. Fredrikson, S. Jha, and T. Ristenpart, “Model inversion attacks
that exploit confidence information and basic countermeasures,” in
Proceedings of the 22nd ACM SIGSAC Conference on Computer and
Communications Security, Denver, CO, USA, October 12-16, 2015,
I. Ray, N. Li, and C. Kruegel, Eds. ACM, 2015, pp. 1322–1333.
[Online]. Available: https://doi.org/10.1145/2810103.2813677
[21] I. Myrtveit, E. Stensrud, and M. Shepperd, “Reliability and validity in
comparative studies of software prediction models,” IEEE Transactions
on Software Engineering, vol. 31, no. 5, pp. 380–391, 2005.
[22] B. R. Aditya, R. Ferdiana, and P. I. Santosa, “Toward modern it
audit- current issues and literature review,” in 2018 4th International
Conference on Science and Technology (ICST), 2018, pp. 1–6.
[23] F. Dölitzscher, T. Rübsamen, T. Karbe, M. Knahl, C. Reich, and
N. Clarke, “Sun behind clouds - on automatic cloud security audits
and a cloud audit policy language,” vol. 06.2013, no. 1 & 2, pp. 1 – 16,
2013.
[24] F. Chollet et al., “Keras,” https://keras.io, 2015.
[25] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S.
Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow,
A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser,
M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray,
C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar,
P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals,
P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng,
“TensorFlow: Large-scale machine learning on heterogeneous systems,”
2015, software available from tensorflow.org. [Online]. Available:
https://www.tensorflow.org/
[26] S. Holland, A. Hosny, S. Newman, J. Joseph, and K. Chmielinski, “The
Dataset Nutrition Label: A Framework To Drive Higher Data Quality
Standards,” arXiv:1805.03677 (cs), May 2018, arXiv: 1805.03677.
[Online]. Available: http://arxiv.org/abs/1805.03677
[27] X. Xie, L. Ma, F. Juefei-Xu, M. Xue, H. Chen, Y. Liu, J. Zhao,
B. Li, J. Yin, and S. See, “DeepHunter: a coverage-guided fuzz
testing framework for deep neural networks,” in Proceedings of the
28th ACM SIGSOFT International Symposium on Software Testing
and Analysis, ser. ISSTA 2019. Beijing, China: Association for
Computing Machinery, Jul. 2019, pp. 146–157. (Online).
Available: https://doi.org/10.1145/3293882.3330579
[28] S. Burton, L. Gauerhof, B. B. Sethy, I. Habli, and R. Hawkins,
“Confidence Arguments for Evidence of Performance in Machine
Learning for Highly Automated Driving Functions,” in International
Conference on Computer Safety, Reliability, and Security. Springer,
2019, pp. 365–377.
[29] “Steel plates faults data set.” (Online). Available:
https://archive.ics.uci.edu/ml/datasets/Steel Plates Faults
Jan Stodt is a member of the Institute for Data
Science, Cloud Computing and IT-security and a
member of the faculty of computer science at
the University of Applied Science in Furtwangen
(HFU). He received his B. Sc. degree in computer
science from the University of Applied Science in
Furtwangen (HFU) in 2017 and his M. Sc. degree
in computer science for the University of Applied
Science in Furtwangen (HFU) in 2019.
Christoph Reich is professor (since 2002) at
the faculty of computer science at the university
of applied science in Furtwangen (HFU) and
teaches in the field of network technologies,
IT protocols, IT security, Cloud Computing,
and Machine Learning. He is CIO of the
HFU Information- and Media Centre, that is
the scientific director for the IT data centre,
Online-Services, Learning-Services, and library
department. He is head of the Institute of
Data Science, Cloud Computing and IT-Security
(IDACUS; idacus.hs-furtwangen.de). Several founded research projects (FP7
EU, BMBF, MWK) have been accomplished successfully.
World Academy of Science, Engineering and Technology
International Journal of Computer and Information Engineering
Vol:15, No:3, 2021
193
International Scholarly and Scientific Research & Innovation 15(3) 2021 ISNI:0000000091950263
Open
Science
Index,
Computer
and
Information
Engineering
Vol:15,
No:3,
2021
publications.waset.org/10011898/pdf