This document provides background on Pooyan Jamshidi's research related to learning software performance models for dynamic and uncertain environments. It summarizes his past work developing techniques for modeling and optimizing performance across different systems and environments, including using transfer learning to reuse performance data from related sources to build more accurate models with fewer measurements. It also outlines opportunities for using transfer learning to adapt performance models to new environments and systems.
Narayanan Sundaram, Research Scientist, Intel Labs at MLconf SF - 11/13/15MLconf
GraphMat: Bridging the Productivity-Performance Gap in Graph Analytics: With increasing interest in large-scale distributed graph analytics for machine learning and data mining, more data scientists and developers are struggling to achieve high performance without sacrificing productivity on large graph problems. In this talk, I will discuss our solution to this problem: GraphMat. Using generalized sparse matrix-based primitives, we are able to achieve performance that is very close to hand-optimized native code, while allowing users to write programs using the familiar vertex-centric programming paradigm. I will show how we optimized GraphMat to achieve this performance on distributed platforms and provide programming examples. We have integrated GraphMat with Apache Spark in a manner that allows the combination to outperform all other distributed graph frameworks. I will explain the reasons for this performance and show that our approach achieves very high hardware efficiency in both single-node and distributed environments using primitives that are applicable to many machine learning and HPC problems. GraphMat is open source software and available for download.
Narayanan Sundaram, Research Scientist, Intel Labs at MLconf SF - 11/13/15MLconf
GraphMat: Bridging the Productivity-Performance Gap in Graph Analytics: With increasing interest in large-scale distributed graph analytics for machine learning and data mining, more data scientists and developers are struggling to achieve high performance without sacrificing productivity on large graph problems. In this talk, I will discuss our solution to this problem: GraphMat. Using generalized sparse matrix-based primitives, we are able to achieve performance that is very close to hand-optimized native code, while allowing users to write programs using the familiar vertex-centric programming paradigm. I will show how we optimized GraphMat to achieve this performance on distributed platforms and provide programming examples. We have integrated GraphMat with Apache Spark in a manner that allows the combination to outperform all other distributed graph frameworks. I will explain the reasons for this performance and show that our approach achieves very high hardware efficiency in both single-node and distributed environments using primitives that are applicable to many machine learning and HPC problems. GraphMat is open source software and available for download.
A Review on Scheduling in Cloud Computingijujournal
Cloud computing is the requirement based on clients that this computing which provides software,
infrastructure and platform as a service as per pay for use norm. The scheduling main goal is to achieve
the accuracy and correctness on task completion. The scheduling in cloud environment which enables the
various cloud services to help framework implementation. Thus the far reaching way of different type of
scheduling algorithms in cloud computing environment surveyed which includes the workflow scheduling
and grid scheduling. The survey gives an elaborate idea about grid, cloud, workflow scheduling to
minimize the energy cost, efficiency and throughput of the system.
Featuring a brief overview of fault-tolerant mechanisms across various Big Data systems such as Google File system (GFS), Amazon Dynamo, Bigtable, Hadoop - Map Reduce, Facebook Cassandra along with description of an existing fault tolerant model
Separating Hype from Reality in Deep Learning with Sameer FarooquiDatabricks
Deep Learning is all the rage these days, but where does the reality of what Deep Learning can do end and the media hype begin? In this talk, I will dispel common myths about Deep Learning that are not necessarily true and help you decide whether you should practically use Deep Learning in your software stack.
I’ll begin with a technical overview of common neural network architectures like CNNs, RNNs, GANs and their common use cases like computer vision, language understanding or unsupervised machine learning. Then I’ll separate the hype from reality around questions like:
• When should you prefer traditional ML systems like scikit learn or Spark.ML instead of Deep Learning?
• Do you no longer need to do careful feature extraction and standardization if using Deep Learning?
• Do you really need terabytes of data when training neural networks or can you ‘steal’ pre-trained lower layers from public models by using transfer learning?
• How do you decide which activation function (like ReLU, leaky ReLU, ELU, etc) or optimizer (like Momentum, AdaGrad, RMSProp, Adam, etc) to use in your neural network?
• Should you randomly initialize the weights in your network or use more advanced strategies like Xavier or He initialization?
• How easy is it to overfit/overtrain a neural network and what are the common techniques to ovoid overfitting (like l1/l2 regularization, dropout and early stopping)?
UnaCloud is an opportunistic based cloud infrastructure
(IaaS) that allows to access on-demand computing
capabilities using commodity desktops. Although UnaCloud
tried to maximize the use of idle resources to deploy virtual
machines on them, it does not use energy-efficient resource
allocation algorithms. In this paper, we design and implement
different energy-aware techniques to operate in an energyefficient
way and at the same time guarantee the performance
to the users. Performance tests with different algorithms and
scenarios using real trace workloads from UnaCloud, show how
different policies can change the energy consumption patterns
and reduce the energy consumption in opportunistic cloud
infrastructures. The results show that some algorithms can
reduce the energy-consumption power up to 30% over the
percentage earned by opportunistic environment.
Spine net learning scale permuted backbone for recognition and localizationDevansh16
Convolutional neural networks typically encode an input image into a series of intermediate features with decreasing resolutions. While this structure is suited to classification tasks, it does not perform well for tasks requiring simultaneous recognition and localization (e.g., object detection). The encoder-decoder architectures are proposed to resolve this by applying a decoder network onto a backbone model designed for classification tasks. In this paper, we argue encoder-decoder architecture is ineffective in generating strong multi-scale features because of the scale-decreased backbone. We propose SpineNet, a backbone with scale-permuted intermediate features and cross-scale connections that is learned on an object detection task by Neural Architecture Search. Using similar building blocks, SpineNet models outperform ResNet-FPN models by ~3% AP at various scales while using 10-20% fewer FLOPs. In particular, SpineNet-190 achieves 52.5% AP with a MaskR-CNN detector and achieves 52.1% AP with a RetinaNet detector on COCO for a single model without test-time augmentation, significantly outperforms prior art of detectors. SpineNet can transfer to classification tasks, achieving 5% top-1 accuracy improvement on a challenging iNaturalist fine-grained dataset. Code is at: this https URL.
Simulation of Heterogeneous Cloud InfrastructuresCloudLightning
During the last years, except from the traditional CPU based hardware servers, hardware accelerators are widely used in various HPC application areas. More specifically, Graphics Processing Units (GPUs), Many Integrated Cores (MICs) and Field-Programmable Gate Arrays (FPGAs) have shown a great potential in HPC and have been widely mobilised in supercomputing and in HPC-Clouds. This presentation focuses on the development of a cloud simulation framework that supports hardware accelerators. The design and implementation of the framework are also discussed.
This presentation was given by Dr. Konstantinos Giannoutakis (CERTH) at the CloudLightning Conference on 11th April 2017.
Slides for the hands on PyData workshop.
Cover three main topics:
- Current state of NLP models at Walmart
- Steps we took to optimize serving BERT
- how we serve models with Facebook’s TorchServe.
Corresponding repo for notebooks for handson:
https://bit.ly/pytorch-workshop-2021
Fault Tolerance in Big Data Processing Using Heartbeat Messages and Data Repl...IJSRD
Big data is a popular term used to define the exponential evolution and availability of data, includes both structured and unstructured data. The volatile progression of demands on big data processing imposes heavy burden on computation, communication and storage in geographically distributed data centers. Hence it is necessary to minimize the cost of big data processing, which also includes fault tolerance cost. Big Data processing involves two types of faults: node failure and data loss. Both the faults can be recovered using heartbeat messages. Here heartbeat messages acts as an acknowledgement messages between two servers. This paper depicts about the study of node failure and recovery, data replication and heartbeat messages.
A Review on Scheduling in Cloud Computingijujournal
Cloud computing is the requirement based on clients that this computing which provides software,
infrastructure and platform as a service as per pay for use norm. The scheduling main goal is to achieve
the accuracy and correctness on task completion. The scheduling in cloud environment which enables the
various cloud services to help framework implementation. Thus the far reaching way of different type of
scheduling algorithms in cloud computing environment surveyed which includes the workflow scheduling
and grid scheduling. The survey gives an elaborate idea about grid, cloud, workflow scheduling to
minimize the energy cost, efficiency and throughput of the system.
Featuring a brief overview of fault-tolerant mechanisms across various Big Data systems such as Google File system (GFS), Amazon Dynamo, Bigtable, Hadoop - Map Reduce, Facebook Cassandra along with description of an existing fault tolerant model
Separating Hype from Reality in Deep Learning with Sameer FarooquiDatabricks
Deep Learning is all the rage these days, but where does the reality of what Deep Learning can do end and the media hype begin? In this talk, I will dispel common myths about Deep Learning that are not necessarily true and help you decide whether you should practically use Deep Learning in your software stack.
I’ll begin with a technical overview of common neural network architectures like CNNs, RNNs, GANs and their common use cases like computer vision, language understanding or unsupervised machine learning. Then I’ll separate the hype from reality around questions like:
• When should you prefer traditional ML systems like scikit learn or Spark.ML instead of Deep Learning?
• Do you no longer need to do careful feature extraction and standardization if using Deep Learning?
• Do you really need terabytes of data when training neural networks or can you ‘steal’ pre-trained lower layers from public models by using transfer learning?
• How do you decide which activation function (like ReLU, leaky ReLU, ELU, etc) or optimizer (like Momentum, AdaGrad, RMSProp, Adam, etc) to use in your neural network?
• Should you randomly initialize the weights in your network or use more advanced strategies like Xavier or He initialization?
• How easy is it to overfit/overtrain a neural network and what are the common techniques to ovoid overfitting (like l1/l2 regularization, dropout and early stopping)?
UnaCloud is an opportunistic based cloud infrastructure
(IaaS) that allows to access on-demand computing
capabilities using commodity desktops. Although UnaCloud
tried to maximize the use of idle resources to deploy virtual
machines on them, it does not use energy-efficient resource
allocation algorithms. In this paper, we design and implement
different energy-aware techniques to operate in an energyefficient
way and at the same time guarantee the performance
to the users. Performance tests with different algorithms and
scenarios using real trace workloads from UnaCloud, show how
different policies can change the energy consumption patterns
and reduce the energy consumption in opportunistic cloud
infrastructures. The results show that some algorithms can
reduce the energy-consumption power up to 30% over the
percentage earned by opportunistic environment.
Spine net learning scale permuted backbone for recognition and localizationDevansh16
Convolutional neural networks typically encode an input image into a series of intermediate features with decreasing resolutions. While this structure is suited to classification tasks, it does not perform well for tasks requiring simultaneous recognition and localization (e.g., object detection). The encoder-decoder architectures are proposed to resolve this by applying a decoder network onto a backbone model designed for classification tasks. In this paper, we argue encoder-decoder architecture is ineffective in generating strong multi-scale features because of the scale-decreased backbone. We propose SpineNet, a backbone with scale-permuted intermediate features and cross-scale connections that is learned on an object detection task by Neural Architecture Search. Using similar building blocks, SpineNet models outperform ResNet-FPN models by ~3% AP at various scales while using 10-20% fewer FLOPs. In particular, SpineNet-190 achieves 52.5% AP with a MaskR-CNN detector and achieves 52.1% AP with a RetinaNet detector on COCO for a single model without test-time augmentation, significantly outperforms prior art of detectors. SpineNet can transfer to classification tasks, achieving 5% top-1 accuracy improvement on a challenging iNaturalist fine-grained dataset. Code is at: this https URL.
Simulation of Heterogeneous Cloud InfrastructuresCloudLightning
During the last years, except from the traditional CPU based hardware servers, hardware accelerators are widely used in various HPC application areas. More specifically, Graphics Processing Units (GPUs), Many Integrated Cores (MICs) and Field-Programmable Gate Arrays (FPGAs) have shown a great potential in HPC and have been widely mobilised in supercomputing and in HPC-Clouds. This presentation focuses on the development of a cloud simulation framework that supports hardware accelerators. The design and implementation of the framework are also discussed.
This presentation was given by Dr. Konstantinos Giannoutakis (CERTH) at the CloudLightning Conference on 11th April 2017.
Slides for the hands on PyData workshop.
Cover three main topics:
- Current state of NLP models at Walmart
- Steps we took to optimize serving BERT
- how we serve models with Facebook’s TorchServe.
Corresponding repo for notebooks for handson:
https://bit.ly/pytorch-workshop-2021
Fault Tolerance in Big Data Processing Using Heartbeat Messages and Data Repl...IJSRD
Big data is a popular term used to define the exponential evolution and availability of data, includes both structured and unstructured data. The volatile progression of demands on big data processing imposes heavy burden on computation, communication and storage in geographically distributed data centers. Hence it is necessary to minimize the cost of big data processing, which also includes fault tolerance cost. Big Data processing involves two types of faults: node failure and data loss. Both the faults can be recovered using heartbeat messages. Here heartbeat messages acts as an acknowledgement messages between two servers. This paper depicts about the study of node failure and recovery, data replication and heartbeat messages.
IT 8003 Cloud ComputingFor this activi.docxvrickens
IT 8003 Cloud Computing
For this activity you need to divide your class in groups
1
Group Activity 1 “SuperTAX Software”
2
SuperTax Overview
Did you know President Abraham Lincoln, one of America's most beloved leaders, also instituted one of its least liked obligations - the income tax? In this brief history of taxes, see the historical events which shaped income taxes in the United States today.
SuperTax is an American tax preparation software package developed in the mid-1980s.
SuperTax Corporation is headquartered in Mountain View, California.
2
Group Activity 1 “SuperTAX Software”
3
SuperTax Information
Desktop Software.
Support MS Windows and Mac OS.
Software method: CD/DVD media format.
Different versions:
SuperTAX Basic, Deluxe, Premier, and Home & Business.
Used by millions of users and organizations.
Group Activity 1 “SuperTAX Software”
4
SuperTAX Project
SuperTAX has hired your group as a consultant to move their Desktop Software to a Traditional IT Hosted Software, available Online.
Group Activity 1 “SuperTAX Software”
5
For Discussion:
Find the challenges that your team will encounter attempting to move SuperTAX Software to the new platform.
Prepared a presentation for the class.
On your Group you will need to define positions.
For example:
Project Manager, Senior Project Network, Senior Project Engineer, etc.
Group Activity 1 “SuperTAX Software”
6
Infrastructure
Software Development
Software Testing
Marketing & Business Model
Project Management
CHALLENGES
Group Activity 1 “SuperTAX Software”
7
Infrastructure
No more test in a single machine. (CD/DVD format model)
Test in a production cluster. (20, 30 users?)
A larger cluster can bring problems. (1000’s of users)
Testing must be done for different clients (mobile, desktops, OS)
Small performance bottleneck. Slow performance.
CHALLENGES
Group Activity 1 “SuperTAX Software”
8
Marketing & Business Model
One time fixed cost vs. subscription model
Before a CD was sold, now a subscription model.
Maintenance and replacement of cooling, power, and server is required
CHALLENGES
Group Activity 1 “SuperTAX Software”
9
Project Management
Project can take many months to years for Software Development cycle.
What model is appropriate for Hosted application. (Agile vs. waterfall)
Ability to try new features faster.
CHALLENGES
RUNNING HEAD: INTERSESSION 5 FINAL PROJECT PROJECTION 1
INTERSESSION 5 FINAL PROJECT PROJECTION 5
INTERSESSION 5 FINAL PROJECT PROJECTION
Shalini Kantamneni
Ottawa University
Intersession 5 Final Project Projection
The Design Process
This process involves the formulation of a model to be used in deriving a comprehensive cloud application. In this case, the model-view-controller design pattern will be used. This type of design pattern partitions the logic of the application into three distinct domains that are to be interconnected to provide a working cloud application (Jailia et al., 2016). ...
Model-driven Framework for Dynamic Deployment and Reconfiguration of Componen...Madjid KETFI
Model-driven Framework for Dynamic Deployment and Reconfiguration of Component-based Software Systems
M. Ketfi and N. Belkhatir
Published in the proceedings of the Metainformatics Symposium, MIS 2005, Esbjerg, Denmark, November 08-11, 2005 (20 pages).
DESIGN PATTERNS IN THE WORKFLOW IMPLEMENTATION OF MARINE RESEARCH GENERAL INF...AM Publications
This paper proposes the use of design patterns in a marine research general information platform. The development of the platform refers to a design of complicated system architecture. Creation and execution of the research workflow nodes and designing of visualization library suited for marine users play an important role in the whole software architecture. This paper studies the requirements characteristic in marine research fields and has implemented a series of framework to solve these problems based on object-oriented and design patterns techniques. These frameworks make clear the relationship in all directions between modules and layers of software, which communicate through unified abstract interface and reduce the coupling between modules and layers. The building of these frameworks is importantly significant in advancing the reusability of software and strengthening extensibility and maintainability of the system.
Dynamic Component Deployment and (Re) Configuration Using a Unified FrameworkMadjid KETFI
Dynamic Component Deployment and (Re) Configuration Using a Unified Framework
M. Ketfi and N. Belkhatir
Proceedings of the ISCA 18th International Conference on Computer Applications in Industry and Engineering, CAINE 2005, November 9-11, 2005, Honolulu, Hawaii, USA. ISBN 1-880843-57-9 (pages 85-90).
The Cloud computing paradigm emerged by establishing new resources provisioning and consumption models. Together with the improvement of resource management techniques, these models have contributed to an increase in the number of application developers that are strong supporters of partially or completely migrating their application to a highly scalable and pay-per-use infrastructure. In this paper we derive a set of functional and non-functional requirements and propose a process-based approach to support the optimal distribution of an application in the Cloud in order to handle fluctuating over time workloads. Using the TPC-H workload as the basis, and by means of empirical workload analysis and characterization, we evaluate the application persistence layer's performance under different deployment scenarios using generated workloads with particular behavior characteristics.
The Cloud computing paradigm emerged by establishing innovative resources provisioning and consumption models. Together with the improvement of resource management techniques, these models have contributed to an increase in the number of application developers that are strong supporters of partially or completely migrating their application to a highly scalable and pay-per-use infrastructure. However, due to the continuous growth of Cloud providers and Cloud offerings, Cloud application developers nowadays must face additional application design challenges related to the efficient selection of such offerings to optimally distribute the application in a Cloud infrastructure. Focusing on the performance aspects of the application, additional challenges arise, as application workloads fluctuate over time, and therefore produce a variation of the infrastructure resources demands. In this research work we aim to define and realize the underpinning concepts towards supporting the optimal (re-)distribution of an application in the Cloud in order to handle fluctuating over time workloads.
To Get any Project for CSE, IT ECE, EEE Contact Me @ 09666155510, 09849539085 or mail us - ieeefinalsemprojects@gmail.com-Visit Our Website: www.finalyearprojects.org
To Get any Project for CSE, IT ECE, EEE Contact Me @ 09666155510, 09849539085 or mail us - ieeefinalsemprojects@gmail.com-Visit Our Website: www.finalyearprojects.org
Learning LWF Chain Graphs: A Markov Blanket Discovery ApproachPooyan Jamshidi
LWF Chain graphs were introduced by Lauritzen, Wermuth, and Frydenberg as a generalization of graphical models based on undirected graphs and DAGs. From the causality point of view, in an LWF CG: Directed edges represent direct causal effects. Undirected edges represent causal effects due to interference, which occurs when an individual’s outcome is influenced by their social interaction with other population members, e.g., in situations that involve contagious agents, educational programs, or social networks. The construction of chain graph models is a challenging task that would be greatly facilitated by automation.
Markov blanket discovery has an important role in structure learning of Bayesian network. It is surprising, however, how little attention it has attracted in the context of learning LWF chain graphs. In this work, we provide a graphical characterization of Markov blankets in chain graphs. The characterization is different from the well-known one for Bayesian networks and generalizes it. We provide a novel scalable and sound algorithm for Markov blanket discovery in LWF chain graphs. We also provide a sound and scalable constraint-based framework for learning the structure of LWF CGs from faithful causally sufficient data. With the use of our algorithm, the problem of structure learning is reduced to finding an efficient algorithm for Markov blanket discovery in LWF chain graphs. This greatly simplifies the structure-learning task and makes a wide range of inference/learning problems computationally tractable because our approach exploits locality.
A Framework for Robust Control of Uncertainty in Self-Adaptive Software Conn...Pooyan Jamshidi
We enable reliable and dependable self‐adaptations of component connectors in unreliable environments with imperfect monitoring facilities and conflicting user opinions about adaptation policies by developing a framework which comprises: (a) mechanisms for robust model evolution, (b) a method for adaptation reasoning, and (c) tool support that allows an end‐to‐end application of the developed techniques in real‐world domains.
Machine Learning Meets Quantitative Planning: Enabling Self-Adaptation in Aut...Pooyan Jamshidi
Modern cyber-physical systems (e.g., robotics systems) are typically composed of physical and software components, the characteristics of which are likely to change over time. Assumptions about parts of the system made at design time may not hold at run time, especially when a system is deployed for long periods (e.g., over decades). Self-adaptation is designed to find reconfigurations of systems to handle such run-time inconsistencies. Planners can be used to find and enact optimal reconfigurations in such an evolving context. However, for systems that are highly configurable, such planning becomes intractable due to the size of the adaptation space. To overcome this challenge, in this paper we explore an approach that (a) uses machine learning to find Pareto-optimal configurations without needing to explore every configuration and (b) restricts the search space to such configurations to make planning tractable. We explore this in the context of robot missions that need to consider task timeliness and energy consumption. An independent evaluation shows that our approach results in high-quality adaptation plans in uncertain and adversarial environments.
Paper: https://arxiv.org/abs/1903.03920
Ensembles of Many Diverse Weak Defenses can be Strong: Defending Deep Neural ...Pooyan Jamshidi
Despite achieving state-of-the-art performance across many domains, machine learning systems are highly vulnerable to subtle adversarial perturbations. Although defense approaches have been proposed in recent years, many have been bypassed by even weak adversarial attacks. Previous studies showed that ensembles created by combining multiple weak defenses (i.e., input data transformations) are still weak. In this talk, I will show that it is indeed possible to construct effective ensembles using weak defenses to block adversarial attacks. However, to do so requires a diverse set of such weak defenses. Based on this motivation, I will present Athena, an extensible framework for building effective defenses to adversarial attacks against machine learning systems. I will talk about the effectiveness of ensemble strategies with a diverse set of many weak defenses that comprise transforming the inputs (e.g., rotation, shifting, noising, denoising, and many more) before feeding them to target deep neural network classifiers. I will also discuss the effectiveness of the ensembles with adversarial examples generated by various adversaries in different threat models. In the second half of the talk, I will explain why building defenses based on the idea of many diverse weak defenses works, when it is most effective, and what its inherent limitations and overhead are.
Transfer Learning for Performance Analysis of Configurable Systems:A Causal ...Pooyan Jamshidi
Modern systems (e.g., deep neural networks, big data analytics, and compilers) are highly configurable, which means they expose different performance behavior under different configurations. The fundamental challenge is that one cannot simply measure all configurations due to the sheer size of the configuration space. Transfer learning has been used to reduce the measurement efforts by transferring knowledge about performance behavior of systems across environments. Previously, research has shown that statistical models are indeed transferable across environments. In this work, we investigate identifiability and transportability of causal effects and statistical relations in highly-configurable systems. Our causal analysis agrees with previous exploratory analysis~\cite{Jamshidi17} and confirms that the causal effects of configuration options can be carried over across environments with high confidence. We expect that the ability to carry over causal relations will enable effective performance analysis of highly-configurable systems.
Integrated Model Discovery and Self-Adaptation of RobotsPooyan Jamshidi
Machine learn models efficiently under budget constraints to adapt to perturbations such as environmental changes or changes in the internal resources.
Modern software-intensive systems are composed of components that are likely to change their behaviour over time (e.g., adding/removing components).
For software to continue to operate under such changes, the assumptions about parts of the system made at design time may not hold at runtime due to uncertainty.
Mechanisms must be put in place that can dynamically learn new models of these assumptions and use them to make decisions about missions, configurations, etc.
Transfer Learning for Performance Analysis of Highly-Configurable SoftwarePooyan Jamshidi
A wide range of modern software-intensive systems (e.g., autonomous systems, big data analytics, robotics, deep neural architectures) are built configurable. These systems offer a rich space for adaptation to different domains and tasks. Developers and users often need to reason about the performance of such systems, making tradeoffs to change specific quality attributes or detecting performance anomalies. For instance, developers of image recognition mobile apps are not only interested in learning which deep neural architectures are accurate enough to classify their images correctly, but also which architectures consume the least power on the mobile devices on which they are deployed. Recent research has focused on models built from performance measurements obtained by instrumenting the system. However, the fundamental problem is that the learning techniques for building a reliable performance model do not scale well, simply because the configuration space is exponentially large that is impossible to exhaustively explore. For example, it will take over 60 years to explore the whole configuration space of a system with 25 binary options.
In this talk, I will start motivating the configuration space explosion problem based on my previous experience with large-scale big data systems in industry. I will then present my transfer learning solution to tackle the scalability challenge: instead of taking the measurements from the real system, we learn the performance model using samples from cheap sources, such as simulators that approximate the performance of the real system, with a fair fidelity and at a low cost. Results show that despite the high cost of measurement on the real system, learning performance models can become surprisingly cheap as long as certain properties are reused across environments. In the second half of the talk, I will present empirical evidence, which lays a foundation for a theory explaining why and when transfer learning works by showing the similarities of performance behavior across environments. I will present observations of environmental changes‘ impacts (such as changes to hardware, workload, and software versions) for a selected set of configurable systems from different domains to identify the key elements that can be exploited for transfer learning. These observations demonstrate a promising path for building efficient, reliable, and dependable software systems. Finally, I will share my research vision for the next five years and outline my immediate plans to further explore the opportunities of transfer learning.
Related Papers:
https://arxiv.org/pdf/1709.02280
https://arxiv.org/pdf/1704.00234
https://arxiv.org/pdf/1606.06543
Architectural Tradeoff in Learning-Based SoftwarePooyan Jamshidi
In classical software development, developers write explicit instructions in a programming language to hardcode the explicit behavior of software systems. By writing each line of code, the programmer instructs the software to have the desirable behavior by exploring a specific point in program space.
Recently, however, software systems are adding learning components that, instead of hardcoding an explicit behavior, learn a behavior through data. The learning-intensive software systems are written in terms of models and their parameters that need to be adjusted based on data. In learning-enabled systems, we specify some constraints on the behavior of a desirable program (e.g., a data set of input–output pairs of examples) and use the computational resources to search through the program space to find a program that satisfies the constraints. In neural networks, we restrict the search to a continuous subset of the program space.
This talk provides experimental evidence of making tradeoffs for deep neural network models, using the Deep Neural Network Architecture system as a case study. Concrete experimental results are presented; also featured are additional case studies in big data (Storm, Cassandra), data analytics (configurable boosting algorithms), and robotics applications.
Sensitivity Analysis for Building Adaptive Robotic SoftwarePooyan Jamshidi
P. Jamshidi, M. Velez, C. Kästner, N. Siegmund, and P. Kawthekar. Transfer learning for improving model predictions in highly configurable software. Int’l Symp. Software Engineering for Adaptive and Self-Managing Systems (SEAMS), 2017.
Transfer Learning for Improving Model Predictions in Highly Configurable Soft...Pooyan Jamshidi
Modern software systems are now being built to be used in dynamic environments utilizing configuration capabilities to adapt to changes and external uncertainties. In a self-adaptation context, we are often interested in reasoning about the performance of the systems under different configurations. Usually, we learn a black-box model based on real measurements to predict the performance of the system given a specific configuration. However, as modern systems become more complex, there are many configuration parameters that may interact and, therefore, we end up learning an exponentially large configuration space. Naturally, this does not scale when relying on real measurements in the actual changing environment. We propose a different solution: Instead of taking the measurements from the real system, we learn the model using samples from other sources, such as simulators that approximate performance of the real system at low cost.
Transfer Learning for Improving Model Predictions in Robotic SystemsPooyan Jamshidi
Modern software systems are now being built to be used in dynamic environments utilizing configuration capabilities to adapt to changes and external uncertainties. In a self-adaptation context, we are often interested in reasoning about the performance of the systems under different configurations. Usually, we learn a black-box model based on real measurements to predict the performance of the system given a specific configuration. However, as modern systems become more complex, there are many configuration parameters that may interact and, therefore, we end up learning an exponentially large configuration space. Naturally, this does not scale when relying on real measurements in the actual changing environment. We propose a different solution: Instead of taking the measurements from the real system, we learn the model using samples from other sources, such as simulators that approximate performance of the real system at low cost.
An Uncertainty-Aware Approach to Optimal Configuration of Stream Processing S...Pooyan Jamshidi
https://arxiv.org/abs/1606.06543
Finding optimal configurations for Stream Processing Systems (SPS) is a challenging problem due to the large number of parameters that can influence their performance and the lack of analytical models to anticipate the effect of a change. To tackle this issue, we consider tuning methods where an experimenter is given a limited budget of experiments and needs to carefully allocate this budget to find optimal configurations. We propose in this setting Bayesian Optimization for Configuration Optimization (BO4CO), an auto-tuning algorithm that leverages Gaussian Processes (GPs) to iteratively capture posterior distributions of the configuration spaces and sequentially drive the experimentation. Validation based on Apache Storm demonstrates that our approach locates optimal configurations within a limited experimental budget, with an improvement of SPS performance typically of at least an order of magnitude compared to existing configuration algorithms.
Pushing the limits of ePRTC: 100ns holdover for 100 daysAdtran
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdfPeter Spielvogel
Building better applications for business users with SAP Fiori.
• What is SAP Fiori and why it matters to you
• How a better user experience drives measurable business benefits
• How to get started with SAP Fiori today
• How SAP Fiori elements accelerates application development
• How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities
• How SAP Fiori paves the way for using AI in SAP apps
Enhancing Performance with Globus and the Science DMZGlobus
ESnet has led the way in helping national facilities—and many other institutions in the research community—configure Science DMZs and troubleshoot network issues to maximize data transfer performance. In this talk we will present a summary of approaches and tips for getting the most out of your network infrastructure using Globus Connect Server.
Le nuove frontiere dell'AI nell'RPA con UiPath Autopilot™UiPathCommunity
In questo evento online gratuito, organizzato dalla Community Italiana di UiPath, potrai esplorare le nuove funzionalità di Autopilot, il tool che integra l'Intelligenza Artificiale nei processi di sviluppo e utilizzo delle Automazioni.
📕 Vedremo insieme alcuni esempi dell'utilizzo di Autopilot in diversi tool della Suite UiPath:
Autopilot per Studio Web
Autopilot per Studio
Autopilot per Apps
Clipboard AI
GenAI applicata alla Document Understanding
👨🏫👨💻 Speakers:
Stefano Negro, UiPath MVPx3, RPA Tech Lead @ BSP Consultant
Flavio Martinelli, UiPath MVP 2023, Technical Account Manager @UiPath
Andrei Tasca, RPA Solutions Team Lead @NTT Data
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
The Metaverse and AI: how can decision-makers harness the Metaverse for their...Jen Stirrup
The Metaverse is popularized in science fiction, and now it is becoming closer to being a part of our daily lives through the use of social media and shopping companies. How can businesses survive in a world where Artificial Intelligence is becoming the present as well as the future of technology, and how does the Metaverse fit into business strategy when futurist ideas are developing into reality at accelerated rates? How do we do this when our data isn't up to scratch? How can we move towards success with our data so we are set up for the Metaverse when it arrives?
How can you help your company evolve, adapt, and succeed using Artificial Intelligence and the Metaverse to stay ahead of the competition? What are the potential issues, complications, and benefits that these technologies could bring to us and our organizations? In this session, Jen Stirrup will explain how to start thinking about these technologies as an organisation.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
2. My background
5
RobusT2Scale/FQL4KE [PhD]
✓ Engineering / Technical
✓ Auto-scaling in cloud
(RobusT2Scale)
✓ Self-learning controller for
cloud auto-scaling (FQL4KE)
BO4CO/TL4CO [Postdoc1@Imperial]
✓ Mathematical modeling
✓ Configuration optimization for big
data (BO4CO)
✓ Performance-aware DevOps
(TL4CO)
Transfer Learning [Postdoc2@CMU]
✓ Empirical
✓ Learns accurate and reliable models
from “related” sources
✓ Reuse learning across
environmental changes
Software industry [2003-2010]: Pre-PhD
Close collaborations with Intel and Microsoft [PhD]
3 EU projects: MODAClouds (cloud), DICE (big data), Human Brain (clustering) [Postdoc1@Imperial]
1 DARPA project: BRASS (Robotics) [Postdoc2@CMU]
4. Cloud applications
7
* JupiterResearch ** Amazon ***Google
• 82% of end-users give up on a lost payment transaction*
• 25% of end-users leave if load time > 4s**
• 1% reduced sale per 100ms load time**
• 20% reduced income if 0.5s longer load time***
flash-crowds
failures
capacity
shortage
slow
application
[Credit to Cristian Klein, Brownout]
6. Common characteristics of the systems
• Moderns systems are
increasingly configurable
• Modern systems are
deployed in dynamic and
uncertain environments
• Modern systems can be
adapted on the fly
9
Hey, You Have Given Me Too Many Knobs!
Understanding and Dealing with Over-Designed Configuration in System Software
Tianyin Xu*, Long Jin*, Xuepeng Fan*‡, Yuanyuan Zhou*,
Shankar Pasupathy† and Rukma Talwadker†
*University of California San Diego, ‡Huazhong Univ. of Science & Technology, †NetApp, Inc
{tixu, longjin, xuf001, yyzhou}@cs.ucsd.edu
{Shankar.Pasupathy, Rukma.Talwadker}@netapp.com
ABSTRACT
Configuration problems are not only prevalent, but also severely
mpair the reliability of today’s system software. One fundamental
eason is the ever-increasing complexity of configuration, reflected
y the large number of configuration parameters (“knobs”). With
undreds of knobs, configuring system software to ensure high re-
ability and performance becomes a daunting, error-prone task.
This paper makes a first step in understanding a fundamental
uestion of configuration design: “do users really need so many
nobs?” To provide the quantitatively answer, we study the con-
guration settings of real-world users, including thousands of cus-
omers of a commercial storage system (Storage-A), and hundreds
f users of two widely-used open-source system software projects.
Our study reveals a series of interesting findings to motivate soft-
ware architects and developers to be more cautious and disciplined
n configuration design. Motivated by these findings, we provide
few concrete, practical guidelines which can significantly reduce
he configuration space. Take Storage-A as an example, the guide-
nes can remove 51.9% of its parameters and simplify 19.7% of
he remaining ones with little impact on existing users. Also, we
tudy the existing configuration navigation methods in the context
f “too many knobs” to understand their effectiveness in dealing
with the over-designed configuration, and to provide practices for
uilding navigation support in system software.
Categories and Subject Descriptors: D.2.10 [Software Engineer-
7/2006 7/2008 7/2010 7/2012 7/2014
0
100
200
300
400
500
600
700
Storage-A
Numberofparameters
Release time
1/1999 1/2003 1/2007 1/2011
0
100
200
300
400
500
5.6.2
5.5.0
5.0.16
5.1.3
4.1.0
4.0.12
3.23.0
1/2014
MySQL
Numberofparameters
Release time
1/1998 1/2002 1/2006 1/2010 1/2014
0
100
200
300
400
500
600
1.3.14
2.2.14
2.3.4
2.0.35
1.3.24
Numberofparameters
Release time
Apache
1/2006 1/2008 1/2010 1/2012 1/2014
0
40
80
120
160
200
2.0.0
1.0.0
0.19.0
0.1.0
Hadoop
Numberofparameters
Release time
MapReduce
HDFS
Figure 1: The increasing number of configuration parameters with
software evolution. Storage-A is a commercial storage system from a ma-
jor storage company in the U.S.
all the customer-support cases in a major storage company in the
U.S., and were the most significant contributor (31%) among all
[Credit to Tianyin Xu, Too Many Knobs]
11. An Example of Auto-scaling Rule These values are
required to be
determined by users
Þ requires deep
knowledge of
application (CPU,
memory)
Þ requires
performance
modeling expertise
(how to scale)
Þ A unified opinion
of user(s) is
required
Amazon auto scaling
Microsoft Azure Watch
14
Microsoft Azure Auto-
scaling Application Block
36. GP for modeling black box response function
true function
GP mean
GP variance
observation
selected point
true
minimum
making. The other reason is that all the computations in
this framework are based on tractable linear algebra.
In our previous work [21], we proposed BO4CO that ex-
ploits single-task GPs (no transfer learning) for prediction of
posterior distribution of response functions. A GP model is
composed by its prior mean (µ(·) : X ! R) and a covariance
unction (k(·, ·) : X ⇥ X ! R) [41]:
y = f(x) ⇠ GP(µ(x), k(x, x0
)), (2)
where covariance k(x, x0
) defines the distance between x
and x0
. Let us assume S1:t = {(x1:t, y1:t)|yi := f(xi)} be
the collection of t experimental data (observations). In this
ramework, we treat f(x) as a random variable, conditioned
on observations S1:t, which is normally distributed with the
ollowing posterior mean and variance functions [41]:
µt(x) = µ(x) + k(x)|
(K + 2
I) 1
(y µ) (3)
2
t (x) = k(x, x) + 2
I k(x)|
(K + 2
I) 1
k(x) (4)
where y := y1:t, k(x)|
= [k(x, x1) k(x, x2) . . . k(x, xt)],
µ := µ(x1:t), K := k(xi, xj) and I is identity matrix. The
shortcoming of BO4CO is that it cannot exploit the observa-
tions regarding other versions of the system and as therefore
cannot be applied in DevOps.
3.2 TL4CO: an extension to multi-tasks
TL4CO 1
uses MTGPs that exploit observations from other
previous versions of the system under test. Algorithm 1
defines the internal details of TL4CO. As Figure 4 shows,
TL4CO is an iterative algorithm that uses the learning from
other system versions. In a high-level overview, TL4CO: (i)
selects the most informative past observations (details in
Section 3.3); (ii) fits a model to existing data based on kernel
earning (details in Section 3.4), and (iii) selects the next
configuration based on the model (details in Section 3.5).
In the multi-task framework, we use historical data to fit a
better GP providing more accurate predictions. Before that,
we measure few sample points based on Latin Hypercube De-
n Optimization here is that it o↵ers a framework
asoning can be not only based on mean estimates
he variance, providing more informative decision
The other reason is that all the computations in
work are based on tractable linear algebra.
revious work [21], we proposed BO4CO that ex-
e-task GPs (no transfer learning) for prediction of
istribution of response functions. A GP model is
by its prior mean (µ(·) : X ! R) and a covariance
k(·, ·) : X ⇥ X ! R) [41]:
y = f(x) ⇠ GP(µ(x), k(x, x0
)), (2)
ariance k(x, x0
) defines the distance between x
et us assume S1:t = {(x1:t, y1:t)|yi := f(xi)} be
on of t experimental data (observations). In this
, we treat f(x) as a random variable, conditioned
tions S1:t, which is normally distributed with the
posterior mean and variance functions [41]:
= µ(x) + k(x)|
(K + 2
I) 1
(y µ) (3)
= k(x, x) + 2
I k(x)|
(K + 2
I) 1
k(x) (4)
y1:t, k(x)|
= [k(x, x1) k(x, x2) . . . k(x, xt)],
t), K := k(xi, xj) and I is identity matrix. The
g of BO4CO is that it cannot exploit the observa-
ding other versions of the system and as therefore
applied in DevOps.
4CO: an extension to multi-tasks
uses MTGPs that exploit observations from other
ersions of the system under test. Algorithm 1
e internal details of TL4CO. As Figure 4 shows,
an iterative algorithm that uses the learning from
em versions. In a high-level overview, TL4CO: (i)
Motivations:
1- mean estimates + variance
2- all computations are linear algebra
3- good estimations when few data
k(x)|
= [k(x, x1) k(x, x2) . . . k(x, xt)], I is
y matrix and
K :=
2
6
4
k(x1, x1) . . . k(x1, xt)
...
...
...
k(xt, x1) . . . k(xt, xt)
3
7
5 (7)
models have shown to be effective for performance
ions in data scarce domains [20]. However, as we
demonstrated in Figure 2, it may becomes inaccurate
the samples do not cover the space uniformly. For
configurable systems, we require a large number of
ations to cover the space uniformly, making GP models
tive in such situations.
del prediction using transfer learning
ansfer learning, the key question is how to make accu-
edictions for the target environment using observations
ther sources, Ds. We need a measure of relatedness not
etween input configurations but between the sources
l. The relationships between input configurations was
ed in the GP models using the covariance matrix that
efined based on the kernel function in Eq. (7). More
cally, a kernel is a function that computes a dot product
sure of “similarity”) between two input configurations.
e kernel helps to get accurate predictions for similar
urations. We now need to exploit the relationship be-
the source and target functions, g, f, using the current
ations Ds, Dt to build the predictive model ˆf. To capture
ationship, we define the following kernel function:
k(f, g, x, x0
) = kt(f, g) ⇥ kxx(x, x0
), (8)
the kernels kt represent the correlation between source
rget function, while kxx is the covariance function for
Typically, kxx is parameterized and its parameters are
by maximizing the marginal likelihood of the model
An overview of a self-optimization solution is depicted
in Figure 3 following the well-know MAPE-K framework
[9], [23]. We consider the GP model as the K (knowledge)
component of this framework that acts as an interface to which
other components can query the performance under specific
configurations or update the model given a new observation.
We use transfer learning to make the knowledge more accurate
using observations that are taken from a simulator or any
other cheap sources. For deciding how many observations
and from what source to transfer, we use the cost model
that we have introduced earlier. At runtime, the managed
system is Monitored by pulling the end-to-end performance
metrics (e.g., latency, throughput) from the corresponding
sensors. Then, the retrieved performance data needs to be
Analysed and the mean performance associated to a specific
setting of the system will be stored in a data repository.
Next, the GP model needs to be updated taking into account
the new performance observation. Having updated the GP
model, a new configuration may be Planned to replace the
current configuration. Finally, the new configuration will be
enacted by Executing appropriate platform specific operations.
This enables model-based knowledge evolution using machine
learning [2], [21]. The underlying GP model can now be
updated not only when a new observation is available but
also by transferring the learning from other related sources.
So at each adaptation cycle, we can update our belief about
the correct response given data from the managed system and
other related sources, accelerating the learning process.
IV. EXPERIMENTAL RESULTS
We evaluate the effectiveness and applicability of our
transfer learning approach for learning models for highly-
configurable systems, in particular, compared to conventional
non-transfer learning. Specifically, we aim to answer the
following three research questions:
RQ1: How much does transfer learning improve the prediction
ernal details about the system; the learning process can
ied in a black-box fashion using the sampled perfor-
measurements. In the GP framework, it is also possible
rporate domain knowledge as prior, if available, which
hance the model accuracy [20].
rder to describe the technical details of our transfer
g methodology, let us briefly describe an overview of
del regression; a more detailed description can be found
ere [35]. GP models assume that the function ˆf(x) can
rpreted as a probability distribution over functions:
y = ˆf(x) ⇠ GP(µ(x), k(x, x0
)), (4)
µ : X ! R is the mean function and k : X ⇥ X ! R
covariance function (kernel function) which describes
tionship between response values, y, according to the
e of the input values x, x0
. The mean and variance of
model predictions can be derived analytically [35]:
x) = µ(x) + k(x)|
(K + 2
I) 1
(y µ), (5)
x) = k(x, x) + 2
I k(x)|
(K + 2
I) 1
k(x), (6)
k(x)|
= [k(x, x1) k(x, x2) . . . k(x, xt)], I is
matrix and
K :=
2
6
4
k(x1, x1) . . . k(x1, xt)
...
...
...
k(xt, x1) . . . k(xt, xt)
3
7
5 (7)
models have shown to be effective for performance
ons in data scarce domains [20]. However, as we
emonstrated in Figure 2, it may becomes inaccurate
he samples do not cover the space uniformly. For
configurable systems, we require a large number of
tions to cover the space uniformly, making GP models
a standard method [35]. After learning the parameters of kxx,
we construct the covariance matrix exactly the same way as in
Eq. 7 and derive the mean and variance of predictions using
Eq. (5), (6) with the new K. The main essence of transfer
learning is, therefore, the kernel that capture the source and
target relationship and provide more accurate predictions using
the additional knowledge we can gain via the relationship
between source and target.
D. Transfer learning in a self-adaptation loop
Now that we have described the idea of transfer learning for
providing more accurate predictions, the question is whether
such an idea can be applied at runtime and how the self-
adaptive systems can benefit from it. More specifically, we
now describe the idea of model learning and transfer learning
in the context of self-optimization, where the system adapts
its configuration to meet performance requirements at runtime.
The difference to traditional configurable systems is that we
learn the performance model online in a feedback loop under
time and resource constraints. Such performance reasoning is
done more frequently for self-adaptation purposes.
An overview of a self-optimization solution is depicted
in Figure 3 following the well-know MAPE-K framework
[9], [23]. We consider the GP model as the K (knowledge)
component of this framework that acts as an interface to which
other components can query the performance under specific
configurations or update the model given a new observation.
We use transfer learning to make the knowledge more accurate
using observations that are taken from a simulator or any
other cheap sources. For deciding how many observations
and from what source to transfer, we use the cost model
that we have introduced earlier. At runtime, the managed
system is Monitored by pulling the end-to-end performance
44. -1.5 -1 -0.5 0 0.5 1 1.5
-4
-3
-2
-1
0
1
2
3
(a) 3 sample response functions
configuration domain
responsevalue
(1)
(2)
(3)
observations
(b) GP fit for (1) ignoring observations for (2),(3)
LCB
not informative
(c) multi-task GP fit for (1) by transfer learning from (2),(3)
highly informative
GP prediction mean
GP prediction variance
probability distribution
of the minimizers
Transfer learning improves optimization
65. Transferable knowledge
71
Target (Learn)Source (Given)
DataModel
Transferable
Knowledge
II. INTUITION
Understanding the performance behavior of configurable
oftware systems can enable (i) performance debugging, (ii)
erformance tuning, (iii) design-time evolution, or (iv) runtime
daptation [11]. We lack empirical understanding of how the
erformance behavior of a system will vary when the environ-
ment of the system changes. Such empirical understanding will
rovide important insights to develop faster and more accurate
earning techniques that allow us to make predictions and
ptimizations of performance for highly configurable systems
n changing environments [10]. For instance, we can learn
erformance behavior of a system on a cheap hardware in a
ontrolled lab environment and use that to understand the per-
ormance behavior of the system on a production server before
hipping to the end user. More specifically, we would like to
now, what the relationship is between the performance of a
ystem in a specific environment (characterized by software
onfiguration, hardware, workload, and system version) to the
ne that we vary its environmental conditions.
In this research, we aim for an empirical understanding of
A. Preliminary concepts
In this section, we prov
cepts that we use through
enable us to concisely con
1) Configuration and e
the i-th feature of a confi
enabled or disabled and o
configuration space is ma
all the features C = Do
Dom(Fi) = {0, 1}. A
a member of the configu
all the parameters are as
range (i.e., complete instan
We also describe an env
e = [w, h, v] drawn from
W ⇥H ⇥V , where they re
values for workload, hard
2) Performance model:
configuration space F and
formance model is a blac
given some observations o
combination of system’s
NTUITION
mance behavior of configurable
(i) performance debugging, (ii)
gn-time evolution, or (iv) runtime
pirical understanding of how the
stem will vary when the environ-
Such empirical understanding will
develop faster and more accurate
ow us to make predictions and
e for highly configurable systems
10]. For instance, we can learn
ystem on a cheap hardware in a
nd use that to understand the per-
em on a production server before
ore specifically, we would like to
is between the performance of a
ment (characterized by software
kload, and system version) to the
mental conditions.
or an empirical understanding of
A. Preliminary concepts
In this section, we provide formal definitions of fo
cepts that we use throughout this study. The formal n
enable us to concisely convey concept throughout the
1) Configuration and environment space: Let Fi
the i-th feature of a configurable system A which i
enabled or disabled and one of them holds by defa
configuration space is mathematically a Cartesian pro
all the features C = Dom(F1) ⇥ · · · ⇥ Dom(Fd)
Dom(Fi) = {0, 1}. A configuration of a system
a member of the configuration space (feature space
all the parameters are assigned to a specific value
range (i.e., complete instantiations of the system’s para
We also describe an environment instance by 3 v
e = [w, h, v] drawn from a given environment spa
W ⇥H ⇥V , where they respectively represent sets of
values for workload, hardware and system version.
2) Performance model: Given a software system
configuration space F and environmental instances E
formance model is a black-box function f : F ⇥ E
given some observations of the system performance
combination of system’s features x 2 F in an envi
space F and environmental instances E, a per-
odel is a black-box function f : F ⇥ E ! R
observations of the system performance for each
of system’s features x 2 F in an environment
construct a performance model for a system A
ation space F, we run A in environment instance
ious combinations of configurations xi 2 F, and
sulting performance values yi = f(xi) + ✏i, xi 2
⇠ N (0, i). The training data for our regression
en simply Dtr = {(xi, yi)}n
i=1. In other words, a
ction is simply a mapping from the input space to
performance metric that produces interval-scaled
e assume it produces real numbers).
mance distribution: For the performance model,
and associated the performance response to each
, now let introduce another concept where we
ironment and we measure the performance. An
rformance distribution is a stochastic process,
(R), that defines a probability distribution over
measures for each environmental conditions. To
ware
o the
g of
med
rfor-
uited
we
arch
ans-
and
ried
able
ider
t of
vari-
ance
tand
be
configuration space F and environmental instances E, a
formance model is a black-box function f : F ⇥ E !
given some observations of the system performance for
combination of system’s features x 2 F in an environm
e 2 E. To construct a performance model for a system
with configuration space F, we run A in environment inst
e 2 E on various combinations of configurations xi 2 F,
record the resulting performance values yi = f(xi) + ✏i, x
F where ✏i ⇠ N (0, i). The training data for our regres
models is then simply Dtr = {(xi, yi)}n
i=1. In other wor
response function is simply a mapping from the input spac
a measurable performance metric that produces interval-sc
data (here we assume it produces real numbers).
3) Performance distribution: For the performance mo
we measured and associated the performance response to
configuration, now let introduce another concept where
vary the environment and we measure the performance
empirical performance distribution is a stochastic pro
pd : E ! (R), that defines a probability distribution
performance measures for each environmental conditions
Extract Reuse
f(·,es) f(·, et)
We hypothesize that
we can
- learn and transfer
- different forms of
knowledge
- across
environments,
- while so far only
simple transfers are
attempted!
68. Subject systems
74
TABLE I: Overview of the real-world subject systems.
System Domain d |C| |H| |W| |V |
SPEAR SAT solver 14 16 384 3 4 2
x264 Video encoder 16 4 000 2 3 3
SQLite Database 14 1 000 2 14 2
SaC Compiler 50 71 267 1 10 1
d: configuration options; C: configurations; H: hardware environments; W : analyzed
workload; V : analyzed versions.
IV. PERFORMANCE BEHAVIOR CONSISTENCY (RQ1)
Here, we investigate the relatedness of the source and target
environments in the entire configuration space. We start by
testing the strongest assumptions about the relatedness of
environm
reuse me
predict th
that virtu
not be ab
Worklo
correlatio
SAT prob
the differ
across en
slightly
for other
instance
69. Level of relatedness between source and
target is important
10
20
30
40
50
60
AbsolutePercentageError[%]
Sources s s1 s2 s3 s4 s5 s6
noise-level 0 5 10 15 20 25 30
corr. coeff. 0.98 0.95 0.89 0.75 0.54 0.34 0.19
µ(pe) 15.34 14.14 17.09 18.71 33.06 40.93 46.75
Fig. 6: Prediction accuracy of the model learned with samples
from different sources of different relatedness to the target.
GP is the model without transfer learning.
TABLE
column
datasets
measure
1
2
3
4
5
6
predictio
system,
as the e
pled for
• Model becomes more accurate
when the source is more
related to the target
• Even learning from a source
with a small correlation is
better than no transfer
75
70. 76
5 10 15 20 25
number of particles
5
10
15
20
25
numberofrefinements
5
10
15
20
25
30
5 10 15 20 25
number of particles
5
10
15
20
25
numberofrefinements
10
12
14
16
18
20
22
24
5 10 15 20 25
number of particles
5
10
15
20
25
numberofrefinements
10
15
20
25
5 10 15 20 25
number of particles
5
10
15
20
25
numberofrefinements
10
15
20
25
5 10 15 20 25
number of particles
5
10
15
20
25
numberofrefinements
6
8
10
12
14
16
18
20
22
24
(a) (b) (c)
(d) (e) 5 10 15 20 25
number of particles
5
10
15
20
25
numberofrefinements
12
14
16
18
20
22
24
(f)
CPU usage [%]
CPU usage [%] CPU usage [%]
CPU usage [%] CPU usage [%] CPU usage [%]
Less related->
Less accurate
More related->
More accurate
79. Implications for transfer learning
• When and why TL works for performance modeling
• Small environmental changes -> Performance behavior is consistent
• A linear transformation of performance models provide a good approximation
• Large environmental changes -> Individual options and interactions
may stay consistent
• A non-linear mapping between performance behavior across environments
• Severe environmental changes -> Found transferable knowledge
• Invalid configurations providing opportunities for avoiding measurements
• Intuitive judgments about transferability of knowledge
• Without deep knowledge about the configuration or implementation
85
81. Future research opportunities
• Sampling strategies
• More informative samples
• Exploiting the importance of specific regions or avoiding invalid regions
• Learning mechanisms
• Learning either a linear or non-linear associations
• Performance testing and debugging
• Transferring interesting test cases that cover interactions between options
• Performance tuning and optimization
• Identifying the interacting options
• Importance sampling exploiting feature interactions
87
82. Selecting from multiple sources
88
Source Simulator Target Simulator
Source Robot Target Robot
C1
C3
C2
- Different cost
associated to the sources
- Problem is to take
sample from appropriate
source to gain more
information given limited
budget
86. Recap of my previous work
92
RobusT2Scale/FQL4KE [PhD]
✓ Engineering / technical
✓ Maintains application
responsiveness
✓ Handles environmental
uncertainties
✓ Enables knowledge evolution
✗ Learns slowly when situation is
changed
BO4CO/TL4CO [Postdoc1@Imperial]
✓ Mathematical modeling
✓ Finds optimal configuration given a
measurement budget
✓ Step-wise active learning
✗ Does’nt scale well to high dimensions
✗ Expensive to learn
Goal
✔ Industry relevant research
✔ Learn accurate/reliable/cheap performance model
✔ Learned model is used for performance
tuning/debugging/optimization/runtime adaptation
Transfer Learning [Postdoc2@CMU]
✓ Empirical
✓ Learns accurate and reliable models
from “related” sources
✓ Reuse learning across
environmental changes
✗ For severe environmental changes,
transfer is limited, but possible!
[SEAMS14,QoSA16,CCGrid17, TAAS] [MASCOTS16,WICSA16] [SEAMS17]