Automatic Load Test Verification Using Control Charts

•Download as PPTX, PDF•

0 likes•139 views

The document proposes using control charts to automatically verify the results of load testing. It discusses how control charts can be used to determine normal performance thresholds and detect when new releases perform outside of acceptable ranges, potentially isolating problematic components. Two case studies examining an enterprise system and small e-commerce site show that control charts are effective at detecting performance regressions and identifying the source of issues.

Software

Automatic Load Test Verification
Using Control Charts
Thanh Nguyen, Bram Adams, ZhenMing Jiang, Ahmed E. Hassan
Queen’s University, Kingston, Canada
Mohamed Nasser, Parminder Flora
Research in Motion, Waterloo, Canada
1

3
Development
Functional
Testing
Load
Testing

Importance of software performance:
More users
4

Load testing is regression based
5
Apply operational
load e.g. 2000
requests/min
Existing version
e.g. version 1.2
New release
e.g. version 1.3
Baseline
Performance counters
Target
Performance counters
Compare the counters
to detect
performance
regression
Code changes
Automated
e.g. HP LoadRunner,
MS Visual Studio, …
Still a manual
process

Challenge in load testing: Lots of data
6
There are thousands of counters with millions
of samples
Frontend BackendClient
Client
Client
Frontend
Client
Frontend
Client
Backend
Backend
Load
balancer
CPU%
User CPU%
Privileged CPU%
IO data bytes/sec
IO read bytes/sec
IO write bytes/sec
Page faults/sec
Pool paged bytes
Private bytes
Virtual bytes

Manufacturing process also requires
automatic verification
7

Control charts example
8
730
735
740
745
750
755
760
765
770
775
0 5 10 15 20 25
Milk Filled (ml)
CL
LCL
UCL
Violation ratio: 0% 20% 20% 60%

Using control charts to verify load test
results
9
Baseline
Performance counters
Target
Performance counters
Determine the UCL,
CL, LCL
730
740
750
760
770
780
0 5 10 15 20 25
Performance counter

Using control charts to verify load test
results
10
Baseline
Performance counters
Target
Performance counters
Determine the UCL,
CL, LCL
730
740
750
760
770
780
0 5 10 15 20 25
Performance counter
Violation
ratio

11
Baseline
Performance counters
Target
Performance counters
Target
Performance counters
730
740
750
760
770
780
0 10 20 30
Performance counter
Baseline
Performance counters
720
730
740
750
760
770
780
0 10 20 30
Performance counter
Low
violation
ratio
High
violation
ratio

We conduct two case studies
• Enterprise system:
– Large and complex
– Multi-tiers
• Dell DVD store
– Small and simple
– 3-tiers
12

13
Type Description
Front end
problems
CPU hog a) 50% and b) 75% of CPU on
the front end server
Back end
problems
CPU hog a) 50% and b) 75% of CPU on
one of the back end servers
Busy waiting Busy waiting in the front end server
code
Query Extra query to the database back end
Normal
Performance counters
Problems
Performance counters
Type Description
Normal Runs of existing version without
performance problems
RQ1: Can control charts detect a
problematic run?

14
Enterprise’s Response Time
0
10
20
30
40
50
60
70
80
90
100
Normal Front 50% Front 75% Back 50% Back 75%
Violation ratio

15
DVD Store’s Response Time
0
10
20
30
40
50
60
70
80
90
100
Normal Front 50% Front 75% Back 50% Back 75% Busy Query
Violation Ratio

RQ2: Can control charts identify the
problematic component?
16
Frontend BackendClient
Client
Client
Frontend
Client
Frontend
Client
Backend
Backend
Load
balancer
High violation in
frontend’s
counters
High violation in
backend’s
counters

Backend 50% stress
17
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
F B B B B F O B F O
Violation ratioBaseline
Target

Filter out unstable counters
18
Baseline
Target
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
F F F F B B B B B B
Violation ratio
These counters are always out-of-control  We filter then
out

Backend 50% stress
19
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
B B B B B O
Violation ratioBaseline
Target

Frontend 50% stress
20
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
F F F F F F O
Violation ratioBaseline
Target

1) The document discusses using control charts to detect performance regressions in software. Control charts can identify when performance counters exceed their expected upper and lower control limits. 2) An approach is proposed to apply control charts to a baseline version's performance counters and compare them to a target version's counters. Out-of-control counters may indicate performance regressions. 3) Experiments show control charts can effectively detect regressions when the number of out-of-control counters is high or the average violation ratio exceeds expectations. The method helps identify where in the system regressions occur.

Automated Detection of Performance Regressions Using Statistical Process Cont...

SAIL_QU

This document proposes using statistical process control techniques like control charts to detect performance regressions. It acknowledges challenges like unstable inputs across test runs and multiple influencing factors. It then presents solutions like scaling counters based on input metrics and isolating influential factors. Finally, it evaluates the approach through two case studies, showing it can accurately detect regressions with a low average violation ratio and achieve high precision and recall.

An Industrial Case Study of Automatically Identifying Performance Regression-...

SAIL_QU

1) Performance regressions in software can be caused by changes that impact resources like CPU usage, memory, disk I/O, and network I/O. 2) A study of performance regressions in industrial software identified the most common causes, like adding frequently executed database queries or mismatched database indices. 3) The study leveraged a repository of performance data across versions to analyze regression causes by comparing counter values and finding patterns associated with known causes.

Understanding the Rationale for Updating a Function's Comment

SAIL_QU

This document discusses a study that used machine learning to predict whether a comment associated with a code function would be updated based on characteristics of the function, change, and development process. The study analyzed comment update histories from four open source projects over 39 years. It found that comments were more likely to be updated for complex functions with many comments, changes that fixed bugs or had many dependent changes, and on certain weekdays. Combining data from all projects improved the predictive model's performance, with an overall misclassification rate of around 20%.

Test automation

vidisha Agrawal

This document presents information on test automation. It discusses what software testing is, the need for testing, different types of testing, and compares manual vs automated testing. It outlines which test cases are best to automate, the essential needs of test automation, and the process of automating tests. Automated testing provides benefits like eliminating human error, reusability, speed and cost reduction compared to manual testing, but has disadvantages like high upfront investment and effort needed.

Empircal Studies of Performance Bugs & Performance Analysis Approaches for La...

SAIL_QU

This document summarizes three empirical studies on software performance bugs: 1. A quantitative study found performance bugs have different characteristics than other bugs, such as taking longer to fix, but findings were not consistent across projects. 2. A qualitative study of bug reports found performance bugs have a higher impact, more context in reports, and require more collaborative fixing. 3. A user-centric performance analysis study found examining performance from users' perspectives provided a complementary view to traditional scenario-centric analyses. Considering individual users revealed different performance trends and consistency than aggregate analyses.

Log Engineering: Towards Systematic Log Mining to Support the Development of ...

SAIL_QU

This document presents the thesis work on systematic log mining to support the development of ultra-large scale systems. There are five key findings from prior log mining research: 1) Little focus on logs in source code; 2) Little use of logs from development; 3) Ad hoc log transformation; 4) Lack of scalability; 5) Limited use for software development activities. The thesis proposes two parts: 1) Study challenges of understanding and evolving logs; 2) Approaches using logs to support development like testing and deployment verification. Evaluation shows logs provide insights into defects and an approach using execution sequences from logs precisely verifies big data application deployments with 86% effort reduction.

Hello

Pallavi Batra

This document discusses using a genetic algorithm to develop a machine learning model for predicting fault-prone software classes. It begins by introducing software reliability and fault prediction. It then explains that a genetic algorithm is a search technique that evaluates potential solutions, keeps the best ones, and generates new solutions iteratively. The algorithm uses software metrics like coupling, cohesion, inheritance, and size as inputs to classify classes as faulty or fault-free with 80.14% accuracy, helping to identify areas for improvement.

Applying code changes to software systems and testing these code changes can be a complex task that involves many different types of software testing strategies, e.g. system and integration tests. However, not all test failures reported during code integration are hinting towards code defects. Testing large systems such as the Microsoft Windows operating system requires complex test infrastructures, which may lead to test failures caused by faulty tests and test infrastructure issues. Such false test alarms are particular annoying as they raise engineer attention and require manual inspection without providing any benefit. The goal of this work is to use empirical data to minimize the number of false test alarms reported during system and integration testing. To achieve this goal, we use association rule learning to identify patterns among failing test steps that are typically for false test alarms and can be used to automatically classify them. A successful classification of false test alarms is particularly valuable for the product teams as manual test failure inspection is an expensive and time-consuming process that not only costs engineering time and money but also slows down product development. We evaluating our approach on system and integration tests executed during Windows 8.1 and Microsoft Dynamics AX development. Performing more than 10,000 classifications for each product, our model shows a mean precision between 0.85 and 0.90 predicting between 34% and 48% of all false test alarms.

The Myths and Facts Surrounding Software Testing

Software Testing Solution

The importance and seriousness of Software Testing is well known. Much has been discussed and evaluated and the bottom line is that mistakes happen generally because humans tend to overlook possibilities and probable errors. A software not tested with due seriousness can lead to major blunders putting the clients of the systems into a tight spot and resulting in nightmares of sort. It is then only prudent and wise to analyze and predict errors and conduct timely rectifications to avoid any embarrassing situations in the future and to deliver a stable and reliable system. Like many other things, there are Myths surrounding Software Testing Services, but Facts remain Facts. Read More At: http://softwaretestingsolution.com/blog/the-myths-and-facts-surrounding-software-testing/

Case study on Test Automation under RUP

Oak Systems

This case study describes an automation project for testing an insurance software with stringent quality goals and short development cycles. The client previously struggled with manual testing being laborious and redundant. Oak Systems solved this by automating over 1300 test cases using a testing tool. They verified test cases and reorganized data, developed automation scripts that were dynamic and reusable, and synchronized regression testing with releases. This automation helped achieve quality goals through faster testing and higher quality, improving client and user satisfaction.

Solano Labs presented at MassTLC's automated testing

MassTLC

[HCMC STC Jan 2015] Risk-Based Software Testing Approaches

Ho Chi Minh City Software Testing Club

Predictive Analytics in Software Testing

Pavan Kumar Kodedela

The document discusses challenges faced by companies with both in-house and outsourced software testing. It introduces predictive analytics as a solution to address common challenges like managing multiple releases and tools, measuring productivity, and generating customized reports. Predictive analytics uses models to analyze test data and predict issues, risks, delays and determine how to optimize testing. Integrating predictive analytics into a testing framework can help reduce costs, improve quality and make better decisions.

[Paul Holland] Bad Metrics and What You Can Do About It

Ho Chi Minh City Software Testing Club

Software testing metrics are used extensively by many organizations to determine the status of their projects and whether or not their products are ready to ship. Unfortunately most, if not all, of the metrics being used are so flawed that they are not only useless but are possibly dangerous—misleading decision makers, inadvertently encouraging unwanted behavior, or providing overly simplistic summaries out of context. Paul Holland identifies four characteristics that will enable you to recognize the bad metrics in your organization. Despite showing how the majority of metrics used today are “bad”, all is not lost as Paul shows the collection of information he has developed that is more effective. Learn how to create a status report that provides details sought after by upper management while avoiding the problems that bad metrics cause.

Machine learning in software testing

Thoughtworks

This document discusses how machine learning can be applied to various activities in software testing. It describes how machine learning works using training and test data to make predictions. Supervised and unsupervised learning techniques are discussed. Specific applications mentioned include software defect prediction, test planning, test case management, debugging, and refining blackbox test specifications. Challenges include availability of past data and finding predictable patterns, while potential steps forward include expanding machine learning to more blackbox techniques, identifying the right patterns for different test activities, algorithm analysis, and crowdsourcing.

Automation in the Bug Flow - Machine Learning for Triaging and Tracing

Markus Borg

Issue management is a costly part of software development. In large projects, the continuous inflow of issue reports contributes to the information overload in a project, i.e., "a state where individuals do not have time or capacity to process all available information". In issue triaging, an initial step in issue management, a developer must be able to overview existing issue reports and easily navigate the software engineering project landscape. In this presentation, we present support for two work tasks involved in issue management: 1) issue assignment and 2) change impact analysis. We use machine learning to harness the ever-growing number of issue reports, by training recommendation systems on previous issues. Our industrial evaluations on 50,000+ issue reports in two large software development organizations indicate that automated issue assignment performs in line with current manual work. Moreover, we present how traceability from already resolved issue reports to various artifacts can be reused to jump start change impact analyses for newly submitted issues. Finally, we speculate on future ways to tame information overload into helpful software engineering recommendations.

How Manual Testers Can Break into Automation Without Programming Skills

Ranorex

Adoption of automating tests has not happened as quickly as organizations need. As more companies move toward implementing agile development as their software development lifecycle, more features are being implemented and released more quickly. This leaves less time for full regression testing of the system, nonetheless this should still be done. Manual testers need to transform into test automation testers as well. Learn how to make this jump as a manual tester and focus on the right areas first e.g. automation test structure, object recognition and results interpretation.

Agile, DevOps, & Hardware

David Evans

AfterTest Madrid March 2016 - DevOps and Testing Introduction

Peter Marshall

This document discusses continuous testing in the context of DevOps. It defines continuous testing as including automated testing, managing production and non-production environments, application monitoring, and evaluating business objectives. Continuous testing relies on DevOps activities like automating builds, infrastructure, deployments, and monitoring. It advocates for smaller, more frequent deliveries through practices like test automation, infrastructure as code, and treating testing as integral to the software delivery process. The conclusion emphasizes automation, configuration management, and obtaining frequent feedback to enable continuous testing.

When is a project ready for Software Automation_NEW

Mike Christesen

This document discusses when a project is ready for software automation. It begins with introductions and an agenda that will cover misconceptions about automation, selecting the right tools, and real-world gains from automation. Some key points made include the benefits of automation being quick runs, lower costs, and finding more bugs, while manual testing has difficulties like being repetitive. It suggests automating repetitive, data-intensive, high risk tests and testing early. Selecting the right tools that cover all technologies and maintaining resilient tests is also discussed. Metrics like time savings and reduced defects are presented from an IDC study on automation ROI.

Risk based testing a new case study

Bassam Al-Khatib

This document provides an overview of risk-based testing (RBT) including the process, examples, and benefits. It discusses how to identify risks, analyze them by considering likelihood and impact, and then mitigate risks through testing. The key aspects of RBT covered include building a risk matrix, prioritizing testing based on risk level, and applying RBT at different scales from large changes to small quick tasks. Overall, RBT is presented as a systematic approach to make testing more efficient and ensure the most critical risks are addressed.

Seven testing principles

Vaibhav Dash

The document outlines 7 testing principles: 1) Testing finds defects but finding none does not mean none exist, 2) Exhaustive testing is impossible so smarter testing is needed, 3) Early testing saves time and money and makes customers happy, 4) Defects tend to cluster together, 5) Test cases must be updated periodically to avoid outdated "pesticide" tests, 6) Testing methods vary depending on the software context, and 7) Software should be stable before testing to avoid false negatives from instability.

Software Testing Principles and Techniques

suresh ramanujam

The document contains a session plan for a software testing principles and techniques course. The session objectives are to define various software testing terms and concepts, differentiate between different types of testing, and learn about the testing process. The session would include slides, demonstrations of testing software, and discussions. Test cases for an example ATM system are also provided to demonstrate initial functional testing.

[Europe merge world tour] Coverity Development Testing

Perforce

Development testing can reduce costs, accelerate development, and protect brands by: 1) Finding defects earlier in the development process before they escape to production through continuous integration and static analysis. 2) Prioritizing testing of critical code and ensuring all code impacted by changes is tested. 3) Optimizing developer workflows by integrating testing into the development process and minimizing redundant testing.

Software testing and test environment

adhirasable

4.4.2013 Software Quality - Regression Testing Automated and Manual - RFT/RQM

IBM Rational

Testing 3: Types Of Tests That May Be Required

ArleneAndrews2

Adding Value in the Cloud with Performance Test

Rodolfo Kohn

This document discusses the importance of performance testing cloud applications and outlines best practices for defining performance requirements, testing methodology, and identifying issues. It provides examples of performance problems found in databases, applications, operating systems, and networks. The key goals of performance testing are to understand system behavior under load, find bottlenecks and hidden bugs, and verify that requirements are met.

The experiences of migrating a large scale, high performance healthcare network

george.james

Partners Healthcare migrated their large-scale Caché database from a mixed Windows and UNIX environment to a new highly available UNIX architecture using HP servers. They took a phased approach, first benchmarking performance on test systems, then migrating the database tier and finally the application tier. Benchmarking revealed optimizations that improved performance in production. The new environment provided improved availability, scalability and reduced maintenance needs to support continued rapid growth of the healthcare network.

What's hot

Empirically Detecting False Test Alarms Using Association Rules @ ICSE 2015

Kim Herzig

The Myths and Facts Surrounding Software Testing

Software Testing Solution

Case study on Test Automation under RUP

Oak Systems

Solano Labs presented at MassTLC's automated testing

MassTLC

[HCMC STC Jan 2015] Risk-Based Software Testing Approaches

Ho Chi Minh City Software Testing Club

Predictive Analytics in Software Testing

Pavan Kumar Kodedela

[Paul Holland] Bad Metrics and What You Can Do About It

Ho Chi Minh City Software Testing Club

Machine learning in software testing

Thoughtworks

Automation in the Bug Flow - Machine Learning for Triaging and Tracing

Markus Borg

How Manual Testers Can Break into Automation Without Programming Skills

Ranorex

Agile, DevOps, & Hardware

David Evans

AfterTest Madrid March 2016 - DevOps and Testing Introduction

Peter Marshall

When is a project ready for Software Automation_NEW

Mike Christesen

Risk based testing a new case study

Bassam Al-Khatib

Seven testing principles

Vaibhav Dash

Software Testing Principles and Techniques

suresh ramanujam

[Europe merge world tour] Coverity Development Testing

Perforce

Software testing and test environment

adhirasable

4.4.2013 Software Quality - Regression Testing Automated and Manual - RFT/RQM

IBM Rational

Testing 3: Types Of Tests That May Be Required

ArleneAndrews2

What's hot (20)

Empirically Detecting False Test Alarms Using Association Rules @ ICSE 2015

The Myths and Facts Surrounding Software Testing

Case study on Test Automation under RUP

Solano Labs presented at MassTLC's automated testing

[HCMC STC Jan 2015] Risk-Based Software Testing Approaches

Predictive Analytics in Software Testing

[Paul Holland] Bad Metrics and What You Can Do About It

Machine learning in software testing

Automation in the Bug Flow - Machine Learning for Triaging and Tracing

How Manual Testers Can Break into Automation Without Programming Skills

Agile, DevOps, & Hardware

AfterTest Madrid March 2016 - DevOps and Testing Introduction

When is a project ready for Software Automation_NEW

Risk based testing a new case study

Seven testing principles

Software Testing Principles and Techniques

[Europe merge world tour] Coverity Development Testing

Software testing and test environment

4.4.2013 Software Quality - Regression Testing Automated and Manual - RFT/RQM

Testing 3: Types Of Tests That May Be Required

Similar to Automatic Load Test Verification Using Control Charts

Adding Value in the Cloud with Performance Test

Rodolfo Kohn

The experiences of migrating a large scale, high performance healthcare network

george.james

DevOps: Find Solutions, Not More Defects

TechWell

The promise of DevOps is that we can push new ideas out to market faster while avoiding delivering serious defects into production. Andreas Grabner explains that testers are no longer measured by the number of defect reports they enter, nor are developers measured by the lines of code they write. As a team, you are measured by how fast you can deploy high quality functionality to the end user. Achieving this goal requires testers to increase their skills. It’s all about finding solutions—not just problems. Testers must transition from reporting “app crashes” to providing details such as “memory leak caused by bad cache implementation.” Instead of reporting “it’s slow,” testers must discover “wrong hibernate configuration causes too much traffic from the database.” Using three real-life examples, Andreas illustrates what it takes for testing teams to become part of the DevOps transformation—bringing more value to the entire organization.

Open source: Top issues in the top enterprise packages

Rogue Wave Software

Accelerate Develoment with VIrtual Data

Kyle Hailey

This document summarizes best practices for application development using data virtualization to remove data as a constraint. It discusses how data management currently does not scale with agile development and is a major bottleneck. The solution presented is using a data virtualization appliance to create thin clones from production data for development, QA, and test environments. This allows for self-service provisioning of environments and parallel development. It provides use cases showing how virtual data improves development throughput, shifts testing left to find bugs earlier, and enables continuous delivery of features to production.

Become a Performance Diagnostics Hero

TechWell

Andreas Grabner maintains that most performance and scalability problems don’t need a large or long running performance test or the expertise of a performance engineering guru. Don’t let anybody tell you that performance is too hard to practice because it actually is not. You can take the initiative and find these often serious defects. Andreas analyzed and spotted the performance and scalability issues in more than 200 applications last year. He shares his performance testing approaches and explores the top problem patterns that you can learn to spot in your apps. By looking at key metrics found in log files and performance monitoring data, you will learn to identify most problems with a single functional test and a simple five-user load test. The problem patterns Andreas explains are applicable to any type of technology and platform. Try out your new skills in your current testing project and take the first step toward becoming a performance diagnostic hero.

Exposing and Fixing Common App Performance Problems

Riverbed Technology

Real Time Network Monitoring System

Girish Naik

This document describes the design and implementation of a real-time network monitoring system. The system allows a network administrator to monitor network resources in real-time from both client and server interfaces. It was developed using a waterfall software engineering model and uses technologies like Java, MySQL, and Linux to enable cross-platform functionality with low hardware requirements. Testing was conducted and future enhancements are proposed to expand the system's monitoring capabilities.

T3 Consortium's Performance Center of Excellence

veehikle

XenApp Load Balancing

Denis Gundarev

The document discusses load balancing and intelligent load balancing. It covers load balancing architecture, how the data collector and dynamic store work, and how performance counters are used. Intelligent load balancing techniques like load throttling are explained. Potential issues that could cause load imbalances like the "black hole effect" or failing to read performance counters are also reviewed. Troubleshooting techniques for resolving common problems are provided.

WebSphere Technical University: Introduction to the Java Diagnostic Tools

Chris Bailey

IBM provides a number of free tools to assist in monitoring and diagnosing issues when running any Java application - from Hello World to IBM or third-party, middleware-based applications. This session introduces attendees to those tools, highlights how they have been extended with IBM middleware product knowledge, how they have been integrated into IBM’s development tools, and how to use them to investigate and resolve real-world problem scenarios Presented at the WebSphere Technical University 2014, Dusseldorf

Top Java Performance Problems and Metrics To Check in Your Pipeline

Andreas Grabner

Using ai and automation to build resiliency into azure dev ops

Rob Jahn

The document discusses cloud native applications and continuous delivery pipelines. It summarizes the results of a survey that found the majority of applications are not fully cloud native yet. It then outlines several capabilities needed for cloud native applications and continuous delivery pipelines, including automated operations, deployment, quality processes, and monitoring across platforms and environments. The key differentiators are described as an "unbreakable continuous delivery blue print" with automated monitoring, deployment, quality processes, and self-healing capabilities.

The experiences of migrating a large scale, high performance healthcare network

george.james

Virtualising Tier 1 Apps

Iwan Rahabok

This document discusses virtualizing tier 1 applications. It begins by showing how virtualization adoption has increased significantly for mission critical applications. It then discusses specific steps and considerations for virtualizing tier 1 applications, including: 1. Ensuring the platform (hardware, virtualization software, etc.) can adequately support the application. 2. Ensuring the people and processes are in place to design, implement, operate and troubleshoot the virtualized application. This includes discussing skills, support models, change management and monitoring. 3. Reviewing the application itself and existing reference architectures to understand virtualization best practices and sizing for that application. The goal is to virtualize at the application layer rather than physical server layer.

Service Virtualization - Next Gen Testing Conference Singapore 2013

Min Fang

Most major enterprises have invested millions of dollars on soIware performance lab infrastructure, that develop recurring maintenance effort and costs, unstable environments, and conflicts over constrained resources. To combat this drain and deliver value, companies need to find ways to optimize. Virtualizing the behavior and performance characteristics of test lab dependencies, CA LISA Service Virtualization has helped many enterprises achieving highly available labs that allow earlier performance testing with greater flexibility, at a much lower cost. Techniques discussed in this session include: Removing capacity constraints from performance testing lab Optimizing performance management by decomposing SLAs Shift-Left: conducting incremental, iterative performance testing

Performance testing virtualized systems v5

Mentora

The document discusses server virtualization technologies and performance testing best practices. It provides details on: 1) The anatomy of a virtual system including hardware virtualization, type 1 and type 2 hypervisors, and allocating VM resources. 2) Key factors for performance testers including modeling virtual workloads, identifying bottlenecks, and effective testing techniques. 3) A case study where a company consolidated 18 servers onto 3 virtual hosts, reduced costs, and performance testing showed maintained or improved performance.

Optimizing performance

Zend by Rogue Wave Software

This is episode 4 of the building the perfect PHP app for the enterprise webinar series. Nothing is faster than a frustrated user clicking away from your site or abandoning your service. Avoid attrition by learning how to tune your applications towards lightning-fast page loads and response times. Learn: the basic principles behind enterprise PHP performance management; How to optimize workloads through background jobs and caching; How to measure performance and make data-driven decisions.

Will it Scale? The Secrets behind Scaling Stream Processing Applications

Navina Ramesh

Industrial Automation and Modern Connectivity

PlantEngineering

This document provides an overview of industrial automation and modern connectivity. It discusses how connecting factory equipment to other networks allows data transfer and optimization of production processes. Three case studies are presented: an automotive story demonstrating automated data collection and reporting; a candy production story showing standardized recipe management across global plants; and an integrator story using remote desktop support. Basic security recommendations are provided, such as backing up software, network isolation, and using recommended practices guides.

Similar to Automatic Load Test Verification Using Control Charts (20)

Adding Value in the Cloud with Performance Test

The experiences of migrating a large scale, high performance healthcare network

DevOps: Find Solutions, Not More Defects

Open source: Top issues in the top enterprise packages

Accelerate Develoment with VIrtual Data

Become a Performance Diagnostics Hero

Exposing and Fixing Common App Performance Problems

Real Time Network Monitoring System

T3 Consortium's Performance Center of Excellence

XenApp Load Balancing

WebSphere Technical University: Introduction to the Java Diagnostic Tools

Top Java Performance Problems and Metrics To Check in Your Pipeline

Using ai and automation to build resiliency into azure dev ops

The experiences of migrating a large scale, high performance healthcare network

Virtualising Tier 1 Apps

Service Virtualization - Next Gen Testing Conference Singapore 2013

Performance testing virtualized systems v5

Optimizing performance

Will it Scale? The Secrets behind Scaling Stream Processing Applications

Industrial Automation and Modern Connectivity

More from SAIL_QU

Studying the Integration Practices and the Evolution of Ad Libraries in the G...

SAIL_QU

In-app advertisements have become a major revenue for app developers in the mobile app economy. Ad libraries play an integral part in this ecosystem as app developers integrate these libraries into their apps to display ads. However, little is known about how app developers integrate these libraries with their apps and how these libraries have evolved over time. In this thesis, we study the ad library integration practices and the evolution of such libraries. To understand the integration practices of ad libraries, we manually study apps and derive a set of rules to automatically identify four strategies for integrating multiple ad libraries. We observe that integrating multiple ad libraries commonly occurs in apps with a large number of downloads and ones in categories with a high percentage of apps that display ads. We also observe that app developers prefer to manage their own integrations instead of using off the shelf features of ad libraries for integrating multiple ad libraries. To study the evolution of ad libraries, we conduct a longitudinal study of the 8 most popular ad libraries. In particular, we look at their evolution in terms of size, the main drivers for releasing a new ad library version, and their architecture. We observe that ad libraries are continuously evolving with a median release interval of 34 days. Some ad libraries have grown exponentially in size (e.g., Facebook Audience Network ad library), while other libraries have worked to reduce their size. To study the main drivers for releasing an ad library version, we manually study the release notes of the eight studied ad libraries. We observe that ad library developers continuously update their ad libraries to support a wider range of Android versions (i.e., to ensure that more devices can use the libraries without errors). Finally, we derive a reference architecture for ad libraries and study how the studied ad libraries diverged from this architecture during our study period. Our findings can assist ad library developers to understand the challenges for developing ad libraries and the desired features of these libraries.

Studying the Dialogue Between Users and Developers of Free Apps in the Google...

SAIL_QU

Improving the testing efficiency of selenium-based load tests

SAIL_QU

Studying User-Developer Interactions Through the Distribution and Reviewing M...

SAIL_QU

This document discusses studying user-developer interactions through the distribution and reviewing mechanisms of the Google Play Store. It analyzes emergency updates made by developers to fix issues, the dialogue between users and developers through reviews and responses, and how the reviewing mechanism can help identify good and bad updates. The study found that responding to reviews is six times more likely to increase an app's rating, with 84% of rating increases going to four or five stars. Three common patterns of developer responses were identified: responding to negative or long reviews, only negative reviews, and reviews shortly after an update.

Studying online distribution platforms for games through the mining of data f...

SAIL_QU

Our studies of Steam platform data provided insights into online game distribution: 1) Urgent game updates were used to fix crashes, balance issues, and functionality; frequent updaters released more 0-day patches. 2) The Early Access model attracted indie developers and increased game participation; reviews were more positive during Early Access. 3) Game reviews were typically short and in English; sales increased review volume more than new updates; negative reviews came after longer play.

Understanding the Factors for Fast Answers in Technical Q&A Websites: An Empi...

SAIL_QU

This study analyzed factors that impact the speed of questions receiving accepted answers on four popular Stack Exchange websites: Stack Overflow, Mathematics, Ask Ubuntu, and Super User. The researchers examined question, answerer, asker, and answer factors from over 150,000 questions. They built classification models and found that key factors for fast answers included the past speed of answerers, length of the question, and past speed of answers for the question's tags. The models achieved AUCs of 0.85-0.95. Fast answers relied heavily on answerers, especially frequent answerers. The study suggests improving incentives for non-frequent and more difficult questions to attract diverse answerers.

Investigating the Challenges in Selenium Usage and Improving the Testing Effi...

SAIL_QU

Selenium is a popular tool for browser-based automation testing. The author analyzes challenges in using Selenium by mining Selenium questions on Stack Overflow. Programming language-related questions, especially for Java and Python, are most common and growing fastest. Less than half of questions receive accepted answers, and questions about browsers and components take longest. In the second part, the author develops an approach to improve efficiency of Selenium-based load testing by sharing browsers among user instances. This increases the number of error-free users by 20-22% while reducing memory usage.

Mining Development Knowledge to Understand and Support Software Logging Pract...

SAIL_QU

This document summarizes Heng Li's PhD thesis on mining development knowledge to understand and support software logging practices. It discusses how logging code is used to record runtime information but can be difficult for developers to maintain. The thesis aims to understand current logging practices and develop tools by mining change history, source code, issue reports, and other development knowledge. It presents research that analyzes logging-related issues to identify developers' logging concerns, uses code topics and structure to predict where logging statements should be added, leverages code changes to suggest when logging code needs updating, and applies machine learning models to recommend appropriate log levels.

Which Log Level Should Developers Choose For a New Logging Statement?

SAIL_QU

The document discusses choosing an appropriate log level when adding a new logging statement. It finds that an ordinal regression model can effectively model log levels, achieving an AUC of 0.76-0.81 in within-project evaluation and 0.71-0.8 in cross-project evaluation. The most influential factors for determining log levels vary between projects and include metrics related to the logging statement, containing code block, and file as well as code change and historical change metrics.

Towards Just-in-Time Suggestions for Log Changes

SAIL_QU

The document presents a study on providing just-in-time suggestions for log changes when developers make code changes. The researchers analyzed over 32,000 log changes from 4 systems. They found 20 reasons for log changes that fall into 4 categories: block changes, log improvements, dependence-driven changes, and logging issues. A random forest classifier using 25 software metrics related to code changes, history, and complexity achieved 0.84-0.91 AUC in predicting whether a log change is needed. Change metrics and product metrics were the most influential factors. The study aims to help developers make better logging decisions for failure diagnosis.

The Impact of Task Granularity on Co-evolution Analyses

SAIL_QU

The document discusses how task granularity at different levels (e.g. commits, pull requests, work items) can impact analyses of co-evolution in software projects. It finds that analyzing at the commit-level can overlook relationships between tasks that span multiple commits. Work item level analysis is recommended to provide a more complete view of co-evolution, as median of 29% of work items consist of multiple commits, and analyzing at the commit level would miss 24% of co-changed files and inability to group 83% of related commits.

A Framework for Evaluating the Results of the SZZ Approach for Identifying Bu...

SAIL_QU

How are Discussions Associated with Bug Reworking? An Empirical Study on Open...

SAIL_QU

1) Initial bug fix discussions with more comments and more developers participating are more likely to experience later bug reworking through re-opening or re-patching of the bug. 2) Manual analysis found that defective initial fixes and failure to reach consensus in discussions contributed to later reworking. 3) For re-opened bugs, initial discussions focused on addressing a particular problem through a burst of comments, while re-patched bugs lacked thorough code review and testing during the initial fix period.

A Study of the Relation of Mobile Device Attributes with the User-Perceived Q...

SAIL_QU

This study examined the relationship between mobile device attributes and user-perceived quality of Android apps. The researchers analyzed 150,373 star ratings from Google Play across 30 devices and 280 apps. They found that the perceived quality of apps varies across devices, and having better characteristics of an attribute does not necessarily correlate with higher quality. Device OS version, resolution, and CPU showed significant relationships with ratings, as did some app attributes like lines of code and number of inputs. However, some device attributes had stronger relationships than app attributes.

A Large-Scale Study of the Impact of Feature Selection Techniques on Defect C...

SAIL_QU

This document presents the results of a large-scale study on the impact of feature selection techniques on defect classification models. The study used expanded scopes including multiple datasets from NASA and PROMISE with different feature types, more classification techniques from different paradigms, and additional feature selection techniques. The results show that correlation-based feature subset selection techniques like FS1 and FS2 consistently appear in the top ranks across most of the datasets, projects within the datasets, and classification techniques. The document concludes that future defect classification studies should consider applying correlation-based feature selection techniques.

Studying the Dialogue Between Users and Developers of Free Apps in the Google...

SAIL_QU

The study analyzes user-developer interactions through reviews and responses on the Google Play Store. It finds that responding to reviews has a significant positive impact, with 84% of rating increases due to the developer addressing the issue or providing guidance. Three common response patterns were identified: only negative reviews, negative or longer reviews, and reviews shortly after an update. Developers most often thank the user, ask for details, provide guidance, or ask for an endorsement. Guidance responses can address common issues through FAQs. The analysis considered over 2,000 apps, 355,000 review changes, 128,000 responses, and 4 million reviews.

What Do Programmers Know about Software Energy Consumption?

SAIL_QU

This document summarizes the results of a survey of 122 programmers about their knowledge of software energy consumption. The survey found that programmers have limited awareness of energy consumption and how to reduce it. They were unaware of the main causes of high energy usage. Programmers lacked knowledge about how to properly rank the energy consumption of different hardware components and were unfamiliar with strategies to improve efficiency, such as minimizing I/O and avoiding polling. The study concludes that programmers would benefit from more education on software energy usage and its causes.

Threshold for Size and Complexity Metrics: A Case Study from the Perspective ...

SAIL_QU

Revisiting the Experimental Design Choices for Approaches for the Automated R...

SAIL_QU

Prior research on automated duplicate issue report retrieval focused on improving performance metrics like recall rate. The author revisits experimental design choices from four perspectives: needed effort, data changes, data filtration, and evaluation process. The thesis contributions are: 1) Showing the importance of considering needed effort in performance measurement. 2) Proposing a "realistic evaluation" approach and analyzing prior findings with it. 3) Developing a genetic algorithm to filter old issue reports and improve performance. 4) Highlighting the impact of "just-in-time" features on evaluation. The findings help better understand benefits and limitations of prior work in this area.

Measuring Program Comprehension: A Large-Scale Field Study with Professionals

SAIL_QU

The document summarizes a large-scale field study that tracked the program comprehension activities of 78 professional developers over 3,148 hours. The study found that: 1) Program comprehension accounted for approximately 58% of developers' time on average, with navigation and editing making up the remaining portions. 2) Developers frequently used web browsers and document editors to aid comprehension beyond just IDEs. 3) Interviews and observations revealed that insufficient documentation, unclear code, and complex inheritance hierarchies contributed to long comprehension sessions.

More from SAIL_QU (20)