This document describes a methodology for pinpointing the subsystems responsible for performance deviations during load testing of large software systems. The methodology involves: 1) preprocessing performance counter data, 2) crafting performance signatures for each subsystem using principal component analysis, 3) identifying deviations between a baseline and load tests, and 4) pinpointing the subsystems most likely causing deviations based on correlation between signature counters. The methodology is demonstrated through load tests on a Dell DVD store system, accurately pinpointing affected subsystems under different stress conditions.
Symantec NetBackup 7.6 benchmark comparison: Data protection in a large-scale...Principled Technologies
In an enterprise environment, a data center VM footprint can grow quickly; large-scale deployments of thousands of virtual machines are becoming increasingly common. Risk of failure grows proportionally to the number of systems deployed and critical failures are unavoidable. Your ability to offer data protection from a backup solution is critical to business continuity. Elongated, inefficient protection windows can create resource contention with production environments, therefore, it is critical to execute system backup in a finite window of time.
The Symantec NetBackup Integrated Appliance running NetBackup 7.6 offered application protection to 1,000 VMs in 80.3 percent less time in SAN testing and used NetApp array-based snapshots to create recovery points in 93.8 percent less time than Competitor “C.” In addition, the Symantec NetBackup Integrated Appliance with NetBackup 7.6 created backup images that offered granular recovery without additional steps and in a backup window 69.0 percent shorter than the backup window needed for Competitor “C.” These time savings can scale as your VM footprint grows, allowing you to execute both system protection and user-friendly, simplified recovery.
NetBackup 7.6 with the NetBackup Integrated Appliance provided more scalable data protection than a competitor's solution for large virtual environments. With 1,000 VMs, NetBackup completed a SAN transport backup in 80.3% less time. For NAS backups of 1,000 VMs using storage snapshots, NetBackup was 93.8% faster. Testing showed NetBackup had superior scalability for protecting the largest virtual server deployments compared to the competitor.
The document describes approaches for automatically detecting performance deviations in load testing of large scale systems. It presents four approaches: three unsupervised using clustering, random sampling, and PCA; and one supervised using WRAPPER. A case study evaluates the approaches on an open-source ecommerce system and industrial telecom system, finding the supervised WRAPPER and unsupervised PCA approaches most effective with high precision and recall. The WRAPPER approach allows real-time analysis but requires more manual overhead during training.
ANSYS SCADE Usage for Unmanned Aircraft VehiclesAnsys
SCADE on-board the UAS P.1HH HammerHead
The Use of SCADE to develop the P.1HH Vehicle Control & Management System (Integrated Modular Avionics System) greatly reduced development time and effort.
Learn more about ANSYS SCADE Solutions for Aerospace & Defense http://bit.ly/1EdcsOJ
The document discusses ways to improve the performance of Oracle SOA Suite 11g. Some key points include:
1. Upgrading to Oracle SOA Suite 11g Patch Set 3 and switching from Sun JDK to JRockit JDK can provide significant performance boosts of up to 32%.
2. Optimizing logging levels and audit settings, such as changing the audit level from Development to Production, can improve performance by 46-92%.
3. Increasing the number of Mediator worker threads for asynchronous services results in a 30% performance improvement.
Sec Process Control Past Present and Future2222Roger Colee
This document summarizes improvements made to secondary wastewater treatment processes at two treatment plants. At the Northside plant, instrumentation was improved in the aeration basin, including adding dissolved oxygen probes and correlating data to optimize solids retention time and wasting. At the Southside plant, process changes like increasing dissolved oxygen and return activated sludge flow enabled some ammonia removal without dedicated nitrogen removal infrastructure. Both plants saw performance benefits from improved data monitoring, process control, and solids inventory management.
This document discusses a concept for avionics systems that provides fault tolerance through rapid start-up and restart capabilities using self-checking pairs (SCP). The concept was developed for unmanned vehicles and involves building the system from a rapid start/restart SCP building block. The next phase of research will involve building a demonstration system to test rapid start-ups in response to simulated faults and developing a prototype SCP building block.
Mechanical Simulations for Electronic ProductsAnsys
As electronic devices become smaller and more ubiquitous, the printed circuit boards and components that drive them face increasing power densities and evermore complexity. To ensure product reliability and performance, accurate and detailed analysis methodologies are necessary.
Symantec NetBackup 7.6 benchmark comparison: Data protection in a large-scale...Principled Technologies
In an enterprise environment, a data center VM footprint can grow quickly; large-scale deployments of thousands of virtual machines are becoming increasingly common. Risk of failure grows proportionally to the number of systems deployed and critical failures are unavoidable. Your ability to offer data protection from a backup solution is critical to business continuity. Elongated, inefficient protection windows can create resource contention with production environments, therefore, it is critical to execute system backup in a finite window of time.
The Symantec NetBackup Integrated Appliance running NetBackup 7.6 offered application protection to 1,000 VMs in 80.3 percent less time in SAN testing and used NetApp array-based snapshots to create recovery points in 93.8 percent less time than Competitor “C.” In addition, the Symantec NetBackup Integrated Appliance with NetBackup 7.6 created backup images that offered granular recovery without additional steps and in a backup window 69.0 percent shorter than the backup window needed for Competitor “C.” These time savings can scale as your VM footprint grows, allowing you to execute both system protection and user-friendly, simplified recovery.
NetBackup 7.6 with the NetBackup Integrated Appliance provided more scalable data protection than a competitor's solution for large virtual environments. With 1,000 VMs, NetBackup completed a SAN transport backup in 80.3% less time. For NAS backups of 1,000 VMs using storage snapshots, NetBackup was 93.8% faster. Testing showed NetBackup had superior scalability for protecting the largest virtual server deployments compared to the competitor.
The document describes approaches for automatically detecting performance deviations in load testing of large scale systems. It presents four approaches: three unsupervised using clustering, random sampling, and PCA; and one supervised using WRAPPER. A case study evaluates the approaches on an open-source ecommerce system and industrial telecom system, finding the supervised WRAPPER and unsupervised PCA approaches most effective with high precision and recall. The WRAPPER approach allows real-time analysis but requires more manual overhead during training.
ANSYS SCADE Usage for Unmanned Aircraft VehiclesAnsys
SCADE on-board the UAS P.1HH HammerHead
The Use of SCADE to develop the P.1HH Vehicle Control & Management System (Integrated Modular Avionics System) greatly reduced development time and effort.
Learn more about ANSYS SCADE Solutions for Aerospace & Defense http://bit.ly/1EdcsOJ
The document discusses ways to improve the performance of Oracle SOA Suite 11g. Some key points include:
1. Upgrading to Oracle SOA Suite 11g Patch Set 3 and switching from Sun JDK to JRockit JDK can provide significant performance boosts of up to 32%.
2. Optimizing logging levels and audit settings, such as changing the audit level from Development to Production, can improve performance by 46-92%.
3. Increasing the number of Mediator worker threads for asynchronous services results in a 30% performance improvement.
Sec Process Control Past Present and Future2222Roger Colee
This document summarizes improvements made to secondary wastewater treatment processes at two treatment plants. At the Northside plant, instrumentation was improved in the aeration basin, including adding dissolved oxygen probes and correlating data to optimize solids retention time and wasting. At the Southside plant, process changes like increasing dissolved oxygen and return activated sludge flow enabled some ammonia removal without dedicated nitrogen removal infrastructure. Both plants saw performance benefits from improved data monitoring, process control, and solids inventory management.
This document discusses a concept for avionics systems that provides fault tolerance through rapid start-up and restart capabilities using self-checking pairs (SCP). The concept was developed for unmanned vehicles and involves building the system from a rapid start/restart SCP building block. The next phase of research will involve building a demonstration system to test rapid start-ups in response to simulated faults and developing a prototype SCP building block.
Mechanical Simulations for Electronic ProductsAnsys
As electronic devices become smaller and more ubiquitous, the printed circuit boards and components that drive them face increasing power densities and evermore complexity. To ensure product reliability and performance, accurate and detailed analysis methodologies are necessary.
The document discusses load balancing and intelligent load balancing. It covers load balancing architecture, how the data collector and dynamic store work, and how performance counters are used. Intelligent load balancing techniques like load throttling are explained. Potential issues that could cause load imbalances like the "black hole effect" or failing to read performance counters are also reviewed. Troubleshooting techniques for resolving common problems are provided.
This document discusses designing applications for resiliency in cloud environments. It defines resiliency, high availability, and disaster recovery. It describes why resiliency is important given the transient faults that can occur in cloud systems. The document outlines a process for improving resiliency that includes planning, designing, implementing, testing, deploying, monitoring, and responding to failures. It provides examples of resiliency techniques like load balancing, failover/failback, data replication, retries, circuit breakers, and deployment strategies.
This document discusses the Bottlenecks project which aims to identify system bottlenecks in OPNFV infrastructure through testing. It provides updates on Release B which implemented throughput testing and virtual switch testing. Release C planning includes refactoring deployment using Puppet for increased reusability, implementing multi-scenario testing for different configurations, and cooperating with other projects to integrate requirements and tools. The goal is to automatically detect bottlenecks during staging to prevent issues in production.
How to make a Load Testing with Visual Studio 2012Chen-Tien Tsai
About Load Testing
Performance testing
Stress testing
Load testing
Create Visual Studio Load Testing Solution
Add Web Perfomance Testing
Create by Wizard(record action)
Create by Code
Add Load Testing
Configure Scenario setting
Configuration Run setting
Open Load Testing Report
Tips for setting
One more thing about Load Testing
Successful Software Development with Apache CassandraDataStax Academy
The document provides an overview of software development with Cassandra, including data modeling, writing code, testing, reviewing code, and managing environments. It discusses using the Java driver for writing code, preparing statements, batches, and lightweight transactions. Testing recommendations include using naming schemes, grouping tests into logical suites, and integrating unit and integration tests. The document also covers tracing queries, instrumentation with metrics, and reviewing code.
Tracing allows you to see the path a query takes through the Cassandra cluster. It shows details like which nodes are queried, how long each step takes, and can help identify performance bottlenecks. The tracing information can be accessed via the Java driver, cqlsh, or DevCenter and provides a detailed timeline of query execution. Reviewing traces is recommended during development to catch unexpected query behavior.
The document provides information on upgrading Oracle E-Business Suite Release 11i to Release 12, including planning, preparing, performing the upgrade, and post-upgrade tasks. Key steps include applying the latest 11i patches, running the TUMS utility, upgrading the database to at least Oracle 10g Release 2, laying down the new Release 12 technology stack, and running the upgrade driver to migrate the applications to Release 12. The document outlines important tasks for each phase of the upgrade process.
Microsoft Solutions Partner in USA DynaTechhenrryfor680
DynaTech Systems, a leading technology company based in the USA, has recently achieved a significant milestone by attaining Microsoft Solution Partner status. This accomplishment marks a pivotal moment in DynaTech's journey, solidifying its position as a trusted provider of innovative solutions in the dynamic landscape of technology. In this comprehensive overview, we delve into the significance of this achievement, its implications for DynaTech, and the benefits it offers to its clientele and the broader tech community.
The document provides best practices for load testing Oracle applications. It discusses recording scripts using names rather than numbers, setting up interfaces between Oracle and third party applications like iSupplier, creating test data, configuring runtime settings like pacing and think time, ensuring uniform load distribution, and avoiding issues like controller crashes. Scripting standards are also covered such as parameterization, exception handling, and reducing jar file downloads.
UPGRADING FROM ORACLE ENTERPRISE MANAGER 10G TO CLOUD CONTROL 12C WITH ZERO D...Leighton Nelson
A step-by-step description of using the 2-System Method to upgrade from Oracle Enterprise Manager 10g to Enterprise Manager Cloud Control 12c while upgrading database and migrating platforms with near zero downtime.
Webinar: Automating the Creation and Use of Virtual Testing Environments Skytap Cloud
Skytap provides on-demand cloud environments to help enterprises deliver better software faster by removing inefficiencies in the product development lifecycle. Traditional on-premise environments cause delays, restrict parallel development, and lead to defects due to inconsistent environments. Skytap's self-service cloud environments eliminate these problems by providing on-demand, pre-configured environments that developers can access immediately, reducing the time spent on environment management and reproducing defects.
Presentation delivered by Matt Done, Head Of Platform Development at expanz Pty. Ltd. during DDD Sydney event on 2 July 2011.
Matt demonstrates what it takes to setup a highly sophisticated load test, using the Azure environment and how to use the results to optimise a fully blown application development platform and application server running on Azure.
Recording of this presentation can be found at www.youtube.com/expanzTV
Adding Value in the Cloud with Performance TestRodolfo Kohn
This document discusses the importance of performance testing cloud applications and outlines best practices for defining performance requirements, testing methodology, and identifying issues. It provides examples of performance problems found in databases, applications, operating systems, and networks. The key goals of performance testing are to understand system behavior under load, find bottlenecks and hidden bugs, and verify that requirements are met.
This document discusses continuous performance testing (CPT) and introduces the Jagger CPT solution. It provides an overview of why performance testing is important, outlines the principles and goals of CPT, and describes the key parts of the Jagger CPT platform including load generation, metrics collection, test data management, and environment management. It also provides an example customer success story where Jagger was used for continuous performance testing of a large ecommerce site.
This document provides guidance for planning an upgrade from Oracle 9i, 10g, or 11g to Oracle Database 12cR1, including reviewing documentation, choosing an upgrade method, applying necessary patches, performing quality assurance testing on a test database, and defining recovery procedures to validate the upgrade plan before upgrading a production database. Thorough planning and testing are emphasized to help ensure a successful database upgrade.
VMworld 2013: Moving Enterprise Application Dev/Test to VMware’s Internal Pri...VMworld
VMworld 2013
Thirumalesh Reddy, VMware
Padmaja Vrudhula, VMware
Learn more about VMworld and register at http://www.vmworld.com/index.jspa?src=socmed-vmworld-slideshare
VMworld 2013: The Missing Link: Storage Visibility In Virtualized Environments VMworld
VMworld 2013
Matt Cowger, EMC
Mahesh Kumar, VMware
Learn more about VMworld and register at http://www.vmworld.com/index.jspa?src=socmed-vmworld-slideshare
Studying the Integration Practices and the Evolution of Ad Libraries in the G...SAIL_QU
In-app advertisements have become a major revenue for app developers in the mobile app economy. Ad libraries play an integral part in this ecosystem as app
developers integrate these libraries into their apps to display ads. However, little is known about how app developers integrate these libraries with their apps and how these libraries have evolved over time.
In this thesis, we study the ad library integration practices and the evolution of such libraries. To understand the integration practices of ad libraries, we manually study apps and derive a set of rules to automatically identify four strategies for integrating
multiple ad libraries. We observe that integrating multiple ad libraries commonly occurs in apps with a large number of downloads and ones in categories with a high percentage of apps that display ads. We also observe that app developers prefer to manage their own integrations instead of using off the shelf features of ad libraries for integrating multiple ad libraries.
To study the evolution of ad libraries, we conduct a longitudinal study of the 8 most popular ad libraries. In particular, we look at their evolution in terms of size, the main drivers for releasing a new ad library version, and their architecture. We observe that ad libraries are continuously evolving with a median release interval of 34 days. Some ad libraries have grown exponentially in size (e.g., Facebook Audience Network ad library), while other libraries have worked to reduce their size. To study the main drivers for releasing an ad library version, we manually study the release notes of the eight studied ad libraries. We observe that ad library developers continuously update their ad libraries to support a wider range of Android versions (i.e., to ensure that more devices can use the libraries without errors). Finally, we derive a reference architecture for ad libraries and study how the studied ad libraries diverged from this architecture during our study period.
Our findings can assist ad library developers to understand the challenges for developing ad libraries and the desired features of these libraries.
The document discusses load balancing and intelligent load balancing. It covers load balancing architecture, how the data collector and dynamic store work, and how performance counters are used. Intelligent load balancing techniques like load throttling are explained. Potential issues that could cause load imbalances like the "black hole effect" or failing to read performance counters are also reviewed. Troubleshooting techniques for resolving common problems are provided.
This document discusses designing applications for resiliency in cloud environments. It defines resiliency, high availability, and disaster recovery. It describes why resiliency is important given the transient faults that can occur in cloud systems. The document outlines a process for improving resiliency that includes planning, designing, implementing, testing, deploying, monitoring, and responding to failures. It provides examples of resiliency techniques like load balancing, failover/failback, data replication, retries, circuit breakers, and deployment strategies.
This document discusses the Bottlenecks project which aims to identify system bottlenecks in OPNFV infrastructure through testing. It provides updates on Release B which implemented throughput testing and virtual switch testing. Release C planning includes refactoring deployment using Puppet for increased reusability, implementing multi-scenario testing for different configurations, and cooperating with other projects to integrate requirements and tools. The goal is to automatically detect bottlenecks during staging to prevent issues in production.
How to make a Load Testing with Visual Studio 2012Chen-Tien Tsai
About Load Testing
Performance testing
Stress testing
Load testing
Create Visual Studio Load Testing Solution
Add Web Perfomance Testing
Create by Wizard(record action)
Create by Code
Add Load Testing
Configure Scenario setting
Configuration Run setting
Open Load Testing Report
Tips for setting
One more thing about Load Testing
Successful Software Development with Apache CassandraDataStax Academy
The document provides an overview of software development with Cassandra, including data modeling, writing code, testing, reviewing code, and managing environments. It discusses using the Java driver for writing code, preparing statements, batches, and lightweight transactions. Testing recommendations include using naming schemes, grouping tests into logical suites, and integrating unit and integration tests. The document also covers tracing queries, instrumentation with metrics, and reviewing code.
Tracing allows you to see the path a query takes through the Cassandra cluster. It shows details like which nodes are queried, how long each step takes, and can help identify performance bottlenecks. The tracing information can be accessed via the Java driver, cqlsh, or DevCenter and provides a detailed timeline of query execution. Reviewing traces is recommended during development to catch unexpected query behavior.
The document provides information on upgrading Oracle E-Business Suite Release 11i to Release 12, including planning, preparing, performing the upgrade, and post-upgrade tasks. Key steps include applying the latest 11i patches, running the TUMS utility, upgrading the database to at least Oracle 10g Release 2, laying down the new Release 12 technology stack, and running the upgrade driver to migrate the applications to Release 12. The document outlines important tasks for each phase of the upgrade process.
Microsoft Solutions Partner in USA DynaTechhenrryfor680
DynaTech Systems, a leading technology company based in the USA, has recently achieved a significant milestone by attaining Microsoft Solution Partner status. This accomplishment marks a pivotal moment in DynaTech's journey, solidifying its position as a trusted provider of innovative solutions in the dynamic landscape of technology. In this comprehensive overview, we delve into the significance of this achievement, its implications for DynaTech, and the benefits it offers to its clientele and the broader tech community.
The document provides best practices for load testing Oracle applications. It discusses recording scripts using names rather than numbers, setting up interfaces between Oracle and third party applications like iSupplier, creating test data, configuring runtime settings like pacing and think time, ensuring uniform load distribution, and avoiding issues like controller crashes. Scripting standards are also covered such as parameterization, exception handling, and reducing jar file downloads.
UPGRADING FROM ORACLE ENTERPRISE MANAGER 10G TO CLOUD CONTROL 12C WITH ZERO D...Leighton Nelson
A step-by-step description of using the 2-System Method to upgrade from Oracle Enterprise Manager 10g to Enterprise Manager Cloud Control 12c while upgrading database and migrating platforms with near zero downtime.
Webinar: Automating the Creation and Use of Virtual Testing Environments Skytap Cloud
Skytap provides on-demand cloud environments to help enterprises deliver better software faster by removing inefficiencies in the product development lifecycle. Traditional on-premise environments cause delays, restrict parallel development, and lead to defects due to inconsistent environments. Skytap's self-service cloud environments eliminate these problems by providing on-demand, pre-configured environments that developers can access immediately, reducing the time spent on environment management and reproducing defects.
Presentation delivered by Matt Done, Head Of Platform Development at expanz Pty. Ltd. during DDD Sydney event on 2 July 2011.
Matt demonstrates what it takes to setup a highly sophisticated load test, using the Azure environment and how to use the results to optimise a fully blown application development platform and application server running on Azure.
Recording of this presentation can be found at www.youtube.com/expanzTV
Adding Value in the Cloud with Performance TestRodolfo Kohn
This document discusses the importance of performance testing cloud applications and outlines best practices for defining performance requirements, testing methodology, and identifying issues. It provides examples of performance problems found in databases, applications, operating systems, and networks. The key goals of performance testing are to understand system behavior under load, find bottlenecks and hidden bugs, and verify that requirements are met.
This document discusses continuous performance testing (CPT) and introduces the Jagger CPT solution. It provides an overview of why performance testing is important, outlines the principles and goals of CPT, and describes the key parts of the Jagger CPT platform including load generation, metrics collection, test data management, and environment management. It also provides an example customer success story where Jagger was used for continuous performance testing of a large ecommerce site.
This document provides guidance for planning an upgrade from Oracle 9i, 10g, or 11g to Oracle Database 12cR1, including reviewing documentation, choosing an upgrade method, applying necessary patches, performing quality assurance testing on a test database, and defining recovery procedures to validate the upgrade plan before upgrading a production database. Thorough planning and testing are emphasized to help ensure a successful database upgrade.
VMworld 2013: Moving Enterprise Application Dev/Test to VMware’s Internal Pri...VMworld
VMworld 2013
Thirumalesh Reddy, VMware
Padmaja Vrudhula, VMware
Learn more about VMworld and register at http://www.vmworld.com/index.jspa?src=socmed-vmworld-slideshare
VMworld 2013: The Missing Link: Storage Visibility In Virtualized Environments VMworld
VMworld 2013
Matt Cowger, EMC
Mahesh Kumar, VMware
Learn more about VMworld and register at http://www.vmworld.com/index.jspa?src=socmed-vmworld-slideshare
Studying the Integration Practices and the Evolution of Ad Libraries in the G...SAIL_QU
In-app advertisements have become a major revenue for app developers in the mobile app economy. Ad libraries play an integral part in this ecosystem as app
developers integrate these libraries into their apps to display ads. However, little is known about how app developers integrate these libraries with their apps and how these libraries have evolved over time.
In this thesis, we study the ad library integration practices and the evolution of such libraries. To understand the integration practices of ad libraries, we manually study apps and derive a set of rules to automatically identify four strategies for integrating
multiple ad libraries. We observe that integrating multiple ad libraries commonly occurs in apps with a large number of downloads and ones in categories with a high percentage of apps that display ads. We also observe that app developers prefer to manage their own integrations instead of using off the shelf features of ad libraries for integrating multiple ad libraries.
To study the evolution of ad libraries, we conduct a longitudinal study of the 8 most popular ad libraries. In particular, we look at their evolution in terms of size, the main drivers for releasing a new ad library version, and their architecture. We observe that ad libraries are continuously evolving with a median release interval of 34 days. Some ad libraries have grown exponentially in size (e.g., Facebook Audience Network ad library), while other libraries have worked to reduce their size. To study the main drivers for releasing an ad library version, we manually study the release notes of the eight studied ad libraries. We observe that ad library developers continuously update their ad libraries to support a wider range of Android versions (i.e., to ensure that more devices can use the libraries without errors). Finally, we derive a reference architecture for ad libraries and study how the studied ad libraries diverged from this architecture during our study period.
Our findings can assist ad library developers to understand the challenges for developing ad libraries and the desired features of these libraries.
Improving the testing efficiency of selenium-based load testsSAIL_QU
Slides for a paper published at AST 2019:
Shahnaz M. Shariff, Heng Li, Cor-Paul Bezemer, Ahmed E. Hassan, Thanh H. D. Nguyen, and Parminder Flora. 2019. Improving the testing efficiency of selenium-based load tests. In Proceedings of the 14th International Workshop on Automation of Software Test (AST '19). IEEE Press, Piscataway, NJ, USA, 14-20. DOI: https://doi.org/10.1109/AST.2019.00008
Studying User-Developer Interactions Through the Distribution and Reviewing M...SAIL_QU
This document discusses studying user-developer interactions through the distribution and reviewing mechanisms of the Google Play Store. It analyzes emergency updates made by developers to fix issues, the dialogue between users and developers through reviews and responses, and how the reviewing mechanism can help identify good and bad updates. The study found that responding to reviews is six times more likely to increase an app's rating, with 84% of rating increases going to four or five stars. Three common patterns of developer responses were identified: responding to negative or long reviews, only negative reviews, and reviews shortly after an update.
Studying online distribution platforms for games through the mining of data f...SAIL_QU
Our studies of Steam platform data provided insights into online game distribution:
1) Urgent game updates were used to fix crashes, balance issues, and functionality; frequent updaters released more 0-day patches.
2) The Early Access model attracted indie developers and increased game participation; reviews were more positive during Early Access.
3) Game reviews were typically short and in English; sales increased review volume more than new updates; negative reviews came after longer play.
Understanding the Factors for Fast Answers in Technical Q&A Websites: An Empi...SAIL_QU
This study analyzed factors that impact the speed of questions receiving accepted answers on four popular Stack Exchange websites: Stack Overflow, Mathematics, Ask Ubuntu, and Super User. The researchers examined question, answerer, asker, and answer factors from over 150,000 questions. They built classification models and found that key factors for fast answers included the past speed of answerers, length of the question, and past speed of answers for the question's tags. The models achieved AUCs of 0.85-0.95. Fast answers relied heavily on answerers, especially frequent answerers. The study suggests improving incentives for non-frequent and more difficult questions to attract diverse answerers.
Investigating the Challenges in Selenium Usage and Improving the Testing Effi...SAIL_QU
Selenium is a popular tool for browser-based automation testing. The author analyzes challenges in using Selenium by mining Selenium questions on Stack Overflow. Programming language-related questions, especially for Java and Python, are most common and growing fastest. Less than half of questions receive accepted answers, and questions about browsers and components take longest. In the second part, the author develops an approach to improve efficiency of Selenium-based load testing by sharing browsers among user instances. This increases the number of error-free users by 20-22% while reducing memory usage.
Mining Development Knowledge to Understand and Support Software Logging Pract...SAIL_QU
This document summarizes Heng Li's PhD thesis on mining development knowledge to understand and support software logging practices. It discusses how logging code is used to record runtime information but can be difficult for developers to maintain. The thesis aims to understand current logging practices and develop tools by mining change history, source code, issue reports, and other development knowledge. It presents research that analyzes logging-related issues to identify developers' logging concerns, uses code topics and structure to predict where logging statements should be added, leverages code changes to suggest when logging code needs updating, and applies machine learning models to recommend appropriate log levels.
Which Log Level Should Developers Choose For a New Logging Statement?SAIL_QU
The document discusses choosing an appropriate log level when adding a new logging statement. It finds that an ordinal regression model can effectively model log levels, achieving an AUC of 0.76-0.81 in within-project evaluation and 0.71-0.8 in cross-project evaluation. The most influential factors for determining log levels vary between projects and include metrics related to the logging statement, containing code block, and file as well as code change and historical change metrics.
Towards Just-in-Time Suggestions for Log ChangesSAIL_QU
The document presents a study on providing just-in-time suggestions for log changes when developers make code changes. The researchers analyzed over 32,000 log changes from 4 systems. They found 20 reasons for log changes that fall into 4 categories: block changes, log improvements, dependence-driven changes, and logging issues. A random forest classifier using 25 software metrics related to code changes, history, and complexity achieved 0.84-0.91 AUC in predicting whether a log change is needed. Change metrics and product metrics were the most influential factors. The study aims to help developers make better logging decisions for failure diagnosis.
The Impact of Task Granularity on Co-evolution AnalysesSAIL_QU
The document discusses how task granularity at different levels (e.g. commits, pull requests, work items) can impact analyses of co-evolution in software projects. It finds that analyzing at the commit-level can overlook relationships between tasks that span multiple commits. Work item level analysis is recommended to provide a more complete view of co-evolution, as median of 29% of work items consist of multiple commits, and analyzing at the commit level would miss 24% of co-changed files and inability to group 83% of related commits.
How are Discussions Associated with Bug Reworking? An Empirical Study on Open...SAIL_QU
1) Initial bug fix discussions with more comments and more developers participating are more likely to experience later bug reworking through re-opening or re-patching of the bug.
2) Manual analysis found that defective initial fixes and failure to reach consensus in discussions contributed to later reworking.
3) For re-opened bugs, initial discussions focused on addressing a particular problem through a burst of comments, while re-patched bugs lacked thorough code review and testing during the initial fix period.
A Study of the Relation of Mobile Device Attributes with the User-Perceived Q...SAIL_QU
This study examined the relationship between mobile device attributes and user-perceived quality of Android apps. The researchers analyzed 150,373 star ratings from Google Play across 30 devices and 280 apps. They found that the perceived quality of apps varies across devices, and having better characteristics of an attribute does not necessarily correlate with higher quality. Device OS version, resolution, and CPU showed significant relationships with ratings, as did some app attributes like lines of code and number of inputs. However, some device attributes had stronger relationships than app attributes.
A Large-Scale Study of the Impact of Feature Selection Techniques on Defect C...SAIL_QU
This document presents the results of a large-scale study on the impact of feature selection techniques on defect classification models. The study used expanded scopes including multiple datasets from NASA and PROMISE with different feature types, more classification techniques from different paradigms, and additional feature selection techniques. The results show that correlation-based feature subset selection techniques like FS1 and FS2 consistently appear in the top ranks across most of the datasets, projects within the datasets, and classification techniques. The document concludes that future defect classification studies should consider applying correlation-based feature selection techniques.
Studying the Dialogue Between Users and Developers of Free Apps in the Google...SAIL_QU
The study analyzes user-developer interactions through reviews and responses on the Google Play Store. It finds that responding to reviews has a significant positive impact, with 84% of rating increases due to the developer addressing the issue or providing guidance. Three common response patterns were identified: only negative reviews, negative or longer reviews, and reviews shortly after an update. Developers most often thank the user, ask for details, provide guidance, or ask for an endorsement. Guidance responses can address common issues through FAQs. The analysis considered over 2,000 apps, 355,000 review changes, 128,000 responses, and 4 million reviews.
What Do Programmers Know about Software Energy Consumption?SAIL_QU
This document summarizes the results of a survey of 122 programmers about their knowledge of software energy consumption. The survey found that programmers have limited awareness of energy consumption and how to reduce it. They were unaware of the main causes of high energy usage. Programmers lacked knowledge about how to properly rank the energy consumption of different hardware components and were unfamiliar with strategies to improve efficiency, such as minimizing I/O and avoiding polling. The study concludes that programmers would benefit from more education on software energy usage and its causes.
Revisiting the Experimental Design Choices for Approaches for the Automated R...SAIL_QU
Prior research on automated duplicate issue report retrieval focused on improving performance metrics like recall rate. The author revisits experimental design choices from four perspectives: needed effort, data changes, data filtration, and evaluation process.
The thesis contributions are: 1) Showing the importance of considering needed effort in performance measurement. 2) Proposing a "realistic evaluation" approach and analyzing prior findings with it. 3) Developing a genetic algorithm to filter old issue reports and improve performance. 4) Highlighting the impact of "just-in-time" features on evaluation. The findings help better understand benefits and limitations of prior work in this area.
Measuring Program Comprehension: A Large-Scale Field Study with ProfessionalsSAIL_QU
The document summarizes a large-scale field study that tracked the program comprehension activities of 78 professional developers over 3,148 hours. The study found that:
1) Program comprehension accounted for approximately 58% of developers' time on average, with navigation and editing making up the remaining portions.
2) Developers frequently used web browsers and document editors to aid comprehension beyond just IDEs.
3) Interviews and observations revealed that insufficient documentation, unclear code, and complex inheritance hierarchies contributed to long comprehension sessions.
Measuring Program Comprehension: A Large-Scale Field Study with Professionals
Issre2010 malik
1. Pinpointing the Subsystems Responsible for
Performance Deviations In a Load Test
1
Haroon Malik, Bram Adams & Ahmed E. Hassan
Software Analysis and Intelligence Lab (SAIL)
Queen’s University, Kingston, Canada
14. WE CAN HELP ANALYSTS:
Decide if a performance test passed or failed
CSMR 2010
Identify the subsystems which violated the
performance COMPSAC 2010
Pinpoint the subsystems that are likely the
cause of performance violation ISSRE 2010
15
18. Principal Component Analysis (PCA)
Explains most of the counter data with minimal
information loss
Removes the noise in the counter data
Influential Counters
Counter Elimination: Norman cut-off criteria
Counter Ranking
2. CRAFTING PERFORMANCE SIGNATURES
19
25. DELL DVD STORE
26
Components of Test Environment
Load
GeneratorDatabase
Server
Web Server (B)
Web Server (C)
Web Server (A)
LoadGenerators
Performance
Logs
Performance Monitoring Tool
26. Base 4-X Dev %
Web-1 0.87 0.72 0.15 17.1
Web-2 0.88 0.82 0.05 6.46
Web-3 0.89 0.83 0.06 7.03
DB 0.78 0.73 0.05 6.92
EXPERIMENT-1
4X- LOAD ON WEB-1
Base CPU Dev %
Web-1 0.87 0.69 0.18 21.1
Web-2 0.88 0.80 0.08 9.29
Web-3 0.89 0.80 0.09 10.6
DB 0.78 0.73 0.05 7.24
Base CPU Dev %
Web-1 0.87 0.83 0.03 4.28
Web-2 0.88 0.83 0.04 5.04
Web-3 0.89 0.84 0.05 6.14
DB 0.78 0.78 0.08 10.4
Base MEM Dev %
Web-1 0.87 0.81 0.06 7.42
Web-2 0.88 0.75 0.13 14.9
Web-3 0.89 0.81 0.08 9.49
DB 0.78 0.7 0.087 11.0
EXPERIMENT-2
CPU STRESS ON WEB-1
EXPERIMENT-3
CPU STRESS ON DB
EXPERIMENT-4
MEMORY STRESS ON WEB-2
27
29. WHY ARE TEST-D & E DIFFERENT?
PINPOINTED – Among 6 subsystems for each
test, web server is likely the cause of
performance deviation in both tests
Signature counters of web server notably
deviated from the baseline:
The packet Outbound Discarded,
Packet Sent/Sec
Message Queue Length
Network Problem – No connectivity to the
database
Web server Under Stress 30
30. LIMITATIONS
Our methodology can only point *A*
subsystem that is likely the cause of
performance deviation for a load test.
Our methodology can not be generalized to
other domains such as network traffic and
security monitoring.
31
So at SAIL lab, we thrive investigating approaches and creating techniques to support practitioners who are producing, maintaining and evolving large scale complex software systems. We aim to develop tools to make the lives of these practitioners more productive, more cheerful and more predictable.
Today's LSS such as Google, eBay ,Facebook and Amazon are composed of many underlying components and subsystems.
These LSS are tremendously growing in size to handle growing traffic, complex services and business critical functionality.
It is very important to periodically measure the performance of such LLS to satisfy the stakeholders and customers high demands on system quality, availability and responsiveness.
-Load testing is an important weapon in LSS development to uncover functional and performance problems of a system under load
Performance of the LSS is calibrated using Load test before it becomes a field or post-deployment problem.
-Performance problems include an application not responding fast enough, crashing or hanging under heavy load or not meeting the desired service level agreements (SLA).
Environment-Setup:
First but the most important phase of the load testing. As the most common load test failures occurs due to improper environment setup for load tests.
The environment setup includes installing the applications and load testing tools on different machines and possibly on different operating systems
Load generators, which emulates the users interaction with the systems, need to carefully configured to match the real workload in field.
Load Test Execution:
It involves starting the components of the system under test, i.e., starting required services, hardware resources and tools ( load generators and performance monitors).
Performance counter are recorded in this step too.
Load Test Analysis:
This step involves comparing the results of a load test against an other load tests results or against predefined thresh holds as baselines.
Unlike functional and unit testing, which results in pass of failure classification for each test; load testing requires additional quantitative metrics like response time, throughput and hardware resources utilization to summarize results.
The performance analysts selects few of the important performance counters among thousands collected. Based on his experience and domain knowledge performance analyst manually compare the selected performance counters with those of past runs to look for evidence of performance deviations, for example using plots and performing correlations tests.
Report Generation:
Includes filing the performance deviations, if found, based on the personal judgment of an analysts. Mostly the results produced are verified by an experience analysts.
Based on the extent of performance deviation and its relevance to team responsible for handling the subsystems i.e., (database, application, web system etc.)
Environment-Setup:
First but the most important phase of the load testing. As the most common load test failures occurs due to improper environment setup for load tests.
The environment setup includes installing the applications and load testing tools on different machines and possibly on different operating systems
Load generators, which emulates the users interaction with the systems, need to carefully configured to match the real workload in field.
Load Test Execution:
It involves starting the components of the system under test, i.e., starting required services, hardware resources and tools ( load generators and performance monitors).
Performance counter are recorded in this step too.
Load Test Analysis:
This step involves comparing the results of a load test against an other load tests results or against predefined thresh holds as baselines.
Unlike functional and unit testing, which results in pass of failure classification for each test; load testing requires additional quantitative metrics like response time, throughput and hardware resources utilization to summarize results.
The performance analysts selects few of the important performance counters among thousands collected. Based on his experience and domain knowledge performance analyst manually compare the selected performance counters with those of past runs to look for evidence of performance deviations, for example using plots and performing correlations tests.
Report Generation:
Includes filing the performance deviations, if found, based on the personal judgment of an analysts. Mostly the results produced are verified by an experience analysts.
Based on the extent of performance deviation and its relevance to team responsible for handling the subsystems i.e., (database, application, web system etc.)
Environment-Setup:
First but the most important phase of the load testing. As the most common load test failures occurs due to improper environment setup for load tests.
The environment setup includes installing the applications and load testing tools on different machines and possibly on different operating systems
Load generators, which emulates the users interaction with the systems, need to carefully configured to match the real workload in field.
Load Test Execution:
It involves starting the components of the system under test, i.e., starting required services, hardware resources and tools ( load generators and performance monitors).
Performance counter are recorded in this step too.
Load Test Analysis:
This step involves comparing the results of a load test against an other load tests results or against predefined thresh holds as baselines.
Unlike functional and unit testing, which results in pass of failure classification for each test; load testing requires additional quantitative metrics like response time, throughput and hardware resources utilization to summarize results.
The performance analysts selects few of the important performance counters among thousands collected. Based on his experience and domain knowledge performance analyst manually compare the selected performance counters with those of past runs to look for evidence of performance deviations, for example using plots and performing correlations tests.
Report Generation:
Includes filing the performance deviations, if found, based on the personal judgment of an analysts. Mostly the results produced are verified by an experience analysts.
Based on the extent of performance deviation and its relevance to team responsible for handling the subsystems i.e., (database, application, web system etc.)
Environment-Setup:
First but the most important phase of the load testing. As the most common load test failures occurs due to improper environment setup for load tests.
The environment setup includes installing the applications and load testing tools on different machines and possibly on different operating systems
Load generators, which emulates the users interaction with the systems, need to carefully configured to match the real workload in field.
Load Test Execution:
It involves starting the components of the system under test, i.e., starting required services, hardware resources and tools ( load generators and performance monitors).
Performance counter are recorded in this step too.
Load Test Analysis:
This step involves comparing the results of a load test against an other load tests results or against predefined thresh holds as baselines.
Unlike functional and unit testing, which results in pass of failure classification for each test; load testing requires additional quantitative metrics like response time, throughput and hardware resources utilization to summarize results.
The performance analysts selects few of the important performance counters among thousands collected. Based on his experience and domain knowledge performance analyst manually compare the selected performance counters with those of past runs to look for evidence of performance deviations, for example using plots and performing correlations tests.
Report Generation:
Includes filing the performance deviations, if found, based on the personal judgment of an analysts. Mostly the results produced are verified by an experience analysts.
Based on the extent of performance deviation and its relevance to team responsible for handling the subsystems i.e., (database, application, web system etc.)
- Unfortunately, the current practice to analyze load test is costly, time consuming and error prone.
This is due to the fact that the load test analysis practices have not kept pace with the rapid growth in size and complexity of the large enterprise systems.
In practice, the dominant tools and techniques to analyze large distributed systems have remained unchanged for over twenty years
Most of the research had focus on the automatic generation of load testing suits rather then load test analysis
- There are many challenges and limitation associated with the current practices of load test analysis that remains unsolved
- Last from computer of hours to several days.
They generate performance logs that can be of terra bytes in size
Even logging all counters on typical machine at 1Hz generates about 8.6 million values in a single weeks
A cluster of 12 machine over a week –13 TB of performance counter data per week. Assuming 64 bit representation for each counter value.
Analysis of such large counter log is still a bit challenge in load tests.
Performance analysts in LSS have only limited time to reach and complete diagnostics on performance counter logs and to make necessary configuration changes.
Load testing is usually the last step in an already tight and usually delayed release schedule. Hence, managers are always eager to reduce the time allocated for performance testing.
Error prone because of manual process involved in analyzing performance counter data in current practice
Impossible for an analyst to skim through large volume of log data, indeed they analyst use few key performance counters know to them from past practices, performance experts and domain trends as ‘ rule of thumbs’.
With large scale system that are continuously being evlved by adding new functionalities, applying same rules of thumb can mislead performance issues.
Due to these challenges, we believe the current practice to perform load test analysis is neither effective nor sufficient to uncover performance deviations accurately and in limited time.
-
1) The performance log obtained from a load test do not suffice for direct analysis by our methodology. These logs need to be prepared to make them suitable for statistical techniques employed by our methodology. In this step take care of data sanitization (missing counter variables and incomplete counter variables) and pre-treatment of data such as standardization and data scaling to remove the biasing of variance depended techniques.