This document presents an approach for automatically detecting performance regressions in heterogeneous environments. It uses association rule mining on performance counter data from past tests to generate performance rules. These rules are then used to detect violation metrics in a new test by identifying significant changes in rule confidence values. Results are combined from multiple heterogeneous lab environments using a weighted voting method based on environment similarities. The approach is evaluated on real-world systems using F-measure and is shown to outperform single model and bagging methods for detecting performance regressions.
Docker–Grid (A On demand and Scalable dockerized selenium grid architecture)STePINForum
by Yogit Thakral, Senior Test Engineer & Kandeel Chauhan, Testing Lead, Naukri.com at STeP-IN SUMMIT 2018 15th International Conference on Software Testing on August 31, 2018 at Taj, MG Road, Bengaluru
When designing a bridge, an architect has al sorts of tools & techniques to verify it won't collapse.
Us software developers... We build. And with a bit of luck, we'll even test. And then we pray it doesn’t collapse :)
Can we measure software? Can we compare code objectively? Can we predict problems?
The Business Case for Test Environment Management ServicesCognizant
In the software development lifecycle, application testing is crucial - but often given short shrift by companies allocating temporary resources. Test environment management services (TEMS) is a perfect solution, reducing testing costs and errors while conducting test/QA environment management, monitoring and maintenance and cloud infrastructure provisioning.
Software analytics (for software quality purpose) is a statistical or machine learning classifier that is trained to identify defect-prone software modules. The goal of software analytics is to help software engineers prioritize their software testing effort on the most-risky modules and understand past pitfalls that lead to defective code. While the adoption of software analytics enables software organizations to distil actionable insights, there are still many barriers to broad and successful adoption of such analytics systems. Indeed, even if software organizations can access such invaluable software artifacts and toolkits for data analytics, researchers and practitioners often have little knowledge to properly develop analytics systems. Thus, the accuracy of the predictions and the insights that are derived from analytics systems is one of the most important challenges of data science in software engineering.
In this work, we conduct a series of empirical investigation to better understand the impact of experimental components (i.e., class mislabelling, parameter optimization of classification techniques, and model validation techniques) on the performance and interpretation of software analytics. To accelerate a large amount of compute-intensive experiment, we leverage the High-Performance-Computing (HPC) resources of Centre for Advanced Computing (CAC) from Queen’s University, Canada. Through case studies of systems that span both proprietary and open- source domains, we demonstrate that (1) realistic noise does not impact the precision of software analytics; (2) automated parameter optimization for classification techniques substantially improve the performance and stability of software analytics; and (3) the out-of- sample bootstrap validation technique produces a good balance between bias and variance of performance estimates. Our results lead us to conclude that the experimental components of analytics modelling impact the predictions and associated insights that are derived from software analytics. Empirical investigations on the impact of overlooked experimental components are needed to derive practical guidelines for analytics modelling.
Docker–Grid (A On demand and Scalable dockerized selenium grid architecture)STePINForum
by Yogit Thakral, Senior Test Engineer & Kandeel Chauhan, Testing Lead, Naukri.com at STeP-IN SUMMIT 2018 15th International Conference on Software Testing on August 31, 2018 at Taj, MG Road, Bengaluru
When designing a bridge, an architect has al sorts of tools & techniques to verify it won't collapse.
Us software developers... We build. And with a bit of luck, we'll even test. And then we pray it doesn’t collapse :)
Can we measure software? Can we compare code objectively? Can we predict problems?
The Business Case for Test Environment Management ServicesCognizant
In the software development lifecycle, application testing is crucial - but often given short shrift by companies allocating temporary resources. Test environment management services (TEMS) is a perfect solution, reducing testing costs and errors while conducting test/QA environment management, monitoring and maintenance and cloud infrastructure provisioning.
Software analytics (for software quality purpose) is a statistical or machine learning classifier that is trained to identify defect-prone software modules. The goal of software analytics is to help software engineers prioritize their software testing effort on the most-risky modules and understand past pitfalls that lead to defective code. While the adoption of software analytics enables software organizations to distil actionable insights, there are still many barriers to broad and successful adoption of such analytics systems. Indeed, even if software organizations can access such invaluable software artifacts and toolkits for data analytics, researchers and practitioners often have little knowledge to properly develop analytics systems. Thus, the accuracy of the predictions and the insights that are derived from analytics systems is one of the most important challenges of data science in software engineering.
In this work, we conduct a series of empirical investigation to better understand the impact of experimental components (i.e., class mislabelling, parameter optimization of classification techniques, and model validation techniques) on the performance and interpretation of software analytics. To accelerate a large amount of compute-intensive experiment, we leverage the High-Performance-Computing (HPC) resources of Centre for Advanced Computing (CAC) from Queen’s University, Canada. Through case studies of systems that span both proprietary and open- source domains, we demonstrate that (1) realistic noise does not impact the precision of software analytics; (2) automated parameter optimization for classification techniques substantially improve the performance and stability of software analytics; and (3) the out-of- sample bootstrap validation technique produces a good balance between bias and variance of performance estimates. Our results lead us to conclude that the experimental components of analytics modelling impact the predictions and associated insights that are derived from software analytics. Empirical investigations on the impact of overlooked experimental components are needed to derive practical guidelines for analytics modelling.
The government of Bangladesh has set a target to bring the whole country under electricity coverage by 2021. Bangladesh requires multibillion dollar investments in the power sector. Bangladesh government is increasingly diversifying the sources of investment. Innovative funds- Export Credit Agency (ECA) financing is one of the big innovative ways. Power Infrastructure is the backbone of development for a country.
AIHT is one of the best institute of Hospitality, Tourism and hotel management in India. The presentation on “ Eye Contact ", To know detail about important part of eye contact using effective communication skills read full presentation.
Singapore startup ecosystem and entrepreneur toolbox - Aug 2015Arnaud Bonzom
Summary:
1. Startup Ecosystem
2. Dynamics and Trends
3. Government
4. Startups in Singapore
5. 100 People You Should Know
6. Networking
7. Hackathons & Startups Competitions
8. Incubators & Accelerators
9. Fund Raising
10. Coworking Spaces & Offices
11. Talents & Recruitment
12. Visa
13. Outsourcing
14. Thank you
15. Bonus
16. We Don't Have The Paypal Mafia, but...
Add more information to your uploadAdd more information to your uploadAdd more information tAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd morAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to youAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadr uploadAdd more information to your uploade information to your uploado your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more information to your uploadAdd more infAdd more info
Application migration process presentation by t2 tech groupKevin Torf
T2 Tech Group's proven methodology helps organizations migrate applications during a data center relocation. The strategy uses a best-practices approach to efficiently coordinate the following phases of a move: the perquisite stage, mock migration, failover testing, implementation and go-live, and project closeout. Using this approach to coordinate an application migration allows teams to efficiently build a solid application strategy; validate the strategy through testing; and go-live with minimal business interruption, a minimized project cost, and an optimal result.
This presentation discusses:
- Best practices regarding the need to ensure that all transformers used for metering circuits are properly sized
- The ability to optimize revenue regardless of customer usage over time
- Best practices to ensure that transformers and meters have been installed correctly in the field and continue to perform in the same fashion
- Best practices for certifying the accuracy class of the transformers and how to best set up a shop testing and field testing/verification program.
- What the costs of implementing such a system and what the costs of not implementing such a system can be.
2011-05-02 - VU Amsterdam - Testing safety critical systemsJaap van Ekris
Presentation about the steps required for Verifying and Vlaidating safety critical systems, as well as the test approach used. Contains examples of real-life IEC 61508 SIL 4 systems.
Similar to An Industrial Case Study on the Automated Detection of Performance Regressions in Heterogeneous Environments (20)
Studying the Integration Practices and the Evolution of Ad Libraries in the G...SAIL_QU
In-app advertisements have become a major revenue for app developers in the mobile app economy. Ad libraries play an integral part in this ecosystem as app
developers integrate these libraries into their apps to display ads. However, little is known about how app developers integrate these libraries with their apps and how these libraries have evolved over time.
In this thesis, we study the ad library integration practices and the evolution of such libraries. To understand the integration practices of ad libraries, we manually study apps and derive a set of rules to automatically identify four strategies for integrating
multiple ad libraries. We observe that integrating multiple ad libraries commonly occurs in apps with a large number of downloads and ones in categories with a high percentage of apps that display ads. We also observe that app developers prefer to manage their own integrations instead of using off the shelf features of ad libraries for integrating multiple ad libraries.
To study the evolution of ad libraries, we conduct a longitudinal study of the 8 most popular ad libraries. In particular, we look at their evolution in terms of size, the main drivers for releasing a new ad library version, and their architecture. We observe that ad libraries are continuously evolving with a median release interval of 34 days. Some ad libraries have grown exponentially in size (e.g., Facebook Audience Network ad library), while other libraries have worked to reduce their size. To study the main drivers for releasing an ad library version, we manually study the release notes of the eight studied ad libraries. We observe that ad library developers continuously update their ad libraries to support a wider range of Android versions (i.e., to ensure that more devices can use the libraries without errors). Finally, we derive a reference architecture for ad libraries and study how the studied ad libraries diverged from this architecture during our study period.
Our findings can assist ad library developers to understand the challenges for developing ad libraries and the desired features of these libraries.
Improving the testing efficiency of selenium-based load testsSAIL_QU
Slides for a paper published at AST 2019:
Shahnaz M. Shariff, Heng Li, Cor-Paul Bezemer, Ahmed E. Hassan, Thanh H. D. Nguyen, and Parminder Flora. 2019. Improving the testing efficiency of selenium-based load tests. In Proceedings of the 14th International Workshop on Automation of Software Test (AST '19). IEEE Press, Piscataway, NJ, USA, 14-20. DOI: https://doi.org/10.1109/AST.2019.00008
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...Globus
Large Language Models (LLMs) are currently the center of attention in the tech world, particularly for their potential to advance research. In this presentation, we'll explore a straightforward and effective method for quickly initiating inference runs on supercomputers using the vLLM tool with Globus Compute, specifically on the Polaris system at ALCF. We'll begin by briefly discussing the popularity and applications of LLMs in various fields. Following this, we will introduce the vLLM tool, and explain how it integrates with Globus Compute to efficiently manage LLM operations on Polaris. Attendees will learn the practical aspects of setting up and remotely triggering LLMs from local machines, focusing on ease of use and efficiency. This talk is ideal for researchers and practitioners looking to leverage the power of LLMs in their work, offering a clear guide to harnessing supercomputing resources for quick and effective LLM inference.
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Globus
The U.S. Geological Survey (USGS) has made substantial investments in meeting evolving scientific, technical, and policy driven demands on storing, managing, and delivering data. As these demands continue to grow in complexity and scale, the USGS must continue to explore innovative solutions to improve its management, curation, sharing, delivering, and preservation approaches for large-scale research data. Supporting these needs, the USGS has partnered with the University of Chicago-Globus to research and develop advanced repository components and workflows leveraging its current investment in Globus. The primary outcome of this partnership includes the development of a prototype enterprise repository, driven by USGS Data Release requirements, through exploration and implementation of the entire suite of the Globus platform offerings, including Globus Flow, Globus Auth, Globus Transfer, and Globus Search. This presentation will provide insights into this research partnership, introduce the unique requirements and challenges being addressed and provide relevant project progress.
top nidhi software solution freedownloadvrstrong314
This presentation emphasizes the importance of data security and legal compliance for Nidhi companies in India. It highlights how online Nidhi software solutions, like Vector Nidhi Software, offer advanced features tailored to these needs. Key aspects include encryption, access controls, and audit trails to ensure data security. The software complies with regulatory guidelines from the MCA and RBI and adheres to Nidhi Rules, 2014. With customizable, user-friendly interfaces and real-time features, these Nidhi software solutions enhance efficiency, support growth, and provide exceptional member services. The presentation concludes with contact information for further inquiries.
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...Juraj Vysvader
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I didn't get rich from it but it did have 63K downloads (powered possible tens of thousands of websites).
SOCRadar Research Team: Latest Activities of IntelBrokerSOCRadar
The European Union Agency for Law Enforcement Cooperation (Europol) has suffered an alleged data breach after a notorious threat actor claimed to have exfiltrated data from its systems. Infamous data leaker IntelBroker posted on the even more infamous BreachForums hacking forum, saying that Europol suffered a data breach this month.
The alleged breach affected Europol agencies CCSE, EC3, Europol Platform for Experts, Law Enforcement Forum, and SIRIUS. Infiltration of these entities can disrupt ongoing investigations and compromise sensitive intelligence shared among international law enforcement agencies.
However, this is neither the first nor the last activity of IntekBroker. We have compiled for you what happened in the last few days. To track such hacker activities on dark web sources like hacker forums, private Telegram channels, and other hidden platforms where cyber threats often originate, you can check SOCRadar’s Dark Web News.
Stay Informed on Threat Actors’ Activity on the Dark Web with SOCRadar!
Into the Box Keynote Day 2: Unveiling amazing updates and announcements for modern CFML developers! Get ready for exciting releases and updates on Ortus tools and products. Stay tuned for cutting-edge innovations designed to boost your productivity.
In software engineering, the right architecture is essential for robust, scalable platforms. Wix has undergone a pivotal shift from event sourcing to a CRUD-based model for its microservices. This talk will chart the course of this pivotal journey.
Event sourcing, which records state changes as immutable events, provided robust auditing and "time travel" debugging for Wix Stores' microservices. Despite its benefits, the complexity it introduced in state management slowed development. Wix responded by adopting a simpler, unified CRUD model. This talk will explore the challenges of event sourcing and the advantages of Wix's new "CRUD on steroids" approach, which streamlines API integration and domain event management while preserving data integrity and system resilience.
Participants will gain valuable insights into Wix's strategies for ensuring atomicity in database updates and event production, as well as caching, materialization, and performance optimization techniques within a distributed system.
Join us to discover how Wix has mastered the art of balancing simplicity and extensibility, and learn how the re-adoption of the modest CRUD has turbocharged their development velocity, resilience, and scalability in a high-growth environment.
Understanding Globus Data Transfers with NetSageGlobus
NetSage is an open privacy-aware network measurement, analysis, and visualization service designed to help end-users visualize and reason about large data transfers. NetSage traditionally has used a combination of passive measurements, including SNMP and flow data, as well as active measurements, mainly perfSONAR, to provide longitudinal network performance data visualization. It has been deployed by dozens of networks world wide, and is supported domestically by the Engagement and Performance Operations Center (EPOC), NSF #2328479. We have recently expanded the NetSage data sources to include logs for Globus data transfers, following the same privacy-preserving approach as for Flow data. Using the logs for the Texas Advanced Computing Center (TACC) as an example, this talk will walk through several different example use cases that NetSage can answer, including: Who is using Globus to share data with my institution, and what kind of performance are they able to achieve? How many transfers has Globus supported for us? Which sites are we sharing the most data with, and how is that changing over time? How is my site using Globus to move data internally, and what kind of performance do we see for those transfers? What percentage of data transfers at my institution used Globus, and how did the overall data transfer performance compare to the Globus users?
Software Engineering, Software Consulting, Tech Lead.
Spring Boot, Spring Cloud, Spring Core, Spring JDBC, Spring Security,
Spring Transaction, Spring MVC,
Log4j, REST/SOAP WEB-SERVICES.
How to Position Your Globus Data Portal for Success Ten Good PracticesGlobus
Science gateways allow science and engineering communities to access shared data, software, computing services, and instruments. Science gateways have gained a lot of traction in the last twenty years, as evidenced by projects such as the Science Gateways Community Institute (SGCI) and the Center of Excellence on Science Gateways (SGX3) in the US, The Australian Research Data Commons (ARDC) and its platforms in Australia, and the projects around Virtual Research Environments in Europe. A few mature frameworks have evolved with their different strengths and foci and have been taken up by a larger community such as the Globus Data Portal, Hubzero, Tapis, and Galaxy. However, even when gateways are built on successful frameworks, they continue to face the challenges of ongoing maintenance costs and how to meet the ever-expanding needs of the community they serve with enhanced features. It is not uncommon that gateways with compelling use cases are nonetheless unable to get past the prototype phase and become a full production service, or if they do, they don't survive more than a couple of years. While there is no guaranteed pathway to success, it seems likely that for any gateway there is a need for a strong community and/or solid funding streams to create and sustain its success. With over twenty years of examples to draw from, this presentation goes into detail for ten factors common to successful and enduring gateways that effectively serve as best practices for any new or developing gateway.
Gamify Your Mind; The Secret Sauce to Delivering Success, Continuously Improv...Shahin Sheidaei
Games are powerful teaching tools, fostering hands-on engagement and fun. But they require careful consideration to succeed. Join me to explore factors in running and selecting games, ensuring they serve as effective teaching tools. Learn to maintain focus on learning objectives while playing, and how to measure the ROI of gaming in education. Discover strategies for pitching gaming to leadership. This session offers insights, tips, and examples for coaches, team leads, and enterprise leaders seeking to teach from simple to complex concepts.
Designing for Privacy in Amazon Web ServicesKrzysztofKkol1
Data privacy is one of the most critical issues that businesses face. This presentation shares insights on the principles and best practices for ensuring the resilience and security of your workload.
Drawing on a real-life project from the HR industry, the various challenges will be demonstrated: data protection, self-healing, business continuity, security, and transparency of data processing. This systematized approach allowed to create a secure AWS cloud infrastructure that not only met strict compliance rules but also exceeded the client's expectations.
Your Digital Assistant.
Making complex approach simple. Straightforward process saves time. No more waiting to connect with people that matter to you. Safety first is not a cliché - Securely protect information in cloud storage to prevent any third party from accessing data.
Would you rather make your visitors feel burdened by making them wait? Or choose VizMan for a stress-free experience? VizMan is an automated visitor management system that works for any industries not limited to factories, societies, government institutes, and warehouses. A new age contactless way of logging information of visitors, employees, packages, and vehicles. VizMan is a digital logbook so it deters unnecessary use of paper or space since there is no requirement of bundles of registers that is left to collect dust in a corner of a room. Visitor’s essential details, helps in scheduling meetings for visitors and employees, and assists in supervising the attendance of the employees. With VizMan, visitors don’t need to wait for hours in long queues. VizMan handles visitors with the value they deserve because we know time is important to you.
Feasible Features
One Subscription, Four Modules – Admin, Employee, Receptionist, and Gatekeeper ensures confidentiality and prevents data from being manipulated
User Friendly – can be easily used on Android, iOS, and Web Interface
Multiple Accessibility – Log in through any device from any place at any time
One app for all industries – a Visitor Management System that works for any organisation.
Stress-free Sign-up
Visitor is registered and checked-in by the Receptionist
Host gets a notification, where they opt to Approve the meeting
Host notifies the Receptionist of the end of the meeting
Visitor is checked-out by the Receptionist
Host enters notes and remarks of the meeting
Customizable Components
Scheduling Meetings – Host can invite visitors for meetings and also approve, reject and reschedule meetings
Single/Bulk invites – Invitations can be sent individually to a visitor or collectively to many visitors
VIP Visitors – Additional security of data for VIP visitors to avoid misuse of information
Courier Management – Keeps a check on deliveries like commodities being delivered in and out of establishments
Alerts & Notifications – Get notified on SMS, email, and application
Parking Management – Manage availability of parking space
Individual log-in – Every user has their own log-in id
Visitor/Meeting Analytics – Evaluate notes and remarks of the meeting stored in the system
Visitor Management System is a secure and user friendly database manager that records, filters, tracks the visitors to your organization.
"Secure Your Premises with VizMan (VMS) – Get It Now"
Advanced Flow Concepts Every Developer Should KnowPeter Caitens
Tim Combridge from Sensible Giraffe and Salesforce Ben presents some important tips that all developers should know when dealing with Flows in Salesforce.
Developing Distributed High-performance Computing Capabilities of an Open Sci...Globus
COVID-19 had an unprecedented impact on scientific collaboration. The pandemic and its broad response from the scientific community has forged new relationships among public health practitioners, mathematical modelers, and scientific computing specialists, while revealing critical gaps in exploiting advanced computing systems to support urgent decision making. Informed by our team’s work in applying high-performance computing in support of public health decision makers during the COVID-19 pandemic, we present how Globus technologies are enabling the development of an open science platform for robust epidemic analysis, with the goal of collaborative, secure, distributed, on-demand, and fast time-to-solution analyses to support public health.
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...informapgpstrackings
Keep tabs on your field staff effortlessly with Informap Technology Centre LLC. Real-time tracking, task assignment, and smart features for efficient management. Request a live demo today!
For more details, visit us : https://informapuae.com/field-staff-tracking/
Prosigns: Transforming Business with Tailored Technology SolutionsProsigns
Unlocking Business Potential: Tailored Technology Solutions by Prosigns
Discover how Prosigns, a leading technology solutions provider, partners with businesses to drive innovation and success. Our presentation showcases our comprehensive range of services, including custom software development, web and mobile app development, AI & ML solutions, blockchain integration, DevOps services, and Microsoft Dynamics 365 support.
Custom Software Development: Prosigns specializes in creating bespoke software solutions that cater to your unique business needs. Our team of experts works closely with you to understand your requirements and deliver tailor-made software that enhances efficiency and drives growth.
Web and Mobile App Development: From responsive websites to intuitive mobile applications, Prosigns develops cutting-edge solutions that engage users and deliver seamless experiences across devices.
AI & ML Solutions: Harnessing the power of Artificial Intelligence and Machine Learning, Prosigns provides smart solutions that automate processes, provide valuable insights, and drive informed decision-making.
Blockchain Integration: Prosigns offers comprehensive blockchain solutions, including development, integration, and consulting services, enabling businesses to leverage blockchain technology for enhanced security, transparency, and efficiency.
DevOps Services: Prosigns' DevOps services streamline development and operations processes, ensuring faster and more reliable software delivery through automation and continuous integration.
Microsoft Dynamics 365 Support: Prosigns provides comprehensive support and maintenance services for Microsoft Dynamics 365, ensuring your system is always up-to-date, secure, and running smoothly.
Learn how our collaborative approach and dedication to excellence help businesses achieve their goals and stay ahead in today's digital landscape. From concept to deployment, Prosigns is your trusted partner for transforming ideas into reality and unlocking the full potential of your business.
Join us on a journey of innovation and growth. Let's partner for success with Prosigns.
Code reviews are vital for ensuring good code quality. They serve as one of our last lines of defense against bugs and subpar code reaching production.
Yet, they often turn into annoying tasks riddled with frustration, hostility, unclear feedback and lack of standards. How can we improve this crucial process?
In this session we will cover:
- The Art of Effective Code Reviews
- Streamlining the Review Process
- Elevating Reviews with Automated Tools
By the end of this presentation, you'll have the knowledge on how to organize and improve your code review proces
An Industrial Case Study on the Automated Detection of Performance Regressions in Heterogeneous Environments
1. An Industrial Case Study on the Automated
Detection of Performance Regressions
in Heterogeneous Environments
2. Flickr outage impacted
89 million users
(05/24/13)
Most field problems for large-scale
systems are rarely functional,
instead they are load-related
One hour global outage
lost $7.2 million in revenue
(02/24/09)
3. Performance Regression Testing
Mimics multiple users repeatedly performing the same tasks
Take hours or even days
Produces GB/TB of data that must be analyzed
10. Initial Attempt
Test N
(tN)
Test 1
(t1)
New Test
(tnew)
.
.
.
Association
Rule Mining
Test 2
(t2)
Perf. Rules
(M)
Detecting
Violation Metric
Violated
Metric Set
(VM)
16. Deriving Frequent Itemset from
Past Test # 1
Time DB read/sec Throughput Request Queue Size
10:00 Medium Medium Low
10:03 Medium Medium Low
10:06 Low Medium Medium
10:09 Medium Medium Low
10:12 Medium Medium Low
10:15 Medium Medium Low
17. Deriving Frequent Itemset from
Past Test # 1
Time DB read/sec Throughput Request Queue Size
10:00 Medium Medium Low
10:03 Medium Medium Low
10:06 Low Medium Medium
10:09 Medium Medium Low
10:12 Medium Medium Low
10:15 Medium Medium Low
Throughput
Medium
DB read/sec
Medium
Request
Queue size
Low
18. Deriving Performance Rules
from Past Test # 1
Throughput
Medium
DB read/sec
Medium
Request
Queue size
Low
Request
Queue Size
Low
DB read/sec
Medium
Throughput
Medium
Throughput
Medium
Request
Queue Size
Low
DB read/sec
Medium
Throughput
Medium
DB read/sec
Medium
Request
Queue size
Low
19. Pruning Performance Rules
• Rules with low support and confidence values are pruned
Throughput
Medium
DB read/sec
Medium
Request
Queue size
Low
Premise Consequence
( 0.5 , 0.9 )
(support, confidence)
Web Server
CPU
Medium
DB read/sec
Medium
Web Server
Memory
High
( 0.1 , 0.7 )
Web Server
CPU
Medium
Web Server
Memory
Medium
Throughput
High
( 0.2 , 0.2 )
20. Detecting Violation Metrics in the
Current Test
Time DB read/sec Throughput Request Queue Size
08:00 Medium Medium High
08:03 Medium Medium High
08:06 Low Medium Medium
08:09 Medium Medium Low
08:12 Medium Medium Low
08:15 Medium Medium High
Throughput
Medium
DB read/sec
Medium
Request
Queue size
Low
21. Detecting Violation Metrics in the
Current Test
Time DB read/sec Throughput Request Queue Size
08:00 Medium Medium High
08:03 Medium Medium High
08:06 Low Medium Medium
08:09 Medium Medium Low
08:12 Medium Medium Low
08:15 Medium Medium High
Throughput
Medium
DB read/sec
Medium
Request
Queue size
Low
Request
Queue size
High
• Rules with significant changes in confidence values are
flagged as “anomalous”
25. v1.71 v5.10v1.71 v5.50 v1.71 v5.50
Perf Lab A Perf Lab B Perf Lab C
Assigning Weights to Past Tests
1 2.2 2
𝒘 𝟏 =
𝟏
𝟏+𝟐.𝟐+𝟐
= 0.20 𝒘 𝟐 =
𝟐.𝟐
𝟏+𝟐.𝟐+𝟐
= 0.42 𝒘 𝟑 =
𝟐
𝟏+𝟐.𝟐+𝟐
= 0.38
T1, T2 T3 T4
0.20 0.42 0.38
26. Combining Results
Throughput
at t0
Throughput
at t1
Throughput
at t2
M1 M2 M3 M4 Anomalous?
Stacking
1.00 v.s. 0.20
0.38 v.s 0.82
(0.20 vote) (0.20 vote) (0.42 vote) (0.38 vote)
0.58 v.s. 0.62
New Test
27. Case Study
Types of Systems Experiments
Dell DVD Store Open Source Benchmark
Application
Bug Injection
JPetStore Open Source Re-
implementation of Oracle’s
Pet Store
Bug Injection
A Large Enterprise
System
Closed Source Large-Scale
Telephony System
Performance Regression
Repository
29. A Large Enterprise System
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
E1 E2 E3
F-measure
Single Bagging Stacking
Editor's Notes
If a system suffers from load-related failures, usually the consequence would result in huge financial losses and impacts large amount of users. Here we show two examples.
http://highscalability.com/latency-everywhere-and-it-costs-you-sales-how-crush-it
http://www.infoworld.com/slideshow/107783/the-worst-cloud-outages-of-2013-so-far-221831
http://techcrunch.com/2013/05/24/flickr-suffers-outage-four-days-after-major-revamp/
Load testing in general assess the system behavior under load to detect load-related problems.
This figure illustrates a typical setup of a load test …
The question facing load testing professionals every day is: “So is the system ready for release”?
How do we dig through these data to find out problems?
Mention NO TEST ORACLE!
Current practices is ad-hoc and involve many high-level checks
Why costly? (many human hours for manually checking the data)
Mainly reply on manual (time consuming and error-prone) approaches for analyzing performance regression tests
Relay mainly on domain knowledge and the results of prior test runs to manually look for large deviations of counter values (e.g., high CPU)
Organizations currently maintain different lab environments to execute performance
There are benchmarks for CPU (e.g., SPEC) to map and compare CPU utilizations among different configurations. But not for other types of resources
Can we automate these kind of manual analysis or maybe even perform deeper analysis?
As one of my colleagues always said, “let the machines work harder, so that our humans don’t”
Typically a timer series like data (periodically sampling data: can be a snapshot, can be an average, or can aggregate [e.g. total number of packets received])
They are usually resource usage or system level metrics (e.g., # of input/output requests).
Lots of counters and follows different trend over time
We had come up with an approach which mines previous past data to flag anomalous performance behavior in the current test.
This is an example of the resulting perf regression report. This kind of report are interactive html files, which can be zipped and sent to developers for future investigation.
Severity is marked by # of violation periods
To make matters worse, some of these differences could be unintentional (e.g., due to automated updates in software systems). Hence, some of these changes can be unnoticed.
Before discretization, we also need to do normalization
Machines in different labs have different names, leading to variations of metric names:
\\DB1\Process(_Total)\%Processor Time
\\DB2\Process(_Total)\%Processor Time
These variations are normalized:
\\DB\Process(_Total)\%Processor Time
The probability with which an association rule holds can be characterized by its support and confidence measures.
Support measures the ratio of times the rule holds (i.e., counters in the premise and consequent are observed together with the specified values). Low support means that the association rules may have been found simply due to chance.
Confidence measures the probability that the rule’s premise leads to the consequent (i.e., how often the consequent holds when the premise does).
Min support = 0.3
Confidence = 0.9
The probability with which an association rule holds can be characterized by its support and confidence measures.
Support measures the ratio of times the rule holds (i.e., counters in the premise and consequent are observed together with the specified values). Low support means that the association rules may have been found simply due to chance.
Confidence measures the probability that the rule’s premise leads to the consequent (i.e., how often the consequent holds when the premise does).
Min support = 0.3
Confidence = 0.9
Stacking
Similar to bagging but uses Weighted Majority Voting
The weight of each rule set is defined by how similar the prior test that is used to generate the particular rule set is to the new test in terms of software and hardware configurations. The closer the configurations, the heavier the weight will be for a particular rule set
SIW System Information for Windows (SIWInfo) is the tool for getting all the software and hardware configuration information
SIW System Information for Windows (SIWInfo) is the tool for getting all the software and hardware configuration information
SIW System Information for Windows (SIWInfo) is the tool for getting all the software and hardware configuration information
Stacking
Similar to bagging but uses Weighted Majority Voting
The weight of each rule set is defined by how similar the prior test that is used to generate the particular rule set is to the new test in terms of software and hardware configurations. The closer the configurations, the heavier the weight will be for a particular rule set
Each of these are 8-hrs long with more than 2000 counters collected.
=> Some of the analysts issues are good comments. They might have wrong comments or missing issues.
In E1, we have found that total 13 counters (out of 2000) has issues, which are not flagged by performance analysts. Our original approach flags 6 counters (6/13). Bagging shows 18 counters (13/18). Stacking flags 13 counters (7 common with original, 2 are false positive) => 11/13
For E2, we have flagged 15 unique counters with true performance regressions. Original 6/7 flagged counters are good. Bagging has flagged 15/20 flagged counters are good (rest of 15 are all good). Stacking flagged 14 counters, 13/14 are good.
For E3, the DB transaction rate is within the historical rate. Hence, we did not consider this as a problem. Both single and stacking shows that there is no problem. However, bagging flags 8 counters, which are all bad (e.g., slightly higher workload, but within historical ranges)
Q/A: Why not VMs?
VMs’ performance can fluctuate due to “noisy neighbors”. Swipe recently moved from Amzon’s EC2 to real-hardware due to its headache.
http://highscalability.com/blog/2015/3/16/how-and-why-swiftype-moved-from-ec2-to-real-hardware.html