The document discusses data-driven approaches to optimizing software testing processes at Microsoft. It describes how historical test and code data can be analyzed to determine which tests are most valuable and cost-effective to run, in order to reduce total test execution time without negatively impacting code quality. Simulation results on Windows 8.1 data show the potential for significant test reduction (up to 60%) while maintaining bug finding ability. This could improve development processes by lowering machine costs and increasing developer satisfaction.
Strategies to Avoid Test Fixture Smells durin Software EvolutionMichaela Greiler
An important challenge in creating automated tests
is how to design test fixtures, i.e., the setup code that initializes the
system under test before actual automated testing can start. Test
designers have to choose between different approaches for the
setup, trading off maintenance overhead with slow test execution.
Over time, test code quality can erode and test smells can develop,
such as the occurrence of overly general fixtures, obscure inline
code and dead fields. In this paper, we investigate how
fixture-related test smells evolve over time by analyzing several
thousand revisions of five open source systems. Our findings
indicate that setup management strategies strongly influence the
types of test fixture smells that emerge in code, and that several
types of fixture smells often emerge at the same time. Based on
this information, we recommend important guidelines for setup
strategies, and suggest how tool support can be improved to help
in both avoiding the emergence of such smells as well as how to
refactor code when test smells do appear.
EuroSTAR Software Testing Conference 2009 presentation on Incremental Scenario Testing by Mattias Ratert. See more at conferences.eurostarsoftwaretesting.com/past-presentations/
Strategies to Avoid Test Fixture Smells durin Software EvolutionMichaela Greiler
An important challenge in creating automated tests
is how to design test fixtures, i.e., the setup code that initializes the
system under test before actual automated testing can start. Test
designers have to choose between different approaches for the
setup, trading off maintenance overhead with slow test execution.
Over time, test code quality can erode and test smells can develop,
such as the occurrence of overly general fixtures, obscure inline
code and dead fields. In this paper, we investigate how
fixture-related test smells evolve over time by analyzing several
thousand revisions of five open source systems. Our findings
indicate that setup management strategies strongly influence the
types of test fixture smells that emerge in code, and that several
types of fixture smells often emerge at the same time. Based on
this information, we recommend important guidelines for setup
strategies, and suggest how tool support can be improved to help
in both avoiding the emergence of such smells as well as how to
refactor code when test smells do appear.
EuroSTAR Software Testing Conference 2009 presentation on Incremental Scenario Testing by Mattias Ratert. See more at conferences.eurostarsoftwaretesting.com/past-presentations/
A Study: The Analysis of Test Driven Development And Design Driven TestEditor IJMTER
There are various types of testing methods. In this paper, we have discussed TDD and
DDT. Test driven development method is a type of agile methodology. In this method, test drives the
design. Basically, TDD is a unit testing. It is a test-first approach method[15]. This method can be a
never ending process, because we are writing the test first, then writing the code to make the test
pass. If test fails then again we will change the code and then we will test again. Using this method,
defect rate can be reduced to 50%. But DDT cannot be a never ending process, because we design
first, then we implement that design any way. TDD is more complex to implement, while DDT is a
bit easier
My talk at #TestConVilnius 19.10.2017 about what is exploratory testing (ET), how can you make it work and what is there in the future of exploratory testing: test management tool support, testing with tools, AI support, ET as standard way of testing in especially agile and DevOps projects
Klaus Olsen - Agile Test Management Using ScrumTEST Huddle
EuroSTAR Software Testing Conference 2008 presentation on Agile Test Management Using Scrum by Klaus Olsen. See more at conferences.eurostarsoftwaretesting.com/past-presentations/
'Continuous Quality Improvements – A Journey Through The Largest Scrum Projec...TEST Huddle
In this presentation you will learn about how the testing process and continuous quality improvements are aligned to the scrum process in a large software project. We hope that our hands -on experience will give you inspiration on how to tailor the test process in an agile environment. The project has been running for more than two years, with six successful releases to end users. We would like to share our experiences with managing test processes in a large scrum project – our do’s and don’ts, our success stories and also our lessons learned. The project is the largest scrum project in Norway to date.
The project scope is to implement system support for managing a new pension reform for all inhabitants in Norway that are members of the pension fund, and replacing existing system due to outdated technology. Approximately 750 000 project hours will be spent and between 100-180 people are involved in the project: thirteen scrum teams, plus two project management and acceptance testing teams, and one business expert team. Each scrum team contains all the knowledge and expertise needed for developing high quality software: Scrum master, business expert, technical architect, UX designer, developers, build/deploy responsible, and of course, dedicated test resources.
Each software delivery in this project contains five sprints. Each sprint is three weeks, followed by acceptance testing before the delivery is shipped. Test driven development is used in all levels of development, from unit tests all the way up to functional system testing. All test levels up to system integration testing is performed during the development sprint by the scrum teams. We tried to automate UI tests, but this was not successful. However, tests in all other levels are successfully automated, and after each delivery, a fully automated regression test suite is shipped with the code.
Mats Grindal - Risk-Based Testing - Details of Our Success TEST Huddle
EuroSTAR Software Testing Conference 2009 presentation on Risk-Based Testing - Details of Our Success by Mats Grindal. See more at conferences.eurostarsoftwaretesting.com/past-presentations/
QUALITY METRICS OF TEST SUITES IN TESTDRIVEN DESIGNED APPLICATIONSijseajournal
New techniques for writing and developing software have evolved in recent years. One is Test-Driven
Development (TDD) in which tests are written before code. No code should be written without first having
a test to execute it. Thus, in terms of code coverage, the quality of test suites written using TDD should be
high.
In this work, we analyze applications written using TDD and traditional techniques. Specifically, we
demonstrate the quality of the associated test suites based on two quality metrics: 1) structure-based
criterion, 2) fault-based criterion. We learn that test suites with high branch test coverage will also have
high mutation scores, and we especially reveal this in the case of TDD applications. We found that TestDriven
Development is an effective approach that improves the quality of the test suite to cover more of the
source code and also to reveal more.
International Journal of Engineering Research and Applications (IJERA) is an open access online peer reviewed international journal that publishes research and review articles in the fields of Computer Science, Neural Networks, Electrical Engineering, Software Engineering, Information Technology, Mechanical Engineering, Chemical Engineering, Plastic Engineering, Food Technology, Textile Engineering, Nano Technology & science, Power Electronics, Electronics & Communication Engineering, Computational mathematics, Image processing, Civil Engineering, Structural Engineering, Environmental Engineering, VLSI Testing & Low Power VLSI Design etc.
Julian Harty - Alternatives To Testing - EuroSTAR 2010TEST Huddle
EuroSTAR Software Testing Conference 2010 presentation on "Presentation Title" by "Speaker Name". See more at: http://conference.eurostarsoftwaretesting.com/past-presentations/
Ken Johnston - Big Bugs That Got Away - EuroSTAR 2010TEST Huddle
EuroSTAR Software Testing Conference 2010 presentation on Big Bugs That Got Away by Ken Johnston . See more at: http://conference.eurostarsoftwaretesting.com/past-presentations/
AI improves software testing to be more fault tolerant, focused and efficientKari Kakkonen
My slides as Finnish Testing Meetup 28.2.2024
AI supports testing in multiple ways: test automation becomes more autonomous and fault-tolerant, test selection is easier after AI analysis and recommendation, and manual and test automation productivity goes up with generative AI. Some background too on AI in general.
AI improves software testing through test automation, test creation and test ...Kari Kakkonen
The early 2024 version of my How AI improve testing presentation that I delivery since 5 years ago in different conference, webinars and customer events abroad and in Finland.
It gives some background on AI and talks about AI opportunities: in short test automation getting more fault tolerant, generative AI bringing efficiency to test creation, and advanced test selection with AI.
A Study: The Analysis of Test Driven Development And Design Driven TestEditor IJMTER
There are various types of testing methods. In this paper, we have discussed TDD and
DDT. Test driven development method is a type of agile methodology. In this method, test drives the
design. Basically, TDD is a unit testing. It is a test-first approach method[15]. This method can be a
never ending process, because we are writing the test first, then writing the code to make the test
pass. If test fails then again we will change the code and then we will test again. Using this method,
defect rate can be reduced to 50%. But DDT cannot be a never ending process, because we design
first, then we implement that design any way. TDD is more complex to implement, while DDT is a
bit easier
My talk at #TestConVilnius 19.10.2017 about what is exploratory testing (ET), how can you make it work and what is there in the future of exploratory testing: test management tool support, testing with tools, AI support, ET as standard way of testing in especially agile and DevOps projects
Klaus Olsen - Agile Test Management Using ScrumTEST Huddle
EuroSTAR Software Testing Conference 2008 presentation on Agile Test Management Using Scrum by Klaus Olsen. See more at conferences.eurostarsoftwaretesting.com/past-presentations/
'Continuous Quality Improvements – A Journey Through The Largest Scrum Projec...TEST Huddle
In this presentation you will learn about how the testing process and continuous quality improvements are aligned to the scrum process in a large software project. We hope that our hands -on experience will give you inspiration on how to tailor the test process in an agile environment. The project has been running for more than two years, with six successful releases to end users. We would like to share our experiences with managing test processes in a large scrum project – our do’s and don’ts, our success stories and also our lessons learned. The project is the largest scrum project in Norway to date.
The project scope is to implement system support for managing a new pension reform for all inhabitants in Norway that are members of the pension fund, and replacing existing system due to outdated technology. Approximately 750 000 project hours will be spent and between 100-180 people are involved in the project: thirteen scrum teams, plus two project management and acceptance testing teams, and one business expert team. Each scrum team contains all the knowledge and expertise needed for developing high quality software: Scrum master, business expert, technical architect, UX designer, developers, build/deploy responsible, and of course, dedicated test resources.
Each software delivery in this project contains five sprints. Each sprint is three weeks, followed by acceptance testing before the delivery is shipped. Test driven development is used in all levels of development, from unit tests all the way up to functional system testing. All test levels up to system integration testing is performed during the development sprint by the scrum teams. We tried to automate UI tests, but this was not successful. However, tests in all other levels are successfully automated, and after each delivery, a fully automated regression test suite is shipped with the code.
Mats Grindal - Risk-Based Testing - Details of Our Success TEST Huddle
EuroSTAR Software Testing Conference 2009 presentation on Risk-Based Testing - Details of Our Success by Mats Grindal. See more at conferences.eurostarsoftwaretesting.com/past-presentations/
QUALITY METRICS OF TEST SUITES IN TESTDRIVEN DESIGNED APPLICATIONSijseajournal
New techniques for writing and developing software have evolved in recent years. One is Test-Driven
Development (TDD) in which tests are written before code. No code should be written without first having
a test to execute it. Thus, in terms of code coverage, the quality of test suites written using TDD should be
high.
In this work, we analyze applications written using TDD and traditional techniques. Specifically, we
demonstrate the quality of the associated test suites based on two quality metrics: 1) structure-based
criterion, 2) fault-based criterion. We learn that test suites with high branch test coverage will also have
high mutation scores, and we especially reveal this in the case of TDD applications. We found that TestDriven
Development is an effective approach that improves the quality of the test suite to cover more of the
source code and also to reveal more.
International Journal of Engineering Research and Applications (IJERA) is an open access online peer reviewed international journal that publishes research and review articles in the fields of Computer Science, Neural Networks, Electrical Engineering, Software Engineering, Information Technology, Mechanical Engineering, Chemical Engineering, Plastic Engineering, Food Technology, Textile Engineering, Nano Technology & science, Power Electronics, Electronics & Communication Engineering, Computational mathematics, Image processing, Civil Engineering, Structural Engineering, Environmental Engineering, VLSI Testing & Low Power VLSI Design etc.
Julian Harty - Alternatives To Testing - EuroSTAR 2010TEST Huddle
EuroSTAR Software Testing Conference 2010 presentation on "Presentation Title" by "Speaker Name". See more at: http://conference.eurostarsoftwaretesting.com/past-presentations/
Ken Johnston - Big Bugs That Got Away - EuroSTAR 2010TEST Huddle
EuroSTAR Software Testing Conference 2010 presentation on Big Bugs That Got Away by Ken Johnston . See more at: http://conference.eurostarsoftwaretesting.com/past-presentations/
AI improves software testing to be more fault tolerant, focused and efficientKari Kakkonen
My slides as Finnish Testing Meetup 28.2.2024
AI supports testing in multiple ways: test automation becomes more autonomous and fault-tolerant, test selection is easier after AI analysis and recommendation, and manual and test automation productivity goes up with generative AI. Some background too on AI in general.
AI improves software testing through test automation, test creation and test ...Kari Kakkonen
The early 2024 version of my How AI improve testing presentation that I delivery since 5 years ago in different conference, webinars and customer events abroad and in Finland.
It gives some background on AI and talks about AI opportunities: in short test automation getting more fault tolerant, generative AI bringing efficiency to test creation, and advanced test selection with AI.
Testing As A Bottleneck - How Testing Slows Down Modern Development Processes...TEST Huddle
We often claim the purpose of testing is to verify that software meets a desired level of quality. Frequently, the term “testing” is associated with checking for functional correctness. However, in large, complex software systems with an established user-base, it is also important to verify system constraints such as backward compatibility, reliability, security, accessibility, usability. Kim Herzig from Microsoft explores these issues with the latest webinar on test Huddle.
Test-Driven Development in the Corporate WorkplaceAhmed Owian
What is TDD, and why is it giving traditional software development practices a run for their money? This presentation answers these questions, while focusing on a popular agile methodology, Extreme Programming (XP). It places a particular emphasis on the exploratory programming nature of XP and its testing practice, TDD. The paper also summarizes prior research on TDD and includes the results from a research survey conducted to compare TDD with traditional testing practices.
Machine learning has become an important tool in the modern software toolbox, and high-performing organizations are increasingly coming to rely on data science and machine learning as a core part of their business. eBay introduced machine learning to its commerce search ranking and drove double-digit increases in revenue. Stitch Fix built a multibillion dollar clothing retail business in the US by combining the best of machines with the best of humans. And WeWork is bringing machine-learned approaches to the physical office environment all around the world. In all cases, algorithmic techniques started simple and slowly became more sophisticated over time. This talk will use these examples to derive an agile approach to machine learning, and will explore that approach across several different dimensions. We will set the stage by outlining the kinds of problems that are most amenable to machine-learned approaches as well as describing some important prerequisites, including investments in data quality, a robust data pipeline, and experimental discipline. Next, we will choose the right (algorithmic) tool for the right job, and suggest how to incrementally evolve the algorithmic approaches we bring to bear. Most fancy cutting-edge recommender systems in the real world, for example, started out with simple rules-based techniques or basic regression. Finally, we will integrate machine learning into the broader product development process, and see how it can help us to accelerate business results
This is the presentation we gave in 2009 during Agile Testing Days in Berlin. Even though it is already more than 2 years old, many things we said during the talk are very valid today. Some things did not change at all.
I believe that our existing models of testing are not fit for purpose – they are inconsistent, controversial, partial, proprietary and stuck in the past. They are not going to support us in the rapidly emerging technologies and approaches. The certification schemes that should represent the interests and integrity of our profession don’t, and we are left with schemes that are popular, but have low value, lower esteem and attract harsh criticism. My goal in proposing the New Model is to stimulate new thinking in this area.
eurostarconferences.com
testhuddle.com
I believe that our existing models of testing are not fit for purpose – they are inconsistent, controversial, partial, proprietary and stuck in the past. They are not going to support us in the rapidly emerging technologies and approaches. The certification schemes that should represent the interests and integrity of our profession don’t, and we are left with schemes that are popular, but have low value, lower esteem and attract harsh criticism. My goal in proposing the New Model is to stimulate new thinking in this area.
eurostarconferences.com
testhuddle.com
Bridging the Gap: from Data Science to ProductionFlorian Wilhelm
A recent but quite common observation in industry is that although there is an overall high adoption of data science, many companies struggle to get it into production. Huge teams of well-payed data scientists often present one fancy model after the other to their managers but their proof of concepts never manifest into something business relevant. The frustration grows on both sides, managers and data scientists.
In my talk I elaborate on the many reasons why data science to production is such a hard nut to crack. I start with a taxonomy of data use cases in order to easier assess technical requirements. Based thereon, my focus lies on overcoming the two-language-problem which is Python/R loved by data scientists vs. the enterprise-established Java/Scala. From my project experiences I present three different solutions, namely 1) migrating to a single language, 2) reimplementation and 3) usage of a framework. The advantages and disadvantages of each approach is presented and general advices based on the introduced taxonomy is given.
Additionally, my talk also addresses organisational as well as problems in quality assurance and deployment. Best practices and further references are presented on a high-level in order to cover all facets of data science to production.
With my talk I hope to convey the message that breakdowns on the road from data science to production are rather the rule than the exception, so you are not alone. At the end of my talk, you will have a better understanding of why your team and you are struggling and what to do about it.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
2. Data-driven software engineering @Microsoft
•How can we optimize the testing process?
•Do code reviews make a difference?
•Is coding velocity and quality always a tradeoff?
•What’s the optimal way to organize work on a large team?
MSR Redmond/TSE:
Michaela GreilerJacek CzerwonkaWolfram SchulteSuresh Thummalapenta
MSR Redmond:
Christian BirdKathryn McKinleyNachi NagappanThomas Zimmermann
MSR Cambridge: Brendan MurphyKim Herzig
14. GMQ vs. Opportunistic data collection
•Easily available ≠ what’s needed
•Determine the needed data
•Find proxy measures if needed
•Know the analysis before collecting the data
Otherwise, data is not usable for the intended purpose
•Goal –Question –Metric
•Check for completeness, cleanness/ noise and usefulness
•Data background
•How was data generated?
•Why was it generated?
•Who consumes the data?
•What about outliers?
•How was the data processed?
16. Tools, processes,
practices and policies.
Release schedule
Time
Engineers
What roles exist?
Who does what?
Responsibilities?
M1
M2
Beta
Organization of code bases
Team structure and culture.
18. Engineers want to understand the nitty-gritty
•How do you calculate the recommended reviewers?
•Why was that person recommended?
•Why is Lisa not recommended?
19. Simplicity first
Files
without
bugs
Files
with
bugs
Files withoutbugs: main contributor made > 50% of all edits
Files withbugs: main contributor made < 60% of all edits
Ownership metric:
Proportion of edits of all edits for the contributor with the most edits
Reporting vs. Prediction
Comprehension
vs. automation
If you can do it with a decision tree… do it…
20. Iterative process with very close involvement of product teams and domain experts.
It’s a dialog
It’s a back and forth
21. Mixed Method Research
Is a research approach or methodology
•for questions that call for real-life contextual understandings;
•employing rigorous quantitative research assessing magnitude and frequency of constructs and
•rigorous qualitative researchexploring the meaning and understanding of constructs;
DR. MARGARET-ANNESTOREY
Professor of Computer Science University of Victoria
All methods are inherently flawed!
Generalizability
Precision
Realism
DR. ARIEVANDEURSEN
Professor of Software Engineering Delft University of Technology
22. Foundations of Mixed
Methods Research
Designing
Social Inquiry
Qualitative Research: Mixed Method Research
•Interviews
•Observations
•Focus groups
•Contextual Inquiry
•Grounded Theory
•…
23. A Grounded Theory Study
23
Systematic procedure to discover a theory from (qualitative) data
S. Adolph, W. Hall, Ph. Kruchten. Using Grounded theory to study the experience of software development. Empirical Software Engineering,2011.
B. Glaser and J. Holton. Remodeling grounded theory. Forum Qualitative Res., 2004.
Glaser and Strauss
24. Deductiveversus inductive
A deductive approach is concerned with developing a hypothesis (or hypotheses) based on existing theory, and then designing a research strategy to test the hypothesis (Wilson, 2010, p.7)
Inductive approach starts with observations. Theories emerge towards the end of the research and as a result of careful examination of patterns in observations (Goddard and Melville, 2004).
Theory
Hypotheses
Observation
Confirm/Reject
Observation
Patterns
Theory
25. All models are wrong but some are useful
(George E. P. Box)
26. Theo: Test Effectiveness Optimization from History
Kim Herzig*, Michaela Greiler+, Jacek Czerwonka+, Brendan Murphy*
*Microsoft Research, Cambridge
+Microsoft Corporation, US
27. Improving Development Processes
Product /
Service
Legacy
changes
New product
features
Technology
changes
Development Environment
$
Speed
R
Cost
Quality / Risk
(should be well balanced)
Microsoft aims for shorter release cycles
Empirical data to support & drive decisions
• Speed up development processes (e.g. code velocity)
• More frequent releases
• Maintaining / increasing product quality
Joint effort by MSR & product teams
• MSR Cambridge: Brendan Murphy, Kim Herzig
• TSE Redmond: Jacek Czerwonka, Michaela Greiler
• MSR Redmond: Tom Zimmermann, Chris Bird, Nachi Nagappan
• Windows, Windows Phone, Office, Dynamics product teams
28. Software Testing for Windows
Winmain (main branch)
Quality gate
(system testing)
Quality gate
(system & component testing)
Quality gate
(component testing)
time
Development branch
Multiple area branches
Multiple component branches
Software testing is very expensive
• Thousands test suites executed, millions test cases executed
• On different branches, architectures, languages, etc.
• We tend to repeat the same tests over and over again
• Too many false alarms (failures due to test and infrastructure issues)
• Each test failures slows down product development
• Aims to find code issues as early as possible
• At the cost of slower product development
Actual problem
Current process aims for maximal protection
{Simplified illustration}
29. Software Testing for Office
Software testing is very expensive
• Thousands test suites executed, millions test cases executed
• On different branches, architectures, languages, etc.
• We tend to repeat the same tests over and over again
• Too many false alarms (failures due to test and infrastructure issues)
• Each test failures slows down product development
• Aims to find code issues as early as possible
• At the cost of slower product development
Actual problem
Current process aims for maximal protection
Dev Inner Loop
BVT and CVT
on main
Dog food
Different
• Branching structure
• Development process
• Testing process
• Release schedules
• …
{Simplified illustration}
30. Goal
Reduce the number of test executions …
… without sacrificing code quality
Dynamic, self-adaptive optimization model
31. Solution
Reduce the number of test executions …
•Runevery test at least once beforeintegrating code change into main branch (e.g., winmain).
•We eventually find all code issues but take riskof finding them later (on higher level branches).
… without sacrificing code quality
High cost, unknown value
$$$$$
High cost, low value$$$$
Low cost,
low value$
Low cost, good value$$
How likely is a test causing:
1)false positivesor
2)finding code issues?
Analyzehistoric data:
-Test Events
-Builds
-Code Integrations
Analyzepast test results
-Passing tests, false alarms, detected code issues
33. Solution
Using cost function to model risk.
푪풐풔풕푬풙풆풄풖풕풊풐풏>푪풐풔풕푺풌풊풑?suspend∶executetest
퐶표푠푡퐸푥푒푐푢푡푖표푛=퐶표푠푡푀푎푐ℎ푖푛푒/푇푖푚푒∗푇푖푚푒퐸푥푒푐푢푡푖표푛+"Costofpotentialfalsealarm"
=퐶표푠푡푀푎푐ℎ푖푛푒/푇푖푚푒∗푇푖푚푒퐸푥푒푐푢푡푖표푛+(푃푟표푏퐹푃∗퐶표푠푡퐷푒푣푒푙표푝푒푟/푇푖푚푒∗푇푖푚푒푇푟푖푎푔푒)
퐶표푠푡푆푘푖푝="Potentialcostoffindingadefectlater"
=푃푟표푏푇푃∗퐶표푠푡퐷푒푣푒푙표푝푒푟/푇푖푚푒∗푇푖푚푒퐹푟푒푒푧푒푏푟푎푛푐ℎ∗#퐷푒푣푒푙표푝푒푟푠퐵푟푎푛푐ℎ
Test
Costto run a test.
Valueof output.
35. Dynamic, Self-Adaptive
Decision points are connected to each other
Skipping tests influences the risk factorsof higher level branches
We re-enable testsif code quality drops (e.g. different milestone)
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
relative test reduction rate
Time (Windows 8.1)
Training period
36. Bug Finding Performance of Tests
How many test executions fail?
#failed test exec
Branch level
Number of test executions
How many of the failed test executions result in bug reports?
FP
TP test-unspecific
TP test-specific
Branch level
37. Impact on Development Process
Secondary Improvements
•Machine Setup: we may lower the number of machines allocated to testing process
•Developer satisfaction: Removing false test failures increases confidence in testing process
…hard to estimate speed improvement through simulation
“We used the data […] to cut a bunch of bad content and are running a much leaner BVT system […] we’re panning out to scale about 4x and run in well under 2 hours” (Jason Means, Windows BVT PM)