The document discusses planning and managing test automation. It covers responsibilities of testers and automators, the importance of a pilot project to establish processes and standards, and setting objectives for test automation efforts. Good objectives include demonstrating value through a pilot, gaining experience with a tool, and establishing internal standards and conventions. Measures like maintenance effort and the number of defects found are also discussed for assessing automation efforts.
Many organizations never achieve the significant benefits that are promised from automated test execution. Surprisingly often, this is not due to technical factors but to management issues. Dot Graham describes the most important management issues you must address for test automation success, and helps you understand and choose the best approaches for your organization—no matter which automation tools you use or your current state of automation. Dot explains how automation affects staffing, who should be responsible for which automation tasks, how managers can best support automation efforts leading to success, and what return on investment means in automated testing and what you can realistically expect. Dot also reviews the key technical issues that can make or break the automation effort. Come away with an example set of automation objectives and measures, and a draft test automation strategy that you can use to plan or improve your own automation.
Why Automation Fails—in Theory and PracticeTechWell
Testers face common challenges in automation. Unfortunately, these challenges often lead to subsequent failures. Jim Trentadue explains a variety of automation perceptions and myths―the perception that a significant increase in time and people is needed to implement automation; the myth that, once automation is achieved, testers will not be needed; the myth that scripted automation will serve all the testing needs for an application; the perception that developers and testers can add automation to a project without additional time, resources, or training; the belief that anyone can implement automation. The testing organization must ramp up quickly on the test automation process and the prep-work analysis that needs to be done including when to start, how to structure the tests, and what system to start with. Learn how to respond to these common challenges by developing a solid business case for increased automation adoption by engaging manual testers in the testing organization, being technology agnostic, and stabilizing test scripts regardless of applications changes.
6 Traits of a Successful Test Automation ArchitectureErdem YILDIRIM
The sector demands that software development life cycle to be delivered faster and cheaper with increasing quality and reliability. TLC (testing life cycle) is a crucuel part of the time, cost and quality level for AUT (Application Under Test). Market got to point that all long ornate talks can be summed up in one word: EFFICIENCY. In quality aspect, automating testing activities had already been came forward to reduce development cycle times, cost, resources allocated with traditional test along past years. It's OK that automation increased the efficiency of the test process, so what about the efficiency of automation itself? Why most of the test automation projects fail (even if you're not aware of it is actually failing)? Because, automating without good test architecture may result in a lot of activity, but little value (if you are lucky). We will talk about following 6 main traits to build a successful test automation architecture; selection/implementation of test levels to be automated, design principles/patterns, locater strategy, tools / framework selection (aside from SeWD / Java), methodology (E2E Testing, TDD, BDD, Continuous Testing) and OOP pillars.
Building a Test Automation Strategy for SuccessLee Barnes
Choosing an appropriate tool and building the right framework are typically thought of as the main challenges in implementing successful test automation. However, long term success requires that other key questions must be answered including:
- What are our objectives?
- How should we be organized?
- Will our processes need to change?
- Will our test environment support test automation?
- What skills will we need?
- How and when should we implement?
In this workshop, Lee will discuss how to assess your test automation readiness and build a strategy for long term success. You will interactively walk through the assessment process and build a test automation strategy based on input from the group. Attend this workshop and you will take away a blue print and best practices for building an effective test automation strategy in your organization.
• Understand the key aspects of a successful test automation function
• Learn how to assess your test automation readiness
• Develop a test automation strategy specific to your organization
What are the Key drivers for automation? What are the Challenges in Agile automation and How to deal with them? How to automate? Who will automate? Which tool to select? Commercial or open source? What to automate? Which features? Here is what our experience says
When implementing test automation, many people encounter problems: where to start with automation, high maintenance costs for the automated tests, or unrealistic management expectations. The good news is that solutions to these problems exist and have been effectively used by many. A “pattern” is a general reusable solution to a commonly occurring problem. Patterns have been popular in software development for many years, but they are not commonly recognized in system-level test automation. Dorothy Graham shares a collection of common problems (issues) and their solutions (patterns) which she and others are now developing as a wiki. To help resolve typical issues, Dot gives you a brief guided tour of some patterns—from Maintainable Testware and Domain-Driven Testing to Fail Gracefully and Kill the Zombies. Dot helps you recognize test automation issues and shows you how to identify appropriate patterns to help solve them.
A brief introduction to test automation covering different automation approaches, when to automate and by whom, commercial vs. open source tools, testability, and so on.
Many organizations never achieve the significant benefits that are promised from automated test execution. Surprisingly often, this is not due to technical factors but to management issues. Dot Graham describes the most important management issues you must address for test automation success, and helps you understand and choose the best approaches for your organization—no matter which automation tools you use or your current state of automation. Dot explains how automation affects staffing, who should be responsible for which automation tasks, how managers can best support automation efforts leading to success, and what return on investment means in automated testing and what you can realistically expect. Dot also reviews the key technical issues that can make or break the automation effort. Come away with an example set of automation objectives and measures, and a draft test automation strategy that you can use to plan or improve your own automation.
Why Automation Fails—in Theory and PracticeTechWell
Testers face common challenges in automation. Unfortunately, these challenges often lead to subsequent failures. Jim Trentadue explains a variety of automation perceptions and myths―the perception that a significant increase in time and people is needed to implement automation; the myth that, once automation is achieved, testers will not be needed; the myth that scripted automation will serve all the testing needs for an application; the perception that developers and testers can add automation to a project without additional time, resources, or training; the belief that anyone can implement automation. The testing organization must ramp up quickly on the test automation process and the prep-work analysis that needs to be done including when to start, how to structure the tests, and what system to start with. Learn how to respond to these common challenges by developing a solid business case for increased automation adoption by engaging manual testers in the testing organization, being technology agnostic, and stabilizing test scripts regardless of applications changes.
6 Traits of a Successful Test Automation ArchitectureErdem YILDIRIM
The sector demands that software development life cycle to be delivered faster and cheaper with increasing quality and reliability. TLC (testing life cycle) is a crucuel part of the time, cost and quality level for AUT (Application Under Test). Market got to point that all long ornate talks can be summed up in one word: EFFICIENCY. In quality aspect, automating testing activities had already been came forward to reduce development cycle times, cost, resources allocated with traditional test along past years. It's OK that automation increased the efficiency of the test process, so what about the efficiency of automation itself? Why most of the test automation projects fail (even if you're not aware of it is actually failing)? Because, automating without good test architecture may result in a lot of activity, but little value (if you are lucky). We will talk about following 6 main traits to build a successful test automation architecture; selection/implementation of test levels to be automated, design principles/patterns, locater strategy, tools / framework selection (aside from SeWD / Java), methodology (E2E Testing, TDD, BDD, Continuous Testing) and OOP pillars.
Building a Test Automation Strategy for SuccessLee Barnes
Choosing an appropriate tool and building the right framework are typically thought of as the main challenges in implementing successful test automation. However, long term success requires that other key questions must be answered including:
- What are our objectives?
- How should we be organized?
- Will our processes need to change?
- Will our test environment support test automation?
- What skills will we need?
- How and when should we implement?
In this workshop, Lee will discuss how to assess your test automation readiness and build a strategy for long term success. You will interactively walk through the assessment process and build a test automation strategy based on input from the group. Attend this workshop and you will take away a blue print and best practices for building an effective test automation strategy in your organization.
• Understand the key aspects of a successful test automation function
• Learn how to assess your test automation readiness
• Develop a test automation strategy specific to your organization
What are the Key drivers for automation? What are the Challenges in Agile automation and How to deal with them? How to automate? Who will automate? Which tool to select? Commercial or open source? What to automate? Which features? Here is what our experience says
When implementing test automation, many people encounter problems: where to start with automation, high maintenance costs for the automated tests, or unrealistic management expectations. The good news is that solutions to these problems exist and have been effectively used by many. A “pattern” is a general reusable solution to a commonly occurring problem. Patterns have been popular in software development for many years, but they are not commonly recognized in system-level test automation. Dorothy Graham shares a collection of common problems (issues) and their solutions (patterns) which she and others are now developing as a wiki. To help resolve typical issues, Dot gives you a brief guided tour of some patterns—from Maintainable Testware and Domain-Driven Testing to Fail Gracefully and Kill the Zombies. Dot helps you recognize test automation issues and shows you how to identify appropriate patterns to help solve them.
A brief introduction to test automation covering different automation approaches, when to automate and by whom, commercial vs. open source tools, testability, and so on.
Synthesizing Continuous Deployment Practices in Software DevelopmentAkond Rahman
Continuous deployment speeds up the process of existing agile methods, such as Scrum, and Extreme Programming (XP) through the automatic deployment of software changes to end-users upon passing of automated tests. Continuous deployment has become an emerging software engineering process amongst numerous software companies, such as Facebook, Github, Netflix, and Rally Software. A systematic analysis of software practices used in continuous deployment can facilitate a better understanding of continuous deployment as a software engineering process. Such analysis can also help software practitioners in having a shared vocabulary of practices and in choosing the software practices that they can use to implement continuous deployment. The goal of this paper is to aid software practitioners in implementing continuous deployment through a systematic analysis of software practices that are used by software companies. We studied the continuous deployment practices of 19 software companies by performing a qualitative analysis of Internet artifacts and by conducting follow-up inquiries. In total, we found 11 software practices that are used by 19 software companies. We also found that in terms of use, eight of the 11 software practices are common across 14 software companies. We observe that continuous deployment necessitates the consistent use of sound software engineering practices such as automated testing, automated deployment, and code review.
7 Deadly Sins of Agile Software Test AutomationAdrian Smith
Automated software testing is a key enabler for teams wanting to build high quality software that can be progressively enhanced and continuously released. To ensure development practices are sustainable, automated testing must be treated as a first-class citizen and not all approaches are created equal. Some approaches can accumulate technical debt, cause duplication of effort and even team dysfunctions.
The seven deadly sins of automated software testing are a set of common anti-patterns that have been found to erode the value of automated testing resulting in long term maintenance issues and ultimately affecting the ability of development teams to respond to change and continuously deliver.
Taking the classic seven sins (Gluttony, Sloth, Lust, Envy, Rage, Pride, Greed) as they might be applied to test automation we will discuss how to identify each automated sin and more importantly provide guidance on recommended solutions and how to avoid them in the first place.
Many organizations never achieve the significant benefits that are promised from automated test execution. Surprisingly often, this is due not to technical factors but to management issues, especially at system testing level. Surprisingly often, this is due not to technical factors but to management issues. Dot Graham describes the most important management concerns the test manager must address for test automation success, and helps you understand and choose the best approaches for your organization—no matter which automation tools you use or your current state of automation. Dot explains how automation affects staffing, who should be responsible for which automation tasks, how managers can best support automation efforts leading to success, and why return on investment can be dangerous and what you can realistically expect. Dot also reviews a few key technical issues that can make or break the automation effort. Come away with an example set of automation objectives and measures, and a draft test automation strategy that you can use to plan or improve your own automation.
Test automation and Agile software developmentBas Dijkstra
Slides for my workshop on test automation, creating realistic expectations around it and what the role of test automation in Agile software development is
This is a presentation given at the Hangzhou Scrum Forum 2009, sponsored by Perficient, China. The topic is how to incorporate automated functional testing into an agile project, and also some best practices, tips, and warnings.
www.perficient.com
Ho Chi Minh City Software Testing Conference January 2015
Software Testing in the Agile World
Website: www.hcmc-stc.org
Author: Tho Thanh Quang
As testing is increasingly incurring a substantial cost in software development, there are many attempts made to automate the testing process. One notable approach is automatic generation of test cases. Recent research has suggested generating test cases from UML-based diagrams, which are over-formal to be applied effectively in industry. In this talk we introduce a framework, known as FATS (Framework for Automated Testing Scenarios), to counter this problem. In FATS, we suggest representing user-defined use-cases by a markup language, therefore activity graphs and test scenarios can be developed accordingly in an automatic manner.
It Seemed a Good Idea at the Time: Intelligent Mistakes in Test AutomationTechWell
Some test automation ideas seem very sensible at first glance but contain pitfalls and problems that can and should be avoided. Dot Graham describes five of these “intelligent mistakes”—1. Automated tests will find more bugs quicker. (Automation doesn’t find bugs, tests do.) 2. Spending a lot on a tool must guarantee great benefits. (Good automation does not come “out of the box” and is not automatic.) 3. Let’s automate all of our manual tests. (This may not give you better or faster testing, and you will miss out on some benefits.) 4. Tools are expensive so we have to show a return on investment. (This is not only surprisingly difficult but may actually be harmful.) 5. Because they are called “testing tools,” they must be tools for testers to use. (Making testers become test automators may be damaging to both testing and automation.) Join Dot for a rousing discussion of “intelligent mistakes”—so you can be smart enough to avoid them.
System-Level Test Automation: Ensuring a Good StartTechWell
Many organizations invest a lot of effort in test automation at the system level but then have serious problems later on. As a leader, how can you ensure that your new automation efforts will get off to a good start? What can you do to ensure that your automation work provides continuing value? This tutorial covers both “theory” and “practice”. Dot Graham explains the critical issues for getting a good start, and Chris Loder describes his experiences in getting good automation started at a number of companies. The tutorial covers the most important management issues you must address for test automation success, particularly when you are new to automation, and how to choose the best approaches for your organization—no matter which automation tools you use. Focusing on system level testing, Dot and Chris explain how automation affects staffing, who should be responsible for which automation tasks, how managers can best support automation efforts to promote success, what you can realistically expect in benefits and how to report them. They explain—for non-techies—the key technical issues that can make or break your automation effort. Come away with your own clarified automation objectives, and a draft test automation strategy to use to plan your own system-level test automation.
STARCANADA 2013 Keynote: Cool! Testing’s Getting Fun AgainTechWell
The last exciting era in testing was in the late ‘90s when the web turned technology on its ear, the agile movement overthrew our processes, and the rise of open source gave us accessible and innovative tools. However, since then, Jonathan Kohl finds it has been a lot of the same-old, same-old: more web applications, variants of agile, and testing tool debates. After a while, you start to feel like you’ve “Been there, Done that, Got that t-shirt.” However, testing doesn’t have to be a rehash of all the things you’ve done before. It is an exciting time to be a tester! Major changes are happening all at once. Smartphones and tablets have upended computing. Continuous release movements like DevOps are challenging our processes. BigData, the Cloud, and other disruptive technologies are fostering the development of new tools. These innovations are pushing testers in new directions. While this upheaval is challenging, it also offers enormous opportunity for growth. Jonathan challenges you to embrace these changes—and have fun doing it. Discover today’s amazing new opportunities, and how we can learn, adapt, and ride the wave.
Many organizations struggle with transforming from the old-style specialized silos of skills into agile teams with generalized specialists. Without this pivot, we get sub-optimal agile/Scrum environments. Howard Deiner describes what can go wrong when integrating testers properly into an agile organization and how to fix that. Without a proper agile mindset, an organization will “revert to form” and return to their old practices after a frustrating failure to adapt agile. Howard examines the real role of testers in the organization and identifies where they truly add value in the production of quality code. He speaks frankly about the skills that agile testers must master and the issues that organizations have that complicate testers’ lives. Finally, Howard discusses exactly what testers need to do to add value to the software development process and how they integrate in the DevOps model that is a contemporary solution to an age old issue.
Synthesizing Continuous Deployment Practices in Software DevelopmentAkond Rahman
Continuous deployment speeds up the process of existing agile methods, such as Scrum, and Extreme Programming (XP) through the automatic deployment of software changes to end-users upon passing of automated tests. Continuous deployment has become an emerging software engineering process amongst numerous software companies, such as Facebook, Github, Netflix, and Rally Software. A systematic analysis of software practices used in continuous deployment can facilitate a better understanding of continuous deployment as a software engineering process. Such analysis can also help software practitioners in having a shared vocabulary of practices and in choosing the software practices that they can use to implement continuous deployment. The goal of this paper is to aid software practitioners in implementing continuous deployment through a systematic analysis of software practices that are used by software companies. We studied the continuous deployment practices of 19 software companies by performing a qualitative analysis of Internet artifacts and by conducting follow-up inquiries. In total, we found 11 software practices that are used by 19 software companies. We also found that in terms of use, eight of the 11 software practices are common across 14 software companies. We observe that continuous deployment necessitates the consistent use of sound software engineering practices such as automated testing, automated deployment, and code review.
7 Deadly Sins of Agile Software Test AutomationAdrian Smith
Automated software testing is a key enabler for teams wanting to build high quality software that can be progressively enhanced and continuously released. To ensure development practices are sustainable, automated testing must be treated as a first-class citizen and not all approaches are created equal. Some approaches can accumulate technical debt, cause duplication of effort and even team dysfunctions.
The seven deadly sins of automated software testing are a set of common anti-patterns that have been found to erode the value of automated testing resulting in long term maintenance issues and ultimately affecting the ability of development teams to respond to change and continuously deliver.
Taking the classic seven sins (Gluttony, Sloth, Lust, Envy, Rage, Pride, Greed) as they might be applied to test automation we will discuss how to identify each automated sin and more importantly provide guidance on recommended solutions and how to avoid them in the first place.
Many organizations never achieve the significant benefits that are promised from automated test execution. Surprisingly often, this is due not to technical factors but to management issues, especially at system testing level. Surprisingly often, this is due not to technical factors but to management issues. Dot Graham describes the most important management concerns the test manager must address for test automation success, and helps you understand and choose the best approaches for your organization—no matter which automation tools you use or your current state of automation. Dot explains how automation affects staffing, who should be responsible for which automation tasks, how managers can best support automation efforts leading to success, and why return on investment can be dangerous and what you can realistically expect. Dot also reviews a few key technical issues that can make or break the automation effort. Come away with an example set of automation objectives and measures, and a draft test automation strategy that you can use to plan or improve your own automation.
Test automation and Agile software developmentBas Dijkstra
Slides for my workshop on test automation, creating realistic expectations around it and what the role of test automation in Agile software development is
This is a presentation given at the Hangzhou Scrum Forum 2009, sponsored by Perficient, China. The topic is how to incorporate automated functional testing into an agile project, and also some best practices, tips, and warnings.
www.perficient.com
Ho Chi Minh City Software Testing Conference January 2015
Software Testing in the Agile World
Website: www.hcmc-stc.org
Author: Tho Thanh Quang
As testing is increasingly incurring a substantial cost in software development, there are many attempts made to automate the testing process. One notable approach is automatic generation of test cases. Recent research has suggested generating test cases from UML-based diagrams, which are over-formal to be applied effectively in industry. In this talk we introduce a framework, known as FATS (Framework for Automated Testing Scenarios), to counter this problem. In FATS, we suggest representing user-defined use-cases by a markup language, therefore activity graphs and test scenarios can be developed accordingly in an automatic manner.
It Seemed a Good Idea at the Time: Intelligent Mistakes in Test AutomationTechWell
Some test automation ideas seem very sensible at first glance but contain pitfalls and problems that can and should be avoided. Dot Graham describes five of these “intelligent mistakes”—1. Automated tests will find more bugs quicker. (Automation doesn’t find bugs, tests do.) 2. Spending a lot on a tool must guarantee great benefits. (Good automation does not come “out of the box” and is not automatic.) 3. Let’s automate all of our manual tests. (This may not give you better or faster testing, and you will miss out on some benefits.) 4. Tools are expensive so we have to show a return on investment. (This is not only surprisingly difficult but may actually be harmful.) 5. Because they are called “testing tools,” they must be tools for testers to use. (Making testers become test automators may be damaging to both testing and automation.) Join Dot for a rousing discussion of “intelligent mistakes”—so you can be smart enough to avoid them.
System-Level Test Automation: Ensuring a Good StartTechWell
Many organizations invest a lot of effort in test automation at the system level but then have serious problems later on. As a leader, how can you ensure that your new automation efforts will get off to a good start? What can you do to ensure that your automation work provides continuing value? This tutorial covers both “theory” and “practice”. Dot Graham explains the critical issues for getting a good start, and Chris Loder describes his experiences in getting good automation started at a number of companies. The tutorial covers the most important management issues you must address for test automation success, particularly when you are new to automation, and how to choose the best approaches for your organization—no matter which automation tools you use. Focusing on system level testing, Dot and Chris explain how automation affects staffing, who should be responsible for which automation tasks, how managers can best support automation efforts to promote success, what you can realistically expect in benefits and how to report them. They explain—for non-techies—the key technical issues that can make or break your automation effort. Come away with your own clarified automation objectives, and a draft test automation strategy to use to plan your own system-level test automation.
STARCANADA 2013 Keynote: Cool! Testing’s Getting Fun AgainTechWell
The last exciting era in testing was in the late ‘90s when the web turned technology on its ear, the agile movement overthrew our processes, and the rise of open source gave us accessible and innovative tools. However, since then, Jonathan Kohl finds it has been a lot of the same-old, same-old: more web applications, variants of agile, and testing tool debates. After a while, you start to feel like you’ve “Been there, Done that, Got that t-shirt.” However, testing doesn’t have to be a rehash of all the things you’ve done before. It is an exciting time to be a tester! Major changes are happening all at once. Smartphones and tablets have upended computing. Continuous release movements like DevOps are challenging our processes. BigData, the Cloud, and other disruptive technologies are fostering the development of new tools. These innovations are pushing testers in new directions. While this upheaval is challenging, it also offers enormous opportunity for growth. Jonathan challenges you to embrace these changes—and have fun doing it. Discover today’s amazing new opportunities, and how we can learn, adapt, and ride the wave.
Many organizations struggle with transforming from the old-style specialized silos of skills into agile teams with generalized specialists. Without this pivot, we get sub-optimal agile/Scrum environments. Howard Deiner describes what can go wrong when integrating testers properly into an agile organization and how to fix that. Without a proper agile mindset, an organization will “revert to form” and return to their old practices after a frustrating failure to adapt agile. Howard examines the real role of testers in the organization and identifies where they truly add value in the production of quality code. He speaks frankly about the skills that agile testers must master and the issues that organizations have that complicate testers’ lives. Finally, Howard discusses exactly what testers need to do to add value to the software development process and how they integrate in the DevOps model that is a contemporary solution to an age old issue.
Organizations worldwide collect data about customers, users, products, and services. Striving to get the most out of collected data, they use it to fuel many day-to-day processes including software testing, development, and personnel training. The majority of this collected data is sensitive and falls under specific government regulations or industry standards that define policies for privacy and generally limit or prohibit using the data for these secondary purposes. Data masking solves this problem. It replaces sensitive information with data that looks real and is structurally similar to the actual information but is useless to anyone trying to obtain the real data. Learn about the process, pros and cons of static and dynamic data masking architectures, subsetting, randomization, generalization, shuffling, and other basic techniques used to set up data masking. Discover how to start data masking and learn about common challenges on data masking projects.
In today's economy, the Creative Economy, businesses face a disrupted, highly competitive, and constantly changing landscape. Robie and Jody Wood say that to thrive in the Creative Economy, team members, managers, and executives need to become and remain agile. Improvisational theater provides a proven model for developing agility skills since the characteristics of “being agile”—engaging people, learning, making decisions in the midst of uncertainty and ambiguity, and adapting—are the very skills that improv artists work to develop with every exercise they perform. This session is about “Being Agile,” developing the mindset and behaviors that grow great abilities in communication, collaboration, inspiring others, building on others’ ideas, learning, adapting, and evolving. This workshop will engage delegates in experiential learning exercises from Improvisational Theater that will have immediate impact in improving agile mindset and behavior. The workshop participants will find the exercises lively, inspiring, fun, life changing—and an experience they will never forget.
Twelve Heuristics for Solving Tough Problems Faster and BetterTechWell
As infants, we begin our lives as problem-solving machines, learning to navigate a strange and complex world in which others communicate in ways we don’t understand. Initially, we hone our problem-solving talents. Then many of us find our explorations thwarted and eventually stop using—and then begin losing—our natural problem-solving ability. It doesn’t have to be that way. Psychologists tell us that people can regain lost skills and learn new ones to become better problem solvers. Payson Hall shares techniques and skills that apply to situations in real life. Specifically, learn techniques to better define problems and explore twelve heuristics for generating solutions that can help when you and your team are staring at a blank paper, struggling to find candidate solutions for further consideration. Learn when random search is appropriate, how binary search can help with diagnostics, strategies for identifying and overcoming opposition, and when transferring the problem to someone else just might be the best strategy of all.
How to Actually DO High-volume Automated TestingTechWell
In high volume automated testing (HiVAT), the test tool generates the test, runs it, evaluates the results, and alerts a human to suspicious results that need further investigation. What makes it simple is its oracle—run the program until it crashes or fails in some other extremely obvious way. More powerful HiVAT approaches are more sensitive to more types of errors. They are particularly useful for testing combinations of many variables and for hunting hard-to-replicate bugs that involve timing or corruption of memory or data. Cem Kaner presents a new strategy for teaching HiVAT testing. Instead of describing what has been done, Cem is creating open source examples of the techniques applied to real (open source) applications. These examples are written in Ruby, making the code readable and reusable by snapping in code specific to your own application. Join Cem Kaner and Carol Oliver as they describe three HiVAT techniques, their associated code, and how you can customize them.
Performance Testing Web 2.0 Applications—in an Agile WorldTechWell
Agile methodologies bring new complexities and challenges to traditional performance engineering practices, especially with Web 2.0 technologies that implement more and more functionality on the client side. Mohit Verma presents a Scrum-based performance testing lifecycle for Web 2.0 applications. Mohit explains when performance engineers need to participate in the project, discusses how important it is for the performance engineer to understand the technical architecture, and explores the importance of testing early to identify design issues. Find out how to create the non-functional requirements that are critical for building accurate and robust performance test scenarios. Learn how to implement practices for continuous collaboration between test engineers and developers to help identify performance bottlenecks early. Learn about the tools available today to help you address the testing and tuning of your Web 2.0 applications. Leave with a new appreciation and new approaches to ensure that your Web 2.0-based applications are ready for prime time from day one.
What Hollywood Can Teach Us about Software TestingTechWell
If we observe the world through the lens of software testing, we discover that there are lessons all around us we can apply on the job, and one venue that’s packed with these tidbits is the movie theater. Bernie Berger gives examples of a few unlikely yet credible lessons from the language of some classic Hollywood movies—and how we can apply them as we test. Learn about the primary root cause of many software bugs from Office Space; see how the characters in Saving Private Ryan enacted ideas of exploratory testing; discover how to apply the skills of situational awareness from Sneakers; learn about the context of discovery from All the President’s Men; and see a great example of inattentional blindness from Star Trek II: The Wrath of Khan. In addition to learning how these concepts are fundamentally related to testing software, learn how to apply lessons from your everyday life to testing challenges.
Zorro Circles: Retrospectives for ExcellenceTechWell
Have you wondered how to progressively harness your agile team’s energy, focus on important goals, and improve outcomes? Woody Zuill said, “If you could adopt only one agile practice, then let it be retrospectives. Everything else will follow.” Retrospectives help individuals and teams adjust to today’s constant change and establish a sustainable pace to deliver complex products. Zorro Circles is a framework for designing retrospectives that employs proven techniques to gather and analyze information required to collectively solve problems. Aakash Srinivasan and Vivek Angiras introduce the four key steps—Pave the Way, Develop a Shared Understanding, Know What to Accomplish, and Executable Actions—to using Zorro Circles to create powerful retrospectives. Discover the essentials of designing retrospectives by understanding the multilayered circles of projects’ and people’s characteristics. Take away new techniques to design moments of inspection and adaptation in your organization’s ecosystem and enable excellence in execution, behaviors, and tactics.
User Stories: Across the Seven Product DimensionsTechWell
User stories are a powerful technique agile teams used to communicate requirements. Yet all too often, the stories are poorly written or even incomprehensible. Some stories are too big and overlap across delivery cycles. Others are too small and don’t deliver sufficient details for developers. Join Paul Reed to learn the Seven Product Dimensions—the 7 D’s—which yield “just right” stories that users and product owners can write and developers can understand. Explore and experience the Seven Dimensions: user, interface, action, data, control, quality, and environment. Learn to identify options for high value business and user needs, and then assemble them into cohesive user stories. As you slice the options, you’ll see ways to leverage analysis models to quickly visualize and discuss options. Practice employing structured conversations about user stories to engage customers while taking business and technology perspectives into consideration. Find out how to establish acceptance criteria based on the value considerations to make your stories more valuable. Leave with a practical framework for writing “just right” stories.
Many teams have a relatively easy time adopting the tactical aspects of agile methodologies. Usually a few classes, some tools introduction, and a bit of practice lead teams toward a fairly efficient and effective adoption. However, these teams often get “stuck” and begin to regress or simply start going through the motions—neither maximizing their agile performance nor delivering as much value as they could. Borrowing from his experience and lean software development methods, Bob Galen examines essential patterns—the thinking models of mature agile teams—so you can model them within your own teams. Along the way, you’ll examine patterns for large-scale emergent architecture, relentless refactoring, quality on all fronts, pervasive product owners, lean work queues, providing total transparency, saying No, and many more. Bob also explores why there is still the need for active and vocal leadership in defending, motivating, and holding agile teams accountable.
Model-Based Testing: Concepts, Tools, and TechniquesTechWell
For decades, software development tools and methods have evolved with an emphasis on modeling. Standards like UML and SysML are now used to develop some of the most complex systems in the world. However, test design remains a largely manual, intuitive process. Now, a significant opportunity exists for testing organizations to realize the benefits of modeling. Adam Richards describes how to leverage model-based testing to dramatically improve both test coverage and efficiency—and lower the overall cost of quality. Adam provides an overview of the basic concepts and process implications of model-based testing, including its role in agile. A survey of model types and techniques shows different model-based solutions for different kinds of testing problems. Explore tool integrations and weigh the pros and cons of model-based test development against a variety of system and project-level factors. Gain a working knowledge of the concepts, tools, and techniques needed to introduce model-based testing to your organization.
Accessibility Standards and Testing Techniques: Be Inclusive or Be Left BehindTechWell
While Information and Communication Technology (ICT) accessibility for a wider spectrum of users—including the blind—and their interfaces is being required by law across more jurisdictions, testing for it remains limited, naïve, and too late. The consequences of staying ignorant include increased exposure to litigation, penalties, and loss of contracts and revenue. Join David Best, Sandy Feldman, and Rob Harvie to learn why accessibility is now becoming a valued, integral part of the design process and much different from usability of twenty years ago. Ensure compliance for your organization and clients by familiarizing yourself with the regional and international standards and their criteria, and find out what testing tools and inclusive design practices you can use. Take away an understanding of the three core guidelines for accessibility; components of authoring tools, web content, and user agent accessibility for mobile, web browsers, and media players—and understand their impact on assistive technologies.
It seems impossible for a DevOps team to even attempt planning its work. The team deals with customers’ never-ending requests and constantly-changing priorities. And don’t forget those unfriendly infrastructure errors that always seem to show up at the worst possible time. Best to live day-by-day and try to keep our heads above water, right? Maybe not. Through a case study of a department that helps make great Electronic Arts games, Ben Vining illustrates how a metrics-driven DevOps team can become reliable, responsive, and predictable—with happy staff and delighted customers. Ben details a process of short sprints tied into longer milestones and regular communication with partner teams, which allows his team to successfully plan their work across a number of different projects. Along the way, Ben demonstrates how tracking metrics related to his team’s work and the reliability of their systems help drive this process. Ben hopes to inspire you to try these ideas, improve the health of your team, and increase the value you add to your customers.
Combinatorial Black-Box Testing with Classification TreesTechWell
A basic problem in software testing often is choosing a subset from the near infinite number of possible test cases. Consider the challenges of testing multiple browsers, multiple mobile devices, mobile applications, or use case paths. Testers must select test cases to design, create, and then execute to obtain sufficient coverage—all while managing the time it takes to test relative to risks. Even though test resources are limited, you still want to select the best possible set of tests. Peter Kruse shares his experiences designing test cases with TESTONA, the most popular tool for systematic test design of classification tree-based tests. Peter shows how to integrate expected test outcomes and how to obtain executable test scripts directly from the test specification or user stories. If you are looking to jumpstart your systematic test design and want to avoid unnecessary tests and overhead, this session is for you!
Many organizations never achieve the significant benefits that are promised from automated test execution. Surprisingly often, this is due not to technical factors but to management issues. Dot Graham describes the most important management concerns the test manager must address for test automation success, and helps you understand and choose the best approaches for your organization—no matter which automation tools you use or your current state of automation. Dot explains how automation affects staffing, who should be responsible for which automation tasks, how managers can best support automation efforts leading to success, and what return on investment means in automated testing and what you can realistically expect. Dot also reviews the key technical issues that can make or break the automation effort. Come away with an example set of automation objectives and measures, and a draft test automation strategy that you can use to plan or improve your own automation.
Many organizations never achieve the significant benefits that are promised from automated test execution. Surprisingly often, this is not due to technical factors but to management issues. Dot Graham describes the most important management issues you must address for test automation success, and helps you understand and choose the best approaches for your organization—no matter which automation tools you use or your current state of automation. Dot explains how automation affects staffing, who should be responsible for which automation tasks, how managers can best support automation efforts leading to success, and what return on investment means in automated testing and what you can realistically expect. Dot also reviews the key technical issues that can make or break the automation effort. Come away with an example set of automation objectives and measures, and a draft test automation strategy that you can use to plan or improve your own automation.
A number of test automation ideas that at first glance seem very sensible actually contain pitfalls and problems that you should avoid. Dot Graham describes five of these “intelligent mistakes”—automated tests will find more bugs more quickly; spending a lot on a tool must guarantee great benefits; it’s necessary to automate all of our manual tests; tools are expensive so we have to show a substantial return on investment; and testing tools must be used by the testers. Dot points out that automation doesn’t find bugs; tests do. Good automation does not come out of the box and is not automatic. Automating everything may not give you better (or faster) testing. Determining the actual rate of return is not only surprisingly difficult but may actually be harmful. Turning testers into test automators may waste their skills and talents. Join Dot for a rousing discussion of intelligent mistakes—so you can be smart enough to avoid them.
In chess, the word blunder means a very bad move by someone who should know better. Even though functional test automation has been around for a long time, people still make some very bad moves and serious blunders. The most common misconception in automation is thinking that manual testing is the same as automated testing. And this misguided thinking accounts for most of the blunders in system level test automation. Dorothy Graham takes you on a tour of these blunders, including the Stable-Application Myth (you can’t start automating until the application is stable), Inside-the-Box Thinking (automating only the obvious test execution), and the Project/Non-Project Dilemma (failing to treat automation like a project by not funding or resourcing it and treating automation as only a project). Other blunders include Testing–Tools–Test, Silver Bullet, Automating the Wrong Thing, Who Needs GPS, How Hard Can It Be, and Isolationism. New skills, approaches, and objectives are needed or you’ll end up with inefficient automation, high-maintenance costs, and wasted effort. Join Dot to discover how you can avoid these common blunders and achieve valuable test automation.
In chess, the word blunder means a very bad move by someone who should know better. Even though functional test automation has been around for a long time, people still make some very bad moves and serious blunders. The most common misconception in automation is thinking that manual testing is the same as automated testing. And this thinking accounts for most of the blunders in system level test automation. Dorothy Graham takes us on a tour of these blunders, including: the Stable-Application Myth (you can’t start automating until the application is stable), Inside-the-Box Thinking (automating only the obvious test execution), the Project/Non-Project Dilemma (failing to treat automation like a project by not funding or resourcing it, and treating automation as only a project). Other blunders include Testing-Tools-Test, Silver Bullet, Automating the Wrong Thing, Who Needs GPS, How Hard Can It Be, and Isolationism. Different skills, approaches, and objectives are needed or you’ll end up with inefficient automation, high maintenance costs, and wasted effort. Join Dot to discover how you can avoid these common blunders and achieve valuable test automation.
In chess, the word blunder means a very bad move by someone who should know better. Even though functional test automation has been around for a long time, people still make some very bad moves and serious blunders. The most common misconception in automation is thinking that manual testing is the same as automated testing. And this thinking accounts for most of the blunders in system level test automation. Dorothy Graham takes us on a tour of these blunders, including: the Stable-Application Myth (you can’t start automating until the application is stable), Inside-the-Box Thinking (automating only the obvious test execution), the Project/Non-Project Dilemma (failing to treat automation like a project by not funding or resourcing it, and treating automation as only a project). Other blunders include Testing-Tools-Test, Silver Bullet, Automating the Wrong Thing, Who Needs GPS, How Hard Can It Be, and Isolationism. Different skills, approaches, and objectives are needed or you’ll end up with inefficient automation, high maintenance costs, and wasted effort. Join Dot to discover how you can avoid these common blunders and achieve valuable test automation.
Successful Test Automation: A Manager’s ViewTechWell
Many organizations invest substantial time and effort in test automation but do not achieve the significant returns they expected. Some blame the tool they used; others conclude test automation just doesn't work in their situation. The truth, however, is often very different. These organizations are typically doing many of the right things but they are not addressing key issues that are vital to long term test automation success. Describing the most important issues that you must address, Mark Fewster helps you understand and choose the best approaches for your organization—no matter which automation tools you use. We’ll discuss both management issues—responsibilities, automation objectives, and return on investment—and technical issues—testware architecture, pre- and post-processing, and automated comparison techniques. If you are involved with managing test automation and need to understand the key issues in making test automation successful, join Mark for this enlightening tutorial.
Challenges in automation which testers face often lead to subsequent failures. Learn how to respond to these common challenges by developing a solid business case for increased automation adoption by engaging manual testers in the testing organization, being technology agnostic, and stabilizing test scripts regardless of applications changes.
In this quality assurance training session, you will learn introduction to automation testing. Topics covered in this course are:
• Introduction
• Why Automated Testing?
• What can I Automate?
• Test Automation Process
• Automation Tool
• Automation Framework
To know more, visit this link: https://www.mindsmapped.com/courses/quality-assurance/software-testing-quality-assurance-qa-training-with-hands-on-exercises/
Top Ten Tips for Tackling Test Automation Webinar Presentation.pptxInflectra
Inflectra and Checkpoint Technologies co-hosted the webinar: Top Ten Tips for Tackling Test Automation. In this webinar, Adam Sandman (Inflectra) and Bob Crews (Checkpoint Technologies) explored the challenges surrounding test automation and offered their tips on overcoming them.
Find the recording of the Webinar on our YouTube channel: https://www.youtube.com/watch?v=vY1MbW4qWnQ
Webinar Agenda:
-Top 10 challenges of test automation with impact and solutions
-Impacts: potential risks if challenges are not overcome
-Solutions: tips to overcoming the challenges
-Automated functional testing
-Criteria of an Automation Assessment
-Addressing several challenges with Inflectra's Spira and Rapise
Webinar Presenters:
Adam Sandman is the Founder and CEO of Inflectra. He has been working in the IT industry for the past 25+ years. His areas of expertise span software architecture to agile development, software testing, test automation, and project management. He is interested in technology, business, and enabling people to follow their passions. At Inflectra, Adam is responsible for researching the tools, technologies, and processes in the software testing and quality assurance space. Adam is a prolific speaker whose speaking engagements range from StarEast, and Eurostar to STPcons, DevGeekWeek, Swiss Testing Day, NDIA, STARCanada, TestingMind, Agile DevOps West, StarWest, testCon, JFTL, and many more.
Bob Crews, Co-Founder and CEO of Checkpoint Technologies, is a consultant with 34 years of IT experience in full life-cycle development and software testing. Bob and his organization provide services and solutions focused on QA with a concentration in functional, performance and application security testing. He’s assisted organizations such as Harvard University, Raymond James, the FBI, and the Department of Veterans Affairs in developing teams, processes, and solutions to help organizations deliver higher quality software faster. He’s consulted for over 290 organizations on QA, effective software testing, strategic test planning, enhanced test automation, and risk-based testing. He’s exceptionally enthusiastic about the future of IT and software testing and believes “The best is yet ahead!”
Automated Software Testing Framework Training by Quontra SolutionsQuontra Solutions
Learn through Experience -- We differentiate our training and development program by delivering Role-Based training instead of Product-based training. Ultimately, our goal is to deliver the best IT Training to our clients.
In this training, attendees learn:
Introduction to Automation
• What is automation
• Advantages of automation & Disadvantages of automation
• Different types of Automation Tools
• What to automate in projects
• When to start automation. Scope for automation testing in projects
• About open-source automation tools
Introduction to Selenium
• What is selenium
• Why selenium
• Advantage and Disadvantages of selenium
Selenium components
• Selenium IDE
• Selenium RC
• Selenium WebDriver
• Selenium Grid
Selenium IDE
• Introduction to IDE
• IDE Installation
• Installation and uses of Firepath, Firebug & Debug bar
• Property & value of elements
• Selenium commands
• Assertions & Verification
• Running, pausing and debugging script
• Disadvantages of selenium IDE
• How to convert selenium IDE Scripts into other languages
Locators
• Tools to identify elements/objects
• Firebug
• IE Developer tools
• Google Chrome Developer tools
• Locating elements by ID
• Finding elements by name
• Finding elements by link text
• Finding elements by XPath
• Finding Elements by using CSS
• Summary
Selenium RC
• What is selenium RC
• Advantages of RC, Architecture
• What is Eclipse/IntelliJ, Selenium RC configure with Eclipse/IntelliJ
• Creating, running & debugging RC scripts
Java Concepts
• Introduction to OOPs concepts and Java
• Installation: Java, Eclipse/IntelliJ, selenium, TestNg/JUnit
• operators in java
• Data types in java
• Conditional statements in java
• Looping statements in java
• Output statements in java
• Classes & Objects
• Collection Framework
• Regular Expressions
• Exception Handling
• Packages, Access Specifiers /Modifiers
• String handling
• Log4J for logging
Selenium Web Driver with Java
• Introduction to WebDriver
• Advantages
• Different between RC and WebDriver
• Selenium WebDriver- commands
• Generate scripts in Eclipse/IntelliJ. Run Test Scripts.
• Debugging Test Script
• Database Connections
• Assertions, validations
• Working with Excel
• Pass the data from Excel
• Working with multiple browser
• Window Handling, Alert/confirm & Popup Handling
• Mouse events
• Wait mechanism
• Rich Web Handling: Calendar handing, Auto suggest, Ajax, browser forward/back navigation, keyboard events, certificate handling, event listeners
TestNg/JUnit Framework
• What is TestNg/JUnit
• Integrate the Selenium Scripts and Run from TestNg/JUnit
• Reporting Results and Analysis
• Run Scripts from multiple programs
• Parallel running using TestNg/JUnit
Automation Framework development in Agile testing
• Introduction to Frame W
Similar to Managing Successful Test Automation (20)
Do you ever feel you have lost confidence in your own abilities? Why does this happen? Isabel Evans spends a lot of time painting. Someone once commented, “Why are you doing this, when you are not very good at it?” And gradually she stopped drawing and painting, after being intimidated by a conventional vision of what good art should look like. At the same time, she experienced a parallel loss of confidence in her professional abilities. Attempting creative pursuits like drawing and painting is essential to cognitive, emotional, creative abilities and she began to understand the correlation between her creative activities and her confidence. Making errors, being wrong, failing – that is a generous gift we receive when we practice outside our skill level. By staying in a comfort zone and repeating successes, we stagnate. As Isabel started to create again she thought “I don’t feel good at it, I do feel good doing it” The difference was that she was learning, having ideas and the act of re-engaging with failure, together with the comradeship of friends and colleagues, including at Women Who Test, Isabel has regained her confidence in her professional abilities, and been able to reboot her career and joy. Join Isabel to share a journey from self-perceived failure, to recovery and renewed learning.
Instill a DevOps Testing Culture in Your Team and Organization TechWell
The DevOps movement is here. Companies across many industries are breaking down siloed IT departments and federating them into product development teams. Testing and its practices are at the heart of these changes. Traditionally, IT organizations have been staffed with mostly manual testers and a limited number of automation and performance engineers. To keep pace with development in the new “you build it, you own it” environment, testing teams and individuals must develop new technical skills and even embrace coding to stay relevant and add greater value to the business. DevOps really starts with testing. Join Adam Auerbach as he explains what DevOps is and how it relates to testing. He describes how testing must change from top to bottom and how to access your own environment to identify improvement opportunities. Adam dives into practices like service virtualization, test data management, and continuous testing so you can understand where you are now and identify steps needed to instill a DevOps testing culture in your team and organization.
Test Design for Fully Automated Build ArchitectureTechWell
Imagine this … As soon as any developed functionality is submitted into the code repository, it is automatically subjected to the appropriate battery of tests and then released straight into production. Setting up the pipeline capable of doing just that is becoming more and more common and something you need to know about. But most organizations hit the same stumbling block—just what IS the appropriate battery of tests? Automated build architectures don't always lend themselves well to the traditional stages of testing. In this hands-on tutorial, Melissa Benua introduces you to key test design principles—applicable to organizations both large and small—that allow you to take full advantage of the pipeline's capabilities without introducing unnecessary bottlenecks. Learn how to make highly reliable tests that run fast and preserve just enough information to let testers and developers determine exactly what went wrong and how to reproduce the error locally. Explore ways to reduce overlap while still maintaining adequate test coverage. Take back ideas about which test areas could benefit from being combined into a single suite and which areas could benefit most from being broken out altogether.
Build Your Mobile App Quality and Test StrategyTechWell
Let’s build a mobile app quality and testing strategy together. Whether you have a web, hybrid, or native app, building a quality and testing strategy means (1) knowing what data and tools you have available to make agile decisions, (2) understanding your customers and your competitors, and (3) testing your app under real-world conditions. Jason Arbon guides you through the latest techniques, data, and tools to ensure the awesomeness of your mobile app quality and testing strategy. Leave this interactive session with a strategy for your very own app—or one you pretend to own. The information Jason shares is based on data from Appdiff’s next-gen mobile app testing platform, lessons from Applause/uTest’s crowd, text mining hundreds of millions of app store reviews, and in-depth discussions with top mobile app development teams.
Testing Transformation: The Art and Science for SuccessTechWell
Technologies, testing processes, and the role of the tester have evolved significantly in the past few years with the advent of agile, DevOps, and other new technologies. It is critical that we testing professionals evaluate ourselves and continue to add tangible value to our organizations. In your work, are you focused on the trivial or on real game changers? Jennifer Bonine describes critical elements that help you artfully blend people, process, and technology to create a synergistic relationship that adds value. Jennifer shares ideas on mastering politics, maneuvering core vs. context, and innovating your technology strategies and processes. She explores how new processes can be introduced in an organization, what the role of organizational culture is in determining the success of a project, and how you can know what tools will add value vs. simply adding overhead and complexity. Jennifer reviews critically needed tester skills and discusses a continual learning model to evolve your skills and stay relevant. This discussion can lead you to technologies, processes, and skills you can stake your career on.
We’ve all been there. We work incredibly hard to develop a feature and design tests based on written requirements. We build a detailed test plan that aligns the tests with the software and the documented business needs. And when we put the tests to the software, it all falls apart because the requirements were changed without informing everyone. Mary Thorn says help is at hand. Enter behavior-driven development (BDD), and Cucumber and SpecFlow, tools for running automated acceptance tests and facilitating BDD. Mary explores the nuances of Cucumber and SpecFlow, and shows you how to implement BDD and agile acceptance testing. By fostering collaboration for implementing active requirements via a common language and format, Cucumber and SpecFlow bridge the communication gap between business stakeholders and implementation teams. In this workshop, practice writing feature files with the best practices Mary has discovered over numerous implementations. If you experience developers not coding to requirements, testers not getting requirements updates, or customers who feel out of the loop and don’t get what they ask for, Mary has answers for you.
Develop WebDriver Automated Tests—and Keep Your SanityTechWell
Many teams go crazy because of brittle, high-maintenance automated test suites. Jim Holmes helps you understand how to create a flexible, maintainable, high-value suite of functional tests using Selenium WebDriver. Learn the basics of what to test, what not to test, and how to avoid overlapping with other types of testing. Jim includes both philosophical concepts and hands-on coding. Testers who haven't written code should not be intimidated! We'll pair you up to make sure you're successful. Learn to create practical tests dealing with advanced situations such as input validation, AJAX delays, and working with file downloads. Additionally, discover when you need to work together with developers to create a system that's more easily testable. This tutorial focuses primarily on automating web tests, but many of the same concepts can be applied to other UI environments. Demos and labs will be in C# and Java using WebDriver. Leave this tutorial having learned how to write high-value WebDriver tests—and stay sane while doing so.
DevOps is a cultural shift aimed at streamlining intergroup communication and improving operational efficiency for development and operations groups. Over time, inclusion of other IT groups under the DevOps umbrella has become the norm for many organizations. But even broadening the boundaries of DevOps, the conversation has been largely devoid of the business units’ place at the table. A common mistake organizations make while going through the DevOps transformation is drawing a line at the IT boundary. If that occurs, a larger, more inclusive silo within the organization is created, operating in an informational vacuum and causing operational inefficiency and goal misalignment. Sharing his experiences working on both sides of the fence, Leon Fayer describes the importance of including business units in order to align technology decisions with business goals. Leon discusses inclusion of business units in existing agile processes, benefits of cross-departmental monitoring, and a business-first approach to technology decisions.
Eliminate Cloud Waste with a Holistic DevOps StrategyTechWell
Chris Parlette maintains that renting infrastructure on demand is the most disruptive trend in IT in decades. In 2016, enterprises spent $23B on public cloud IaaS services. By 2020, that figure is expected to reach $65B. The public cloud is now used like a utility, and like any utility, there is waste. Who's responsible for optimizing the infrastructure and reducing wasted expenses? It’s DevOps. The excess expense, known as cloud waste, comprises several interrelated problems: services running when they don't need to be, improperly sized infrastructure, orphaned resources, and shadow IT. There are a few core tenets of DevOps—holistic thinking, no silos, rapid useful feedback, and automation—that can be applied to reducing your cloud waste. Join Chris to learn why you should include continuous cost optimization in your DevOps processes. Automate cost control, reduce your cloud expenses, and make your life easier.
Transform Test Organizations for the New World of DevOpsTechWell
With the recent emergence of DevOps across the industry, testing organizations are being challenged to transform themselves significantly within a short period of time to stay meaningful within their organizations. It’s not easy to plan and approach these changes considering the way testing organizations have remained structured for ages. These challenges start from foundational organizational structures and can cut across leadership influence, competencies, tools strategy, infrastructure, and other dimensions. Sumit Kumar shares his experience assisting various organizations to overcome these challenges using an organized DevOps enablement framework. The framework includes radical restructuring, turning the tools strategy upside down, a multidimensional workforce enablement supported by infrastructure changes, redeveloped collaborations models, and more. From his real world experiences Sumit shares tips for approaching this journey and explains the roadmap for testing organizations to transform themselves to lead the quality in DevOps.
The Fourth Constraint in Project Delivery—LeadershipTechWell
All too often, the triple constraints—time, cost, and quality—are bandied about as if they are the be-all, end-all. While they are important, leadership—the fourth and larger underpinning constraint—influences the first three. Statistics on project success and failure abound, and these measurements are usually taken against the triple constraints. According to the Project Management Institute, only 53 percent of projects are completed within budget, and only 49 percent are completed on time. If so many projects overrun budget and are late, we can’t really say, “Good, fast, or cheap—pick two.” Rob Burkett talks about leadership at every level of a team. He shares his insights and stories gleaned from his years of IT and project management experience. Rob speaks to some of the glaring difficulties in the workplace in general and some specifically related to IT delivery and project management. Leave with a clearer understanding of how to communicate with teams and team members, and gain a better understanding of how you can be a leader—up and down your organization.
Resolve the Contradiction of Specialists within Agile TeamsTechWell
As teams grow, organizations often draw a distinction between feature teams, which deliver the visible business value to the user, and component teams, which manage shared work. Steve Berczuk says that this distinction can help organizations be more productive and scale effectively, but he recognizes that not all shared work fits into this model. Some work is best handled by “specialists,” that is people with unique skills. Although teams composed entirely of T-shaped people is ideal, certain skills are hard to come by and are used irregularly across an organization. Since these specialists often need to work closely with teams, rather than working from their own backlog, they don’t fit into the component team model. The use of shared resources presents challenges to the agile planning model. Steve Berczuk shares how teams such as those providing infrastructure services and specialists can fit into a feature+component team model, and how variations such as embedding specialists in a scrum team can both present process challenges and add significant value to both the team and the larger organization.
Pin the Tail on the Metric: A Field-Tested Agile GameTechWell
Metrics don’t have to be a necessary evil. If done right, metrics can help guide us to make better forward-looking decisions, rather than being used for simply managing or monitoring. They can help us identify trade-offs between options for what to do next versus punitive or worse, purely managerial measures. Steve Martin won’t be giving the Top Ten List of field-tested metrics you should use. Instead, in this interactive mini-workshop, he leads you through the critical thinking necessary for you to determine what is right for you to measure. First, Steve explores why you want to measure something—whether it’s for a team, a portfolio, or even an agile transformation. Next, he provides multiple real-life metrics examples to help drive home concepts behind characteristics of good and bad metrics. Finally, Steve shows how to run his field-tested agile game—Pin the Tail on the Metric. Take back this activity to help you guide metrics conversations at your organization.
Agile Performance Holarchy (APH)—A Model for Scaling Agile TeamsTechWell
A hierarchy is an organizational network that has a top and a bottom, and where position is determined by rank, importance, and value. A holarchy is a network that has no top or bottom and where each person’s value derives from his ability, rather than position. As more companies seek the benefits of agile, leaders need to build and sustain delivery capability while scaling agile without introducing unnecessary process and overhead. The Agile Performance Holarchy (APH) is an empirical model for scaling and sustaining agility while continuing to deliver great products. Jeff Dalton designed the APH by drawing from lessons learned observing and assessing hundreds of agile companies and teams. The APH helps implement a holarchy—a system composed of interacting organizational units called holons—centered on a series of performance circles that embody the behaviors of high performing agile organizations. Jeff describes how APH provides guidelines in the areas of leadership, values, teaming, visioning, governing, building, supporting, and engaging within an all-agile organization. Join Jeff to see what the APH is all about and how you can use it in your team and organization.
A Business-First Approach to DevOps ImplementationTechWell
DevOps is a cultural shift aimed at streamlining intergroup communication and improving operational efficiency for development and operations groups. Over time, inclusion of other IT groups under the DevOps umbrella has become the norm for many organizations. But even broadening the boundaries of DevOps, the conversation has been largely devoid of the business units’ place at the table. A common mistake organizations make while going through the DevOps transformation is drawing a line at the IT boundary. If that occurs, a larger, more inclusive silo within the organization is created, operating in an informational vacuum and causing operational inefficiency and goal misalignment. Sharing his experiences working on both sides of the fence, Leon Fayer describes the importance of including business units in order to align technology decisions with business goals. Leon discusses inclusion of business units in existing agile processes, benefits of cross-departmental monitoring, and a business-first approach to technology decisions.
Databases in a Continuous Integration/Delivery ProcessTechWell
DevOps is transforming software development with many organizations adopting lean development practices, implementing continuous integration (CI), and performing regular continuous deployment (CD) to their production environments. However, the database is largely ignored and often seen as a bottleneck in the DevOps process. Steve Jones discusses the challenges of database development and why many developers find the database to be an impediment to the CD process. Steve shares the techniques you can use to fit a database into the DevOps process. Learn how to store database code in a version control system, and the differences between that and application code. Steve demonstrates a CI process with SQL code and uses automated testing frameworks to check the code. Steve then shows how automated releases with manual gates can reduce the stress and risk of database deployments while ensuring consistent, reliable, repeatable releases to QA, UAT, and production.
Mobile Testing: What—and What Not—to AutomateTechWell
Organizations are moving rapidly into mobile technology, which has significantly increased the demand for testing of mobile applications. David Dangs says testers naturally are turning to automation to help ease the workload, increase potential test coverage, and improve testing efficiency. But should you try to automate all things mobile? Unfortunately, the answer is not always clear. Mobile has its own set of complications, compounded by a wide variety of devices and OS platforms. Join David to learn what mobile testing activities are ripe for automation—and those items best left to manual efforts. He describes the various considerations for automating each type of mobile application: mobile web, native app, and hybrid applications. David also covers device-level testing, types of testing, available automation tools, and recommendations for automation effectiveness. Finally, based on his years of mobile testing experience, David provides some tips and tricks to approach mobile automation. Leave with a clear plan for automating your mobile applications.
Cultural Intelligence: A Key Skill for SuccessTechWell
Diversity is becoming the norm in everyday life. However, introducing global delivery models without a proper understanding of intercultural differences can lead to difficulty, frustration, and reduced productivity. Priyanka Sharma and Thena Barry say that in our diverse world, we need teams with people who can cross these boundaries, communicate effectively, and build the diverse networks necessary to avoid problems. We need to learn about cultural intelligence (CI) and cultural quotient (CQ). CI is the ability to relate and work effectively across cultures. CQ is the cognitive, motivational, and behavioral capacity to understand and respond to beliefs, values, attitudes, and behaviors of individuals and groups. Together, CI and CQ can help us build behavioral capacities that aid motivation, behavior, and productivity in teams as well as individuals. Priyanka and Thena show how to build a more culturally intelligent place with tools and techniques from Leading with Cultural Intelligence, as well as content from the Hofstede cultural model. In addition, they illustrate the model with real-life experiences and demonstrate how they adapted in similar circumstances.
Turn the Lights On: A Power Utility Company's Agile TransformationTechWell
Why would a century-old utility with no direct competitors take on the challenge of transforming its entire IT application organization to an agile methodology? In an increasingly interconnected world, the expectations of customers continue to evolve. From smart meters to smart phones, IoT is creating a crisis point for industries not accustomed to rapid change. Glen Morris explains that pizzas can be tracked by the minute and packages at every stop, and customers now expect this same customer service model should exist for all industries—including power. Glen examines how to create momentum and transform non-IT-focused industries to an agile model. If you are struggling with gaining traction in your pursuit of agile within your business, Glen gives you concrete, practical experiences to leverage in your pursuit. Finally, he communicates how to gain buy-in from business partners who have no idea or concern about agile or its methodologies. If your business partners look at you with amusement when you mention the need for a dedicated Product Owner, join Glen as he walks you through the approaches to overcoming agile skepticism.
Scale: The Most Hyped Term in Agile Development TodayTechWell
Scrum is everywhere. More than 90 percent of agile teams use it. But for many organizations wanting to scale agile, one team using Scrum is not enough. Dave West says the Nexus Framework, created by Ken Schwaber, the co-creator of Scrum, provides an exoskeleton for Scrum. Nexus allows multiple teams to work together to produce an integrated increment regularly. It addresses the key challenges of scaling agile development by adding new yet minimal events, artifacts, and roles to the Scrum framework. Dave discusses Nexus, addresses its boundaries, and explains what else is needed for agile to thrive in an organization. Dave explores how organizations have transitioned to agile, and examines their successes and challenges in implementing Scrum, how they envision scaling with Nexus, and goals for creating a Scrum Studio.
Pushing the limits of ePRTC: 100ns holdover for 100 daysAdtran
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
Enhancing Performance with Globus and the Science DMZGlobus
ESnet has led the way in helping national facilities—and many other institutions in the research community—configure Science DMZs and troubleshoot network issues to maximize data transfer performance. In this talk we will present a summary of approaches and tips for getting the most out of your network infrastructure using Globus Connect Server.
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
The Metaverse and AI: how can decision-makers harness the Metaverse for their...Jen Stirrup
The Metaverse is popularized in science fiction, and now it is becoming closer to being a part of our daily lives through the use of social media and shopping companies. How can businesses survive in a world where Artificial Intelligence is becoming the present as well as the future of technology, and how does the Metaverse fit into business strategy when futurist ideas are developing into reality at accelerated rates? How do we do this when our data isn't up to scratch? How can we move towards success with our data so we are set up for the Metaverse when it arrives?
How can you help your company evolve, adapt, and succeed using Artificial Intelligence and the Metaverse to stay ahead of the competition? What are the potential issues, complications, and benefits that these technologies could bring to us and our organizations? In this session, Jen Stirrup will explain how to start thinking about these technologies as an organisation.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex ProofsAlex Pruden
This paper presents Reef, a system for generating publicly verifiable succinct non-interactive zero-knowledge proofs that a committed document matches or does not match a regular expression. We describe applications such as proving the strength of passwords, the provenance of email despite redactions, the validity of oblivious DNS queries, and the existence of mutations in DNA. Reef supports the Perl Compatible Regular Expression syntax, including wildcards, alternation, ranges, capture groups, Kleene star, negations, and lookarounds. Reef introduces a new type of automata, Skipping Alternating Finite Automata (SAFA), that skips irrelevant parts of a document when producing proofs without undermining soundness, and instantiates SAFA with a lookup argument. Our experimental evaluation confirms that Reef can generate proofs for documents with 32M characters; the proofs are small and cheap to verify (under a second).
Paper: https://eprint.iacr.org/2023/1886
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
1. Managing Successful Test Automation
Contents
Session 0: Introduction to the tutorial
Objectives, What we cover (and don’t cover) today
Session 1: Planning and Managing Test Automation
Responsibilities
Pilot project
Test automation objectives (and exercise)
Measures for automation
Return on Investment (ROI) (and exercise)
Session 2: Testware Architecture
Importance of a testware architecture
What needs to be organised
Session 3: Pre- and Post-Processing
Automating more than tests
Test status
Session 4: Scripting Techniques
Objectives of scripting techniques
Different types of scripts
Domain specific test language
Session 5: Automated Comparison
Automated test verification
Test sensitivity
Comparison example
Session 6: Final Advice, Q&A and Direction
Strategy exercise
Final advice
Questions and Answers
Appendix (useful stuff)
That’s no reason to automate (Better Software article)
Effective Automated Testing with a DSTL, Martin Gijsen
Man and Machine, Jonathan Kohl (Better Software)
Technical vs non-technical skills in test automation
104. “Why automate?”
This seems such
an easy question to answer; yet many
people don’t achieve the success they
hoped for. If you are aiming in the wrong
direction, you will not hit your target!
This article explains why some
testing objectives don’t work for automation, even though they may be very
sensible goals for testing in general. We
take a look at what makes a good test
automation objective; then we examine
six commonly held—but misguided—
objectives for test execution automation,
explaining the good ideas behind them,
where they fail, and how these objectives
can be modified for successful test automation.
Good Objectives for Test
Automation
A good objective for test automation
should have a number of characteristics.
First of all, it should be measurable so
that you can tell whether or not you
have achieved it.
Objectives for test automation should
support testing activities but should not
be the same as the objectives for testing.
Testing and automation are different and
distinct activities.
Objectives should be realistic and
achievable; otherwise, you will set yourself up for failure. It is better to have
smaller-scale goals that can be met than
far-reaching goals that seem impossible.
Of course, many small steps can take
you a long way!
Automation objectives should be
both short and long term. The shortterm goals should focus on what can be
achieved in the next month or quarter.
The long-term goals focus on where you
want to be in a year or two.
Objectives should be regularly revised
in the light of experience.
Misguided Objectives for
Test Automation
OBJECTIVE 1: FIND MORE BUGS
Good ideas behind this objective:
tomated testing should find them
quicker.
run more tests and find even more
bugs.
so we should also find bugs in
the parts we weren’t able to test
manually.
Basing the success of automation on
finding bugs—especially the automation of regression tests—is not a good
thing to do for several reasons. First, it
is the quality of the tests that determines
whether or not bugs are found, and this
has very little, if anything, to do with
automation. Second, if tests are first run
manually, any bugs will be found then,
and they may be fixed by the time the
automated tests are run. Finally, it sets
an expectation that the main purpose of
test automation is to find bugs, but this
is not the case: A repeated test is much
less likely to find a new bug than a new
test. If the software is really good, automation may be seen as a waste of time
and resources.
Regression testing looks for unexpected, detrimental side effects in unchanged software. This typically involves running a lot of tests, many of
which will not find any defects. This is
ideal ground for test automation as it
can significantly reduce the burden of
this repetitive work, freeing the testers
to focus on running manual tests where
more defects are likely to be. It is the
testing that finds bugs—not the automation. It is the testers who may be able to
find more bugs, if the automation frees
them from mundane repetitive work.
The number of bugs found is a misleading measure for automation in any
case. A better measure would be the percentage of regression bugs found (compared to a currently known total). This
is known as the defect detection percentage (DDP). See the StickyNotes for
more information.
Sometimes this objective is phrased
in a slightly different way: “Improve
the quality of the software.” But identifying bugs does nothing to improve
software—it is the fixing of bugs that
improves the software, and this is a development task.
If finding more bugs is something that
you want to do, make it an objective for
measuring the value of testing, not for
measuring the value of automation.
Better automation objective: Help teswww.StickyMinds.com
ters find more regression bugs (so fewer
regression failures occur in operation).
This could be measured by increased
DDP for regression bugs, together with
a rating from the testers about how well
the automation has supported their objectives.
OBJECTIVE 2: RUN REGRESSION TESTS
OVERNIGHT AND ON WEEKENDS
Good ideas behind this objective:
nings and weekends).
“while we sleep.”
At first glance, this seems an excellent
objective for test execution automation,
and it does have some good points.
Once you have a good set of automated regression tests, it is a good idea
to run the tests unattended overnight
and on weekends, but resource use is not
the most important thing.
What about the value of the tests
that are being run? If the regression
tests that would be run “off peak” are
really valuable tests, giving confidence
that the main areas of the system are still
working correctly, then this is useful.
But the focus needs to be on supporting
good testing.
It is too easy to meet this stated objective by just running any test, whether it
is worth running or not. For example, if
you ran the same one test over and over
again every night and every weekend,
you would have achieved the goal as
stated, but it is a total waste of time
and electricity. In fact, we have heard of
someone who did just this! (We think he
left the company soon after.)
Of course, automated tests can be run
much more often, and you may want
some evidence of the increased test execution. One way to measure this is using
equivalent manual test effort (EMTE).
For all automated tests, estimate how
long it would have taken to run those
tests manually (even though you have no
intention of doing so). Then each time
the test is run automatically, add that
EMTE to your running total.
Better automation objective: Run the
most important or most useful tests, employing under-used computer resources
when possible. This could be partially
JULY/AUGUST 2009
BETTER SOFTWARE
33
105. measured by the increased use of resources and by EMTE, but should also
include a measure of the value of the
tests run, for example, the top 25 percent of the current priority list of most
important tests (priority determined by
the testers for each test cycle).
OBJECTIVE 3: REDUCE TESTING STAFF
Good ideas behind this objective:
tool, so we should be able to save
elsewhere.
and staff costs are high.
This is an objective that seems to
be quite popular with managers. Some
managers may go even further and think
that the tool will do the testing for them,
so they don’t need the testers—this is
just wrong. Perhaps managers also think
that a tool won’t be as argumentative as
a tester!
It is rare that staffing levels are reduced when test automation is introduced; on the contrary, more staff are
usually needed, since we now need
people with test script development skills
in addition to people with testing skills.
You wouldn’t want to let four testers go
and then find that you need eight test automators to maintain their tests!
Automation supports testing activities; it does not usurp them. Tools cannot
make intelligent decisions about which
tests to run, when, and how often. This
is a task for humans able to assess the
current situation and make the best use
of the available time and resources.
Furthermore, automated testing is
not automatic testing. There is much
work for people to do in building the automated tests, analyzing the results, and
maintaining the testware.
Having tests automated does—or at
least should—make life better for testers.
The most tedious and boring tasks are
the ones that are most amenable for automation, since the computer will happily do repetitive tasks more consistently
and without complaining. Automation
can make test execution more efficient,
but it is the testers who make the tests
themselves effective. We have yet to see
a tool that can think up tests as well as a
human being can!
34
BETTER SOFTWARE
JULY/AUGUST 2009
The objective as stated is a management objective, not an appropriate objective for automation. A better management objective is “Ensure that everyone
is performing tasks they are good at.”
This is not an automation objective
either, nor is “Reducing the cost of
testing.” These could be valid objectives,
but they are related to management, not
automation.
Better automation objective: The total
cost of the automation effort should be
significantly less than the total testing effort saved by the automation. This could
be partially measured by an increase in
tests run or coverage achieved per hour
of human effort.
OBJECTIVE 4: REDUCE ELAPSED TIME
FOR TESTING
Good ideas behind this objective:
way we can save time is good.
testing will help overall.
This one seems very sensible at first
and sometimes it is even quantified—
“Reduce elapsed time by X%”—which
sounds even more impressive. However,
this objective can be dangerous because
of confusion between “testing” and “test
execution.”
The first problem with this objective is that there are much easier ways
to achieve it: run fewer tests, omit long
tests, or cut regression testing. These are
not good ideas, but they would achieve
the objective as stated.
The second problem with this objective is its generality. Reducing the
elapsed time for “testing” gives the impression we are talking about reducing
the elapsed time for testing as a whole.
However, test execution automation
tools are focused on the execution of
the tests (the clue is in the name!) not
the whole of testing. The total elapsed
time for testing may be reduced only if
the test execution time is reduced sufficiently to make an impact on the whole.
What typically happens, though, is that
the tests are run more frequently or
more tests are run. This can result in
more bugs being found (a good thing),
that take time to fix (a fact of life), and
www.StickyMinds.com
increase the need to run the tests again
(an unavoidable consequence).
The third problem is that there are
many factors other than execution that
contribute to the overall elapsed time
for testing: How long does it take to set
up the automated run and clear up after
it? How long does it take to recognize a
test failure and find out what is actually
wrong (test fault, software fault, environment problem)? When you are testing
manually, you know the context—you
know what you have done just before
the bug occurs and what you were doing
in the previous ten minutes. When a tool
identifies a bug, it just tells you about the
actual discrepancy at that time. Whoever
analyzes the bug has to put together the
context for the bug before he or she can
really identify the bug.
In figures 1 and 2, the blocks represent the relative effort for the different
activities involved in testing. In manual
testing, there is time taken for editing
tests, maintenance, set up of tests, executing the tests (the largest component
of manual testing), analyzing failures,
and clearing up after tests have completed. In figure 1, when those same tests
are automated, we see the illusion that
automating test execution will save us
a lot of time, since the relative time for
execution is dramatically reduced. However, figure 2 shows us the true picture—
total elapsed time for testing may actually increase, even though the time for
test execution has been reduced. When
test automation is more mature, then the
total elapsed time for all of the testing
activities may decrease below what it
was initially for manual testing. Note
that this is not to scale; the effects may
be greater than we have illustrated.
We now can see that the total elapsed
time for testing depends on too many
things that are outside the control or influence of the test automator.
The main thing that causes increased
testing time is the quality of the software—the number of bugs that are already there. The more bugs there are,
the more often a test fails, the more bug
reports need to be written up, and the
more retesting and regression testing
are needed. This has nothing to do with
whether or not the tests are automated or
manual, and the quality of the software
106. is the responsibility of the developers,
not the testers or the test automators.
Finally, how much time is spent maintaining the automated tests? Depending
on the test infrastructure, architecture,
or framework, this could add considerably to the elapsed time for testing.
Maintenance of the automated tests for
later versions of the software can consume a lot of effort that also will detract
from the savings made in test execution.
This is particularly problematic when
the automation is poorly implemented,
without thought for maintenance issues
when designing the testware architecture. We may achieve our goal with the
first release of software, but later versions may fail to repeat the success and
may even become worse.
Here is how the automator and tester
should work together: The tester may
request automated support for things
that are difficult or time consuming, for
example, a comparison or ensuring that
files are in the right place before a test
runs. The automator would then provide utilities or ways to do them. But the
automator, by observing what the tester
is doing, may suggest other things that
could be supported and “sell” additional
tool support to the tester. The rationale
is to make life easier for the tester and
to make the testing faster, thus reducing
elapsed time.
Better automation objective: Reduce
the elapsed time for all tool-supported
Figure 1
Figure 2
www.StickyMinds.com
testing activities. This is an ongoing
objective for automation, seeking to
improve both manual and existing automated testing. It could be measured by
elapsed time for specified testing activities, such as maintenance time or failure
analysis time.
OBJECTIVE 5: RUN MORE TESTS
Good ideas behind this objective:
better coverage.
must be better.
More is not better! Good testing is
not found in the number of tests run, but
in the value of the tests that are run. In
fact, the fewer tests for the same value,
the better. It is definitely the quality of
the tests that counts, not the quantity.
Automating a lot of poor tests gives you
maintenance overhead with little return.
Automating the best tests (however many
that is) gives you value for the time and
money spent in automating them.
If we do want to run more tests, we
need to be careful when choosing which
additional tests to run. It may be easier
to automate tests for one area of the
software than for another. However, if it
is more valuable to have automated tests
for this second area than the first, then
automating a few of the more difficult
tests is better than automating many of
the easier (and less useful) tests.
A raw count of the number of automated tests is a fairly useless way of
gauging the contribution of automation
to testing. For example, suppose testers
decide there is a particular set of tests
that they would like to automate. The
real value of automation is not that the
tests are automated but the number of
times they are run. It is possible that the
testers make the wrong choice and end
up with a set of automated tests that
they hardly ever use. This is not the fault
of the automation, but of the testers’
choice of which tests to automate.
It is important that automation is
responsive, flexible, and able to automate different tests quickly as needed.
Although we try to plan which tests to
automate and when, we should always
start automating the most important
tests first. Once we are running the tests,
JULY/AUGUST 2009
BETTER SOFTWARE
35