Top Ten Tips for Tackling Test Automation Webinar Presentation.pptx
•Download as PPTX, PDF•
0 likes•52 views
Inflectra and Checkpoint Technologies co-hosted the webinar: Top Ten Tips for Tackling Test Automation. In this webinar, Adam Sandman (Inflectra) and Bob Crews (Checkpoint Technologies) explored the challenges surrounding test automation and offered their tips on overcoming them. Find the recording of the Webinar on our YouTube channel: https://www.youtube.com/watch?v=vY1MbW4qWnQ Webinar Agenda: -Top 10 challenges of test automation with impact and solutions -Impacts: potential risks if challenges are not overcome -Solutions: tips to overcoming the challenges -Automated functional testing -Criteria of an Automation Assessment -Addressing several challenges with Inflectra's Spira and Rapise Webinar Presenters: Adam Sandman is the Founder and CEO of Inflectra. He has been working in the IT industry for the past 25+ years. His areas of expertise span software architecture to agile development, software testing, test automation, and project management. He is interested in technology, business, and enabling people to follow their passions. At Inflectra, Adam is responsible for researching the tools, technologies, and processes in the software testing and quality assurance space. Adam is a prolific speaker whose speaking engagements range from StarEast, and Eurostar to STPcons, DevGeekWeek, Swiss Testing Day, NDIA, STARCanada, TestingMind, Agile DevOps West, StarWest, testCon, JFTL, and many more. Bob Crews, Co-Founder and CEO of Checkpoint Technologies, is a consultant with 34 years of IT experience in full life-cycle development and software testing. Bob and his organization provide services and solutions focused on QA with a concentration in functional, performance and application security testing. He’s assisted organizations such as Harvard University, Raymond James, the FBI, and the Department of Veterans Affairs in developing teams, processes, and solutions to help organizations deliver higher quality software faster. He’s consulted for over 290 organizations on QA, effective software testing, strategic test planning, enhanced test automation, and risk-based testing. He’s exceptionally enthusiastic about the future of IT and software testing and believes “The best is yet ahead!”
Recommended
Recommended
More Related Content
Similar to Top Ten Tips for Tackling Test Automation Webinar Presentation.pptx
Similar to Top Ten Tips for Tackling Test Automation Webinar Presentation.pptx (20)
More from Inflectra
More from Inflectra (20)
Recently uploaded
Recently uploaded (20)
Top Ten Tips for Tackling Test Automation Webinar Presentation.pptx
- 1. TOP Ten Tips for Tackling Test Automation Webinar October 11th, 2022
- 2. Welcome! Adam Sandman Inflectra CEO / Founder Presenter Bob Crews Checkpoint Technologies CEO / Co-Founder Presenter
- 3. GoToWebinar Instructions Toggle View Window Between Full Screen / Window Mode Questions Can Be Inserted Here 3
- 4. What we will cover… Top 10 challenges with Impact and Solutions Impacts - potential risks if challenges are not overcome Solutions – Tips to overcome challenges Focus will be on automated functional testing Criteria of an Automation Assessment 4
- 5. What we will not cover… • Technical challenges specific to specific automated tools. Each solution has its own limitations. • Challenges pertaining to a programming language • Issues with hardware, security, networking, etc. • Performance nor application security testing 5
- 6. Challenge 1: Utilizing full benefits of data-driving The challenge for most organizations is getting the team to take full advantage of the benefits of data-driving. Description: Storing data relevant to a test case in an external file. A set of data makes up a “record” and, typically, a test is executed once for each record within the file. 6
- 7. Impacts Failure to realize maximum ROI Greater challenges testing multiple conditions Increased maintenance time 7
- 8. Solutions Train and educate your team Implement guidelines & standards Enforce guidelines & standards (peer reviews) Implement strategy for test data creation and management 8
- 9. Challenge 2: Effectively implementing an efficient framework Description: In this challenge focus on “effectively implementing” and “efficient framework”. Organizations lean towards two extremes: No framework at all An overly complex framework that never seems to be ready 9
- 10. Impacts No framework • More difficult to create system test strategy • Reduces ROI by not creating and taking advantage of reusable, modular components • Increases learning curve by not creating standards & guidelines Overly complex framework • Increases learning curve • Slows down use of automation • Decreases ROI Insisting framework 100% complete before executing • Decreases ROI • Delays realization of automation benefits 10
- 11. Solutions Let the framework evolve 1. Start the creation of “Automation Standards & Guidelines” 2. Perform analysis and select tests that are best candidates for automation 3. Automate those tests (and execute). 4. As automation occurs take note of common processes 5. Create modular, reusable common processes 6. Integrate common, automated processes with other automated tests 7. Continue building framework – it’s an on-going process Monitor the creation and evolution of the framework 11
- 12. Challenge 3: Committing to the investment required to realize an ROI Description: Once implemented, it takes effort, time, and money to realize (and continue to realize) a return-on-investment with automation. A commitment is needed: • At the company (managerial) level • At the user level 12
- 13. Impacts With no commitment • Education on the tool, and thus utilization, will suffer • ROI not realized or, worse, a loss will be realized • Automation tool will become “shelfware” • Highest quality (of applications and systems) will not be realized • Frustration will occur 13
- 14. Solutions Understand the commitment needed early: • Time • Money • Resources Understand Be supportive of team and automation initiative Be Ensure team remains open-minded Ensure Provide training, mentoring, and on-going education Provide Set guidelines and standards for test automation Set Involve team in decisions Involve 14
- 15. Challenge 4: Knowing what processes to automate and what to leave manual Description: It’s important to know, and heed, the criteria as to when to automate a test and when to leave manual. 15
- 16. Impacts Automating tests that should not be automated NOT automating tests that should be automated Don’t fall victim to: • Red Durango Mentality (Automate everything!) • Analysis Paralysis (Automate nothing!) • Framework Fumble (Slow to automate anything) 16
- 17. Solutions Tip: Set strict guidelines and conduct peer reviews to check for compliance. What to Automate • Tests that validate requirements/features that will exist for multiple versions to come • Tedious, error-prone tests • Manually intensive • Tests that validate numerous conditions with same actions but different data values (ex: field edits) What to Leave Manual • Tests that validate requirements/features soon to be obsolete • Exploratory tests • Tests for which expected results are unknown • Tests for ad-hoc and emergency processes • Usability testing (subjective) 17
- 18. Challenge 5: Planning and implementing an automation strategy Automated testing: Requires different skills Has different considerations Provides different benefits Should be a strategy within a strategy Description: Automation must be planned and well thought out. It presents a unique set of challenges and requires a different approach. It should NOT be treated as manual testing. 18
- 19. Impacts Automation may grow out-of-control and become the primary testing focus Automation may be “set aside” and not be fully utilized Approached with same strategy as manual testing Decreases benefit of re-usability Redundant tests Excessive number of tests Increase in development/mainte nance time 19
- 20. Solutions Understand & accept that automated testing strategy is very different from manual testing strategy Consider automation to be a PART of your overall testing strategy Early in planning stage have Automation Assessment performed 20
- 21. Automation Assessment Criteria Environment Overview Automated Tool Experience and Training Evaluation Test Automation Process Documentation Evaluation Test Automation Environment Evaluation Automated Test Design Process Evaluation Automated Test Execution Process & Metrics Evaluation 21
- 22. Challenge 6: Developing a versatile, skillful QA team with automation skills • Description: In order to successfully implement and utilize automation an organization must have, or develop, a skillful automation team. • Skills must include: • Utilization of the chosen automated testing solution • Implementation of an automation strategy 22
- 23. Impacts Decrease in realization of ROI Not realizing high level of application quality that can be obtained with better testing Automation tool becomes “shelfware” Longer automation implementation period 23
- 24. Solutions • Assess the automation skill level of your existing team early in the process • Identify those team members that are, or can quickly evolve, into automation leads and/or administrators • Know your options regarding: • Improving the skills of your existing team • Contracting experienced testers • Monitor the skill level of your team to ensure that their skills and knowledge are improving 24
- 25. Composition of Automation Team Always have a Lead and/or Administrator Best teams have mixture of skills What percentage of team should be involved? Depends on size and skill set of QA/QC (software testing team) How technical are your team members? Are you required to “work with what you have” or can you expand the team? 25
- 26. Basic Skill Assessment - Exercise • Technical: • Do you consider yourself to be technical? • What is a “loop”? • Name one conditional statement • What is the difference between a “constant” and a “variable”? • What is an “array”? • Methodology: • Can you describe an automation strategy? • Describe a test that should NOT be automated • Is it better to define expected results within a script or a data-table? • What data in a script would you not parameterize? • What are the benefits of modularization? 26
- 27. Level of Skill 10 9 8 7 6 5 4 3 2 1 1 2 3 4 5 6 7 8 9 10 T E C H N I C A L S K I L L AUTOMATION METHODOLOGY KNOWLEDGE Automation Lead Automation Technical Advisor Automation Tester Automation Administrator 27
- 28. Challenge 7: Executing automated tests unattended and remotely Description: Most automated solutions offer capabilities to schedule tests to run at a specific date/time on remote network machines Many users fail to utilize this feature. Why? • Lack of confidence in automation • Lack of secure environment • Conflict with other processes (i.e., backups) 28
- 29. Impacts Won’t realize as significant an ROI with automation Will have greater challenges testing on different machines with different setups (software, OS, etc.) Can’t create automated tests on local machine while running tests 29
- 30. Solutions Set up Set up a secure, dedicated testing environment with network access Own Always own at least two licenses of automated solution Ensure Ensure team has enough knowledge and skill (with tool) to instill confidence in automation 30
- 31. Challenge 8: Calculation of ROI Description: How can management calculate the return- on-investment to justify costs of automation? •Savings in man-hours •Time for tester to perform other tasks •Greater coverage Direct ROI •Risk avoidance •Decrease in errors •Increase in defect reported Indirect ROI 31
- 32. Impacts If ROI is not calculated or if it’s incorrectly calculated… • Management may abandon automation or • Too much might be automated • Incorrect decisions may be made 32
- 33. Solutions • Report and track relevant data prior to, during, & after automation effort for manual vs. automated testing • Cost of resources • Time to execute manually and automated • Time to develop and maintain automated tests • Number of conditions validated • Number of defects reported • Track “Cost of Failure” data 33
- 34. Cost of Quality Preventive Costs - Costs associated with preventing errors Training Establishing methods and procedures Tool acquisition Appraisal Costs - Costs associated with the detection of errors Inspections Testing Failure Costs - Costs associated with defective products delivered to customers Repairs Staffing Help Desk Damage caused by defect 34
- 35. Challenge 9: Integration of automated tests to create a full system test Description: Tests need to be integrated to validate the system. Tests, standalone, should validate specific, granular requirements. Integrated they should validate the “system”. Biggest challenge typically seems to be test data management. 35
- 36. Impacts • System requirements fail to get validated for those processes up and/or down stream • Fail to maximize ROI (return-on-investment) 36
- 37. Solutions • Define a Test Automation Administrator role who will track progress relative to types of testing – Unit/Component, Integration, System, and Regression • Define a test data lead to handle the generation and management of test data (especially for system testing) • Develop a strategic framework to ease the effort of implementing a fully automated, end-to-end system test 37
- 38. Challenge 10: Tendency to focus on scripting (programming) features Description: Focus on programming and fail to take advantage of tool features. • Recording features, packaged functions, etc. • People with strong development experience tend to be most guilty 38
- 39. Impacts Increased automation script development and maintenance time Scripts become more difficult to maintain More reliant on the person (programmer) that created automated script(s) 39
- 40. Solutions Train all users on automation and automation tool Require all users to learn features of automation tool Conduct peer reviews 40
- 41. Top Ten Challenges in Review 1. Utilizing the full benefits of data-driving automated tests 2. Effectively implementing an efficient framework 3. Committing to the investment required to realize an ROI 4. Knowing what processes to automate and what to leave manual 5. Planning and implementing an automation strategy 6. Developing a versatile, skillful QA team with automation skills 7. Executing automated tests unattended and remotely 8. Calculation of ROI 9. Integration of automated tests to create a full system test 10. Tendency to focus on scripting (programming) features 41
- 42. Open Discussion • Q & A • Thoughts • Experiences 42
- 43. 43 CONFIDENTIAL About Incorporated in January 2003 QA and QC expertise focused on functional, performance and application security testing Proud partners with Inflectra, Keysight (Eggplant), Tricentis, and Kobiton, Micro Focus Services Partner with Absolute Cyber Defense Authorized ASTQB Training Provider
- 44. About Inflectra End to End Capabilities Program Management IT Service Management Process Automation Exploratory Testing Integration Services Agile & DevOps Requirements Management Quality Assurance Risk Management Software Development Project Management Test Automation • 80,000 users in over 5,000 companies worldwide • We Focus on our customers – not pleasing investors
- 45. 45 CONFIDENTIAL THANKS FOR JOINING US! Bob Crews CEO and Co-founder bcrews@checkpointech.com 813.445.7077 LinkedIn: linkedin.com/in/bob-crews-checkpointech Twitter: @BobCrews_CPTech Adam Sandman CEO and Founder Adam.sandman@inflectra.com LinkedIn: linkedin.com/in/adamsandman/ Twitter: @adammarksandman Corporate LinkedIn LinkedIn: linkedin.com/company/checkpoint-technologies
Editor's Notes
- Bob: Hello everyone, my name is Bob Crews. I’m the CEO and co-founder of Checkpoint Technologies. I’m very excited to be moderating todays webinar. With me today is our presenter, Clyde Ford. Clyde is one of the senior leaders on my team, who has over 31 years of experience working in the IT field. I’ll turn it over to Clyde…
- Bob: Here’s an image of the Go To Webinar control panel that usually appears in the upper right-hand corner of your screen. To submit a question, make sure the Questions Pane is expanded and type in your question and click on send. We’re going to do our best to answer all questions today. We’ll email all questions with answers to registrants within the next week or so. And by the way, today’s webinar is being recorded. Just email us to receive a copy of the recording. So let’s get started with the presentation. Take it away Clyde!
- Challenges are based on author’s 22 years in IT and his consulting and training to over 250 organizations on field of Test Automation. Most challenges fall into categories of: Technical People Methodology (process)
- The team might not be taking full advantage of data-driven tests due to: Lack of knowledge and training Lack of confidence in automation or automation tool Challenges with test data management Since automation enables you to test a significantly greater number of conditions it stands to reason that more test data is required. The creation and management of test data can be a burdensome task by itself.
- Remember, these are impacts if you don’t overcome the challenge. Storing test data in data tables, especially expected results, can significantly decrease maintenance time. If an expected result changes, you simply modify the value in the data table.
- Most would agree that an automation framework is essential to fully realize the benefits of automation and maximize the ROI. The challenge comes with “effective implementation” of a framework that is “efficient”.
- Don’t wait until framework is complete before you start executing automated tests. As you automate tests ensure that each test provides value independently and can be executed standalone.
- The company (management) must be committed to: Accepting that expense is required to educate users Understanding that a learning-curve is inherent with automation Supporting the effort to collect and track metrics so that the cost can be justified Enforcing the use of test automation (lest it become “shelf-ware”) Accepting that there will be mistakes and that mistakes are viewed as an opportunity to learn The user must be committed to: Using the automation tool Maintaining a positive attitude and remaining open-minded in the use of automation Learning as much as they can about automation and viewing their mistakes as an opportunity to learn (without getting frustrated) Accepting the fact that not everything can be nor should be automated
- The investment required to implement and maintain automation should be known upfront before the effort begins. While the cost can be significant, the ROI will be even greater when automation is used correctly. Not only will the ROI be in time saved, it will also be in terms of risk avoidance and benefits realized by a higher quality product be delivered to market sooner. Education in the automation tool AND automation methodology is critical. But it must be remembered that it is a two way street. Management should be willing to provide education and users must be open to seeking our education opportunities. There are numerous webinars, seminars, user groups, and conferences as well as residing collateral on the subject of automation. The best way to improve your skills with automation is by using it!
- Red Durango Mentality: Understand the benefits and the limitations of automation. Don’t let the excitement of getting a “new toy” drive you into automating more than you should. Analysis Paralysis: Make it a priority to quickly identify 10 test cases that are good candidates for automation and automate them! Framework Fumble: Building a solid, robust framework has several advantages. It helps: increase ROI, set standards, ease reusability, increase coverage, and decrease script development & maintenance time. However, do NOT make the mistake of insisting that a framework be 100% completely in place before realizing the benefits of automation. A solid framework takes several months to create and, even then, it is a constant work-in-process. There’s no reason any organization cannot realize an ROI from test automation before the framework is completed. Take the time to analyze your test cases on a granular level. (Look at the ROI of the one automated test case presented with the ROI calculation challenge.)
- Think of common requirements for a login process in which the actions to validate are: Invoke application Enter username Enter password Click OK button Check for expected result of successful login or error message The requirements for the login process might be: Valid user name and valid password results in successful login Valid user name and blank password results in error message “Password blank. Please enter password.” being displayed Blank user name and valid password results in error message “User name blank. Please enter user name.” being displayed Valid user name and invalid password results in error message “Password is not valid. Please enter valid password.” being displayed Invalid user name and valid password results in error message “User name is not valid. Please enter valid user name.” being displayed Perfect candidate for automation. All 5 requirements can be validates with one automated, data-driven test.
- Often there’s no strategy. Everybody goes their own way and does their own thing. They approach automated testing based on what they know – which is manual testing.
- In a solid QA and QC process automation should NEVER become the primary focus. It should be a part of the process. One of many different approaches to include: A strong verification strategy (peer reviews and walk-throughs) Well planned manual testing Strategic exploratory testing Extensive user-acceptance testing Robust system testing When automated testing is attacked the same as manual testing certain, negative developments tend to occur. Users end up taking too much ownership of their automated tests and don’t share thus decreasing benefit of “re-usability” and causing the creation of redundant, automated tests. This leads to an increase in script development and maintenance time.
- In my experience I have found that if you can automate 60% to 70% of your test cases you’re doing great. As much as I am a proponent of test automation I am also a big fan of verification (peer reviews and walk-through), exploratory testing, manual testing, etc. Often, when you listen to testing experts that rally against automation, you will find that they do so as though it is being suggested that you automate 100% of your testing efforts and abandon other testing strategies (such as exploratory testing).
- Environment Overview Applications Software Testing Team Automation Tools Automated Tool Experience and Training Evaluation Test Automation Process Documentation Evaluation Test Automation Environment Evaluation Automated Test Design Process Evaluation Integration of Automated Testers, Manual Testers, and the Business Overview of Existing Automation Codebase Evaluation of General Automated Script Code Management Automated Test Execution Process and Metrics Evaluation Process Evaluation Execution Metrics Automated vs. Manual Defect Catch Rate Successful vs. Non-Successful Test Runs
- By “develop” I mean either: Training and educating your team to increase their skills to the required level and/or Contracting expertise on a temporary basis. If you choose to contract expertise it is important that you ensure that knowledge transference is taking place. Use the contracted experts to not only implement an automation plan but also to mentor your team,
- Ideal situation is for an organization to assess the automation skill level of the team before an automation tool has been chosen. Let the team be involved in the evaluation and proof-of-concept. Use a collaborative approach in selecting the tool.
- In deciding the percentage of team members that are involved in the creation of automated scripts first assess each members skill level. If each and every member has strong automation skills take advantage of that by letting them create automated scripts. However, it is critical that a “lead” or “administrator” be designated.
- Give yourself 2 points for each correct answer. Total the points for the “Technical” category and the “Methodology” category. These are just a few of the questions one might ask in evaluating the skill set of an individual. It is certainly not meant to be all-inclusive. I would highly recommend that any assessment be performed with an individual via a verbal conversation rather than written. This would give the individual an opportunity to explain their answers which may be justified and have merit.
- Plot the points from the exercise for your “Technical” score and your “Methodology” score.
- Keep in mind that if you wish to execute automated tests remotely the remote machine must have access to the: Application being tested Test data Automated solution (so that it can run on that machine)
- Only executing automated tests locally (on the machine in front of you) means that: You then cannot use that machine to perform other tasks for 5 hours 20 automated tests, each taking 15 minutes to run, will consume 5 hours of real time on that machine If you’re under a time constraint it will still take, real-time, 5 hours If you take advantage of remote execution then you can distribute the execution among several machines. With the 20 automated tests distributed among 10 remote machines, with 2 automated tests running on each of the 10 (and none running on your local machine): You can use your local machine for other tasks The 20 automated tests, each taking 15 minutes to run, will consume 30 minutes real time on each of the remote machines If you’re under a time constraint it will still take, real-time, 30 minutes If you also take advantage of scheduling the tests to run during “off” hours then, in regards to your time, it will take you 5 – 10 minutes to schedule the tests on the remote machines. All the remote machines can then be used during the normal work day. The test will run during the night and you can then perform the necessary analysis the next day when you start your work day.
- Only executing automated tests locally (on the machine in front of you) means that: You then cannot use that machine to perform other tasks for 5 hours 20 automated tests, each taking 15 minutes to run, will consume 5 hours of real time on that machine If you’re under a time constraint it will still take, real-time, 5 hours If you take advantage of remote execution then you can distribute the execution among several machines. With the 20 automated tests distributed among 10 remote machines, with 2 automated tests running on each of the 10 (and none running on your local machine): You can use your local machine for other tasks The 20 automated tests, each taking 15 minutes to run, will consume 30 minutes real time on each of the remote machines If you’re under a time constraint it will still take, real-time, 30 minutes If you also take advantage of scheduling the tests to run during “off” hours then, in regards to your time, it will take you 5 – 10 minutes to schedule the tests on the remote machines. All the remote machines can then be used during the normal work day. The test will run during the night and you can then perform the necessary analysis the next day when you start your work day.
- Direct ROI is much easier to calculate. It requires that you have relevant data such as: Cost of resources (hourly cost including benefits) Time it takes to manually test process(es) Time it takes to automate process(es) Average number of conditions tested manually Average number of conditions tested when automated
- From QAI’s “Boot Camp for Software Testers” course
- This integration does not need to occur immediately. As stated, each individual, standalone automated test should validate requirements and provide a return-on-investment. Refer to information in this presentation regarding the development of a “framework” (Challenge #2).
- Requirements typically exist to validate data that originated in a previous process or that impacts a subsequent process.
- By development I mean programming – especially if its experience programming in the same language as the automation tool. Many go to great lengths to avoid recording. Recording is a great way to create the initial, baseline script. Record the navigation and initial data entry and THEN enhance as necessary using tool features and, yes, programming.
- Impacts Many automation tools have features that are intended to save from development time. Not using those features increases time to develop and/or maintain script. Scripts become more difficult to maintain especially for those users with less programming experience. Sad but true: Users with extensive programming experience sometimes let their ego get in the way. As a result they may focus on the programming aspects, knowingly creating more complex automated scripts, thus ensuring their position and the team’s need for them (job security). Solutions Users with programming experience tend to designated “advanced users” by default. Require ALL users to get trained on the tool. Don’t trust them to educate themselves. People will go down the “path of least resistance” and stick with what they know (in this case – programming)
- Impacts Many automation tools have features that are intended to save from development time. Not using those features increases time to develop and/or maintain script. Scripts become more difficult to maintain especially for those users with less programming experience. Sad but true: Users with extensive programming experience sometimes let their ego get in the way. As a result they may focus on the programming aspects, knowingly creating more complex automated scripts, thus ensuring their position and the team’s need for them (job security). Solutions Users with programming experience tend to designated “advanced users” by default. Require ALL users to get trained on the tool. Don’t trust them to educate themselves. People will go down the “path of least resistance” and stick with what they know (in this case – programming)
- Bob: Just a little about Checkpoint Technologies. We’re a solutions provider with a focus in Quality Assurance and software testing. We offer software solutions used for functional, performance, and applications security testing and partner with leading organizations such as Micro Focus, Tricentis, Mobile Labs, and Kobiton. We have vast expertise in the Atlassian solutions such as Jira and Confluence. We also have expertise with SAP solutions such as Solution Manager, Focus Build and Focus Run. Our services - with these solutions as well as all areas of QA and software testing – regardless of whether you’re DevOps, Agile, Activate, Waterfall, or somewhere in-between, include…<next slide>
- Bob: Thank you all again for joining us today! Remember, todays’ webinar has been recorded so just email us to request a copy of the recording. And we’ll be responding to all questions by emailing all those who registered a Q&A file. Thanks and stay safe! <Then I’ll come in and thank everyone as well and end the webinar>