Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide


  1. 1. SQA WEBINAR: Q&A --AUGUST 2007 This is the list of all the questions that Truc To and Don Koloski received during our joint QAZone/SQA Forums and SendWordNow Webinar last week. They were great questions, so thank you very much for asking them! Questions for Truc To, Director of QA, SendWordNow 1. For Truc To: Do you envision a balance between automated testing and manual testing? What disadvantages (if any) do you see emerging from the migration to automated testing? a. Automation has helped us reduce testing time, and increase test case coverage here at SendWordNow. Definitely we are striving to have a balance between manual testing and automated testing (we will never have 100% automation, it’s just impossible). We are at 40% now and we might have 50%-50% on the longer term. The main disadvantage of automated testing is the initial investment of time and QA cycles at the beginning, plus training, to get people up to speed on the automation tools. However, the ROI on this initial investment has been worthy for us. We can test more cases and conditions, testing is performed faster –it can even be done off hours--, and it’s more challenging and less tedious for my team. I strongly recommend exploring the possibility of incorporating automation as part of your regular dev/QA cycles. 2. What about generating negative tests? a. We are a small organization, but we always build negative testing in our Test Plans. If you think about our applications, we have a very interconnected infrastructure, and our applications need to be up and running 24 x 7, 365 days, so it’s really important for us to make sure that we perform thorough boundary analysis –and that we consider all types of scenarios with different combinations of pre-conditions and post-conditions. The way we add this intelligence is by customizing our recorded scripts (the ones that we captured with Empirix’s eTester). We can use VBA to add for example logical conditions to our scripts, or for example we can use data banking to perform data driven testing. Typically recording one script is not enough…it’s the starting point, but then you need to customize your scripts so they can truly cover all your different test cases. 3. Coming to automation how can we decide which test case is good and which one is bad? a. We mainly look at 4 things: i. Result is predictable/Reproducible ii. Manually tedious
  2. 2. iii. Doesn’t change often from release to release iv. Doesn’t require human intervention b. In other words, there is always a cost associated with automation. So you have to think: it is worthy to spend time and energy to automate this test case? If you are only going to run that test once or twice, then it’s definitely not worthy to invest on automation. I always like to spend time at the beginning, while we are planning our testing efforts, determining what will be automated. 4. Are the automated test engineers the same people who perform the manual testing? a. In my team yes, we all perform both manual and automated testing. We recently hired Kamlesh, who came with automation experience, and he has really helped out the rest of the team come up to speed with Empirix tools. I am surprised, really, at the high level of automation that we have been able to achieve in a fairly short amount of time. You know, automation can be a little bit intimidating at the beginning, but once you feel more comfortable with the tools, it can really help a lot. 5. What about a test case which is data dependent-should it be automated? Pls. send the presentation slides a. Yes, a test case that is data dependent can be automated as well. I can tell you that in our case we use something that Empirix calls data-banking capabilities (so for example you can create a script that tests how to log into an application by automatically going through a list of usernames and passwords --You can even define what are correct application behavior for each username and password combination). But it all comes back to…is it worthy for you to invest on automating that particular case? 6. How many people do you have devoted to automation vs manual testing at Send Word Now? a. We are 4 people, and we all participate in automation and manual testing. The people with more automation experience are helping the other QA engineers, so we are all learning together and building automation as a team. 7. How do you avoid the "pestaside paradox" scenario with automation? a. No test script is perfect. We have to continually evaluate our automated and manual tests to determine if they are “good enough” and have we “tested enough.” It is always a compromise between available resources (people and timeline) and the level of coverage, to reach an acceptable comfort level with each production release. i. As our applications evolve, some of our scripts have required extensive revisions. Where time and resources permit, we revise or create additional scripts to deal with new bugs or conditions, uncovered, but not tested by the earlier versions.
  3. 3. 8. How did you determine your coverage? a. At the beginning of each new testing project we plan for it. The same way that we determine how many hours/resources we will need to test, we determine what tests are going to be automated. Again, automation is not free…so in my opinion the 2 main questions to ask yourself are really…1- Can it be automated? 2- Does it makes sense to automate? In other words, is the long-term savings going to offset the short-term cost associated with automation? 9. Do u have any statistics comparing the test effort of SOA product and silo product. a. Please elaborate on what is a silo product. 10. When you said you automate 40% of test cases, how do you select those test cases a. I think that I already answered this question in question #3…. 11. Who will decide on when and where to stop the automation process Is there a danger of "Too Much automation"? that would delay objectives? a. Great question…We share our information across the QA team. So we all participate in this decision and process. I think that if you have done your planning right, there is not a danger of too much automation. In my mind automation is an investment, and as such it requires careful consideration and planning. We like to spend time at the beginning of each development cycle evaluating the need for automation for each test case…So far we haven’t felt that it was too much automation…I guess that we have been planning pretty well so far!  12. When to go for automation... its always difficult to maintain them while development is going on? a. Definitely! That’s like trying to hit a moving target. It’s always a matter of communication and finding out the right balance. For example if a product is not stable, you can’t automate. If a product is going to change a lot from release to release it doesn’t make sense to automate either. If it’s a product that’s entering an end of life cycle, you shouldn’t invest on automation. We like to take an approach of divide and conquer, so it’s easier to maintain our scripts, and they are not that much impacted when development changes one feature. For example, if your test case is 5 steps: logging into the application, select contacts, send a message, check the status and results, then log out. You could create 5 scripts and attach them together as needed. So if there is a change on let’s say, a new status code, then you will only need to change one script, and you can reuse the other 4 scripts, instead of having to perform massive revisions on one large script. 13. How many test cases do u have? a. More than I thought we would at this time! Kidding aside, we have built 200+ scripts that are helping us test hundreds of different test cases.
  4. 4. Questions for Dan Koloski, Director of Strategy, Empirix 14. Does e-Load expert support flex based application? a. e-LoadExpert is our hosted load testing service. It makes use of our e-Load Product. e-LoadExpert can be used to load test most flex based applications. Please contact us with specific questions about your application. 15. How does auto-generation of script takes place using e-Test? a. The short answer is that by instrumenting the descriptor file (WSDL in the case of Web Services), we are able to see a menu of all possible inputs and outputs of the Service Under Test. Using that information, we can generate automated scripts to test and validate the Service response to its expected stimuli. 16. Does e-Test suite supports testing WCF services? a. Empirix has been testing Web Services since the introduction of FirstACT in 2001. So we have a good deal of maturity in the toolset. With .NET 3.0, WCF Web Services finally correspond to the universal SOA specs, and so we test them very well. 17. What is the maximum number of users possible to generate via the load tester? a. We have never reached an absolute maximum. The scalability of a load test environment relates to the size of agent machines, available network bandwidth, the types of scripts involved, and of course the application under test. Many of our largest load tests are executed using our e-LoadExpert hosted load test service, and those have scaled to well over 500,000 concurrent users. 18. How do you secure this open and exposed code upon release? a. Security, Governance and Change Management of SOA applications are a big deal, though a bit outside what we were talking about in the web event. The good news is that the SOA standards for security, such as WS-Security, are becoming more mainstream, which means that there’s a good chance that any third-party Services you want to access are making use of them. That takes the pain out of securing transmissions. In terms of access control, there are SOA repository tools available that allow you to manage access to your published services. 19. Can auto-generation be applied to XML files? a. Yes. We can auto-generate Web Services scripts from WSDL files, raw XML files or by proxy recording. 20. How we perform system testing in SOA based applications? a. System testing is testing the integrated, end-to-end system. It is typically the last phase of functional testing and most of what happens in load testing. e-TEST suite and other black-box QA solutions are optimized for system testing, where the goal is to act like real end-users and measure their experience. In a SOA application
  5. 5. 21. How we perform boundary value testing of SOA based applications? a. The same way you do it for regular applications. Parameterizing your scripts allows the scripts to simulate a wide variety of stimulus, including boundary conditions. 22. How we perform integration testing in SOA based applications? a. In a SOA context, integration testing is much like system testing. Our goal will be to emulate end-user, which will then stimulate the various services and test that the coordinated system works as designed. A piece of good news here is that if developers adhere to the SOA standards, the interoperability of individual services shouldn’t be a problem. Of course, not 23. What is the right benefit of using SOA technology? a. That’s a big question. But the short answer I believe is that it allows for code re- use, application integration and ongoing adaptability better than traditional technologies. 24. What is the procedure of building test cases for SOA based applications how we perform unit testing of the SOA application module? How did you use e-Tester for SOA Testing? a. e-TEST suite’s primary mechanism is script generation. We are posting a recorded example of creating Web Services scripts on QAZone since we received a number of inquiries about it. 25. How we perform load and stress testing in SOA based applications? a. We believe that load and stress testing are extensions of functional testing. As such, load should be created using realistic end-user scenarios that will stimulate and validate the applications exactly as real users will. That’s why e-TEST suite creates scripts that can be used for both functional and load testing, whether it’s a SOA application or not. Once you have realistic scripts, then it’s a matter of methodology. We have published a number of resources on our recommended methodology, which is known as Rapid Bottleneck Identification (or RBI). Visit QAZone (http://qazone.empirix.com) for more information. 26. How will convert simple ERP based solutions into SOA architecture? a. SOA Development is outside the scope of our discussion. There are numerous resources available online about SOA development. 27. What about application of e-TEST for testing API of standalone applications? a. e-TEST suite is primarily used for end-user emulation, though testers who want to test at the API level can use our VB or Java programming interfaces to do so. 28. Is this strictly for web applications testing?
  6. 6. a. e-TEST suite is primarily used for testing Web and SOA applications. Look for our upcoming product releases for Modules that can be used to test other kinds of applications. 29. Is XML msg a main tool for communication between services? a. Yes. The SOAP specification provides transport for XML data through HTTP. 30. Is SOA specific to web services or one can use it for automating Web based or desktop apps? a. Web Services are the most common mechanism for SOA applications because of the ubiquity of the protocols. However, there are many other options which are most appropriate for internal-only business applications. 31. Do you have any recommendations for this specific situation? Is it true the WSDL file will only define the basics of the interfact, such as methods and their parameters. The logic behind the interface which drives the majority of test cases is not visible from the WSDL file. So, How can that test creation be automated through the tool? The big challenge is knowing what specific permutations of data should be driving the interactions with the component. Is there any magic with this tool or is it old fashioned requirements and design analysis to generate the test cases. a. You are correct. The WSDL file explains what the Service does. It does not put it into the context of the broader application. There is no magic way to do that – you are correct that understanding the application requirements and mapping them into test cases is still the most important job. What the WSDL file does is allow you to create the automation very quickly. 32. We are familiar with using FileAid from Compuware to generate the permutations of data that drive our scripts by populating Excell spreadsheets that feed the scripts. Do you have a data generation tool like FileAid that populates your databank? a. We don’t sell a tool for the generation of test data. Most of our customers report generating test data via SQL queries to their databases or by using Excel’s built- in features, though of course tools the one you describe can be used also. The Databank doesn’t really care where the data comes from as long as it’s in text format. 33. Recommedations for handling the situation where the development manager feels that QA interacting with developers will delay their ability to deliver so QA must wait for development to be complete to interact with development a. It’s a bummer to hear about that situation, which is far too common. I recently gave a talk which touched this very subject with Theresa Lanowitz of the analyst firm voke (http://www.vokeinc.com). While I’d love to say that positioning yourselves as “all on the same team” is the way to go it sounds like this person is too cynical for that. So, given that, the most important thing is to communicate to the development manager that getting your team visibility into the upstream
  7. 7. process means the product will ship faster. Many development organizations are goaled on product ship dates and view QA as a barrier to getting product out the door. So to them, shortening the time required for testing should accelerate ship dates. But the only way to shorten test cycles is to get planning done before the application is ready, and the only way to do that is to get visibility upstream. So, it’s in the manager’s best interest to encourage communication, not discourage it. 34. Does Empririx support Infragistics controls? a. Yes. .NET Windows Forms controls such as those from Infraguistics are supported using e-Tester’ .NET Option. 35. I believe this tool helps testing web services. If a back end process controls a web service, is there a possibility to schedule the back end process through this tool? Do you follow any specific Automation Framework? a. Yes. If the back-end service takes stimulation through, say, a command-line interface then you would, as the first step of your script, invoke the command-line command to start the process. Then run the rest of the script. 36. You talked about WSDL, is it the contract layer in the service? Empirix provides some UI for entering strings, does Empirix transfer these strings to objects and pass to service contracts? a. Yes you are correct. The UI is built off of the WSDL, which lists the methods and objects the Service can accept. 37. Does it have ability to extract attribute value from XML response (Either using XPath query or other way) a. Yes, there are multiple ways to do so including XPath. 38. How do you handle schema changes - SOAP, etc.? what version of .NET is their tool built on? a. One of the reasons SOA is still in its infancy is that standards have been slow to emerge. We monitor the standards bodies 39. Could u pls elaborate on e-Test. Can we buy it or get the trial at least? a. e-TEST suite is a black-box QA suite designed for Test Management, Functional Testing and Load Testing of Web and SOA applications. Free evaluation downloads are available at the Empirix Download Center at http://www.empirix.com/dlc . 40. Can Java-based Web Services running under Linux be tested using E-Test Suite? a. Yes. e-TEST suite is agnostic to the Web Services development platform. 41. Question for Empirix: We are currently licensed for eTester and eLoad. Is the web services feature a separate license for the suite. a. Yes. Web Services is an option to the e-TEST suite, licensed separately.
  8. 8. 42. Which scripting language mainly e-Tester use ? a. e-TEST suite is designed to eliminate as much manual scripting/coding as possible. When users do require code, e-TEST suite is extensible in Visual Basic for functional testing and Java for load testing. 43. Does this work with https also? a. Yes. e-TEST suite supports https. 44. For both dan and truc: how mnay test cases do u have? a. It’s different for every customers. We have customers who have tens of thousands of test cases that they manage with e-TEST suite. Others have much fewer. 45. Tried Empirix e-tester and e-load both a couple of years back, but one thing that I was not very much comfortable with was there was no way to edit the script, use script contorl statements (if/for etc...statements). Does empirix provide edit script feature now? a. e-TEST suite is designed to eliminate as much manual scripting/coding as possible in order to make testers more efficient. When users do require code (such as conditional logic), e-TEST suite is extensible in Visual Basic for functional testing and Java for load testing.