SQA WEBINAR: Q&A
This is the list of all the questions that Truc To and Don Koloski received during our joint
QAZone/SQA Forums and SendWordNow Webinar last week. They were great questions,
so thank you very much for asking them!
Questions for Truc To, Director of QA, SendWordNow
1. For Truc To: Do you envision a balance between automated testing and manual
testing? What disadvantages (if any) do you see emerging from the migration to
a. Automation has helped us reduce testing time, and increase test case coverage
here at SendWordNow. Definitely we are striving to have a balance between
manual testing and automated testing (we will never have 100% automation, it’s
just impossible). We are at 40% now and we might have 50%-50% on the longer
term. The main disadvantage of automated testing is the initial investment of time
and QA cycles at the beginning, plus training, to get people up to speed on the
automation tools. However, the ROI on this initial investment has been worthy for
us. We can test more cases and conditions, testing is performed faster –it can
even be done off hours--, and it’s more challenging and less tedious for my team.
I strongly recommend exploring the possibility of incorporating automation as part
of your regular dev/QA cycles.
2. What about generating negative tests?
a. We are a small organization, but we always build negative testing in our Test
Plans. If you think about our applications, we have a very interconnected
infrastructure, and our applications need to be up and running 24 x 7, 365 days,
so it’s really important for us to make sure that we perform thorough boundary
analysis –and that we consider all types of scenarios with different combinations
of pre-conditions and post-conditions. The way we add this intelligence is by
customizing our recorded scripts (the ones that we captured with Empirix’s
eTester). We can use VBA to add for example logical conditions to our scripts, or
for example we can use data banking to perform data driven testing. Typically
recording one script is not enough…it’s the starting point, but then you need to
customize your scripts so they can truly cover all your different test cases.
3. Coming to automation how can we decide which test case is good and which one
a. We mainly look at 4 things:
i. Result is predictable/Reproducible
ii. Manually tedious
iii. Doesn’t change often from release to release
iv. Doesn’t require human intervention
b. In other words, there is always a cost associated with automation. So you have
to think: it is worthy to spend time and energy to automate this test case? If you
are only going to run that test once or twice, then it’s definitely not worthy to
invest on automation. I always like to spend time at the beginning, while we are
planning our testing efforts, determining what will be automated.
4. Are the automated test engineers the same people who perform the manual
a. In my team yes, we all perform both manual and automated testing. We recently
hired Kamlesh, who came with automation experience, and he has really helped
out the rest of the team come up to speed with Empirix tools. I am surprised,
really, at the high level of automation that we have been able to achieve in a
fairly short amount of time. You know, automation can be a little bit intimidating at
the beginning, but once you feel more comfortable with the tools, it can really
help a lot.
5. What about a test case which is data dependent-should it be automated? Pls.
send the presentation slides
a. Yes, a test case that is data dependent can be automated as well. I can tell you
that in our case we use something that Empirix calls data-banking capabilities (so
for example you can create a script that tests how to log into an application by
automatically going through a list of usernames and passwords --You can even
define what are correct application behavior for each username and password
combination). But it all comes back to…is it worthy for you to invest on
automating that particular case?
6. How many people do you have devoted to automation vs manual testing at Send
a. We are 4 people, and we all participate in automation and manual testing. The
people with more automation experience are helping the other QA engineers, so
we are all learning together and building automation as a team.
7. How do you avoid the "pestaside paradox" scenario with automation?
a. No test script is perfect. We have to continually evaluate our automated and
manual tests to determine if they are “good enough” and have we “tested
enough.” It is always a compromise between available resources (people and
timeline) and the level of coverage, to reach an acceptable comfort level with
each production release.
i. As our applications evolve, some of our scripts have required extensive
revisions. Where time and resources permit, we revise or create
additional scripts to deal with new bugs or conditions, uncovered, but not
tested by the earlier versions.
8. How did you determine your coverage?
a. At the beginning of each new testing project we plan for it. The same way that we
determine how many hours/resources we will need to test, we determine what
tests are going to be automated. Again, automation is not free…so in my opinion
the 2 main questions to ask yourself are really…1- Can it be automated? 2- Does
it makes sense to automate? In other words, is the long-term savings going to
offset the short-term cost associated with automation?
9. Do u have any statistics comparing the test effort of SOA product and silo
a. Please elaborate on what is a silo product.
10. When you said you automate 40% of test cases, how do you select those test
a. I think that I already answered this question in question #3….
11. Who will decide on when and where to stop the automation process Is there a
danger of "Too Much automation"? that would delay objectives?
a. Great question…We share our information across the QA team. So we all
participate in this decision and process. I think that if you have done your
planning right, there is not a danger of too much automation. In my mind
automation is an investment, and as such it requires careful consideration and
planning. We like to spend time at the beginning of each development cycle
evaluating the need for automation for each test case…So far we haven’t felt that
it was too much automation…I guess that we have been planning pretty well so
12. When to go for automation... its always difficult to maintain them while
development is going on?
a. Definitely! That’s like trying to hit a moving target. It’s always a matter of
communication and finding out the right balance. For example if a product is not
stable, you can’t automate. If a product is going to change a lot from release to
release it doesn’t make sense to automate either. If it’s a product that’s entering
an end of life cycle, you shouldn’t invest on automation. We like to take an
approach of divide and conquer, so it’s easier to maintain our scripts, and they
are not that much impacted when development changes one feature. For
example, if your test case is 5 steps: logging into the application, select contacts,
send a message, check the status and results, then log out. You could create 5
scripts and attach them together as needed. So if there is a change on let’s say,
a new status code, then you will only need to change one script, and you can
reuse the other 4 scripts, instead of having to perform massive revisions on one
13. How many test cases do u have?
a. More than I thought we would at this time! Kidding aside, we have built 200+
scripts that are helping us test hundreds of different test cases.
Questions for Dan Koloski, Director of Strategy, Empirix
14. Does e-Load expert support flex based application?
a. e-LoadExpert is our hosted load testing service. It makes use of our e-Load
Product. e-LoadExpert can be used to load test most flex based applications.
Please contact us with specific questions about your application.
15. How does auto-generation of script takes place using e-Test?
a. The short answer is that by instrumenting the descriptor file (WSDL in the case of
Web Services), we are able to see a menu of all possible inputs and outputs of
the Service Under Test. Using that information, we can generate automated
scripts to test and validate the Service response to its expected stimuli.
16. Does e-Test suite supports testing WCF services?
a. Empirix has been testing Web Services since the introduction of FirstACT in
2001. So we have a good deal of maturity in the toolset. With .NET 3.0, WCF
Web Services finally correspond to the universal SOA specs, and so we test
them very well.
17. What is the maximum number of users possible to generate via the load tester?
a. We have never reached an absolute maximum. The scalability of a load test
environment relates to the size of agent machines, available network bandwidth,
the types of scripts involved, and of course the application under test. Many of
our largest load tests are executed using our e-LoadExpert hosted load test
service, and those have scaled to well over 500,000 concurrent users.
18. How do you secure this open and exposed code upon release?
a. Security, Governance and Change Management of SOA applications are a big
deal, though a bit outside what we were talking about in the web event. The
good news is that the SOA standards for security, such as WS-Security, are
becoming more mainstream, which means that there’s a good chance that any
third-party Services you want to access are making use of them. That takes the
pain out of securing transmissions. In terms of access control, there are SOA
repository tools available that allow you to manage access to your published
19. Can auto-generation be applied to XML files?
a. Yes. We can auto-generate Web Services scripts from WSDL files, raw XML
files or by proxy recording.
20. How we perform system testing in SOA based applications?
a. System testing is testing the integrated, end-to-end system. It is typically the last
phase of functional testing and most of what happens in load testing. e-TEST
suite and other black-box QA solutions are optimized for system testing, where
the goal is to act like real end-users and measure their experience. In a SOA
21. How we perform boundary value testing of SOA based applications?
a. The same way you do it for regular applications. Parameterizing your scripts
allows the scripts to simulate a wide variety of stimulus, including boundary
22. How we perform integration testing in SOA based applications?
a. In a SOA context, integration testing is much like system testing. Our goal will be
to emulate end-user, which will then stimulate the various services and test that
the coordinated system works as designed. A piece of good news here is that if
developers adhere to the SOA standards, the interoperability of individual
services shouldn’t be a problem. Of course, not
23. What is the right benefit of using SOA technology?
a. That’s a big question. But the short answer I believe is that it allows for code re-
use, application integration and ongoing adaptability better than traditional
24. What is the procedure of building test cases for SOA based applications how we
perform unit testing of the SOA application module? How did you use e-Tester for
a. e-TEST suite’s primary mechanism is script generation. We are posting a
recorded example of creating Web Services scripts on QAZone since we
received a number of inquiries about it.
25. How we perform load and stress testing in SOA based applications?
a. We believe that load and stress testing are extensions of functional testing. As
such, load should be created using realistic end-user scenarios that will stimulate
and validate the applications exactly as real users will. That’s why e-TEST suite
creates scripts that can be used for both functional and load testing, whether it’s
a SOA application or not. Once you have realistic scripts, then it’s a matter of
methodology. We have published a number of resources on our recommended
methodology, which is known as Rapid Bottleneck Identification (or RBI). Visit
QAZone (http://qazone.empirix.com) for more information.
26. How will convert simple ERP based solutions into SOA architecture?
a. SOA Development is outside the scope of our discussion. There are numerous
resources available online about SOA development.
27. What about application of e-TEST for testing API of standalone applications?
a. e-TEST suite is primarily used for end-user emulation, though testers who want
to test at the API level can use our VB or Java programming interfaces to do so.
28. Is this strictly for web applications testing?
a. e-TEST suite is primarily used for testing Web and SOA applications. Look for
our upcoming product releases for Modules that can be used to test other kinds
29. Is XML msg a main tool for communication between services?
a. Yes. The SOAP specification provides transport for XML data through HTTP.
30. Is SOA specific to web services or one can use it for automating Web based or
a. Web Services are the most common mechanism for SOA applications because
of the ubiquity of the protocols. However, there are many other options which
are most appropriate for internal-only business applications.
31. Do you have any recommendations for this specific situation? Is it true the WSDL
file will only define the basics of the interfact, such as methods and their
parameters. The logic behind the interface which drives the majority of test cases
is not visible from the WSDL file. So, How can that test creation be automated
through the tool? The big challenge is knowing what specific permutations of data
should be driving the interactions with the component. Is there any magic with
this tool or is it old fashioned requirements and design analysis to generate the
a. You are correct. The WSDL file explains what the Service does. It does not put
it into the context of the broader application. There is no magic way to do that –
you are correct that understanding the application requirements and mapping
them into test cases is still the most important job. What the WSDL file does is
allow you to create the automation very quickly.
32. We are familiar with using FileAid from Compuware to generate the permutations
of data that drive our scripts by populating Excell spreadsheets that feed the
scripts. Do you have a data generation tool like FileAid that populates your
a. We don’t sell a tool for the generation of test data. Most of our customers report
generating test data via SQL queries to their databases or by using Excel’s built-
in features, though of course tools the one you describe can be used also. The
Databank doesn’t really care where the data comes from as long as it’s in text
33. Recommedations for handling the situation where the development manager feels
that QA interacting with developers will delay their ability to deliver so QA must
wait for development to be complete to interact with development
a. It’s a bummer to hear about that situation, which is far too common. I recently
gave a talk which touched this very subject with Theresa Lanowitz of the analyst
firm voke (http://www.vokeinc.com). While I’d love to say that positioning
yourselves as “all on the same team” is the way to go it sounds like this person is
too cynical for that. So, given that, the most important thing is to communicate to
the development manager that getting your team visibility into the upstream
process means the product will ship faster. Many development organizations are
goaled on product ship dates and view QA as a barrier to getting product out the
door. So to them, shortening the time required for testing should accelerate ship
dates. But the only way to shorten test cycles is to get planning done before the
application is ready, and the only way to do that is to get visibility upstream. So,
it’s in the manager’s best interest to encourage communication, not discourage it.
34. Does Empririx support Infragistics controls?
a. Yes. .NET Windows Forms controls such as those from Infraguistics are
supported using e-Tester’ .NET Option.
35. I believe this tool helps testing web services. If a back end process controls a web
service, is there a possibility to schedule the back end process through this tool?
Do you follow any specific Automation Framework?
a. Yes. If the back-end service takes stimulation through, say, a command-line
interface then you would, as the first step of your script, invoke the command-line
command to start the process. Then run the rest of the script.
36. You talked about WSDL, is it the contract layer in the service? Empirix provides
some UI for entering strings, does Empirix transfer these strings to objects and
pass to service contracts?
a. Yes you are correct. The UI is built off of the WSDL, which lists the methods and
objects the Service can accept.
37. Does it have ability to extract attribute value from XML response (Either using
XPath query or other way)
a. Yes, there are multiple ways to do so including XPath.
38. How do you handle schema changes - SOAP, etc.? what version of .NET is their
tool built on?
a. One of the reasons SOA is still in its infancy is that standards have been slow to
emerge. We monitor the standards bodies
39. Could u pls elaborate on e-Test. Can we buy it or get the trial at least?
a. e-TEST suite is a black-box QA suite designed for Test Management, Functional
Testing and Load Testing of Web and SOA applications. Free evaluation
downloads are available at the Empirix Download Center at
40. Can Java-based Web Services running under Linux be tested using E-Test Suite?
a. Yes. e-TEST suite is agnostic to the Web Services development platform.
41. Question for Empirix: We are currently licensed for eTester and eLoad. Is the web
services feature a separate license for the suite.
a. Yes. Web Services is an option to the e-TEST suite, licensed separately.
42. Which scripting language mainly e-Tester use ?
a. e-TEST suite is designed to eliminate as much manual scripting/coding as
possible. When users do require code, e-TEST suite is extensible in Visual
Basic for functional testing and Java for load testing.
43. Does this work with https also?
a. Yes. e-TEST suite supports https.
44. For both dan and truc: how mnay test cases do u have?
a. It’s different for every customers. We have customers who have tens of
thousands of test cases that they manage with e-TEST suite. Others have much
45. Tried Empirix e-tester and e-load both a couple of years back, but one thing that I
was not very much comfortable with was there was no way to edit the script, use
script contorl statements (if/for etc...statements). Does empirix provide edit script
a. e-TEST suite is designed to eliminate as much manual scripting/coding as
possible in order to make testers more efficient. When users do require code
(such as conditional logic), e-TEST suite is extensible in Visual Basic for
functional testing and Java for load testing.