Whats In Your QA Tool Belt?


Published on

Presentation to Dallas Metro-SQA Group Oct, 2005

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Title slide
  • Portals added/modified Hardware and Software changes affected unexpected areas in Production Production changes were not deployed the same way they were in Test. Services were down after changes, “Monday AM triage” Customer complaints Potential violation of SLA terms VP said “There will be testing”; and so there was…
  • Whats In Your QA Tool Belt?

    1. 1. What’s In Your QA Tool Belt? Dallas Metro-SQA Presentation Walter Mamed Oct 12, 2005
    2. 2. <ul><li>Problem #1 – Branded Portal Testing and Migration </li></ul><ul><ul><li>Off-The-Shelf (OTS) automation software & demo </li></ul></ul><ul><li>Problem #2 – “Sunday Sanity” Manual Test Effort </li></ul><ul><ul><li>In-House Built (IHB) automation software & demo </li></ul></ul><ul><li>Problem #3 – Expanding on #1 & #2 </li></ul><ul><ul><li>Open Source S/W (OSS) automation software & demo </li></ul></ul><ul><li>Summary/Benefits </li></ul><ul><li>Q & A </li></ul><ul><li>Networking @12:30 </li></ul>Agenda
    3. 3. <ul><li>Branded Portals Background </li></ul><ul><li>ZixPort base provides configurable, semi-customized brands </li></ul><ul><li>Includes portal branding, images, links, and functionality </li></ul><ul><li>New Customer Portals </li></ul><ul><li>Customer modifications to existing Production Portals </li></ul><ul><li>Portal migration from .net49 to ZixPort 2.0 code base (2 nd Qtr 2004) </li></ul><ul><li>Branded Portal Manual Testing Effort Burden </li></ul><ul><li>Effort: 1 FTE & 1 QA full time, continuously each week </li></ul><ul><li>Frequency: Portals added and modified weekly </li></ul><ul><li>Limitation: QA could only handle max of 6 portals a week </li></ul>Problem #1 Portal Testing
    4. 4. <ul><li>New Branded Portal Automation Requirements </li></ul><ul><li>Handle sales prediction of 20 new portals per week </li></ul><ul><li>Portals added and modified weekly </li></ul><ul><li>Test upgrade of all 80+ portals from .net49 and ZixPort2.0; estimated at 10 weeks manual testing effort, and no new rollouts for 10 weeks during testing </li></ul><ul><li>Goal: Test future ZixPort releases in days; not weeks. </li></ul><ul><li>Tasks Identified to Meet Automation Goals </li></ul><ul><li>Solution required coordination among several departments </li></ul><ul><li>Development created QAPOP.dll to retrieve (pop) email </li></ul><ul><li>‘ Client Services’ created Excel based client worksheets which included conversions to WinRunner data-table format. </li></ul><ul><li>QA developed WinRunner script driven from a data-table. </li></ul>QA Automation Requirements
    5. 5. <ul><li>Branded Portal Automation using WinRunner </li></ul><ul><li>Fast, accurate portal verification for new portals </li></ul><ul><li>Ability to add a greater number of new portals each week </li></ul><ul><li>Verifies logos, bitmaps, images, custom text </li></ul><ul><li>Metrics: 1 Full time person accomplished in 1/10 time for continuous weekly rollouts </li></ul><ul><li>Metrics: 10 week portal upgrade testing effort accomplished in 8 hours!! </li></ul>Solution #1: WinRunner (OTS)
    6. 6. <ul><li>Branded Portal Automation Demo </li></ul><ul><li>WinRunner (Lite Version – Not “Full Monty”) </li></ul>Solution #1: WinRunner (OTS)
    7. 7. QA Manual Test Effort <ul><li>Verify new features pushed to Production </li></ul><ul><li>Regression test existing Production functionality after changes </li></ul><ul><li>Effort: Medium number of tedious test cases across many portals </li></ul><ul><ul><li>Six QA people needed for each Sanity run @ ~1.5 hrs each </li></ul></ul><ul><li>Frequency: Any time changes are made in production </li></ul><ul><ul><li>Hardware, Software, Network, ACL, etc... </li></ul></ul><ul><ul><li>Planned changes each weekend (Sunday Maintenance) </li></ul></ul><ul><ul><li>Hotfixes during week, after 5pm </li></ul></ul><ul><ul><li>As business grew, “Sunday Sanity” was more frequent </li></ul></ul><ul><li>Limitation: Large after hours and weekend effort </li></ul><ul><ul><li>Same tests run over and over </li></ul></ul><ul><ul><li>No consistent logging of dates, times, results, etc. </li></ul></ul>Problem #2 “Sunday Sanity”
    8. 8. <ul><li>With the resource drain, frequency, off hours, volume and limitations… This effort screamed </li></ul><ul><li>“ AUTOMATION!!!” </li></ul>This QA Testing Burden…
    9. 9. <ul><li>WinRunner (Obvious Solution to #2) </li></ul><ul><li>Loses focus in Browser (WinRunner version 7.01) </li></ul><ul><li>Couldn’t attach to apps it started (invoke_application) </li></ul><ul><li>Therefore, not reliable for off-hour testing </li></ul><ul><li>Perl/Mechanize Automation (Actual Solution to #2) </li></ul><ul><li>Uses Perl “Mechanize” package. </li></ul><ul><li>Grabs the HTML directly </li></ul><ul><li>Able to easily follow links, fill in forms, search html </li></ul><ul><li>Also referred to as “screen scraping” </li></ul>Problem #2 “Sunday Sanity”
    10. 10. <ul><li>Pros </li></ul><ul><li>Abstracts out Perl programming </li></ul><ul><li>Set of common, easy to use functions </li></ul><ul><li>Reads in test file (text), executes each command </li></ul><ul><li>Keeps a log of commands and results </li></ul><ul><li>Provides combination of Web and Perl commands </li></ul><ul><li>Able to call other tests </li></ul><ul><li>Cons </li></ul><ul><li>Requires a serious Perl programmer initially (in QA clothing!) </li></ul><ul><li>No automatic redirection </li></ul><ul><li>Cannot specify individual buttons </li></ul><ul><li>Could not use some Windows functions (like file attachments) </li></ul><ul><li>Browser-less testing </li></ul>Solution #2 Perl-Mechanize
    11. 11. <ul><li>Mechanize Code Example </li></ul><ul><li>#, Lexus Sanity </li></ul><ul><li>Get, https://www.lexusfinancialofsc.com </li></ul><ul><li>PrintTitle </li></ul><ul><li>FindText, Register for online access to your LFS , Pass, Fail </li></ul><ul><li>SetField, User, support@zixcorp.com </li></ul><ul><li>SetField, password, ****** </li></ul><ul><li>SubmitForm </li></ul><ul><li>Get, https://www.lexusfinancialofsc.com/my_lfs/access_account/AccountSummary.do </li></ul><ul><li>PrintTitle </li></ul><ul><li>FindText, Welcome To My LFS Account, Pass, Fail </li></ul><ul><li>FollowURL, Mail Center </li></ul><ul><li>FindText, Welcome to the Mail Center, Pass, Fail </li></ul><ul><li>FollowURL, Inbox </li></ul><ul><li>FindText, Status, Pass, Fail </li></ul><ul><li>FindText, Subject, Pass, Fail </li></ul><ul><li>FindText, Date Received, Pass, Fail </li></ul><ul><li>FindText, Expiration Date, Pass, Fail </li></ul><ul><li>FollowURL, Compose </li></ul><ul><li>PrintTitle </li></ul><ul><li>FindText, Email Address:, Pass, Fail </li></ul><ul><li>FindText, Account:, Pass, Fail </li></ul><ul><li>FindText, Subject:, Pass, Fail </li></ul><ul><li>FindText, Message:, Pass, Fail </li></ul>Solution #2 Perl-Mechanize
    12. 12. <ul><li>Results For Problem #2 </li></ul><ul><ul><li>Executes defined well documented “Sunday Sanity” tests </li></ul></ul><ul><ul><li>Utilizes Windows Task Scheduler every day at 6am </li></ul></ul><ul><ul><li>“ Sunday Sanity” accomplished in 1 hour. </li></ul></ul><ul><ul><li>Six person effort accomplished with one person, less than five minutes (when no errors) on Sunday morning </li></ul></ul><ul><ul><li>“ Sunday Sanity” on demand for Hotfixes (rarely needed now) </li></ul></ul><ul><ul><li>Any QA person can run and verify results </li></ul></ul><ul><ul><li>All required functionality checked </li></ul></ul><ul><ul><li>Results logged and saved in time-stamped files </li></ul></ul>Solution #2 Perl-Mechanize
    13. 13. <ul><li>Perl-Mechanize Demonstration of “Sanity Test” </li></ul><ul><li>“ Sunday Sanity” Automated Testing using Mechanize </li></ul><ul><ul><li>Scheduled execution 6am on Sunday </li></ul></ul><ul><ul><li>Scheduled execution 6am on Weekdays (less Billing) </li></ul></ul><ul><ul><li>Example of a test run summary </li></ul></ul><ul><ul><li>Example of drilling down into test results </li></ul></ul>Solution #2 Perl-Mechanize
    14. 14. <ul><li>Combined limitations of WinRunner and TestMech </li></ul><ul><li>WinRunner </li></ul><ul><ul><li>Loses focus in Browser </li></ul></ul><ul><ul><li>Couldn’t attach to apps it started </li></ul></ul><ul><ul><li>Babysitting execution of test runs. </li></ul></ul><ul><ul><li>GUI Map issues with new releases </li></ul></ul><ul><li>TestMech </li></ul><ul><ul><li>Requires a serious Perl programmer initially </li></ul></ul><ul><ul><li>No automatic redirection </li></ul></ul><ul><ul><li>Cannot specify individual buttons </li></ul></ul><ul><ul><li>Could not use some Windows functions (like Msg attachments) </li></ul></ul><ul><ul><li>Browser-less testing </li></ul></ul>Problem #3
    15. 15. <ul><li>Making Fortunate Discoveries by Accident </li></ul><ul><li>Software Quality Engineering Training Programs Brochure </li></ul><ul><ul><li>Script Programming for Testers </li></ul></ul><ul><ul><li>Brett Pettichord (the guy in the cowboy hat – of course!) </li></ul></ul><ul><ul><li>Web Application Testing In Ruby (Watir) – pronounced water </li></ul></ul><ul><ul><li>Ruby Scripting Language </li></ul></ul><ul><li>Danny Faught - Tejas Consulting </li></ul><ul><ul><li>Posted he liked Watir over Samie circa 12/04 </li></ul></ul><ul><li>Curiosity killed the cat... </li></ul><ul><ul><li>Subscribed to Watir User Forums </li></ul></ul><ul><ul><li>Observed a serious commitment by the developers </li></ul></ul><ul><ul><li>Developers AND users responsive to “newbie questions”. </li></ul></ul><ul><ul><li>Feature development prioritized by user community demand </li></ul></ul><ul><ul><li>Releases were fairly steady (~2-3 months) </li></ul></ul>Serendipity Strikes!
    16. 16. <ul><li>Initial Steps </li></ul><ul><li>Read the install procedures – clear and fact full </li></ul><ul><li>Downloaded Ruby and Watir </li></ul><ul><li>Ran unit tests supplied with Watir – Failed tests! </li></ul><ul><li>Read FAQs – how to configure pop-up handling… </li></ul><ul><li>Reran unit tests – All PASSED! </li></ul><ul><li>All this in an 1 ½ hours – I was impressed. </li></ul><ul><li>Within an hour, I had a script that could log into a Production portal. </li></ul><ul><li>Possibilities appeared endless! </li></ul><ul><li>Seeing the unit tests work and examining the code that performed them was very insightful. </li></ul>Watir Investigation
    17. 17. <ul><li>Production Rollouts on the horizon </li></ul><ul><li>New feature releases and fixes for the two largest ZixPort customers, health insurance payors. </li></ul><ul><li>Personal desire to reduce several hours of post production testing and increase test coverage. </li></ul><ul><li>Able to automate Payor#1 pseudo sanity fairly quickly for all portals. </li></ul><ul><li>Able to automate Payor#2 pseudo sanity with exception of two portals (due to hidden objects) </li></ul><ul><li>Watir Bonus! – Log into the two Payor#2 portals manually and Watir could “attach” to the browser and complete automated testing. – way cool! </li></ul>Watir Enthusiasm Grows
    18. 18. <ul><li>Continued Watir Development </li></ul><ul><li>Modularized functions to use against both portals. </li></ul><ul><li>Leveraged Prod tests against Payor#2 Staging Environment </li></ul><ul><li>Added more Staging tests and pushed coverage to Test </li></ul><ul><li>Continued adding test cases in Test Environment </li></ul><ul><li>Reversed trend from Test, Staging & Production with next Prod Release! </li></ul><ul><li>Now the serious coding begins… </li></ul><ul><ul><li>Added Send Mail code </li></ul></ul><ul><ul><li>Added Receive Mail code </li></ul></ul><ul><ul><li>Attachments can now be performed (need to follow-thru) </li></ul></ul>Watir Progress
    19. 19. <ul><li>Pros </li></ul><ul><li>Very reliable, repeatable & dependable </li></ul><ul><li>Execution stability extremely high </li></ul><ul><li>Uses IE Browser </li></ul><ul><li>Browser can be minimized during testing </li></ul><ul><li>Multiple scripts can be run simultaneously on the same machine without interference. </li></ul><ul><li>Other activities can be performed while Watir is running. </li></ul><ul><li>Comes with a great editor SciTE (Scintilla Text Editor) </li></ul><ul><li>Leverages a mature OO platform – Ruby </li></ul><ul><li>Great Developer and User support forum </li></ul><ul><li>Best of all – it’s Open Source (feel free to contribute!) </li></ul>Solution #3 Watir - Pros
    20. 20. <ul><li>Cons </li></ul><ul><li>Cannot perform Netscape and FireFox Browser testing (Selenium to the rescue?) </li></ul><ul><li>Handling of popup dialogue boxes continually in flux (WinHelper, AutoIT & what-next?) </li></ul><ul><li>New Watir Release changes may cause some heartache (migration woes – I desired access to “hidden objects” which required minor coding changes across all scripts) </li></ul><ul><li>Primarily intended for Unit testing, being adopted by QA worldwide for functional and regression testing. </li></ul><ul><li>Several potential logging mechanisms, but none are a clear winner going forward. </li></ul>Solution #3 Watir - Cons
    21. 21. <ul><li>Benefits </li></ul><ul><li>Increased test coverage, reduced test execution time </li></ul><ul><li>Reduces repetition of monotonous tests </li></ul><ul><li>Reduces overtime to a fairly tolerable level. </li></ul><ul><li>Reduced post-deployment test time for large customers (from 4 Hours to 1 Hour) </li></ul><ul><li>No problems have been detected by customer </li></ul><ul><li>Customer confidence in ZixPort products continues to grow </li></ul><ul><li>Walt has fun coding and doing test automation! </li></ul>Solution #3 Watir - Benefits
    22. 22. <ul><li>Crystal Ball (Future plans) </li></ul><ul><li>Continue increasing test coverage </li></ul><ul><li>Need ability to log test results to time-stamped files </li></ul><ul><li>Need ability to post to Test Director </li></ul><ul><li>Modularize and consolidate functions </li></ul>Solution #3: Watir
    23. 23. <ul><li>Watir - Demo </li></ul><ul><li>Single Execution </li></ul><ul><li>Multiple Execution </li></ul><ul><li>Minimized Execution </li></ul><ul><li>All of the above… </li></ul>Solution #3: Watir Demo
    24. 24. <ul><li>Watir home page </li></ul><ul><ul><ul><li>http://wtr.rubyforge.org </li></ul></ul></ul><ul><li>Wtr-general Archives </li></ul><ul><ul><ul><li>http://rubyforge.org/pipermail/wtr-general </li></ul></ul></ul><ul><li>Ruby </li></ul><ul><ul><ul><li>http://rubyforge.org </li></ul></ul></ul><ul><li>Special thanks to our host! – NerdBooks! </li></ul><ul><ul><li>Spread the Word! </li></ul></ul>Watir - References
    25. 25. <ul><li>What’s in the ZixCorp QA Tool Belt? </li></ul><ul><li>Each test automation tool fills a valuable niche. </li></ul><ul><li>Each tool complements the other. </li></ul><ul><li>All three are used on a daily basis. </li></ul><ul><li>All three contribute to increased test coverage and reduced test cycle times. </li></ul><ul><li>ZixCorp is able to meet the level of quality and the high-availability as required by our Customer SLA’s. </li></ul>Summary
    26. 26. <ul><li>Test Automation Benefits: </li></ul><ul><ul><li>Fast </li></ul></ul><ul><ul><li>Accurate </li></ul></ul><ul><ul><li>Repeatable </li></ul></ul><ul><ul><li>Reliable </li></ul></ul><ul><ul><li>More extensive tests </li></ul></ul><ul><ul><li>Results are stored </li></ul></ul><ul><ul><li>Improved use of QA time under existing budget and time constraints </li></ul></ul><ul><li>Q & A </li></ul>Test Automation Benefits