Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Web App Testing - A Practical Approach


Published on

Testing Web Applications: A Practical Approach
Walter Mamed,

Track 3: 11:00 – 12:00

Web-based applications have become the most widely used form of software, not only for e-commerce, but in our personal lives as well. Whether your spouse is booking your next vacation, or you are scheduling an appointment in an acute care facility, responsiveness and reliability are key to your satisfaction and desire to return. The quality assurance group testing these applications faces many challenges, with shorter test cycle times, fewer resources, constantly evolving technology, and instant world wide exposure. Explore how to plan, test, and deploy new or updated websites with confidence using practical, no nonsense methods. Functional and non-functional testing including configuration, usability, performance, and security will be covered. Learn how to use software tools to improve your testing techniques. Automated testing, mobile browsing, and the future of Rich Internet Applications will also be discussed. Take home a new perspective on testing web applications; implement these solutions and reduce your testing anxiety.

About the Speaker…
Walter Mamed is Director of Quality Assurance at JWT (Digital Technology) in Irving, Texas. He has over 30 years experience in a variety of quality assurance and software test engineering development positions, focusing on software and hardware test automation. Walt has been building test automation frameworks for GUI testing and web based applications for over 15 years. His web testing experience includes secure Email, On-boarding, ecommerce and lead generation as well as large-scale automated regression test suites. Walt is very active in the professional community as Director of the Board and Secretary for the Dallas/Ft. Worth (HP) Mercury User Group ( for the last 7 years. He is an ASQ Certified Software Quality Engineer.

  • Be the first to comment

Web App Testing - A Practical Approach

  1. 1.       Friday, April 23rd  Testing Web  Applications: A  Practical Approach   2010   INSTRUCTOR:   WALTER MAMED  COMPANY:    JWT.COM        11:00 AM – 12:00 PM  
  2. 2.   This page intentionally left blank.   
  3. 3. Testing Web Sites & Applications A Practical Approach Walt Mamed JWT, Director, Quality Assurance 02/09/2010 News Story ‐ Stray Mouse Click 2 1
  4. 4. Double Click of Death • On November 14, 2007 at 3:30pm one of Credit  Suisse s trading algorithms suddenly went haywire Suisse’s trading algorithms suddenly went haywire • Sent hundreds of thousands of bogus requests to the  exchange. • Acted like a denial‐of‐service attack on the NYSE • Affected trading of 975 stocks • Caused by a trader who accidentally double‐clicked  an icon in a trading program’s interface. • Credit Suisse assessed a $150,000 fine 3 Overview 1. Test Planning 2. Configuration Management 3. Test Execution 4. Projects – CBT, iPhone, Security, and more 5. Test Tools (Hint: Watch for gold nuggets) 4 2
  5. 5. Test planning for a new or existing web site TEST PLANNING 5 Test Planning Test Planning is one of the keys to project success:  Gathering Requirements G th i R i t Functional Decomposition Risk Based Testing Analysis Develop Test Plans and Procedures 6 3
  6. 6. Requirements Gathering Requirements come in many forms:  Project Plan or Business Requirements Spec P j t Pl B i R i t S Software Requirements Specification Functional Design Document Feature Specification Document Interface Control Document Use Cases Wireframes 7 Functional Decomposition FD ‐ Breaking it down piece by piece: Decompose the intended function into sub‐functions D th i t d d f ti i t b f ti Divide and conquer (split due to volume of effort) Top‐down: if system is fully described FD based on the flow of data or traversal by user  y q Verify all requirements have been covered. It’s easier if you have Use Cases More challenging if you have Business Requirements 8 4
  7. 7. Functional Decomposition Critical or High use of Balancing risk based subsystem, function testing and repetition  or feature or feature Booking of tasks ft k Booking Scenarios Smoke Tests Searches & Loyalty Program Filters Defect Functional 1 Functional 1 Escapes Business Functional … Adjustments Tools supporting Functional …N manual testing 9 Test Planning – RBT Analysis RBT ‐ Risk Based Testing analysis: High use of a subsystem, function or feature. Hi h f b t f ti f t Criticality of a subsystem, function or feature,  including the cost of failure. Prioritize what should be tested first. Not doing so explains why big bugs are found at the  end of a test cycle; its human nature to test the easy  functionality first. 10 5
  8. 8. Test Planning – RBT Analysis Test Design Techniques using software models: Equivalence partitioning E i l titi i − Breakdown elements into classes − Perhaps use a mind map Boundary value analysis − Identify edges or end‐points Decision tables State transition diagrams − Will also help define your negative tests 11 Test Planning ‐ Mindmap 12 6
  9. 9. Test Planning – Test Plans Develop Test Plans and Procedures: Test plans are usually in Word. Test plans are usually in Word Detailed test procedures are usually in Excel. Quality Center – Requirements and Test Plan  modules (great for confirming all requirements  covered). Writing these documents should be easier if the  Writing these documents should be easier if the previous steps (FD & RBT) were performed. IEEE 829 defines many types of test specifications – “If it’s not written down, it didn’t happen.” 13 Test Planning Practical Suggestions for Test Planning :  Web Analytics (for existing websites) W b A l ti (f i ti b it ) Planning Test Automation? – Testability as a  requirement for Development Production Monitoring 14 7
  10. 10. Test Planning Suggestions Web Analytics (for existing websites) Online Business Optimization (Tealeaf, Omniture) O li B i O ti i ti (T l f O it ) – Exit rate, average time on page, contribution to revenue. – Where and why are visitors leaving. – Know how customers are using your site. Browser usage (Cross Browser Test planning) – Browser type used, what version, mobile user type d h bl – Use what you know your customers/visitors use. Behavior Map (page hit frequency) 15 Test Planning ‐ Behavior Map 16 8
  11. 11. Test Planning ‐ Testability Planning Test Automation? Testability T t bilit as a requirement for Development: i tf D l t Provide a unique and meaningful name property for: – Every actionable html object on the page. (entry‐fields,  buttons, radio buttons, dropdown list boxes, images, links,  etc.) – Every table object that requires testing Every table object that requires testing. – Every response that requires testing. The response may be  in tables, spans, divs, lis, etc. 17 Test Planning ‐ Testability Testability as a requirement for Development: Populate the ‘id’ and ‘alt’ tags to give QA more  P l t th ‘id’ d ‘ lt’ t t i QA alternatives to identify an object during scripting. – SEO and 508 Compliance contribute to this  recommendation as well. Use a naming convention that includes the function  or purpose of the given object.  or purpose of the given object Do not change any HTML element property name  (including id & alt tags) from release to release. 18 9
  12. 12. Test Planning ‐ Monitoring Production Monitoring: Ensure your site and applications are performing. Ensure your site and applications are performing Identify, resolve and prevent issues. Develop an escalation policy, triage, remediate, and  confirm resolution. Use automated daily smoke tests to supplement  monitoring from a customer or partner perspective. monitoring from a customer or partner perspective. Discuss this during the requirements phase – What, how, where and who? – The wrong time is the day of deployment. Make sure you know what you are testing. CONFIGURATION MANAGEMENT 20 10
  13. 13. Configuration Management Manage software configurations: Audit configuration after push to QA/Prod A dit fi ti ft h t QA/P d – Use to confirm – Output contains Build Version, Date & Time Establish method to directly access web servers – Avoid round‐robin approach behind load balancers. – h // b http://web#‐ / 21 Let’s get to it! TEST EXECUTION 22 11
  14. 14. Functional Testing Functional testing: Run Smoke, Sanity, Critical Path tests R S k S it C iti l P th t t Check all links and web pages – Site spider • Start at Home page and traverse whole site – (exclude external links) • Check HTTP status 2xx, 3xx, 4xx & 5xx • View pages for gross or cosmetic failures. (more later) – Xenu link sleuth 23 Functional Testing Functional testing: Forms submittal F b itt l Email User profiles Role based access , j Flash, Ajax Back office testing Examine server side logs 24 12
  15. 15. Examine Server Side Logs 25 Usability Inspection Usability Inspection: Navigation N i ti Page Content Intuitive 508 Compliance ‐ accessibility  Search Sitemap Help 26 13
  16. 16. Usability – User Experience Page Download Times and Browser Rendering: No one likes a slow website N lik l b it Load testing and performance usually done late in  the test cycle Measure web page download performance early – Part of Sanity/Smoke test script. – Run multiple times and average. Track page download trends from release to release. – Test script writes download times to csv. 27 SEO Dated Page Download Trend XPIE6 (Single User) 25 38b5 R6 20 42b2 44b4 47b4 15 49b2 10 5 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 28 14
  17. 17. Page Download and Render Time 29 Usability – Drilling Down Page Download Time and HTML elements: Measure HTML element download times M HTML l td l d ti – HttpWatch (works with both Firefox & IE, has an API) – (to demo object downloads) – Yslow (Firefox addon) 30 15
  18. 18. HttpWatch 31 Cross Browser Testing, iPhone automation, Security, Defect Life Cycle TESTING PROJECTS 32 16
  19. 19. Cross Browser Testing Cross Browser Testing project: Created and used a CBT lab. C t d d d CBT l b – Various combinations of FF, IE and Windows OS  • XP/IE6, XP/IE7 • Vista/IE7 • XP/Firefox – Ran automated regression tests on each combination. – Discovered many cosmetic defects. – No functional errors found. – Many companies use Selenium. 33 Cross Browser Testing Cross Browser Testing project (next steps?): Considering an HTML/CSS syntax checker / W3C  C id i HTML/CSS t h k / W3C validator – Many online tools generate considerable output. – HTML Validator (Firefox Add on) – Total Validator (Firefox Add on) – is another consideration Litmusapp com is another consideration 34 17
  20. 20. Automated testing on the iPhone Automated testing on the iPhone: A website was created for the iPhone AH t l b it t d f th iPh Examined test tools to automate testing of iPhone  web site.  Tried SafariWatir on the Mac without success. As a reasonable alternative I used FireWatir (Watir  for Firefox) on a PC and ran automated regression  tests for the iPhone web site. – Minor visual differences vs. Safari 35 Security Testing Security – start simply (perhaps you already do?): Invalid inputs in text entry fields and forms I lid i t i t t t fi ld df SSL– https is used where appropriate (e.g. forms) Internal URLs not accessible (unless logged in) Confirm no access to web server directories XSS – Cross Site Scripting p g 36 18
  21. 21. Security Testing Security: Set everyone's expectations S t ' t ti Gather good tools Look at your application from every perspective Test for underlying weaknesses yy g Go back and verify your scanner findings Manually check for weaknesses Test your source code 37 Security Testing Security – Captcha: Captcha – detects automated scripts in the wild and  C t h d t t t t d i t i th ild d blocks them – Verifying the detection of automated scripts is easy if you  have automated tests – In order to run automated tests in Production, plan to have  a means to disarm Captcha p – Timed re‐arming is preferable such that Captcha is enabled  automatically to protect the site in case you forget.  38 19
  22. 22. Defect Lifecycle Managing the defect lifecycle:  Issues are detailed, descriptive, and concise. I d t il d d i ti d i Ensure severity and priority are appropriate. Ensure there are no unassigned issues. Hold weekly mandatory review meetings between  QA & stakeholders Write a defect, write a test case (if none exists) – Copy steps to reproduce into a new test case. ^C^V – Great way to “beef up” regression test suite. 39 What’s in your QA Tool Belt? TEST TOOLS 40 20
  23. 23. Test Tools – Browser Add‐ons Useful Firefox Add ons: Firebug Fi b FormSaver FireCookie tamperData (view/modify HTTP/HTTPS) g Screengrab Xpather 41 Test Tools – Browser Add‐ons Useful Internet Explorer Add ons: Developer Toolbar D l T lb IECookiesView Fiddler (Watcher – Passive Security Auditor) Webcollect (screen capture) y Web Accessibility Toolbar Mathon (Swiss army knife) 42 21
  24. 24. Test Tools ‐ WATiR Automated Test Tool: WATiR – Web App Testing in Ruby WATiR W b A T ti i R b – Supports your web app no matter what it is developed in – Full featured modern scripting language – Supports multiple browsers on different platforms – It is powerful and easy to use, yet beautifully lightweight – There is an active and growing community behind it There is an active and growing community behind it – It is free Open Source tool.  There are no costs to use the  tool – User for five years (solid, stable, growing functionality) 43 Test Tools ‐ WATiR Automated testing results using Watir: Fully automated the testing F ll t t d th H t l t ti – Sanity testing (page download times too) – Regression Testing (a deployment every week) – Booking tests were data driven (Excel spreadsheet) – Three day test cycle (2 resources) reduced to two hours  end to end.  Exploratory testing added to process. end to end Exploratory testing added to process – Data Center Consolidation (15 app servers, 8 instances on  each, 120 total instances) 44 22
  25. 25. Test Tools ‐ Justification Tips when using automation: Use a widescreen monitor in portrait mode to  U id it i t it d t maximize visibility of the whole page. Use automation to scroll to the bottom of the page. Record all defect #’s detected by automated testing – Application Services (weekly releases) – Projects (web page redesigns) Branch test scripts to mimic Development code Calculate your ROI 45 Test Tools – Software Utilities Test Utilities: Ruby is a fully featured programming language. R b i f ll f t d i l – Gems like NET::SMTP to send mail or pop mail • Verify emails sent from web app; like change password, click link. – NET::SSH tail utility pulls server logs to desktop for viewing – Missed destinations utility (feedback to Dev) – Run SEO tests on web pages (too tedious to do manually) Run SEO tests on web pages (too tedious to do manually) – Site Spider that traverses site starting at the home page • Able to traverse the whole site with minimal scripting time. 46 23
  26. 26. Test Tools ‐ Security Security: – web security testing tools OWASP b it t ti t l Ethical Hacker Network HP Dev Inspect (for programmers) HP QA Inspect (for QA testing) p ( HP Web Inspect (for Production)) Hosted services; McAfee for production security  testing. 47 Test Tools ‐ Performance Social Networking: Facebook – Can’t “load test” in Facebook’s domain. – Created simulateUser.php (randomized actions) • Register new friends • View canvas, tag other friends with characteristics, save profile – Ran apachebench (ab) against simulateUser.php – Facebook application refactored in targeted areas application refactored in targeted areas – Placed database in RAM (limited risk) – 312 to 46K requests/transactions (150 X better  performance ) 24
  27. 27. Test Tools ‐ Flash Flash Testing Apps ‐ Commercial: QTP with plugin (instrument the Flash code) QTP with plugin (instrument the Flash code) TestComplete Ranorex AutoCzar TestPlant ‐ EggPlant (image based) Flash Testing Apps – O Fl h T ti A OpenSource: S T‐Plan Robot  ASUnit 49 Quality Nuggets Before Deployment Day! Run your regression test scripts in Prod – Run your regression test scripts in Prod why? – Deployment failed, troubleshooting focused on new  release, root cause was a pre‐existing condition in Prod. – Discovering issues before deployment eliminates the  confusion and unnecessary troubleshooting from assuming  that a new deployment caused the problem. – By running automated regression the evening before a By running automated regression the evening before a  deployment, several issues have been found since, some  serious. – Content Management System changes 50 25
  28. 28. QUESTIONS? 51 Biography – Walter Mamed Walter Mamed is Director of Quality Assurance at JWT (Digital Technology)  in Irving, Texas.  He has over 30 years experience in a variety of quality  assurance and software test engineering development positions, focusing  on software and hardware test automation. Walt has been building test automation frameworks for GUI testing and  web based applications for over 15 years.  His web testing experience  includes secure Email, On‐boarding, ecommerce and lead generation as  well as large‐scale automated regression test suites. Walt is very active in the professional community as Director of the Board  and Secretary for the Dallas/Ft. Worth (HP) Mercury User Group  / ( for the last 7 years.   He is an ASQ Certified Software  Quality Engineer.  52 26
  29. 29. Acronyms • API – Application Program Interface • CBT – Cross Browser Testingg • CMS – Content Management System • CSS – Cascading Style Sheets • CVS – Concurrent Versioning System • ETL – Extraction, Transformation, and Loading  • FD – Functional Decomposition • RBT – Risk Based Testing • RCS – Revision Control System • SVN – SubVersion • SQL – Structured Query Statements • W3C – World Wide Web Consortium 53 27