Mark meninger-feb-2010

1,034 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,034
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Mark meninger-feb-2010

  1. 1. An Examination of Test Automation Use Cases (and the factors behind the examination) January 19, 2010 Mark Meninger 1
  2. 2. January 20, 2010 An Examination of Test Automation Use Cases Contents Objectives and Perspective • The Speaker • Presentation Objectives Test Automation Philosophy • Objectives of test automation • GUI vs non-GUI • A look at 2 products Test Automation Use Cases How does this all fit in? 2 Confidential McAfee Internal Use Only
  3. 3. January 20, 2010 Objectives and Perspective 3 Confidential McAfee Internal Use Only
  4. 4. January 20, 2010 Objectives and Perspective The Speaker (Mark_Meninger@McAfee.com) • Current role @ McAfee: Driving test automation on Consumer side • Functional GUI automation (support for L10n) across 3 sites • Functional non-GUI automation (development focused) • Automated Performance testing • Previously @ RIM • As test automation manager for end-user BlackBerry testing • RIM Test Automation Conference Chair • A student of test automation and testing • Worked (and work) with some very capable individuals • Made some good mistakes • Learned from my mistakes • Little experience with Agile • Presentations @ StarEast, QAI, KWSQA • My4Blog (as of this Friday): http://automatedtestingblog.blogspot.com Confidential McAfee Internal Use Only
  5. 5. January 20, 2010 Objectives and Perspective Presentation Objectives 1. Talk about my learnings and philosophy of test automation 2. Hear your perspective on what I’m talking about 3. Evaluate test automation use cases/examples (group- exercise) a) Interface complexity b) Speed of test automation c) Speed of execution d) Derived value e) How this fits into the testing cycle 4. In the context of Agile, I would… 5 Confidential McAfee Internal Use Only
  6. 6. January 20, 2010 Test Automation Philosophy Mark’s Test Automation Philosophy 6 Confidential McAfee Internal Use Only
  7. 7. January 20, 2010 Test Automation Philosophy Manual Testing - A path in the trees • Strategy/Plan (resourcing, infrastructure, approach) • Need to learn the SUT: • User-stories • Use Cases • Tools • Technologies • Testing types & cycles (regression, defect, smoke) • Sprints • Apply common techniques • Scripted test cases (with different methodologies), • Exploratory testing • Then testing can begin • I’ve seen most testing organizations do this well to varying degrees 7 Confidential McAfee Internal Use Only
  8. 8. January 20, 2010 Test Automation Philosophy Automated testing – Find a path in the forest • Test automation is less of a cookie-cutter approach than manual testing • Common tasks in test automation • Interface, framework, integration • Very specific considerations: • The technology of the system under test (SUT) • The technologies used to test the SUT • Impact on the development/testing schedule • These details impact the success of the implementation, the derived value, the time to delivery, availability etc • This makes each test automation implementation unique and challenging • Not all orgs do test automation well 8 Confidential McAfee Internal Use Only
  9. 9. January 20, 2010 Test Automation Philosophy Evolution of the Approach to Test Automation In the beginning: Here is a test automation tool… now automate! (seen it) 9 Confidential McAfee Internal Use Only
  10. 10. January 20, 2010 Test Automation Philosophy Evolution of the Approach to Test Automation And Then: Evaluate tools, choose one and automate! (done it) 10 Confidential McAfee Internal Use Only
  11. 11. January 20, 2010 Test Automation Philosophy Evolution of the Approach to Test Automation After much learning (now do it!) : Examine my test automation requirements a) Look at • Objectives • SUT technology, roadmap • Available budget, resources • Available tool technologies, capabilities and constraints • SDLC methodology (Agile, Waterfall etc.) • Development culture • Testing culture • Relationships b) Put together my business case 11 Confidential McAfee Internal Use Only c) Start with a careful plan
  12. 12. January 20, 2010 Test Automation Philosophy Objectives of test automation 1. Executed successfully Test functionality sooner relatively early within the rather than later in the cycle development/testing cycle Interface used is available and stable 2. Reduction in manual testing Automated execution as soon time as a build is released: Evenings, Weekends Automated execution run in parallel Execution across platforms and test suites Limited by availability of infrastructure 12 Confidential McAfee Internal Use Only
  13. 13. January 20, 2010 Test Automation Philosophy Test Automation Objectives 3. Provides reliable results on Tests can be executed repeatedly 1st run with minimal false negatives Regular testing of interface and framework essential Changing interface not breaking tests We can increase coverage of 4. The solution is scalable & manual test cases without excessive maintainable framework growth Grow test case numbers with confidence: 10’s -> 100’s 100’s -> 1000’s 13 Confidential McAfee Internal Use Only
  14. 14. January 20, 2010 Test Automation Philosophy Test Automation Objectives 5. Agile Environment Out-of-the-box, light-weight test automation supports planned sprints Test automation team structured, organized to support agile testing requirements Solution, approach is flexible and expandable 14 Confidential McAfee Internal Use Only
  15. 15. January 20, 2010 Test Automation Philosophy GUI vs non-GUI Technology Considerations Non-GUI GUI (some API) 15 Confidential McAfee Internal Use Only
  16. 16. January 20, 2010 Test Automation Philosophy Test Automation – The Big Question Do we UI start here? UI-Logic Interface Or here? First Business Logic Layer Software Re Abstractions Or here? Another Abstraction And Another Where? The Basement 16 Confidential McAfee Internal Use Only
  17. 17. January 20, 2010 Test Automation Philosophy Test Automation - GUI Fat Client (Generalities) GUI - Fat Client • Interface • Skill-set • Typically complex - more points of failure • Requires solid technical skill-set with • Changes more frequent good design and programming abilities • Increased maintenance • Deeper test automation skill-set and experience required • Automation - dependent upon being stable • More expensive • Technology • Localization • Vendor tools provide better support • Running identical functional tests across localized builds • Scripting technology usually has limited support (i.e. VBScript) • Language independence support is essential to support L10n automation • Performance • Management for unresponsive GUI • GUI adds performance hit each time accessed • Framework can add considerable overhead 17 Confidential McAfee Internal Use Only
  18. 18. January 20, 2010 Test Automation Philosophy Test Automation – GUI Web (Generalities) GUI - Web • Interface • Typically simpler – fewer complexities to manage • Changes frequently over time • Automation – dependent upon being stable • Technology • Good open source (Selenium, Watir) • Performance • Fewer if no performance issues • Good available open source frameworks • Skill-set • Better use of ‘scripters’ rather than developers • Less investment required in more experienced test automation skill-sets • Less expensive • Localization • Language independent testing can be supported 18 Confidential McAfee Internal Use Only
  19. 19. January 20, 2010 Test Automation Philosophy Test Automation – non-GUI (Generalities) Non-GUI (API) Automation • Interface • Performance • Stable and reliable earlier in the • No GUI impact - much faster dev/testing cycle • Choose integrated framework with little • Less changes over time overhead • Fewer points of failure • Skill-set • High-level testing can be supported • Requires knowledge of underlying • Usually needs to be modified for increased business logic, application architecture and testability over time (fits Agile well IMO) design • Technology • More expensive • Choose scripting technology with robust • Localization library support • Issues around language independence • Integrate into development cycle (localized strings integrated into GUI abstractions) 19 Confidential McAfee Internal Use Only
  20. 20. January 20, 2010 Test Automation Philosophy GUI Automation GUI • Automation Implication • Coverage • Good top-to-bottom testing solution • Broader • # of automated test cases dependent upon GUI comlexity • Development (GUI complexity) • Higher maintenance costs • More effort to write tests • Longer development times • Execution • Dependent upon stable GUI • Slower execution times • More expensive (resources, equipment and license cost) • Value-add later in dev/testing cycle for products with major GUI changes 20 Confidential McAfee Internal Use Only
  21. 21. January 20, 2010 Test Automation Philosophy Non-GUI Automation Non-GUI Automation • Automation Implication • Coverage • Not top-to-bottom • Deeper testing • Larger # of automated test cases • Development • Easier to write tests • Shorter development times • Lower maintenance costs • Execution • Ready when the API are ready • Test much sooner in the dev/test • Test results faster; test more often • Gate before release to testing • Cheaper (resources, no licenses) • Add quality paradigm to development organization (code for testability) •21Value-add sooner in dev/testing cycle Confidential McAfee Internal Use Only
  22. 22. API Automation Manual Testing GUI Automation Automated Automated API Testing API Testing No No Tests Tests Pass? Pass? Yes Yes Automated GUI Manual Testing & Manual Quality Testing Testing Assurance Starts ?? !!! Start End 22 January 20, 2010 RTQA Build Release Cycle Confidential McAfee Internal Use Only
  23. 23. January 20, 2010 Test Automation Philosophy Complex GUI - Cost for Coverage • Complex GUI applications will incur an increasing cost to automate larger #’s of test cases • Cost: Effort, Infrastructure, Dollars, Time for Execution Automation Cost Ceiling GUI Costs become prohibitive Automated Testing Coverage 23 Confidential McAfee Internal Use Only
  24. 24. January 20, 2010 Test Automation Philosophy Automation Philosophy Review – Cost for Coverage • Non-GUI Automation has lower cost for coverage • Cost: Effort, Infrastructure, Dollars, Time for Execution Automation Cost Ceiling GUI Less expensive Non- Non- GUI24 Automated Testing Coverage Confidential McAfee Internal Use Only
  25. 25. Consumer Test Automation Framework (CTAF)/QTP/MAGI Lots of VBScript Test Script libraries CTAF External Libraries CTAF Internal to build GUI/ Assert Language Dependent Libraries Unit Test Table (and MPF MVS Helper Global support) r Our own MAGI Framework CTAF Complex Services Core Extensions extensions Functions Framework (yet very HP Quick Test Pro helpful) GUI HTML ID HTML ID HTML ID Control Control Control Confidential McAfee Internal Use Only
  26. 26. Example API Test Automation – Using Python Python Test Script Framework exists Python Python py.test/nose(?) Test Automation which has Framework nicely Python CTAF API Python numerous Libraries Libraries integrates rich and with diverse Python libraries COM High- Level High- Level High- Level Interface Class Class Class Implement IDispatch Confidential McAfee Internal Use Only
  27. 27. January 20, 2010 Test Automation Use Cases A look at some interfaces • Google Search • McAfee Security Center (2010 release) Disclaimer • These thoughts are my own and are not necessary correct • Part of the process would be for me to understand the technology and determine the best approach (Finding a path through the forest I find myself in) 27 Confidential McAfee Internal Use Only
  28. 28. January 20, 2010 Test Automation Use Cases Points to Evaluate • What approach would you take? • What are the risks of doing it this way? • What are the costs/investment requirements of doing this? (be specific) • What are the gains of doing this? • Why would you do this? • How would you integrate this into a testing cycle? • Why would you integrate this into the testing cycle this way? • How could this approach be fit into an Agile/Lean methodology? • What type of buy-in would you need to achieve this? • What relationships would you need to establish to be successful? 28 Confidential McAfee Internal Use Only
  29. 29. January 20, 2010 Test Automation Use Cases What I will focus on • Interface complexity • Speed of test automation • Speed of execution • Derived value • How this fits into the testing cycle 29 Confidential McAfee Internal Use Only
  30. 30. January 20, 2010 30 Confidential McAfee Internal Use Only
  31. 31. January 20, 2010 Test Automation Use Cases – Google Search • Google Search – From: http://en.wikipedia.org/wiki/Google_search – PageRank Logic – synonyms, weather forecasts, time zones, stock quotes, maps, earthquake data, movie showtimes, airports, home listings, and sports scores – prices, temperatures, money/unit conversions ("10.5 cm in inches"), calculations ( 3*4+sqrt(6)-pi/2 ), package tracking, patents, area codes,[6] and rudimentary language translation of displayed pages – ‘I’m feeling lucky’ – Privatization logic (Switzerland) – Ajax code 31 Confidential McAfee Internal Use Only – ….
  32. 32. January 20, 2010 Test Automation Use Cases – Google Search Our job to automate Search logic • Core functionality • Page rank is an algorithm that evaluates an index using supposedly over 200 factors 32 Confidential McAfee Internal Use Only
  33. 33. January 20, 2010 Test Automation Use Cases – Google Search •What I’d do – Back-end (non-gui) automation – Focus just on the engine and it’s data – Evaluate the probability and weighting of each factor – Generate a list of results and probably would fit into some level of confidence of accuracy – Rendering of data could be easily verified manually 33 Confidential McAfee Internal Use Only
  34. 34. January 20, 2010 Test Automation Use Cases – Google Search • How • Work with developers to build a testing engine that could handle ‘plug-ins’ of new testing factors • A common testing pattern would be for each factor • Determine how each factor would return results on its own or in interaction with another factor • Integrate this all into the testing engine • Establish a common mechanism for executing tests, gathering expected results and comparing actual results • Utilize the common testing pattern • Utilize the same pattern for evaluation in the testing engine • Drive a common testing interface across those adding new ranking functionality to the google search engine 34 Confidential McAfee Internal Use Only
  35. 35. January 20, 2010 Test Automation Use Cases Interface complexity • Interface would be simple – an API • The combination of algorithms (factors) would make this complex Speed of test automation delivery • Fast – Could start writing tests for each factor – Could start building in some relationships for each factor • Slower – Integration of everything together – This would also include the test engine 35 Confidential McAfee Internal Use Only
  36. 36. January 20, 2010 Test Automation Use Cases Speed of execution • Very fast • Could run very often Derived value • Substantial • Could fully automate the algorithm • If the integration of factors together could be done successfully, this would be substantial How this fits into the testing cycle • Immediately • Write code, test code 36 Confidential McAfee Internal Use Only • No throw-over to SV&V/QA
  37. 37. January 20, 2010 37 Confidential McAfee Internal Use Only
  38. 38. January 20, 2010 Cialis - erectile dysfunction drug Radical Prostatectomy - is an operation to remove the prostate gland and some of the tissue around it Very fast 38 Confidential McAfee Internal Use Only
  39. 39. January 20, 2010 39 Confidential McAfee Internal Use Only
  40. 40. January 20, 2010 Test Automation Use Cases – McAfee Security Center • McAfee Consumer Product (2010) – Security Center – Firewall, AntiVirus, AntiSpam, Parental Controls… – Focus on AntiVirus – Verify GUI and Functional behaviour 40 Confidential McAfee Internal Use Only
  41. 41. January 20, 2010 Test Automation Use Cases • McAfee Consumer Product (2010) – GUI – thick GUI – Verify GUI and Functional behaviour 41 Confidential McAfee Internal Use Only
  42. 42. January 20, 2010 Test Automation Use Cases • How • Vendor tool 42 Confidential McAfee Internal Use Only
  43. 43. January 20, 2010 Test Automation Use Cases • McAfee Consumer Product (2010) – GUI – thick GUI – Verify GUI and Functional behaviour 43 Confidential McAfee Internal Use Only
  44. 44. January 20, 2010 Test Automation Use Cases • McAfee Consumer Product (2010) – GUI – thick GUI – Verify GUI and Functional behaviour 44 Confidential McAfee Internal Use Only
  45. 45. January 20, 2010 Test Automation Use Cases – McAfee Security Center •What I’d do & How – Get a good GUI automation tool – Find a manageable way to integrate with the GUI DOM • Re-usability • Maintainability – First go: limit my automation to easiest test cases – Find a good framework to integrate with my GUI automation tool • Automate the automation – Aim would be reduce manual testing with an eye to reduce maintenance 45 Confidential McAfee Internal Use Only
  46. 46. January 20, 2010 Test Automation Use Cases – McAfee Security Center Interface complexity • Interface is very complex – Lots of objects to manage (lots of points of failure) – How we work with the interface is complex under the covers (constrained by tool) Speed of test automation delivery • Slow!! – Most off-the-shelf have limited library support – Have common set of libraries across the products – Need to build libraries for each product at the GUI & functional level – Integration into the framework could take even longer 46 Confidential McAfee Internal Use Only
  47. 47. January 20, 2010 Test Automation Use Cases – McAfee Security Center Speed of execution • Slow! • Fat client GUIs are usually slower • Hooks from application driving the GUI adds overhead • Integrating a framework that drives all this adds additional time Derived value • Some! • New GUI; some changes – later in the cycle • Same GUI; little changes – earlier in the cycle • Need more infrastructure to support 47 Confidential McAfee Internal Use Only
  48. 48. January 20, 2010 Test Automation Use Cases – McAfee Security Center How this fits into the testing cycle • If no GUI changes - > Right Away • GUI changes - > Later • Testing cycles will take longer • Slower to execute 48 Confidential McAfee Internal Use Only
  49. 49. January 20, 2010 Test Automation Use Cases – McAfee Security Center Test Automation – The Big Question Do we UI start here? UI-Logic Interface First Business Logic Layer Software Re Abstractions Or here? Another Abstraction And Another The Basement 49 Confidential McAfee Internal Use Only
  50. 50. January 20, 2010 Test Automation Use Cases – McAfee Security Center •What I’d do & How – Examine the layers below the GUI – Can I use any of them to automate testing? – Work to drive this idea within the organization – Most likely some developer has thought the same thing – That is my foothold and I build on that 50 Confidential McAfee Internal Use Only
  51. 51. January 20, 2010 Test Automation Use Cases – McAfee Security Center Interface complexity • Interface is less complex – New technology learning curve – Easier to call some API than manage GUI objects – Less change; more stability Speed of test automation delivery – Fast!! – Available frameworks (don’t need to build your own) (i.e. py.test) – Access to rich scripting environments & libraries (i.e. Python) – don’t need to build your own – Less points of failure to manage 51 Confidential McAfee Internal Use Only
  52. 52. January 20, 2010 Test Automation Use Cases – McAfee Security Center Speed of execution • Fast!! No GUI overhead • Integrated framework will also add little overhead (i.e. TestNG, Py.test) Derived value • Higher value How this fits into the testing cycle • Almost always test earlier in the cycle • Test more frequently • Integrate into the development cycle rather than QA – Quality moving upstream 52 Confidential McAfee Internal Use Only
  53. 53. January 20, 2010 53 Confidential McAfee Internal Use Only
  54. 54. January 20, 2010 Objectives and Perspective How does this all fit in? • Have your test automation specialist begin to develop a methodology that will fit your agile development cycle • Tell her/him what your requirements are and ask for a solution that will meet this • Optimize your test automation execution for Agile! • This will most likely be an out-of-the-box approach to test automation • All pay-offs should be well understood: • Light-weight, quick to execute, easy to develop, not-too-deep solution • Heavier, complex, longer-to execute, harder to develop, deeper test automation solution 54 Confidential McAfee Internal Use Only
  55. 55. January 20, 2010 Objectives and Perspective How does this all fit in? • They should have a good understanding of the available technologies to use and what the trade-offs are for each • What solution will best meet the above requirements? • Let that test automation specialist know what’s needed and why. This will hopefully inspire them to build the solution that meets these needs • Integrate automated testing into your testing cycles • Fit the automated testing for a given sprint/release • Each automated testing approach will have a given set of benefits and costs • Choose the automated testing items that make most sense 55 Confidential McAfee Internal Use Only
  56. 56. January 20, 2010 Consumer Applications QA Test Automation Strategy – Detailed Review Thank-you Mark_Meninger@McAfee.com As of this Friday http://automatedtestingblog.blogspot.com 56 Confidential McAfee Internal Use Only

×