Your SlideShare is downloading. ×
'BIG Testing' with Hans Buwalda
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

'BIG Testing' with Hans Buwalda


Published on

View webinar: …

View webinar:

Large-scale testing projects can stress many of the testing practices we have gotten used to over the years. This can result in less than optimal outcomes. A number of ideas and concepts have therefore emerged to support industrial-strength testing of big and complex projects. In this excerpt from a larger workshop Hans Buwalda shares experiences and the strategies he and his colleagues have used for testing on large projects, both in Europe and the US. Learn how to design tests specifically for automation and how to successfully incorporate keyword testing. See how to obtain more stable automation, what benefits and issues are of virtualization, and what to expect of global outsourcing. The information presented is based on nineteen years of experience with testing and test automation including projects that have tests executing continuously for many weeks on dozens of machines.

Published in: Technology, Business

1 Like
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide
  • in de VM slides - snapshots take spacedefinitie van "done" is een probleem agree on approach (in het agile verhaal)-betere voorbeelden, zoals bij data driven- gebruik ms verhaal-
  • Transcript

    • 1. Today’s webinar is presented by Hans Buwalda and he will discuss BIG Testing. Welcome to the EuroSTAR July Webinar Series This webinar is due to start at 4pm. Make sure you stick around at the end for the Q&A session and continue the conversation with the speaker on Twitter after the show! @esconfs #esconfs
    • 2. BIG Testing Hans Buwalda, CTO, LogiGear (USA) An internationally recognized expert in testing, Hans Buwalda is the pioneer of keyword-driven test automation, an approach now widely used throughout the testing industry. Working in The Netherlands for CMG (now CFI) he helped develop the successful TestFrameTMframework. He now is CTO of LogiGear Corporation in California, responsible for the keyword-driven Action Based TestingTM method, and the tools that support it: TestArchitectTM and TestArchitect for Visual Studio®. Hans speaks frequently at international conferences and is coauthor of Integrated Test Design and Automation: Using the TestFrame Method. @esconfs #esconfs
    • 3. Q&A If you have a question, the control panel on the right has a Questions box. All questions will be answered at the end of the presentation. The GoToWebinar attendee interface is made up of two parts. The Viewer Window shows the presenter’s screen. The Control Panel is where attendees can interact and ask questions. The webinar will be recorded and you will be notified via email when it is available. @esconfs #esconfs
    • 4. How can I get the slides? Slides and all materials will be posted on the EuroSTAR website as well as emailed to you. @esconfs #esconfs
    • 5. Join the conversation on Twitter #esconfs @esconfs @hansbuwalda Continue the conversation with the speaker on Twitter after the show! @esconfs #esconfs
    • 6. BIG Testing Hans Buwalda LogiGear Tutorial excerpts Keywords, Test Design, Automation, Virt ualization, Globalization
    • 7. © 2013 LogiGear Corporation. All Rights Reserved About LogiGear  Software testing company, around since 1994  Testing and test automation expertise, services and tooling consultancy, training test development and automation services "test integrated" development services  Aims to be thought leader, in particular for large and complex test projects  Products: TestArchitect™, TestArchitect for Visual Studio™ integrating test development with test management and automation based on modularized keyword-driven testing
    • 8. © 2013 LogiGear Corporation. All Rights Reserved What is "BIG"  Big efforts in development, automation, execution and/or follow up  It takes a long time and/or large capacity to run tests (lot of tests, lot of versions, lot of configurations, ...)  Scalability, short term and long term  Complexity, functional, technical  Number and diversity of players and stakeholders pigs, chicken, elephants, ankle biters, ...  Various definitions of "big" possible... and relevant... "10 machines" or "10 acres" "1000 tests" or "1000 weeks of testing"  Big today means: big for you "non trivial", you need to think about it "Windows 8 has undergone more than 1,240,000,000 hours of testing" Steven Sinofsky, Microsoft, 2012
    • 9. © 2013 LogiGear Corporation. All Rights Reserved Keywords, essential for scalability  Distinguish tasks for test development and for automation  The test developer creates tests using "actions". Each action consists of a keyword ("action word") and arguments  Automation focuses on automating the actions, each action is automated only once however…. actions by themselves are just a format, not a method….
    • 10. © 2013 LogiGear Corporation. All Rights Reserved Action Based Testing Test Development Plan Test Cases Test Objectives Test Module 1 Test Cases Test Objectives Test Module 2 Test Cases Test Objectives Test Module N Actions . . . ACTION AUTOMATION Break down Automate
    • 11. © 2013 LogiGear Corporation. All Rights Reserved Test Design  Effective test breakdown (into test modules) make sure test modules (test clusters) have a clear focus keep different kinds and levels of tests separate  Right level of actions as “high level” if possible, hiding as many details as much as possible ...but not if the details are relevant for the test It is my believe that successful automation is not a technical challenge. It is most of all a test design challenge.
    • 12. © 2013 LogiGear Corporation. All Rights Reserved What's the trick...
    • 13. © 2013 LogiGear Corporation. All Rights Reserved What's the trick...  Have or acquire facilities to store and organize you content  Select your stuff  Decide where to put what assign and label the shelves  Put it there  If the organization is not sufficient anymore, add to it or change it
    • 14. © 2013 LogiGear Corporation. All Rights Reserved Example of a business test module  Test a business functionality, doesn't show details that are not relevant to that  Navigation details are avoided, unless they're meant to be tested TEST MODULE Car Rental Payments user start system john TEST CASE TC 01 Rent some cars first name last name car rent car John Doe Ford Escape rent car John Doe Chevvy Volt last name amount check payment Doe 140.4 FINAL close application
    • 15. © 2013 LogiGear Corporation. All Rights Reserved Example of an interaction test module  In this "low level" test details are not hidden, since they are the target of the test  The right level of abstraction depends on the scope of the test, and is an outcome of your test design process TEST MODULE Screen Flow user start system john TEST CASE TC 01 "New Order" button first name control click main new order window check window exists new order FINAL close application
    • 16. © 2013 LogiGear Corporation. All Rights Reserved Eye on the ball, Scope  Always know the scope of the test module  The scope should be unambiguous  The scope determines many things: what the test objectives are which test cases to expect what level of actions to use what the checks are about and which events should generate a warning or error (if a “lower” functionality is wrong)
    • 17. © 2013 LogiGear Corporation. All Rights Reserved What I have seen not work  "Over-Checking": having checks that don't fit the scope of the test  Forcing data driven: making all tests data driven (variables, data files) without clear reason  Combinatorial explosions: test all ... for all ... in all ...  All actions high level (or all actions low level)  Many tests for forms and dialogs, little tests for business processes  Abundance of irrelevant comments, and lack of relevant comments
    • 18. © 2013 LogiGear Corporation. All Rights Reserved Choosing actions  Scope of the test determines the actions  As high level as appropriate, but not too high  As little arguments as possible use default values for arguments not relevant to the scope of the test  Clear names (usually verb + noun usually works well) to standardize action names: standardize both the verbs and the nouns, so "check customer" versus "verify client" (or vice versa) tests are not C++ code: avoid "technical habits", like mixed case and underlines  Manage the Actions  Document the Actions  Actions should be by-product of the test design
    • 19. © 2013 LogiGear Corporation. All Rights Reserved Example of using actions In this real world example the first "sequence number" for teller transactions for a given day is retrieved, using a search function • the "#" means an expression, in this case a variable • the ">>" assign to a variable for use later on in the test key key navigate F7 key navigate 3 page tab locate page tab Scan Criteria w indow wait for controls loaded search text check breadcrumb general functions > search w indow control value select search scan direction Backward w indow control value enter value search business date match # bus date source control click search go w indow wait for controls loaded search results w indow control variable get search results sequence number >> seq num
    • 20. © 2013 LogiGear Corporation. All Rights Reserved variable get sequence number >> seq num Example of using actions In this real world example the first "sequence number" for teller transactions for a given day is retrieved, using a search function • the "#" means an expression, in this case a variable • the ">>" assign to a variable for use later on in the test
    • 21. © 2013 LogiGear Corporation. All Rights Reserved Low-level, high-level, mid-level actions  Low-level: detailed interaction with the UI (or API) generic, do not show any functional or business logic examples: "click", "expand tree node", "select menu"  High-level: represent a business function specific to the scope of the test hide the interaction examples: "enter customer", "rent car", "check balance"  Mid-level: auxiliary actions that represent common sequences of low level actions usually to wrap a form or dialog greatly enhance maintainability example: "enter address fields" enter customer enter address fields enter select set . . .. . .
    • 22. © 2013 LogiGear Corporation. All Rights Reserved Test Development and Automation in sprints Test Module Definition (optional) Test Module Development Interface Definition Action Automation Test Execution Sprint ProductsProduct Backlog Test re-use Automation re-use product owner team prod owner & team User stories Documentation Domain understanding Acceptance Criteria PO Questions Situations Relations Agile life cycle Test development Main Level Test Modules Interaction Test Modules Cross over Test Modules
    • 23. © 2013 LogiGear Corporation. All Rights Reserved Some Tips to Get Stable Automation  Make the system under test automation-friendly  Use "active" timing always wait for something, never just wait a fixed amount of time ("passive timing")  Test your automation  Use automation to identify differences between versions of the system under test  Keep an eye on the test design
    • 24. © 2013 LogiGear Corporation. All Rights Reserved  Look for properties a human user can't see, but a test tool can  This approach is a must-do for speedier and more stable automation interface mapping is often bottleneck, and source of maintenance problems with predefined identifying property values an interface map can be created without "spy" tools not sensitive to changes in the system under test not sensitive to languages and localizations  Examples: "id" attribute for HTML elements "name" field for Java controls "AccessibleName" property in .Net controls (see below) Automation-friendly design: hidden properties
    • 25. © 2013 LogiGear Corporation. All Rights Reserved Environments, configurations  Many factors can influence details of automation language, localization hardware version of the system under test system components, like OS or browser  Test design can reflect these certain test modules are more general others are specific, for example for a language  But for tests that do not care about the differences, the automation just needs to "deal" with them shield them from the tests
    • 26. © 2013 LogiGear Corporation. All Rights Reserved Capture variations of the system under test in the actions and interface definitions, rather than in the tests (unless relevant there). Can be a feature in a test playback tool, or something you do with a global variable or setting. Variation Variation Variation "Variations" "Master Switch" Actions, Interface Definitions . . .
    • 27. © 2013 LogiGear Corporation. All Rights Reserved Possible set up of variations linked variation keyworded variation Specify for example in a dialog when you start an execution:
    • 28. © 2013 LogiGear Corporation. All Rights Reserved Virtualization  Virtual machines rather than physical machines allow "guest" systems to operate on a "host" system host can be Windows, Linux, etc, but also a specialized "hypervisor" the hypervisor can be "hosted" or "bare metal"  Main providers: VMWare: ESX and ESXi Microsoft: Hyper-V (readily available in Windows 8) Oracle/Sun: Virtual Box Citrix: Xen (open source)  Hardware support gets common now processor, chipset, i/o for example "slat" (second level address translation, helps map addresses to memory)  For most testing purposes you need virtual clients, not virtual servers most offerings in the market currently target virtual servers, particularly data centers  Virtual clients will become more mainstream with the coming of VM's as part of regular operating systems Windows 8: Hyper-V Linux: KVM
    • 29. © 2013 LogiGear Corporation. All Rights Reserved Virtualization, a testers dream...  In particular for functional testing  Much easier to define and create needed configurations you basically just need storage managing this is your next challenge  One stored configuration can be re-used over and over again  The VM can always start "fresh", in particular with fresh base data (either server or client) specified state, for example to repeat a particular problematic automation situation  Can take "snap shots" of situations, for analysis of problems  Can use automation itself to select and start/stop suitable VM's for example using actions for this or letting an overnight or continuous build take care of this
    • 30. © 2013 LogiGear Corporation. All Rights Reserved Virtualization, bad dream?  Performance, response times, capacities  Virtual machine latency can add timing problems see next slide can be derailing in big test runs  Management of images images can be (very) large, and difficult to store and move around • there can be many, with numbers growing combinatorial style • configuration in the VM can have an impact, like fixed/growing virtual disks distinguish between managed configurations and sandboxes define ownership, organize it IT may be the one giving out (running) VM's, restricting your flexibility  Managing running tests in virtual machines can take additional efforts on top of managing the VM's themselves with the luxury of having VM's the number of executing machines can increase rapidly one approach: let longer running tests report their progress to a central monitoring service (various tools have features for this)
    • 31. © 2013 LogiGear Corporation. All Rights Reserved Virtual machines, capacity  Key to pricing is number of VM's that can run in parallel on a physical machine  An automated test execution will typically keep a VM more busy than human use  Factors in determining VM/PM ratio: memory, for guest OS, AUT, test tooling storage devices (physical devices, not disk images) processors, processor cores specific hardware support (becoming more common) • processor, chipset, I/O We started regression with 140 VMs. Very slow performance of Citrix VM clients.
    • 32. © 2013 LogiGear Corporation. All Rights Reserved Globalization  Three Challenges: other countries, other cultures geographic distances time differences  Seven "Patterns": "Solution" "Push Back" "Time Pressure" "Surprises" "Ownership" "Mythical Man Month" "Cooperation" other country surprise
    • 33. © 2013 LogiGear Corporation. All Rights Reserved Three Challenges  Other country tricky, but can be managed study and prepare (culture, governance, etc) have a clear company culture to complement regional (and professional) culture  Distance cumbersome have good (great) communication features, at all levels  Time difference this is the biggest problem, comes back every day as much as possible have local management and responsibility company culture
    • 34. © 2013 LogiGear Corporation. All Rights Reserved Seven patterns  "Mythical Man Month" more (inexpensive) resources does not mean faster timelines  "Solution" team "solves" problems rather than asking  "Push Back" on-shore side dictates work and practices in detail tell teams to push back if necessary  "Time Pressure" deadlines are quite often imposed without need  "Surprises" bad news is communicated too late  "Ownership" establish clear ownership: shared responsibility is no responsibility  "Cooperation" distance creates confusion and conflict (human nature) tip: work with briefs and agendas for meetings
    • 35. © 2013 LogiGear Corporation. All Rights Reserved Industrial Organization  Large scale testing can move from a "design" to a "production" focus mostly applies to test execution, but also seen for test development this not black and white, both paradigms can occur in the same projects  A production organization is different a development organization different professional culture emphasis more on delivery and scale, "thinking big" discipline rather than creativity, "get stuff done" activities are like planning, control, logistics, information  Responsibilities Keeping the tests running Allocating resources Respond to hick-ups Analyze and address automation issues
    • 36. © 2013 LogiGear Corporation. All Rights Reserved Summary  Not all "big project" challenges are the same  Think before you do. Best results come from combining effective concepts, tricks and tools  Consider tests and automation as products that need planning and design  Virtualization can be a solution, but also a challenge  Globalization can help scalability, but needs knowledge and attention  Big testing projects typically have a "production" focus, in addition to test development and automation
    • 37. Questions? @esconfs #esconfs
    • 38. EuroSTAR Webinar Archive Access almost 70 on-demand software testing webinars by the world's leading testing experts. Topics include…. Agile, Automation, Test Management, Process Improvements, Risk-based Testing, Performance Testing, Test Documentation, Cloud Testing, Session-based Testing, Weekend Testers Visit the archive! @esconfs #esconfs
    • 39. The EuroSTAR Community. Are you a member? Did you know we have an online resource area with almost … 600 software testing presentations, 70 topical webinars, over 40 videos and podcasts and 25 expert eBooks Do you know that membership is completely free and you can access these premium testing resources anytime? Join today! member @esconfs #esconfs
    • 40. Are you coming to Gothenburg? Join us in the Swedish Exhibition & Congress Centre, Gothenburg, Sweden from the 04-07 November for the 21st annual EuroSTAR Conference. Visit for more information on what you can expect to see at Europe’s largest software testing event! @esconfs #esconfs Book before Sept
    • 41. Have you seen the latest competition? You and 3 of your team could be joining us at the 2013 EuroSTAR Conference! Entering is easy! Just send us a short video (2 minutes approx.) demonstrating how your team works together to meet goals or solve problems. Deadline for entries is Wednesday, August 14th! More information here: #esconfs
    • 42. Don’t miss these great savings, book your ticket today! Plus Make even bigger savings when you register for the Group discount where every 5th attendee goes free! For more information on the Conference Programme and pricing options go to #esconfs