Inverting The Testing Pyramid

2. Plan Back in the Stone-age Design Happiness/Excitement Distribute Work in Isolation Integrate Time/Money/Opportunity Cost

4. BAD things were visible too Late...

5. Birth of CI

9. Brian Marick’s Test Categorization Before/While Coding Post Coding Business Facing Supports Programming Critique product Technology/Implementation Facing Copyright © 2013, AgileFAQs. All Rights Reserved.

10. It Helps to Think of Tests this way... Business Facing Acceptance Testing Exploratory Testing Drives Development Critique product Low-ﬁ prototypes UI and Usability Testing Performance Testing Unit Testing System Tests Technology/Implementation Facing Copyright © 2013, AgileFAQs. All Rights Reserved.

12. Lean-Start-up Community Tried something quite disruptive...

16. But...

18. Commercial Break!

20. Tech Talks!

23. Inverting the Testing Pyramid Typical testing strategies lead to an inverted testing pyramid... Manual Checking End-to-End GUI Tests 80-90% Integration 5-15% Tests Unit Tests 1-5% Copyright © 2013, AgileFAQs. All Rights Reserved.

24. Inverting the Testing Pyramid GUI 1% Tests Performance Tests End to End 4% Security Flow Tests Tests One Layer Below GUI Workﬂow Tests 6% Integration Test 9% Biz Logic Acceptance Tests 10% Unit Tests 70% This is the need of the hours... Copyright © 2013, AgileFAQs. All Rights Reserved.

28. Types of Tests in the Testing Pyramid GUI Tests GUI 1% Tests Performance Acceptance Tests Tests End to End 4% Security Flow Tests Tests One Layer Below GUI Unit Tests Workﬂow Tests 6% Small Agile Projects with Single Team Integration Test 9% Biz Logic Acceptance Tests 10% Unit Tests 70% Large Scale Agile Projects with Global Teams Copyright © 2013, AgileFAQs. All Rights Reserved.

29. Types of Tests for Large Agile Project Type Tool Who When Level Objective xUnit, Mockito/ Each class functions correctly in isolation. The During the Iteration, while Unit Tests Moq, Jasmine, Developers coding Class UI components (including JS) can also be unit tested in isolation. HSqlDB, Jumble Biz Logic Cucumber (some TechBAs + Acceptance Criteria are Work-steam specific From a business logic point of view, the functionality is implemented correctly. Works defined before Iteration starts. Acceptance may need Lisa) Developers + Acceptance Tests are written business functionality in isolation with other work-streams/modules. Also good point to do some basic Tests JMeter/LoadRunner Testers (QA) during Iteration, before coding (At component level) performance testing. Contracts are defined before Between dependencies Integration Developers + 2 Components can communicate with each Iteration, tests are (work-streams, other correctly. Best suited for negative path xUnit + Lisa Testers (QA) + Test TechBAs implemented during the Iteration upstream/down-stream testing. Should not test any functional aspects of the application. systems, 3rd Party) SMEs + TechBAs + UX Across a few work- Inside a business operation, a particular step Workflow When a theme is picked (business workflow) works correctly from the Cucumber + Lisa + Developers + Arch (functional group of Iterations) streams (Issuance/ Tests + Testers (QA + UAT) Acceptance) business point of view. Upstream/Downstream dependencies stubbed. When an Internal Milestone is SMEs + TechBAs + UX decided, the E2E flow tests are Ensure a specific type of business operation End to End Cucumber defined & authored. As & when End to end Trade flow, can go thru all the steps successfully. + Developers + Arch + the complete workflow is Flow Tests JMeter/LoadRunner Testers (QA + UAT) implemented these tests are one layer below the UI Very important to do detailed performance testing at this stage. enabled. Usually its takes 3-4 Iterations Selenium/Sahi/ to complete one whole TechBAs + UX + End to end Trade flow Ensure all the basicthrough all the steps types of business GUI Tests Watir/White/ workflow including the UI. operations can go SWTBot Testers (QA) After that the UI tests will be with GUI successfully via a UI. authored and executed. Copyright © 2013, AgileFAQs. All Rights Reserved.

30. “Testing” Touch Points during Project Lifecycle Work- Internal Product Release 1st Release Iteration Iteration Themes Stream Iteration Milestone Release Discovery Planning Planning Review Planning Release ➡Collaborate to ➡Ensure themes ➡Validate Acceptance ➡Show demo (parts) ➡Sanity Testing in create the story map are coherent criteria at story level ➡Confirm that the biz Live environment ➡Help SMEs to ➡Create theme ➡Estimate Tasks requirements are met ➡Participate in define Acceptance level epics with stakeholders Release retros Criteria at each level ➡Author End-to- ➡Participate in in story map end flow tests Iteration retrospectives ➡Define goals & ➡Author Work-flow tests ➡Collaborate with Dev ➡Sanity Testing in ER cycles readiness criteria ➡Collaborate on identifying & Biz to write Automated ➡Confirm user goals are for each Release. dependency and Acceptance Test met with operations team documenting them ➡Exploratory Testing ➡Participate in Internal ➡Create stream level epics ➡Buddy testing Release retrospective ➡Create Integration Test ➡UI automation of previous Iteration tasks ➡Performance & Security Tests ➡ The Continuous Integration and Build Pipeline process will execute all tests as part of the regression pack and report various quality parameters to all stakeholders. ➡ On going Governance & Test Data management Copyright © 2013, AgileFAQs. All Rights Reserved.

31. Inverting the Pyramid How to…?

32. 1 Acceptance Test/Behavior Driven Development

33. Acceptance Test Driven Development Iteration Automated Acceptance Acceptance Tests Criteria User Story Automated Unit Test Automated UI Tests P E Automated R F Acceptance Tests O R M Acceptance E Exploratory Criteria N T Testing C E E S T Copyright © 2013, AgileFAQs. All Rights Reserved. S

34. 2 Acceptance Criteria Driven Discovery

35. Acceptance Criteria Driven Product Discovery Personas Elevator Business Pragmatic User A Pitch Goals Personas Goals c Chartering c Day in Life e Scenarios & p of each Narratives Persona t a Participants n Product Manager/Owners Story Mapping Activity Map c SMEs Operation/Support Reps e Tech BAs UI Sketch Interaction Task Map User Experience Designer Design C App Developers r Architects Planning i Testers (QA+UAT) Reiterating Grouping by Prioritization t Themes e Project Managers r User Story Authoring i Acceptance User Criteria Stories a

36. Story mapping at SEP Photo courtesy of So@ware Engineering Professionals, Indianapolis IN Src: Jeﬀ Pa+on, www.AgileProductDesign.com

37. Organize & Visualize your Product’s Release Roadmap Src: Jeﬀ Pa+on, www.AgileProductDesign.com

38. 3 Continuous Integration Pipeline

40. Dev Box Continuous Integration Build Pipeline Update Local Version Checkout Dev Build Checkin Control ~5 Min Team CI Server ou t Checkout Project/Task Checkout k ec Each build will update Management Ch Smoke Ch Task statues eck Build Tool out ~10 Min Team CI Server Retrieve Functional Artifacts Build ~30 Min Retrieve Dev Env Artifacts Cross ts Publish ifac Artifacts Stream Art (jars) Build d ~60 Min ts she ac SIT Env ti f cts ifa ubli A r t r Theme ed nt A eP b lish nd e Build riev Pu ep e ve ~60 Min eD acts e Artif Ret tri ot hed QA Env Re om ublis r eP Product h/P Retriev Dependency ub lis ed Artifacts Build Management P Retrieve Publish ~60 Min UAT Env UAT Build ~30 Min Staging Env SCM Staging Each build will broadcast build stats and other reports to all Build ~30 Min relevant stakeholders at each Prod Env stage of the build pipeline Live Copyright © 2013, AgileFAQs. All Rights Reserved. Build ~30 Min

41. Local Cross Smoke Func Theme Prod UAT Staging Live Dev Stream Build Build Build Build Build Build Build Build Build Ensure the Ensure each Ensure each Ensure each story developer has Ensure each story’s implemented, implemented end-to- Ensure the product integrates with other Ensure checked in all files, business logic is complete trade end feature works as as as a whole can Ensure basic sanity streams and business Operations can the code compiles, implemented workflow works expected including work with other Deploy the product before the logic continues to execute an end to Objective developer checks all unit tests work correctly and work as expected. as expected. various performance end workflow and systems in the to the live and basic code satisfies the Interact with as and security related enterprise. Make environment. in code. Promote new it meets their quality related doneness criteria at many sub-systems SLAs are met. Also test Go/No-Go produced jars to goals. stats are reported the story level & interfaces as out the migration decision. dependent teams. to the devs. possible. scripts Integrate promoted Clean environment, code with other Clean Migrate existing system Migrate existing Retains the Migrate existing Compiles, Unit test setup database, team modules, runs cross- environment, to the latest version, system to the system to the latest environment. Only and static/dynamic specific biz team workflow & setup dependent setup dependent latest version, version, test the Migrate existing compiles and tests acceptance tests. integration tests in systems, execute systems, execute setup dependent Job locally changed code analysis on an Stubs/Mocks out an incremental env. theme-level product-level systems, let the system with other system to the latest incremental systems during the version and party! code other functional Stubs/Mocks other acceptance tests. acceptance and UI Operations team environment Enterprise Release (incremental). modules or sub- sub-systems. As few stubs/ tests. Almost no stubs/ use the system cycle. systems. Promote jars to mocks as possible. mocks. thoroughly. dependent stream. Duration ~ 5 mins. ~ 10 mins. ~ 30 mins ~ 60 mins ~ 60 mins ~ 60 mins ~ 30 mins ~ 30 mins ~ 30 mins Stake- Developer writing Developers of a Team members of a given team + Teach All functionally affected Teams + All teams + SMEs All teams + SMEs SCM + SMEs + Enterprise Wide World-wide holders the code given team BA SMEs Operations Team On successful On Successful Trigger/ Before checking in code. At least Every checkin. At least every 30 On successful On successful Smoke build. At least Functional build. At Cross-Stream On successful Theme build. At least once Each Iteration Each Enterprise Staging Build Frequency every 30 mins. mins. every 2 hours least twice each day build. At least once each day each week Release Cycle (Manually triggered for now) On developer’s Stating Where workstation Team CI Server Team CI Server Dev Environment SIT Environment QA Environment UAT Environment Environment Production Env Team produced Jars/ Fully functional Configured & packaged Configured & Configured & Team produced Integrated jars Code+Tests Wars. Static+ app with app (war) with packaged app packaged app (war) Artifacts +Config Jars, static code Dynamic Code pushed to workflow level migration scripts & (war) with with migration $$ analysis reports dependent teams Analysis reports reports traceability reports. migration scripts scripts ➡ DevOps Roles (currently missing) & Test Data strategy is very important to get this working. ➡ Overall governance model will also be required to make sure build failures are addressed immediately. Copyright © 2013, AgileFAQs. All Rights Reserved.

44. Automate 100%

45. Testing VS. Checking Testing is explorative, probing and learning oriented. Checking is confirmative (verification and validation of what we already know). The outcome of a check is simply a pass or fail result; the outcome doesn’t require human interpretation. Hence checking should be the first target for automation. James Bach points out that checking does require some element of testing; to create a check requires an act of test design, and to act upon the result of a check requires test result interpretation and learning. But its important to distinguish between the two because when people refer to Testing they really mean Checking. Why is this distinction important? Michael Bolton explains it really well: “A development strategy that emphasizes checking at the expense of testing is one in which we’ll emphasize confirmation of existing knowledge over discovery of new knowledge. That might be okay. A few checks might constitute sufficient testing for some purpose; no testing and no checking at all might even be sufficient for some purposes. But the more we emphasize checking over testing, and the less testing we do generally, the more we leave ourselves vulnerable to the Black Swan.” Copyright © 2013, AgileFAQs. All Rights Reserved.

Inverting The Testing Pyramid

Inverting The Testing Pyramid

Recommended

More Related Content

What's hot

What's hot(20)

Similar to Inverting The Testing Pyramid

Similar to Inverting The Testing Pyramid(20)

More from Naresh Jain

More from Naresh Jain(20)

Recently uploaded

Recently uploaded(20)

Inverting The Testing Pyramid