Tackling Barriers in Multi-Customer Contract Acceptance Testing<br />Maaret Pyhäjärvi, Ilmarinen<br />Email: <maaret@iki.f...
Or “Why Can’t You Just Let Us Test!”<br />Maaret Pyhäjärvi, Ilmarinen<br />Email: <maaret@iki.fi> | Twitter: maaretp<br />...
About the Speaker: Experiences in testing and being a tester on varied basis<br />Started with testing -95: localization t...
The Project Scope<br /><ul><li>One vs. 3-14 owner organizations, multiple contractors
Contract-acceptance testing in the end of the project
Data-intensive system</li></ul>US<br />Procured for us in mid-80’s<br />(Cobol)<br />Procured for us in 90’s<br />(Cobol, ...
The Acceptance Testing<br />Contractually ”must act as a single customer” towards the contractor<br />Needs in use same on...
Wanting to Try Something Different<br />Management in own organization<br /> with limited people hours available, buy-in ...
Framework of Management<br />Vision (“Sandbox”)<br />Current Charter<br />Other Charters<br />Details<br />Coaching<br />Q...
The Pieces in Management Framework<br />A disciplined tester replanning on various levels<br />Session with charter that p...
What We Did <br />Phase 1: Inform and negotiate<br />Suggested test data oriented metrics, different form of documenting (...
Lessons for Improving Our Testing (1/2)<br />Adding and dropping tests<br />Vague idea of number of tests made it easier f...
Lessons for Improving Our Testing (2/2)<br />Working with data that was inaccessible due to rights until the system under ...
The Excuses<br />It’s not us that want these information, it’s your managers<br />Managers did not say how they want they ...
Upcoming SlideShare
Loading in …5
×

Tackling Barriers in Multi-Customer Contract Acceptance Testing (or Why Can't You Just Let Us Test!)

799 views

Published on

Presentation given in CAST 2011 in Seattle 10.8.2011.

Published in: Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
799
On SlideShare
0
From Embeds
0
Number of Embeds
20
Actions
Shares
0
Downloads
11
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Exploratory testing is an effective and efficient way to organize testing working in collaboration with the developers. However, the pension insurance sector is known for long waterfall-like projects, where a customer procures the changes to the system from a contractor with fixed-price contracts. The customer’s contract-acceptance testing happens in a limited timeframe at the end of the project with scripted test cases. In this presentation, I go through a project from the insurance pension sector in Finland, where we made an effort to change multi-customer contract acceptance testing from pre-scripted test cases created from specifications to data-oriented exploratory testing. Within the project we faced several objections and lost many battles of doing things our way. Keeping the goals of sector collaboration and good testing in mind, we sought compromises and learned valuable lessons for improving our testing.James asked me to talk about something I’m passionate about – and wasted effort and bad testing resulting in test case design phases for the acceptance testing efforts is definitely that. That’s the reason I joined Ilmarinen / Pension Insurance sector – to change the testing so that it makes a financial difference on a sector that is known for extreme plan-orientation. Theme: context-driven testing. Not doing things the same when people and needs are essentially different. Understanding the real needs instead of the presumed solution.
  • Allocating responsibilities in relevant ways happened 2/3 into testing. The coordinating organization learns too.
  • Tackling Barriers in Multi-Customer Contract Acceptance Testing (or Why Can't You Just Let Us Test!)

    1. 1. Tackling Barriers in Multi-Customer Contract Acceptance Testing<br />Maaret Pyhäjärvi, Ilmarinen<br />Email: <maaret@iki.fi> | Twitter: maaretp<br />CAST 2011 @ Seattle<br />(2011-08)<br />
    2. 2. Or “Why Can’t You Just Let Us Test!”<br />Maaret Pyhäjärvi, Ilmarinen<br />Email: <maaret@iki.fi> | Twitter: maaretp<br />CAST 2011 @ Seattle<br />(2011-08)<br />
    3. 3. About the Speaker: Experiences in testing and being a tester on varied basis<br />Started with testing -95: localization testing for greek version of an English software system<br />Subcontractor-side on various projects: moving from localizations to functional testing<br />Tried being in a developer role after thinking testers are not respected: the problem is not per role<br />Teaching testing at HUT, giving public presentations and courses on testing<br />Researcher in testing, consulting various organizations, teaching, being a test manager and a tester, with and without test automation<br />At F-Secure with product business 2005 – 2008 <br />Moved to insurance pension sector, first with subcontractor side, now on customer side (Ilmarinen) as test manager / test specialist. <br />Doing testing-related training on the side of a normal day-to-day job to get to meet brilliant testers in Finland. <br />TESTER AND A WALKING TESTING DICTIONARY<br />AGILE PROJECTS @ F-SECURE<br />PLAN-DRIVEN PROJECTS @ ILMARINEN<br />
    4. 4. The Project Scope<br /><ul><li>One vs. 3-14 owner organizations, multiple contractors
    5. 5. Contract-acceptance testing in the end of the project
    6. 6. Data-intensive system</li></ul>US<br />Procured for us in mid-80’s<br />(Cobol)<br />Procured for us in 90’s<br />(Cobol, J2EE)<br />Procurement for 3 customers ongoing (Java/ Soap)<br />Procured since 80’s for 14 customers (Cobol, Java/Soap)<br />
    7. 7. The Acceptance Testing<br />Contractually ”must act as a single customer” towards the contractor<br />Needs in use same on principle level, practice does not follow the principle strictly<br />Testing-wise everyone tests how they use the system for acceptance<br />Shared reporting, based on scripted testing methodology<br />Long test planning phase with reviews, counting numbers of passed / failed / not run tests<br />
    8. 8. Wanting to Try Something Different<br />Management in own organization<br /> with limited people hours available, buy-in for <br /> exploratory testing was not difficult (1:3 investment ratio)<br />- Most relevant factor: reputation for test management<br />Individual contractors<br /> Acceptance testing is customer territory, reporting<br />- Most relevant factor: contract not requiring <br /> detailed test cases with fixed data<br />Customer procurement facilitator organization<br /> no buy-in – request denied. Everyone must work <br />with same methodology and formats. <br />
    9. 9. Framework of Management<br />Vision (“Sandbox”)<br />Current Charter<br />Other Charters<br />Details<br />Coaching<br />Quality Report<br />Perception of quality and coverage<br />Debriefing<br />Charter backlog of the future testing<br />”A day’s work”<br />Past<br />Results<br />Obstacles<br />Outlook<br />Feelings<br />Next in importance!<br />Test Manager<br />Metrics summary<br />#<br />?<br />x<br />Out of budget<br />#, ?, x, +<br />20:20:60<br />Session sheets of the past testing<br />Bug Reports<br />Idea of exploration<br />Tester<br />7<br />Playbooks<br />
    10. 10. The Pieces in Management Framework<br />A disciplined tester replanning on various levels<br />Session with charter that provides a report<br />Classification of information created as metrics<br />Prioritizing of what test idea comes next<br />Supporting reporting by debriefing<br />Supporting skills development by coaching<br />Supporting future needs to remember with playbooks<br />Creating a combined judgement of quality by quality reporting<br />8<br />
    11. 11. What We Did <br />Phase 1: Inform and negotiate<br />Suggested test data oriented metrics, different form of documenting (not from spec but from data)<br />Result: ultimatum - our way or not at all<br />Phase 2a: Scope down the visible scale of testing & documenting<br />Example area: 53 test cases (P1 – visible) and 184 (P2 – not visible)<br />Minimal energy principle in reviews: ”thanks for feedback, we decide if we change or not”<br />Disobeying: ”add test case before writing a bug report”<br />Our written test cases were bad – just as anyone else’s except for one<br />Phase 2b: Defer commitment with test cases<br />Vacation as an excuse, then delivering 2 documented that included 255 individuals after execution<br />Phase 3: Add visibility to the hidden<br />Start showing P2 numbers in reports, without explaining they don’t mean the same<br />One-on-one discussions within customer organizations to avoid duplication<br />Supporting our own manager’s continuously with information of value vs. cost. They needed to know more than usually, playing politics. <br />Phase 4: Followup<br />Continously emphasizing the opportunity cost<br />Explaining what really happened & results<br />
    12. 12. Lessons for Improving Our Testing (1/2)<br />Adding and dropping tests<br />Vague idea of number of tests made it easier for testers to not do things that were not valuable and add things that were needed<br />Testing halfs-of-tests<br />”Start of process” could be tested much earlier in non-data destroying way than the whole process<br />The user interfaces may just not be as ready as they should... <br />One test to find things in the masses to guide other tests<br />Identifying one out of ”essentially same” is different helped in focusing more effort on the one that needed the effort without wasting it equally on all<br />Creating unplanned scenarios with production-like use<br />Identified crashing problems that did not exist until enough times the system was unaccessible for days during testing<br />
    13. 13. Lessons for Improving Our Testing (2/2)<br />Working with data that was inaccessible due to rights until the system under test would deliver it<br />Identified data that we did not seem to ever get<br />Efficiency in test data selection process (avg 2 man-days / data sample) – data outdates easily<br />Spec-based test cases miss relevant information for us<br />Spec  defect – contractor pays ;Other  change – customer(s) pay<br />Regression testing became relevant for us<br />Not focusing on checklist, but check after various changes<br />Replacing ”test case” with a ”test day” <br />Time for testing if did not need to stop for bugs<br />Guiding experts to disagree on scope and resolving the disagreement<br />
    14. 14. The Excuses<br />It’s not us that want these information, it’s your managers<br />Managers did not say how they want they want the information, they just wanted to know if we’ll make it. <br />Lack of creativity in the test management layer is eminent, not enough practical examples how how (without detailed session based testing) you could control it. <br />Coordinator had little idea of how the customer organizations will use the information system provides, could not control based on contents<br />End users as tester’s will not tell what they did if not planned in advance and provided as reference<br />Test cases needed in bug reports<br />
    15. 15. The Real Barriers<br />Collaborating with various groups that are testing simultaneously<br />Controlling the schedule<br />Contractual allocations of responsibilities and costs<br />Many possible solutions without compromising these. <br />
    16. 16. To End This With<br />A long way to go<br />Allowing us to do exploratory testing constantly<br />Enabling our organization not to use the permission as not preparing<br />Enabling other customer organizations to do exploratory testing within their acceptance testing<br />Enabling the contractors to exploratory testing in any scale<br />When asked to do a routine that does not provide value, try first doing less of it<br />

    ×