At the heart of the matter was one word – QUALITY (or Kvalitet in Sweden). “Your software isn’t good enough” was the assertion from the customer. “You’re being unreasonable” was the retort from the vendor. “You are our last chance” was the message we received from the customer’s CEO. “You”, being a small specialist (SWOT) team comprising, a Senior Project Manager, a Systems Architect, a Technical Project Lead, a Senior Business Analyst and a Senior Testing Consultant. This is not an unfamiliar scenario for those of us who have seen the software landscape change from bespoke to prêt-à-porter solutions during the past 15 to 20 years. It is also not an unfamiliar outcome for lawyers to get rich in these situations and for the customer to lose credibility in the marketplace. In May 2009 I was asked to assist with a Transport Ticketing project that had been struggling to Go Live for the previous 2 years (the overall project lifespan was 7 years). The solution was 98% ready - just 360 requirements of the original 16,000 required sign-off. This had been the situation for over 12 months, with no agreed end in sight. How could they be so close and yet so far away? How we achieved (what appeared impossible just seven short months earlier) is the story I will tell at EuroSTAR 2011. How we turned “your software isn’t good enough” into “we’re going live next Saturday” will never be a Nordic legend, but it will be remembered as the project that saved a software vendor from bankruptcy and another major city from a multi-million dollar failure. At the centre of this story is how we re-focused everyone on QUALITY/KVALITET. What’s essential, what’s negotiable, what’s unacceptable? These questions were central to our success. Our team didn’t identify any new requirements or write any new software. We just analysed the available data, spoke to the right people, re-worked the Test Strategy and agreed on a new GO LIVE strategy with an agreed date – and then MADE IT HAPPEN. An interesting Footnote from this Case Study is that the Customer hired/payed us (and expected us to deliver) – so we got little in the way of recognition. The vendor was over the moon, as we had saved them from an expensive lawsuit and (almost certain) bankruptcy. The vendor still calls us. The customer experienced NO significant outages – KVALITET!!

1. You say , I say
Stories behind establishing Quality Benchmarks
© 2011 Capgemini. All rights reserved. 1
Colin Cherry
Director, Testing Services
Capgemini Australia

2. Content
The Background Story
The Story about National & Corporate Cultures
The Story about meeting the Players
The Story about Gathering the Evidence
The Story about Analysing the Risk
The Story about Root Cause Analysis
The Story about a new Belief System
The Story about achieving the Goal
The Story about Changed Perceptions
The Story about a Changed Reality
The Story about thanking my (new) friends
© 2011 Capgemini. All rights reserved. 2

3. 1. A finely balanced project involving a Swedish Transport Authority & a
major Australian Software provider in a strained relationship
2. A Steering Committee running out of patience & a CEO who wanted
one last chance to implement a new SMART transport ticketing system
3. A press highlighting every new delay
4. A project with more than 16,000 (fully documented) requirements that
had been running for over 7 years
5. A solution that had almost gone live twice before
6. A team of 5 (external) Senior Consultants from a leading Systems
Integrator, given 6 months to get the ticketing solution LIVE
7. A phone call from Sydney, a flight to Perth (followed by a slightly longer
one to Stockholm)
The Background Story
© 2011 Capgemini. All rights reserved. 3

4. The Story about National & Corporate Cultures
© 2011 Capgemini. All rights reserved. 4

5. 1. Meet the Vendor, the Client, the Consultants & Identify the Key Players
2. Sort through the (last 7 years) information – what’s important & what’s not
3. Perform a SWAT followed by a Root Cause Analysis
4. Assess the gathered information
5. Develop a Plan of Attack
The Story about meeting the Players
© 2011 Capgemini. All rights reserved. 5

6. The Story about Gathering the Evidence
© 2011 Capgemini. All rights reserved. 6
July 15
Compare Vendor &
Customer Metrics

7. The Story about Gathering the Evidence
© 2011 Capgemini. All rights reserved. 7
HPQC Structure

8. The Story about Analysing the Risk
© 2011 Capgemini. All rights reserved. 8
July 29
Software Version 3.2.3

9. The Story about Root Cause Analysis
© 2011 Capgemini. All rights reserved. 9

10. The Story about Root Cause Analysis
© 2011 Capgemini. All rights reserved. 10
Triage
Process
Install
Process
Variance found &
confirmed by the client
Variance logged in HPQC;
Type =
Observation, Status =
New
Client Vendor Client
HPQC automatically
generates EMAIL to Vendor
DEV MGR (with enough
detail for Vendor assessment
& fix)
Vendor updates HPQC
with comments to accept or
reject Observation; Status
= Rejected or Open
If client disagrees with
rejection, Type = Bug
Status = Open
If Vendor has (existing) Bug
with same
symptoms, Vendor adds
comments to HPQC Bug
record
Rejected
Rejection overturned
Accepted
Vendor Internal Bug /
Fix Management
Process
Vendor fix tested
OK, Status = Vendor
Retest Successful +
HPQC generates EMAIL
(PC) to client Test
Manager
SL & Vendor conduct
daily Bug Status
reviews & update
HPQC accordingly
XX retest; if OK, Status =
XX Retest Successful;
else Status = Retest
Failed (HPQC generates
EMAIL PC to Vendor DEV
MGR)Retest
Failed
Triage Observation; assign
Critical & Major Bugs to
Vendor DEV MGR; Minor +
Trivial Bugs + Observations
stay at SL
Critical Major
Minor & Trivial Bugs
monitored internally until
released to Vendor.
Observations “watched”
Minor&TrivialBugorObservation
Client sets Status to
Closed
Rejection
Confirmed

11. The Story about Root Cause Analysis
© 2011 Capgemini. All rights reserved. 11
Trend: More Re-test Failures as we close in on LIVE Date

12. The Story about a new Belief System
© 2011 Capgemini. All rights reserved. 12
Probability
6 Frequent The risk is likely to occur frequently. The risk
will be experienced hourly or more often.
5 Probable The risk will occur several times. The risk is
expected to occur at least once a week.
4 Occasional The risk is likely to occur several times. The
risk is likely to occur once a month.
3 Remote The risk is likely to occur sometime in the
lifetime of the product; maybe once per 10
years.
2 Improbable The risk is unlikely to occur but is possible;
maybe once per 100 years.
1 Incredible The risk is extremely unlikely to occur; maybe
once in a 1,000 years.
Consequence
4 Significant Immobilizing failure that prevents
a service being provided
3 Major A service failure that must be
rectified to ensure a significant
service outage does not occur
2 Minor A service failure that reduces the
effectiveness of a service
1 Insignificant A failure that does not impact the
provision of a service

13. The Story about achieving the Goal
© 2011 Capgemini. All rights reserved. 13
CodeFreeze
Production Today
Release 3.2.3.1
Production Nov 15
Release “3.2.3.X”
STEP ZERO
Production Sept 1
Release 3.2.3.2
Production Sept 15
Release 3.2.3.3
Acceptance Sept 1
Release 3.2.3.2
+ Patches
Acceptance Sept 15
Release “3.2.3.X-1”
Acceptance Oct 1
Release “3.2.3.X”
Acceptance Today
Release 3.2.3.2
Acceptance Nov 15
Release 3.2.3.X
FinalSAT
RegressionTest
Consolidate Patches
into Single Release

14. The Story about achieving the Goal
© 2011 Capgemini. All rights reserved. 14

15. The Story about Changed Perceptions
© 2011 Capgemini. All rights reserved. 15
We have been LIVE for almost 1 week, but are still finding new Bugs

16. The Story about a Changed Reality
© 2011 Capgemini. All rights reserved. 16
1. We realised that the Vendor & the Client were not comparing Apples with Apples
• The vendor was producing Release Notes, but not explaining to the customer how to
understand & use them (JIRA was the vendor Release Management tool)
• The customer wasn’t forceful enough in asking for the Release Notes to be provided in
a way that was usable (HPQC was the customer Test Management tool)
2. We improved visibility of what was really happening
• We got the Vendor & the Customer to agree on a single source of truth – HPQC
• We turned HPQC into a Release Management tool
3. We got the Vendor to send Testers to the Customer’s site & the Customer’s Test
Manager to visit the Vendor (for the first time)
4. We created a joint vision of what was required for Go Live, built a consolidated
Plan to achieve it & then executed the Plan
5. We provided the GLUE

17. The Story about Thanking my (new) Friends
© 2011 Capgemini. All rights reserved. 17
1. The Consultants:
• Magnus Wikholm, Juraj Lesko, Martin Torebring, Maria Ander
2. The Customer:
• Lars Bromander, Anders Nillson, Thomas Enjin, Torkel Strahle, Milena Haykowska
3. The Vendor:
• Kim Fitzpatrick, Russell Ascott, Sonia Libao, Simon Morley, Brian Roberts, Jerico
Ubalde

18. Thank You…
© 2011 Capgemini. All rights reserved. 18
Colin Cherry
Director
Testing Services
Capgemini Australia Pty Ltd
Level 2, 477 Collins Street
Melbourne VIC 3000 Australia
Tel: +61 3 9613 3138
Mob: +61 412 214 240
colin.cherry@capgemini.com