When the Phoenix Pay system was released in April 2016, it had severe problems affecting some 70% of the people who the system was intended to pay. It has been fraught with issues ever since resulting in under and overpayments of those employees, and in some cases no payments at all. Hundreds of millions of dollars have been and will be spent to correct the pay issues and update the system. This was yet another example of the ineffectiveness of the antiquated approach that the government used to deliver Phoenix. When the 2018 budget was tabled, nearly half a BILLION dollars was allocated to fixing Phoenix! What seemed like a footnote to that was $16 million over two years to study how to build the replacement system. Study. That's what finally broke me. I had been grumbling at the stories of Phoenix for years, but this finally triggered me to do something.

1. From the Ashes of Phoenix

2. Um, who is this guy?
Old enough to be alive before colour was invented
30+ years in software development
15 of those years in federal government
departments and crown corporations
Part of the XP & Agile movement since the turn of
the century
Been there, done that enough to know when an
approach is doomed to failure

3. Once upon a fiasco...
Another $431 Million to deal with issues until a new system
could be completed
$16 Million over two years to study how to implement a replacement
system
New system expected to be completed in 2025

4. Groundhog Day?
Canadian Firearms Program
• $119M estimate in 1995, $688M actual by 2002
• replaced by a system estimated in 2000 to take 8 months
• replacement started in 2002, “redefined” in 2005, delivered in 2006
Global Case Management System
• $195M estimate in 2000, to be completed by 2007
• $387M actual, delivered in 2011
Secure Channel
• $98M estimate in 1999
• $400 actual by 2004 with $100M per year operating costs

5. …or the Definition of Insanity?!

6. Expecting a different outcome using the same approach
Mandate the use of COTS (Peoplesoft in this case)
Spend multiple years dreaming up every possible requirement
Issue a massive RFP for which only massive systems integrators qualify to reply
Take months if not years to evaluate the bids
Select the winning massive integrator who, by the way, low-balled their bid
Begin a massive waterfall project, calling it Agile because standups & sprints
Deal with the tsunami of change requests as new requirements are discovered
during implementation of the system

7. So how’s that workin’ for ya?
Five years of development
Delayed a year at IBM’s request due
to critical issues
Released in March 2016
Cost of $450M vs. original budget of
$50M
By July 2016, 80,000 employees had
issues with their pay

8. What Led to This? Two key assumptions!
“Pay is really hard!”
Over 80,000 business rules according to one source
Integrations with dozens of other systems
“This system is YUGE!”
300,000 people need to be paid by the system

9. Neither of these assumptions
necessitate that the project needs
to be massive, with a massive team
fielded from a massive company,
with an associated massive budget.

10. Pay is Complicated, but not Complex
For a given input you can accurately predict the
output (at least you’d better be!)
The problem has been solved before!

12. Size isn’t everything!
300,000 people need to be paid, so yes the data
volumes are large
That doesn’t mean the system and team needs to
reflect the data!

13. Surely this is problem is unique!

14. This isn’t our first rodeo
Initial version of healthcare.gov in the US launched in October 2013
Only 6 people were able to sign up on the first day
Major aspects of the system simply didn’t work
A large team from large integrators building a large system with a large
budget, and it didn’t work… shocking

15. What did they learn from that?
Faced with failure, leadership challenged the assumption that everything had to
be large
Small group recruited from the private sector to help perform emergency work
to get the system processing enrolments
Second small group took over and replaced applications written by the original
teams

16. A small team was able to replace in
the order of months what a large
team could barely deliver over
years, and they did so at a fraction
of the cost.

17. How Do We Fix This Mess? A Proposal.
Think Small!
Handpicked team of maybe a dozen people with the skills
necessary to deliver
Only grow when there’s enough pain to warrant it
Think “Pink”!
Give that team Autonomy, Mastery and Purpose!

18. Autonomy
Have complete autonomy regarding the process used to deliver
the system;
Have complete autonomy regarding the system architecture and
technologies used;
Be ridiculously transparent in their operation with respect to
their progress and what work is being done

19. Mastery
Have direct access to the people who would be the consumers of the system
in order to ensure that the system would work effectively for them;
Have direct access to the subject matter experts in order to ensure that the
system is properly handling the business rules;
Have direct access to the people responsible for any systems with which the
new system would have to integrate;
Have the constant, unwavering support from management at the Deputy
Minister and Ministerial levels of the public service

20. Purpose
People are dedicated to the team full-time with no other
responsibilities
The goal of the team is clear - deliver a reliable, working
compensation system for the federal public service
Leverage any and all experience to bring creative solutions to the
effort

21. When eating an elephant
take one bite at a time.
-- U.S. Army General Creighton Abrams

22. Eating the Elephant
Start with discovery workshops with as many stakeholders and SMEs as possible
Ideally, hire Jeff Patton to facilitate the workshops!
Yes, that means duplicating the requirements effort
If the requirements were already sound then no change requests would have been
necessary
Exit the workshops with a shared understanding of the work to be done, a high-level
view of the system architecture, and a Story Map for the first significant pieces of work

23. The First Bite
Pay one person. Yes, one. No, really… one.
Build just enough to process the pay for a single employee and iterate
from there
Exposes unknown or unexpected requirements early
Mitigates risks to the project early
Implement automated testing infrastructure and feedback from the start,
dramatically increasing the system quality

24. Ridiculous Transparency
Make progress public, i.e. a publicly accessible web site
Show the metrics, whether they’re good or bad
Provide the option of cancelling the project at any time
Get the system into the hands of its consumers as early and often as
possible
Incorporate feedback as quickly as possible
Build trust.

25. Constant Reflection for Improvement
Examine the team’s process to determine how to improve
Ensure that improvement activities are given proper priority
Quantify the improvements when possible
Articulate the qualitative improvements

26. Real artists ship.
-- Steve Jobs

27. Questions?
Dave Rooney
@daverooneyca on Twitter
https://www.linkedin.com/in/daverooneyagile/
https://medium.com/@daverooneyca