Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

From the Ashes of Phoenix - PrDeliver 2018


Published on

When the Phoenix Pay system was released in April 2016, it had severe problems affecting some 70% of the people who the system was intended to pay. It has been fraught with issues ever since resulting in under and overpayments of those employees, and in some cases no payments at all. Hundreds of millions of dollars have been and will be spent to correct the pay issues and update the system. This was yet another example of the ineffectiveness of the antiquated approach that the government used to deliver Phoenix.

When the 2018 budget was tabled, nearly half a BILLION dollars was allocated to fixing Phoenix! What seemed like a footnote to that was $16 million over two years to study how to build the replacement system.

Study. That's what finally broke me. I had been grumbling at the stories of Phoenix for years, but this finally triggered me to do something.

Published in: Software
  • Be the first to comment

  • Be the first to like this

From the Ashes of Phoenix - PrDeliver 2018

  1. 1. From the Ashes of Phoenix
  2. 2. Um, who is this guy? Old enough to be alive before colour was invented 30+ years in software development 15 of those years in federal government departments and crown corporations Part of the XP & Agile movement since the turn of the century Been there, done that enough to know when an approach is doomed to failure
  3. 3. Once upon a fiasco... Another $431 Million to deal with issues until a new system could be completed $16 Million over two years to study how to implement a replacement system New system expected to be completed in 2025
  4. 4. The Definition of Insanity
  5. 5. ...or, Groundhog Day?
  6. 6. Expecting a different outcome using the same approach Mandate the use of COTS (Peoplesoft in this case) Spend multiple years dreaming up every possible requirement Issue a massive RFP for which only massive systems integrators qualify to reply Take months if not years to evaluate the bids Select the winning massive integrator who, by the way, low-balled their bid Begin a massive waterfall project, calling it Agile because standups & sprints Deal with the tsunami of change requests as new requirements are discovered during implementation of the system
  7. 7. So how’s that workin’ for ya? Five years of development Delayed a year at IBM’s request due to critical issues Released in March 2016 Cost of $450M vs. original budget of $50M By July 2016, 80,000 employees had issues with their pay
  8. 8. What Led to This? Two key assumptions! Pay is really hard! Over 80,000 business rules according to one source Integrations with dozens of other systems This system is HUGE! 300,000 people need to be paid by the system
  9. 9. Neither of these assumptions necessitate that the project needs to be massive, with a massive team fielded from a massive company, with an associated massive budget.
  10. 10. Pay is Complicated, but not Complex For a given input you can accurately predict the output (at least you’d better be!) The problem has been solved before!
  11. 11. Size isn’t everything! 300,000 people need to be paid, so yes the data volumes are large That doesn’t mean the system and team needs to reflect the data!
  12. 12. Surely this is problem is unique!
  13. 13. This isn’t our first rodeo Initial version of in the US launched in October 2013 Only 6 people were able to sign up on the first day Major aspects of the system simply didn’t work A large team from large integrators building a large system with a large budget, and it didn’t work… shocking
  14. 14. What did they learn from that? Faced with failure, leadership challenged the assumption that everything had to be large Small group recruited from the private sector to help perform emergency work to get the system processing enrollments Second small group took over and replaced applications written by the original teams
  15. 15. A small team was able to replace in the order of months what a large team could barely deliver over years, and they did so at a fraction of the cost.
  16. 16. How Do We Fix This Mess? A Proposal. Think Small! Handpicked team of maybe a dozen people with the skills necessary to deliver Only grow when there’s enough pain to warrant it Think Pink! Give that team Autonomy, Mastery and Purpose!
  17. 17. Autonomy Have complete autonomy regarding the process used to deliver the system; Have complete autonomy regarding the system architecture and technologies used; Be ridiculously transparent in their operation with respect to their progress and what work is being done
  18. 18. Mastery Have direct access to the people who would be the consumers of the system in order to ensure that the system would work effectively for them; Have direct access to the subject matter experts in order to ensure that the system is properly handling the business rules; Have direct access to the people responsible for any systems with which the new system would have to integrate; Have the constant, unwavering support from management at the Deputy Minister and Ministerial levels of the public service
  19. 19. Purpose People are dedicated to the team full-time with no other responsibilities The goal of the team is clear - deliver a replacement for Phoenix Leverage any and all experience to bring creative solutions to the effort
  20. 20. When eating an elephant take one bite at a time. -- U.S. Army General Creighton Abrams
  21. 21. Eating the Elephant Start with discovery workshops with as many stakeholders and SMEs as possible Ideally, hire Jeff Patton to facilitate the workshops! Yes, that means duplicating the requirements effort Although, if the requirements were already sound then no change requests would have been necessary Exit the workshops with a shared understanding of the work to be done, a high-level view of the system architecture, and a Story Map for the first significant pieces of work
  22. 22. The First Bite Pay one person. Yes, one. No, really… one. Build just enough to process the pay for a single employee and iterate from there Exposes unknown or unexpected requirements early Mitigates risks to the project early Implement automated testing infrastructure and feedback from the start, dramatically increasing the system quality
  23. 23. Ridiculous Transparency Make progress public, as in a publicly accessible web site Show the metrics, whether they’re good or bad Provide the option of cancelling the project at any time Get the system into the hands of its consumers as early and often as possible Incorporate feedback as quickly as possible Build trust.
  24. 24. Constant Reflection for Improvement Examine the team’s process to determine how to improve Ensure that improvement activities are given proper priority Quantify the improvements when possible Articulate the qualitative improvements
  25. 25. Real artists ship. -- Steve Jobs
  26. 26. Questions? Dave Rooney @daverooneyca on Twitter