Using periodic audits to prevent catastrophic project failure

801 views
690 views

Published on

From May 2008 ICGFM Conference,Paul Doresy, Dulcian on Project Management

Published in: Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
801
On SlideShare
0
From Embeds
0
Number of Embeds
74
Actions
Shares
0
Downloads
14
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Using periodic audits to prevent catastrophic project failure

  1. 1. Dr. Paul Dorsey Dulcian, Inc. May 22, 2008 Using Periodic Audits to Prevent Catastrophic Project Failure
  2. 2. The Vasa <ul><li>Early 17 th century (1628) </li></ul><ul><li>“ The greatest ship of all time” </li></ul><ul><ul><li>3 years to build </li></ul></ul><ul><ul><li>2 gun decks with 64 cannons </li></ul></ul><ul><li>King Gustavus Adolphus of Sweden set the measurements. </li></ul><ul><ul><li>1000 trees were required </li></ul></ul><ul><ul><li>Triple laminated oak walls (18in/46cm thick) </li></ul></ul><ul><ul><li>Mast = 190 ft/57m </li></ul></ul><ul><ul><li>Length = 201 ft/61m </li></ul></ul><ul><li>Cost = 5% of Sweden’s GNP </li></ul>
  3. 3. Maiden Voyage <ul><li>Set sail on a beautiful summer day </li></ul><ul><ul><li>August 10, 1628 </li></ul></ul><ul><ul><li>Passed the royal palace of Stockholm </li></ul></ul><ul><ul><li>Fired proud salutes from cannons </li></ul></ul><ul><ul><li>Sailed 1400 yards </li></ul></ul><ul><ul><li>Gust of wind came </li></ul></ul><ul><ul><li>Ship sank in less than 1 minute </li></ul></ul>
  4. 4. Why is the Vasa relevant? <ul><li>What sank the Vasa sinks FMIS projects. </li></ul><ul><li>We are still building Vasas. </li></ul><ul><li>We can’t stop bad decisions. </li></ul><ul><ul><li>We CAN stop ignoring them. </li></ul></ul><ul><li>If you spend $1M and fail, it is bad. </li></ul><ul><ul><li>If you spend $100M and fail, it impacts the whole organization </li></ul></ul><ul><ul><ul><li>If you spend $1B and fail, it impacts the country. </li></ul></ul></ul><ul><li>If you are going to fail - fail cheap. </li></ul>
  5. 6. The Essence of Project Management <ul><li>Leading kids on a hike </li></ul><ul><li>From time to time, climb a tree </li></ul><ul><li>Check for obstacles </li></ul><ul><li>Adjust direction </li></ul><ul><li>Call everyone together </li></ul><ul><li>&quot;LET'S GO!&quot; </li></ul>
  6. 7. Periodic Audits <ul><li>Critical success factor for failure prevention </li></ul><ul><li>Must be external </li></ul><ul><ul><li>Developers cannot self-assess. </li></ul></ul><ul><li>Big, substantial effort </li></ul><ul><ul><li>Weak audit is worse than useless </li></ul></ul><ul><ul><ul><li>Provides illusion of safety </li></ul></ul></ul>
  7. 8. Audit Costs <ul><li>Delay project </li></ul><ul><li>Expensive </li></ul><ul><ul><li>5%-10% of project cost </li></ul></ul><ul><li>Intrusive </li></ul><ul><li>Annoying </li></ul><ul><li>Politically costly </li></ul>
  8. 9. Team response to audit <ul><li>Project Manager </li></ul><ul><ul><li>&quot;Why don't you trust me?&quot; </li></ul></ul><ul><ul><li>&quot;Waste of time&quot; </li></ul></ul><ul><li>Developers </li></ul><ul><ul><li>&quot;We would rather be coding.&quot; </li></ul></ul><ul><li>Team may feel threatened… </li></ul><ul><ul><li>If the team feels threatened, they probably have reason to be. </li></ul></ul><ul><li>If the team welcomes the audit…. </li></ul><ul><ul><li>Sign of professional maturity </li></ul></ul>
  9. 10. Audit Benefits <ul><li>Early detection of failure </li></ul><ul><ul><li>If a $300 million project fails after $20 million, that is a huge savings. </li></ul></ul><ul><li>Validation of system architecture </li></ul><ul><li>Second set of eyes </li></ul><ul><li>Give team time to take stock </li></ul><ul><li>Mid-project course correction </li></ul>
  10. 11. Auditor Independence <ul><li>Auditor must be told that there will be no chance for follow-on work. </li></ul><ul><ul><li>Otherwise the audit is suspect </li></ul></ul><ul><li>To enforce independence: </li></ul><ul><ul><li>No economic attachment to the results </li></ul></ul><ul><ul><li>No incentive to skew the results </li></ul></ul><ul><ul><li>No relationship to the development team </li></ul></ul><ul><li>During the audit: </li></ul><ul><ul><li>Limit contact with development team </li></ul></ul><ul><ul><ul><li>&quot;Stockholm Syndrome&quot; </li></ul></ul></ul><ul><ul><ul><li>After the audit, auditors can help with the project plan. </li></ul></ul></ul><ul><li>Auditor is the agent of the person who hired him/her (no one else) </li></ul><ul><ul><li>Only reports to the contract owner are objective. </li></ul></ul><ul><ul><li>No professional standard or certification for IT auditors exists. </li></ul></ul><ul><ul><li>Contract creates objectivity. </li></ul></ul>
  11. 12. Audits are never 100% objective <ul><li>Auditors bring their own biases. </li></ul><ul><li>There are IT &quot;religions.&quot; </li></ul><ul><ul><li>.Net vs. Java </li></ul></ul><ul><ul><li>&quot;Thick database&quot; vs. middle tier logic </li></ul></ul><ul><ul><li>Service Oriented Architecture (SOA) </li></ul></ul><ul><ul><li>Repository-based development </li></ul></ul><ul><ul><li>Business rules </li></ul></ul><ul><ul><li>Agile </li></ul></ul><ul><li>A professional with a different perspective can still detect a &quot;Vasa.&quot; </li></ul>
  12. 13. Justifying Audit Cost <ul><li>An extensive audit requires approximately 10% of the total project cost. </li></ul><ul><li>Industry project failure rate is ~ 60%. </li></ul><ul><li>Example: Audit at the halfway point of a $1 million project </li></ul><ul><ul><li>Cost: (50% x 1,000,000) x 10% = $50,000 </li></ul></ul><ul><ul><li>Benefit: (50%) x 1,000,00) x 60% = $300,000 </li></ul></ul><ul><li>Given the high failure rate, audits are very cheap insurance. </li></ul><ul><li>With other benefits, it is obvious that audits are a good deal. </li></ul>
  13. 14. Finding the right auditor <ul><li>Not internal </li></ul><ul><li>Not from the same company as the developers </li></ul><ul><li>Not dedicated auditors </li></ul><ul><ul><li>Must be real developers, DBAs, architects </li></ul></ul><ul><ul><li>Must have built systems of similar scope and subject area </li></ul></ul>
  14. 15. The Audit Team <ul><li>Project Manager </li></ul><ul><ul><li>Experience in the subject area with projects of similar scope </li></ul></ul><ul><li>Database Administrator (DBA) </li></ul><ul><ul><li>Experience with same platform and similar database size </li></ul></ul><ul><li>Application Architect </li></ul><ul><ul><li>Skill in the same area (Java, .Net, Oracle, etc.) </li></ul></ul><ul><li>Security Specialist </li></ul><ul><ul><li>Military, financial system or health care experience </li></ul></ul>
  15. 16. Structure of the Audit <ul><li>Knowledge transfer </li></ul><ul><ul><li>Deep understanding </li></ul></ul><ul><ul><li>As if auditor were taking over the project </li></ul></ul><ul><ul><li>Understand the system before evaluating </li></ul></ul><ul><li>Infrastructure audit </li></ul><ul><ul><li>Evaluate the existing infrastructure to support system </li></ul></ul><ul><ul><li>Every area needs to be assessed. </li></ul></ul><ul><li>Ability to meet current and future user requirements </li></ul><ul><ul><li>Auditor must understand the requirements </li></ul></ul><ul><li>Financial review </li></ul>
  16. 17. Detailed Structure of Audit <ul><li>A. Knowledge Transfer </li></ul><ul><ul><li>Allows auditors to understand the entire system architecture as if they were taking over system development. </li></ul></ul><ul><ul><li>The following areas should be reviewed for the knowledge transfer portion of the audit: </li></ul></ul><ul><ul><ul><li>System Overview </li></ul></ul></ul><ul><ul><ul><li>Data Model Walkthrough </li></ul></ul></ul><ul><ul><ul><li>Review/Identify Transaction Use Cases </li></ul></ul></ul><ul><ul><ul><li>Review User Interface Code Architecture </li></ul></ul></ul><ul><ul><ul><li>Review Reporting Requirements/Architecture </li></ul></ul></ul><ul><ul><ul><li>Review System Architecture </li></ul></ul></ul><ul><ul><ul><li>Review System Installation/Upgrade Mechanism. </li></ul></ul></ul><ul><ul><ul><li>Internal Control review </li></ul></ul></ul><ul><ul><ul><li>User Access review </li></ul></ul></ul><ul><ul><ul><li>Handling system bugs/enhancements </li></ul></ul></ul><ul><ul><ul><li>Multi-Lingual capabilities </li></ul></ul></ul><ul><ul><ul><li>Basic System Requirements </li></ul></ul></ul><ul><ul><ul><li>Process Flows </li></ul></ul></ul><ul><ul><ul><li>Custom Framework </li></ul></ul></ul><ul><ul><ul><li>Performance </li></ul></ul></ul><ul><ul><ul><li>Standards </li></ul></ul></ul><ul><ul><ul><li>Training </li></ul></ul></ul><ul><li>B. Infrastructure Audit </li></ul><ul><ul><li>Examine from technical soundness perspective. </li></ul></ul><ul><ul><li>Compare to current industry best practices; document any discrepancies. </li></ul></ul><ul><ul><ul><li>System and User Documentation </li></ul></ul></ul><ul><ul><ul><li>Data Model Audit </li></ul></ul></ul><ul><ul><ul><li>Database Review </li></ul></ul></ul><ul><ul><ul><li>User Interface (UI) Architecture Review </li></ul></ul></ul><ul><ul><ul><li>Distributed System/ETL Audit </li></ul></ul></ul><ul><ul><ul><li>Security Audit </li></ul></ul></ul><ul><ul><ul><li>Hardware/Software/Networking Review </li></ul></ul></ul><ul><ul><ul><li>Backup/Recovery Procedures </li></ul></ul></ul><ul><ul><ul><li>Appropriateness of system upgrade mechanism </li></ul></ul></ul><ul><li>C. Ability to Meet Current/Future Requirements </li></ul><ul><ul><li>Examine the current system requirements, identify use cases, and review for suitability, specifically: </li></ul></ul><ul><ul><ul><li>Compare documented requirements to the existing use cases and how they are handled. </li></ul></ul></ul><ul><ul><ul><li>Assess user satisfaction with the existing system. </li></ul></ul></ul><ul><ul><ul><li>Are existing backup/recovery procedures sufficient to meet maximum downtime requirements? </li></ul></ul></ul><ul><ul><ul><li>Assess system flexibility to meet new requirements. </li></ul></ul></ul><ul><li>D. Financial review </li></ul><ul><ul><li>Review how resources have been spent over the lifetime of the project including budgeted vs. actual expenditures </li></ul></ul>
  17. 18. Knowledge Transfer <ul><li>&quot;Seek first to understand.&quot; Stephen Covey </li></ul><ul><li>Learn enough to take over the process: </li></ul><ul><ul><li>Architecture </li></ul></ul><ul><ul><li>Data Model </li></ul></ul><ul><ul><li>Use Cases </li></ul></ul><ul><ul><li>Report Audit </li></ul></ul><ul><ul><li>Configuration Management </li></ul></ul><ul><ul><li>Internal Controls </li></ul></ul><ul><ul><li>Documentation </li></ul></ul><ul><ul><li>Training </li></ul></ul><ul><li>System may be bad enough that this is not possible. </li></ul><ul><ul><li>Do it anyway. </li></ul></ul><ul><ul><li>Preventing the Vasa from sailing is hard work. </li></ul></ul>
  18. 19. Infrastructure Audit <ul><li>Compare what was learned in the knowledge transfer with best practices </li></ul><ul><li>Each area needs to be assessed: </li></ul><ul><ul><li>Documentation </li></ul></ul><ul><ul><li>Data model </li></ul></ul><ul><ul><li>Database design </li></ul></ul><ul><ul><li>User interface architecture </li></ul></ul><ul><ul><li>Security </li></ul></ul><ul><ul><li>Backup & Recovery </li></ul></ul><ul><ul><li>Configuration Management </li></ul></ul><ul><ul><li>Internal Controls </li></ul></ul><ul><li>Identify weaknesses in each area: </li></ul><ul><ul><li>Corrective actions </li></ul></ul><ul><ul><li>Exposure </li></ul></ul><ul><ul><li>Controls needed </li></ul></ul><ul><li>One mistake can sink the Vasa. </li></ul><ul><ul><li>System won’t scale </li></ul></ul><ul><ul><li>Security hole </li></ul></ul><ul><ul><li>Inflexible design </li></ul></ul>
  19. 20. Ability to Meet Requirements <ul><li>Requires careful use case documentation </li></ul><ul><li>Assess user satisfaction </li></ul><ul><ul><li>Sit with users </li></ul></ul><ul><ul><li>Survey </li></ul></ul><ul><ul><li>Request queue is not a good measure. If system is poor, users give up. </li></ul></ul><ul><li>Assess each use case </li></ul><ul><ul><li>Functional requirements </li></ul></ul><ul><ul><li>Performance </li></ul></ul><ul><ul><li>Downtime </li></ul></ul><ul><li>Future requirements </li></ul><ul><ul><li>Flexibility </li></ul></ul><ul><ul><li>Deployment </li></ul></ul>
  20. 21. Financial Review <ul><li>Stewardship Audit </li></ul><ul><ul><li>If requirements change, price can balloon. </li></ul></ul><ul><ul><li>Does this make sense? </li></ul></ul><ul><ul><li>Sunk costs are &quot;somewhat&quot; relevant </li></ul></ul><ul><ul><li>Measure decision quality </li></ul></ul><ul><li>Financial History </li></ul>Date Budget $ Spent Mile-stones/ Achieved Notes
  21. 22. COTS Project Audits <ul><li>Not very different from custom projects </li></ul><ul><li>COTS projects fail just as often. </li></ul><ul><li>Review COTS architecture </li></ul><ul><li>Be careful about how well COTS satisfies requirements: </li></ul><ul><ul><li>COTS customizations can be VERY expensive. </li></ul></ul><ul><ul><li>Customized COTS cannot be upgraded. </li></ul></ul>
  22. 23. Audit Report <ul><li>2-5 page Executive Summary Report </li></ul><ul><ul><li>Are we OK? </li></ul></ul><ul><li>10-15 page CIO Report </li></ul><ul><ul><li>Are we OK? Why? </li></ul></ul><ul><li>100 page detailed report </li></ul><ul><ul><li>What we did </li></ul></ul><ul><ul><li>What we found </li></ul></ul><ul><ul><li>What needs to be fixed </li></ul></ul><ul><ul><li>Next steps </li></ul></ul>
  23. 24. Acting on the Audit Report <ul><li>If report concludes &quot;Vasa….&quot; </li></ul><ul><ul><li>Get a second opinion </li></ul></ul><ul><ul><li>Let the development team respond </li></ul></ul><ul><li>Sunk costs are sunk costs. </li></ul><ul><ul><li>The amount of money budgeted for the project is irrelevant. </li></ul></ul><ul><li>&quot;Upgrade&quot; is a way to change direction without admitting failure. </li></ul>
  24. 25. Case Study: Ethiopia FMIS <ul><li>Harvard University Project </li></ul><ul><li>Small part of major financial reform </li></ul><ul><ul><li>$38 M USD over 12 years </li></ul></ul><ul><ul><li>$3 M USD over 3 years for IT (very frugal) </li></ul></ul><ul><li>Harvard was ending the project and wanted to assess quality of system. </li></ul><ul><li>Custom development project </li></ul><ul><li>Dulcian was brought in to do the audit. </li></ul>
  25. 26. Audit Scope <ul><li>Standard audit </li></ul><ul><ul><li>As described in this presentation </li></ul></ul><ul><li>Total knowledge transfer and team back-up </li></ul><ul><ul><li>Support for any “what if?” scenarios </li></ul></ul><ul><ul><li>Total system back-up </li></ul></ul>
  26. 27. Audit Recommendations <ul><li>Maintain current system </li></ul><ul><li>Upgrade system internals </li></ul><ul><ul><li>Business rules approach </li></ul></ul><ul><ul><li>Oracle DBMS </li></ul></ul><ul><ul><li>Ultra-light Web architecture </li></ul></ul>
  27. 28. Audit Results <ul><li>Government and Harvard accepted recommendations. </li></ul><ul><ul><li>Maintain/evolve current system $1.5M USD </li></ul></ul><ul><ul><li>Redesign architecture $1.5M USD </li></ul></ul><ul><li>Dual nature of the audit (audit + handoff) made existing team very uncomfortable. </li></ul><ul><ul><li>Top three IT people resigned. </li></ul></ul>
  28. 29. Conclusions <ul><li>Audits don't prevent failure; they just catch failures earlier in the process. </li></ul><ul><li>Audits don't replace good design. </li></ul><ul><ul><li>The audit may only help fail more small projects rather than one big project. </li></ul></ul><ul><li>Audits are resource intensive, expensive, annoying, politically charged. </li></ul><ul><ul><li>But they are cheaper than not doing them at all in the long run. </li></ul></ul><ul><li>Auditors must be kept independent. </li></ul><ul><ul><li>No follow-on work </li></ul></ul><ul><li>Both COTS and custom projects need audits. </li></ul>
  29. 30. Result of Audit <ul><li>Quite a good design </li></ul><ul><ul><li>Excellent documentation </li></ul></ul><ul><ul><li>Very good developer coding </li></ul></ul><ul><ul><li>Supported current requirements </li></ul></ul><ul><li>Some architecture portions needed modification. </li></ul><ul><ul><li>Database design issues </li></ul></ul><ul><ul><ul><li>No Foreign Keys </li></ul></ul></ul><ul><ul><ul><li>Odd design (inherited from previous team) </li></ul></ul></ul><ul><ul><li>Poor performance for parts of system </li></ul></ul><ul><ul><li>Scalability questions </li></ul></ul><ul><ul><li>Would not work on Ethiopian internet due to slow/unreliable connections </li></ul></ul>
  30. 31. Contact Information <ul><li>Dr. Paul Dorsey – paul_dorsey@dulcian.com </li></ul><ul><li>Dulcian website - www.dulcian.com </li></ul>Latest book: Oracle PL/SQL for Dummies Developer Advanced Forms & Reports Designer Handbook Design Using UML Object Modeling

×