cPrime FBI Agile Success
•
5 likes•1,545 views
Many doubt it to be so, but Agile Development and supporting Agile software DOES have a place among Government Agencies. Tune in to see the successes and failures as the FBI attempted to utilize Agile Development practices
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to cPrime FBI Agile Success
Similar to cPrime FBI Agile Success (20)
More from Cprime
More from Cprime (20)
Recently uploaded
Recently uploaded (20)
cPrime FBI Agile Success
- 1. Agile Project Success and Failure The Story of the FBI Sentinel program & Tools for your Agile Tool box “US Digital Services Playbook” & “Tech-Fars” 3 December 2015 Tom Friend
- 2. Tom Friend PMI-ACP, CSP, PSM, CSM, AHF, FLEX, ATP • Agile-Scrum Trainer - Coach • 12+ Years Agile Scrum Experience • 25+ Year IT Application Development – GAO Agile Advisory Work Group – Carnegie Mellon Agile Government Group – DOD CIO Software Assurance Working Group – Scrum Alliance Veteran Outreach Representative • US Naval Aviator & Air Force “Galaxy” Pilot • Squadron Commander • Air War College / Honors Graduate • Bronze Star, Combat Action Badge
- 3. Learning Objectives 1. Why is this story of value? 2. Stress Systematic Failure 3. Tell the FBI story of Failure and Success of Agile. 4. Tools for Agile in Federal and DOD programs.
- 4. WHY? The Old Way is not working! 1. BDUF programs are train wrecks. 2. Long Cycle time projects can’t react to changing requirements. 3. Base assumptions are not valid. Cost!!!! Time-Effort-Opportunity-Money
- 5. Summary of Federal Failures 1. Federal IT spending is 3.9% of the US Federal Budget. 78 Billion USD 2. 90% of large IT projects in government are unsuccessful. 3. Over half were delayed and over budget didn’t meet user expectations. 4. 41% failed completely. 32 Billion Waste!
- 6. How much is a Billion Dollars? • A Billion is a “thousand millions”. • A billion seconds ago was 1959. • A billion minutes ago, Jesus was alive. • A billion hours ago was the Stone Age. • A billion dollars ago was 9 hours of Federal spending.
- 7. 1 Billion In $100 dollar Bills
- 8. Original Source of FBI Story • Brian Wernham • Director at The Association for Project Management • Delivery of £100m+ IT-enabled programs • Business Case development at Director Level + brian.wernham@gmail.com
- 9. Brian's Research Focus Area • Brian’s research focused on critical, main-stream Federal projects. • Projects that had used various methods. • Based on the FBI case study, here is the question that we will address together today: –What evidence is there that Agile is suitable for large-scale projects?
- 10. Viewpoints on Project Approaches • Normal argument is Waterfall or Agile • The Federal Waterfall approach will tend towards what Kent Beck called ‘Big Design Up Front’ (BDUF) – Due to BDUF being the starting point for waterfall • Brian stresses that we should aim for Enough Design Up- Front (EDUF), not BDUF.
- 11. What is Big Design Up Front (BDUF)? • ‘The Simpsons’ Homer the average American, is the perfect person to design a new car. • The car turns out to be totally unusable and too expensive to produce! • Normal output from BDUF – BIG – DESIGN – UP – FRONT
- 12. Remembering failed DOD BDUF programs We repeat what we forget.
- 13. Kinetic Energy Interceptor (KEI) $1,300,000,000
- 16. Expeditionary Fighting Vehicle (EFV) $3,300,000,000
- 18. Future Combat System (FCS) $19,000,000,000
- 19. Brian’s Case Study : A Story of Agile Success At the FBI • The following FBI projects detail a fascinating case study of project methodologies and how Agile leadership delivered where traditional approaches failed. • Three reasons this cases study is relevant – Size: This was a large project - hundreds of millions of dollars: Agile Scales. – Prior Failure: This Agile project succeeded where two previous waterfall projects using the same technology had failed. – Visibility: This was a high-profile and mission-critical project - in government. Precisely the environment and scale at which Agile is often misconstrued as being inappropriate.
- 20. Case study : History (part 1) • 2001 FBI Agent Robert Hanssen Arrested • 1.4 Million in payments • 20 separate occasions
- 21. Case study : History (part 2) • Three months later, in May 2001, the Oklahoma City bomber was about to be executed. • 700 documents had not been disclosed to the defense. • The FBI had forgot to send and lost evidence. • Stay of execution was granted the FBI came under severe criticism
- 22. FBI looks for a Solution • Root Cause old computer system and outdated manual processes • A secure and reliable set of systems and processes were required. • The FBI sets up a project to build a new Virtual Case File system (VCF). – One $400 contract – Brand new system as no market alternatives were deemed sufficient • (SAIC) won the bid and created a classic waterfall project. – 200 person team spent 6 months creating the requirements – A grand design upfront – Go live at once - a classic big-bang.
- 23. Results of First Attempt over 4 years • Fours CIO’s, one per year • 2002 No Delivery • 2003 No Delivery • 2004 No Delivery • 2005 No Delivery • Four years of no results!
- 24. Scope Change 9/11 • The 9/11 attacks increased political pressure for better homeland security and data sharing between agencies. • Responding to this pressure, the FBI made promises they could not keep. • The FBI received an additional $78m of funding for compliance and also promised to chop off another 6 months from the schedule. • Summary: – Waterfall-style Big Bang rollout – $400 Million – $78 Million Additional Funding – 300 person team for Requirements + 6 months = 600 pages of listed requirements
- 25. Results of First Attempt • The 600 pages of requirements were now the subject of over 400 documented ‘change requests’, and over 700,000 lines of program code had been being written and re-written time and time again. • 700,000 lines of code were cut, then worked and reworked again and again… • The size of the requirements document went on to nearly double to 1,200 pages as the development work tried to keep pace with the changes to this Big Design Up Front.
- 26. Final Disposition Failure • Program Cancelled – A 318 page post-mortem report concluded that the contractor had: “badly bungled the project – it should be abandoned … the software is incomplete, inadequate and (incompetently) designed - essentially unusable under real world conditions.” – Randolph Hite, GAO, said: "When you do a program like this, you need to apply a level of rigor and discipline that's very high.”
- 27. The need is still there…. • That year, yet another FBI agent, Leandro Aragoncillo, is arrested. • The ancient FBI system had failed to spot his suspicious behavior as he fished through the existing case-management systems for over 9 months. • Obviously the need for a new system remained.
- 28. When you fail, blame the vendor and try again with more money! • Lockheed Martin wins the new ‘Sentinel’ system project 3 years 2006 to 2009. The total project budget was $425m. – $305m was budgeted for Lockheed Martin – $120m was allocated for the FBI to run a massive program office to carry out detailed and prescriptive oversight of the work. • That's one quarter of the budget being spent on planning and control of the contractor!
- 29. The Sentinel project Second Try • The new project as Waterfall – $60 million spent just creating a web-based front-end to the already broken system in-place • Some agents called it ‘lipstick on a pig’ • Due to its incompleteness, they stopped using it
- 30. Digress to irrelevance… • Overly optimistic reporting – Status reports were full of facts and statistics - but they never reported even one part of the project as being in trouble. • When it came to final testing, one year late in 2010, the stakeholders rejected the system, even though it was theoretically compliant with the original specifications. • The dream of implementing electronic information sharing before the end of the decade had been shattered.
- 31. Time to get serious - Remove impediments • Project internalized • The FBI CIO takes ownership • Agile is adopted as the project framework – Design is broken into 670 user stories – Self-organizing teams – 45 staff (not 300 as previous) – Product Owner prioritized the work – Two week sprints – Demo every sprint
- 32. Outcome functionality • After a few sprints, it became possible to forecast the rough timescales and start to plan the dates for incremental business change and adoption of releases of the new software. • System delivered using only half of the budget. • Agents used the system on real cases. In the first quarter of its use, over 13,000 agents progressed over 600 cases, meeting or exceeding all expected targets. • The old mainframe system was turned off.
- 33. Outcome in dollars and cents • The three-year Agile project delivered the requested system and improvements. • A success after 10 years of failure and $600 million wasted on the two previous aborted 'Waterfall' attempts. • Total cost of only $99 million.
- 34. What lessons can be learned from this? • Much of the Enterprise Agile literature out there was based on a leap of faith and has not necessarily been evidence- based. • Organizational Leaders need credible, evidence-based argument that puts the business case for Agile to top management. • Consider approaching change in a different way: – You can help your organizations think differently about how objectives are agreed up front on projects. – You can influence your colleagues in procurement, and those that carry out project audits, and assurance work to adapt and harness an Agile - not waterfall, approach.
- 35. Lessons learned were brought to focus. • Patient Protection and Affordable Care Act (PPACA) • Heatcare.gov changed the conversation. • HealthCare.gov went live in 2013,
- 36. US Federal Agile Agency Leaders • FBI Federal Bureau of Investigation • USCIS US Customs and Immigration Service. • Veterans Administration • USPTO Patent and Trademark Office
- 37. Directives • U.S. Digital Services Playbook (Website) • The TechFAR Handbook (GitHub Collaboration) • Leveraging Best Practices to Help Ensure Successful Major Acquisitions (GAO, Feb 2014)
- 39. Purpose of the Playbook 1. Establishing standards to bring the government’s digital services in line with the best private sector services; 2. Identifying common technology patterns that will help us scale services effectively; 3. Collaborating with agencies to identify and address gaps in their capacity to design, develop, deploy and operate excellent citizen-facing services; and 4. Providing accountability to ensure agencies see results.
- 42. In the end, what we need is Agile leadership • We need proof that Agile leadership will bring success. • We need people to lead their projects, not manage them. • 9 Agile Leadership Behaviors – Satisfy the customer – Harness Change – Be very incremental – Create trust through ‘high touch’ leadership – Encourage face to face conversations – Set targets and reword progress – Pursue simplicity, not complexity – Give team space they need to excel
- 43. Retrospective 1. Why is this story of value? 2. Stress Systematic Failure 3. Tell the story of Failure and Success of Agile. 4. Tools for Agile in Federal and DOD programs.
- 45. Sources • The Agile approach saves the FBI Sentinel Project… – Brian Wernham FBCS FAPM • SOFTWARE DEVELOPMENT; Effective Practices and Federal Challenges in Applying Agile Methods – United States Government Accountability Office • David A. Powner, (202) 512-9286 or pownerd@gao.gov • Dr. Nabajyoti Barkakati, (202) 512-4499 or barkakatin@gao.gov
Editor's Notes
- Share observations on systematic failures in procurement of software systems in the Federal Government. I will reinforce that this is a systematic failure with large Procurement software and hardware systems . Introduce tools and resources that can be leveraged in the Federal and DOD Agile Application Development
- Cost!!!! Time Cost!!!! Effort Cost!!!! Opportunity Cost!!!! Money
- Now that we have talked about why this is of value lets get to the story. This is an adaptation of a presentation that was originally given by Brian Wenham at the Agile Business Conference 10th October 2012 held at the Inmarsat Centre, London Main Board Non-Executive Director at Association for Project Management, Europe's largest dedicated Project Management member organization Delivery of £100m+ IT-enabled programmes Business Case development at Director level and above
- However, long-standing congressional interest has contributed to the identification of numerous examples of lengthy IT projects that incurred cost overruns and schedule delays while contributing little to mission-related outcomes. To reduce the risk of such problems, the Office of Management and Budget (OMB) recommends modular software delivery consistent with an approach known as Agile, which calls for producing software in small, short increments.
- In ‘The Simpsons’ there is an episode in which a Detroit car company decides that being an average American, Homer is the perfect person to design a new car. The car turns out to be totally unusable and too expensive to produce. Normal output from BIG DESIGN UP FRONT BDUF
- The Simpson’s may be too abstract so lets take a minute to talk about several billion dollar failures.
- KEI was a part of the emerging Missile Defense Agency’s (MDA) architecture at the turn of the century. However, it was done in by cost growth resulting from MDA’s expanded scope, to include mid-course and terminal phase missions as well as the KEI’s inability to fire from ships due to its size. In 2009, Secretary Gates cancelled the program, but not after MDA spent $1,300,000,000.
- Initiated by the Air Force in 1996, the ABL was designed to be an integral part of the Missile Defense Agency’s architecture for Theater Missile Defense. However, after 15 years and one prototype based on an old Air India 747 air frame. Citing a program that has “significant affordability and technology problems” whose “proposed operational role is highly questionable,” Secretary Gates cancelled the program at the end of 2011, but not before spending $5,000,000,000.
- The Crusader was to be the Army’s next-generation mobile gun system Crusader was conceived in the early 1990s as a powerful new self-propelled howitzer (SPH). While it was designed to be lighter and faster than the existing M109A6 Paladin SPH, it was too similar to the existing, upgraded inventory. Consequently, Secretary Rumsfeld cancelled the program in 2002 after spending $2,000,000,000, making it one of the first RMA-era programs to dissolve. Ironically, many of the Crusader technologies were incorporated in the FCS family of XM1203 Non-Line of Sight (NLOS) cannons, which were subsequently cancelled as well.
- Intended to be a swimming tank for the Marines, the EFV was a hefty 38-ton amphibious assault vehicle designed in the late 1980s. The program’s original use case — beach landings like Normandy and Inchon — was dubious given the emerging threat of long-range, anti-ship missiles. Hampered by delays and cost overruns totaling 270%, the program was cancelled by Secretary Gates in 2011, having already spent $3,300,000,000.
- 22 years, $6,900,000,000 spent and zero helicopters. Originally conceived at the height of the Cold War, it was supposed to become the next-generation of armed reconnaissance air support for the Army, replacing the Huey, Cobra and Kiowa helicopters in the process. A textbook case in technology being superseded by current events, the Comanche also faced serious concerns over its ability to simply get off the ground when fully loaded. The program was cancelled in 2004 with two prototypes now on display.
- First introduced in 1999 by Army Chief of Staff, Eric Shinseki, FCS was supposed to be a family of networked, manned and unmanned vehicles and aircraft for the 21st century battlefield. With the Warfighter Information Network – Tactical (WIN-T) intended to support the FCS, it was supposed to be a wholesale re-envisioning of the ground force. However, the terror attacks of Sept. 11, 2001 short-circuited a 15-year operational pause that the military was hoping for to implement the program. Spiral development and shifting requirements by the Army also resulted in costs ballooning by 25%. Finally, after $19,000,000,000 already spent and the program in the System Design and Demonstration phase, Secretary Gates cancelled the program in 2009, making it the biggest failure of the RMA era.
- The story I want to tell you is a story of Agile success - the FBI Sentinel project. It is one story of many where a large-scale project, with the objective of developing mission critical technology has succeeded where a previous traditional waterfall attempts did not. Let's consider why two previous attempts at Waterfall failed at the FBI, and the consequences. Then let’s investigate how the Agile approach was made to work. Then let's identify the leadership behaviors that were critical to the eventual success when the FBI went Agile. Three reasons this cases study is relevant Size: This was a large project - hundreds of millions of dollars: Agile Scales. Prior Failure: This Agile project succeeded where two previous waterfall projects using the same technology had failed. Visibility: This was a high-profile and mission-critical project - in government. Precisely the environment and scale at which Agile is often misconstrued as being inappropriate.
- So now our story begins In 2001 FBI Agent Robert Hanssen arrested Apprehended dropping a package of secret information under the Foxstone bridge at the Fairfax county park. He was about to pick up a $50,000 payment…which had been left by a Russian spy at another drop site just 10 miles away. Hanssen was using these payments which totaled over $1.4m in cash and diamonds to shower gifts on a particular exotic dancer in a local ‘gentleman’s club’ The FBI had evidence that on 20 separate occasions, Hanssen had left packages for the Russians. He had provided dozens of computer diskettes and more than 6,000 pages of documents. And he had endangered FBI agents in the field and divulged "Top Secret" documents. When the news broke The question pressing question to the FBI from congress was how did one agent get access to so much information? This is nothing compared to what Slowdown did for no money at all. See Hidden slide next page
- Now you may think I am being a bit harsh on Roberts but in his defense I would have to say that he was a pretty good negotiator. If you dollar cost average his price he got around $230 per document. This is In 2001 dollars. Years later Edward Snowden released hundreds of thousands of documents causing billions of dollars in damage for nothing/ Score 1 for the FBI Nill for the CIA
- So to add insult on to injury Just three months later, in May 2001, the Oklahoma City bomber was about to be executed. Then (just one week before the date of the execution) it was revealed that over 700 documents had not been disclosed to the defense. The FBI had forgotten to send materials… and in many cases - had simply lost evidence. The legal process was thrown into turmoil, a stay of execution was granted the FBI came under severe criticism Back to capital hill for a senate inquiry.
- An investigation showed that a combination of an old, rudimentary computer system and outdated manual processes were to blame. A much more secure and reliable set of systems and processes were required. The FBI sets up a project to build a new Virtual Case File system (VCF). This Virtual Case File system project was to be created as one massive contract at a total cost of $400m. Brand new system as no market alternatives were deemed sufficient Science Applications International Corporation (SAIC) created a classic waterfall project. 200 person team spent 6 months creating the requirements A grand design being drawn up before work would start the plan was for the whole system to go live at once - a classic big-bang.
- Well at least they have a consistent track record in their favor right. Everything changed overnight with the September 11 attacks. The 9/11 attacks increased political pressure for better homeland security and data sharing between agencies. Responding to this pressure, the FBI made promises to bring forward deployment of the new VCF case management system by 6 months to December 2002. The FBI received an additional $78m of funding and promised to chop off another 6 months from the schedule. Summary: Waterfall-style Big Bang rollout $400 Million $78 Million Additional Funding 300 person team for Requirements + 6 months = 600 pages of listed requirements
- That year, yet another FBI agent, Leandro Aragoncillo, had been arrested. He had been passing damaging dossiers on the president of the Philippines to opposition politicians who were planning a coup. The ancient FBI system had failed to spot his suspicious behavior as he fished through the existing case-management systems for over 9 months. Obviously the need for a new system remained
- The FBI awarded the next contracted project to Lockheed Martin to develop the newly entitled ‘Sentinel’ system over three years from 2006 to 2009. The total project budget was $425m. $305m was budgeted for Lockheed Martin $120m was allocated for the FBI to run a massive program office to carry out detailed and prescriptive oversight of the work. That's one quarter of the budget being spent on planning and control of the contractor!
- The three-year Agile project has now delivered - at a total cost of only 99 million dollars. A success after 10 years of failure and 597 million dollars wasted on the two previous aborted 'Waterfall' attempts.
- Now lets talk about tools and techniques that came out of this huge success story.
- Patient Protection and Affordable Care Act (PPACA) United States federal statute signed into law by President Barack Obama on March 23, 2010. HealthCare.gov is a health insurance exchange website operated under the United States federal government under the provisions of the Patient Protection and Affordable Care Act (ACA, often known as 'Obamacare') The design of the website was overseen by Medicare and Medicaid Services and built by a number of federal contractors, The original budget for CGI was $93.7 million, however this grew to $292 million prior to launch of the website. Overall cost for building the website had reached over $500 million prior to launch. Office of Inspector General released a report finding that the total cost of the HealthCare.gov website had reached $1.7 billion On July 30, 2014, the Government Accountability Office released a non-partisan study that concluded the administration did not provide "effective planning or oversight practices" in developing the HealthCare.gov website
- USCIS Flexible Agile Development Services (FADS) has fully engaged agile They are on the cutting edge. Went from new Capabilities every couple of years to every quarter will be every month. Want to be releasing new apps and functions on a daily basis. And what is remarkable is that this is being done within the current Federal Acquisition Framework. Those Agencies that had significant failures now are leaders They had a disrupter that challenged their status quo. And were forced to adapt. wave of the internet is now. Will be able to create trillions of dollars. 1st wave 1985 to 2000 first wave Infrastructure and contentions to create the Onramp to information super highway. 2nd wave 2000 to now is connecting people just coming to the end of that era Google face book twitter Netscape aol seamless integration 3rd wave is the transformation of the economy using digital services, food, banking, government healthcare education energy financial services education these 3rd wave sectors are ready for disruption and represent over half of the us economy. Open source has matured, all of the successful companies in the second wave of the internet have adopted open source as their foundations and frameworks. Rise of digital services, digital business models, will drive the next phase of growth in the economy. Government Transformation Paint a picture of what is possible.
- 2012 While the Healthcare.gov site was being built guidance is to provide agencies with contracting guidance to support modular development, as required by Information Technology (IT) Reform Action 15: Issue Contracting Guidance and Templates to Support Modular Development 25 Point Implementation Plan to Reform Federal Information Technology Management. The White House recently launched the United States Digital Service to deliver “customer-focused government through smarter IT.” As part of this announcement the Office of Management and Budget released the Digital Services Playbook and an accompanying TechFAR Handbook "for procuring digital services using Agile Processes,” making it easier for government agencies to buy and implement Agile. In this session, participants will understand the history, intentions and key components of these documents, and have a deeper understanding on how to bringing Agile into government agencies.
- The playbook has 13 key “plays” drawn from successful practices from the private sector and government that, if followed together, will help government build effective digital services. 1 Has the play 2. Checklists to support you 3. Supporting questions that may help you When you find something or learn you update the document. Designed to be a one stop shopping of how to. In the future the contracting review goes through the Digital services playbook in the RFP . Talk about the orals.
- Who has a Git Hub account? It is free, it is all open source Anyone can collaborate and make it better.
- 18 Pages For Release on Delivery February 26, 2014 Information technology (IT) acquisition best practices have been developed by both industry and the federal government. I participate on the General Accounting Office Agile working group. The work with the Carnegie Mellon Software Engineering Institute has developed highly regarded and widely used guidance on best practices, such as requirements development and management, risk management, validation and verification, and project monitoring and control. GAO's own research in IT management best practices led to the development of the Information Technology Investment Management Framework, which describes essential and complementary IT investment management disciplines, such as oversight of system development and acquisition management, and organizes them into a set of critical processes for successful investments. GAO has made numerous recommendations to OMB and agencies on key aspects of IT acquisition management, as well as the oversight and management of those investments.
- Why is this story of value? Because the old way is not working Cost!!!! Time-Effort-Opportunity-Money Stress Systematic Failure IN BIG DESIGNUP FRONT projects Tell the story of Failure and Success of Agile in the case of the FBI Sentinel project. Tools for Agile in Federal and DOD programs. That are worth a look, Digital services play book and Tech Fars on GitHub will get you started.