1. LESSONS LEARNED ADOPTING AGILE FOR
FIXED-PRICE CONTRACTS
By
Glenn R. Fuller
A Master of Science Research Practicum
Submitted to the Faculty of
The Graduate School of Management,
at the University of Dallas
in Partial Fulfillment of the Requirements
for the Master of Science Degree
in Information Technology
November 03, 2009
4. iv
TABLE OF CONTENTS
Page
DEDICATION ……………………………………………………………………. ii
ACKNOWLEDGEMENTS ……………………………………………………….iii
CHAPTER 1: INTRODUCTION..............................................................................1
Agile Software Development Method ...........................................................1
From Predictive to Agile................................................................................2
The Case For Agile........................................................................................6
CHAPTER 2: LITERATURE REVIEW...................................................................8
Contractual Obligations .................................................................................8
Fixed-price Software Development Process..................................................9
Agile Methods................................................................................................11
Scrum.............................................................................................................13
Agile and Fixed-price Software Development ..............................................14
CHAPTER 3: ANALYSIS ........................................................................................17
The Project.....................................................................................................17
The Old Ways ................................................................................................17
The New Ways...............................................................................................19
New Process...................................................................................................20
The Tools .......................................................................................................23
The Results.....................................................................................................24
Lessons Learned.............................................................................................25
CHAPTER 4: CONCLUSION ..................................................................................28
Recommendations..........................................................................................29
Limitations.....................................................................................................31
Areas of Future Work ....................................................................................32
REFERENCES ..........................................................................................................34
5. 1
CHAPTER 1: INTRODUCTION
The holy grail of software development is to deliver the customer’s software on
schedule, within the agreed-upon price or cost, and meeting the original scope; this is
known as the Iron Triangle (Ambler, 2003). Software has evolved over the last 30 years,
from punch cards to reusable code, all in an effort to improve the software development
process. In the last 10 years, Agile software development processes were created and
now claim to reduce cost and cycle time (Turk & Vijayasarathy, 2008).
Agile Software Development Method
Agile was created by a group of software developers that felt the existing software
development methods were inadequate and failed to meet the needs of the customer. An
Agile Manifesto is credited as the starting point for the Agile movement. It states:
We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on
the right, we value the items on the left more. (Beck, et al, 2001).
This new approach eliminates the Big Up Front Design (BUFD) and Big Up Front
Requirements (BUFR) in favor of spending more time developing working software that
can be used by the customer to further refine their requirements. It requires flexibility on
the part of the customer and the development team as the requirements are in flux during
6. 2
the entire development cycle. With frequent deliverables, the customer has the
opportunity to evaluate the product and provide feedback and changes as it is being
developed.
Agile was developed originally within IT projects where the project teams have
flexibility with respect to requirements and deliverables, but due to its success,
commercial software developers have begun to adopt it also. This is a case study of a
team that utilized an Agile software development methodology to develop commercial
software under a fixed-price contract.
From Predictive to Agile
Many attempts have been made to reduce the level of uncertainty and risk in
projects, and this has resulted in the formalized processes called predictive project
management. The predictive approach is premised on using historical estimates for the
effort required to perform similar tasks, or using a bottom up estimate with the
assumption that smaller tasks can be more accurately estimated than larger ones. The
project schedule, theoretically, now becomes more certain, and time and costs can be
accurately estimated. This approach fits well with fixed-price software development, as
customers and developers believe they have reduced their risks by developing and having
the customer approve large amounts of documentation before proceeding to the next
phase.
The predictive model emphasizes the BUFD and BUFR approach, which is a
necessity for a fixed-price software development contract. It dictates that all
requirements are detailed and documented before any work is performed. Only when the
7. 3
detailed requirements documentation has been approved by the customer does the design
phase begin. The design is now created from the requirements and a design document is
generated and approved. Next, the design documentation is used to define and document
the code modules, which are then given to individual developers to create. Any change
to the product requirements at this point creates significant rework in all the
documentation prior to any change to the coded modules.
The most common predictive model, called the Waterfall lifecycle, is associated
with a process defined by Winston Royce of IBM (1970). This process has distinct
phases with strictly-defined hand-offs between each one. These hand-offs are formal
documents that have approval by the owner(s) of the previous phase.
The waterfall phases are:
1. System Requirements
2. Software Requirements
3. Preliminary Design
4. Analysis
5. Program Design
6. Coding
7. Testing
8. Operation
Royce (1970) also included feedback loops from the current phase to the previous
phase. This allowed for changes to be made if an unforeseen problem occurred in the
current phase, to revisit the previous phase and make the necessary changes. Again, all
changes are documented and approved before handing off to the next phase.
The issues of the Waterfall lifecycle software development methodology are well
documented by Craig Larman (2003) when he succinctly states the root cause of the
8. 4
failures of the Big Up Front Design (BUFD) process for software development as;
A “Waterfall” lifecycle, big up-front specifications, estimates, and speculative
plans applicable to predictable manufacturing have been misapplied to software
projects, a domain of inventive, high-change, high-novelty work. (Larman, 2003).
The issues of the Waterfall and BUFD methodologies has led to the exploration of
different approaches to software development in the hopes of more successful results.
Starting in 2001, the Agile software development methodology (Beck, et al, 2001)
was created as a flexible development methodology that allows for changes in scope,
cost, and requirements during the development process, based on changing needs within
the customer’s business process during the software development cycle. Delivery cycles
are reduced to weeks instead of years as features are created and demonstrated to the
customer. This allows the software to evolve with the business requirements rather than
remain stagnant waiting for the software delivery. Since the process is designed to allow
changes, the large up-front documentation effort is reduced to only what is needed within
each delivery cycle.
The benefits of Agile have been well documented over the last eight years as
Agile software development methods have been adopted within IT organizations to
improve software productivity, reduce defects, and increase internal customer satisfaction
(Turk & Vijayasarathy, 2008). Due to the documented benefits of Agile for internal IT
projects, the Agile development methodology has been expanded to projects outside the
IT arena. The attempt to apply Agile methods to commercial software development
projects has been a recent phenomenon, and it is success in the commercial software
9. 5
environment has not been fully explored (Begal & Nagappan, 2006).
Though Agile has been attempted in the commercial software development
environment, the successful application of Agile methods to a fixed-price contract with a
predetermined scope and schedule has not been proven. In fact, many Agile proponents
believe that Agile methods are not compatible with fixed-price contracts.
Agile methods acknowledge the market and project uncertainties and try to adapt
by building software in small and completely done increments, so that the
business could change the project direction up to the point of efficiently killing
the projects at pretty much any moment. Naturally this approach is not very
compatible with the idea of fixing everything upfront. (Marchenko, 2008).
In the Agile Manifesto, the key aspects of the methodology are the flexibility to
“Respond to change over following a plan”, and “Customer collaboration over contract
negotiation” (Beck, et al, 2001). These aspects of Agile are in seeming conflict with the
process of negotiating and signing a contract with a commercial customer for delivery of
a specific set of software requirements for a fixed-price and within a contractual schedule
that typically has penalties for missing the dates or requirements in the contract
(Cockburn, 2005).
Are there benefits to implementing Agile Software Development Methods in a
fixed-price commercial software company, developing software to predefined and
contractually obligated schedule, scope, and cost? Many Agile practitioners believe that
Agile is not a good fit for a fixed-price software development contract due to it is
necessity for up-front estimates (Cockburn, 2005), however, there a some who believe it
may offer an improvement in the software development process over traditional
10. 6
approaches (Miller, 2007).
I don't really think the Agile answer to a fixed bid is any different from any other
process. I do think that Agile practices and project management can give you far
more control and feedback on the "Iron Triangle" of resources, time, and
features. Agile/RUP/CMMI/waterfall whatever, the iron triangle constraints still
apply. If you try to lock all three constraints you're in for either pain,
unhappiness, or protective sandbagging in your estimates. I would still choose to
use Agile delivery for fixed bid projects because I think that is the most efficient
way to execute and allows for the ability to fail "softly" with some fraction of the
features instead of total abject failure to deliver any features on time like a
waterfall project. (Miller, 2007).
The Case For Agile
This case study is about a company that provides software under fixed-price
contracts to state and local government agencies. The requirements are provided by the
agency in the form of a Request For Proposal (RFP). The bidders submit a proposal that
includes scope, schedule, and price to the customer. The agency reviews the bids and
selects one vendor to enter into a fixed-price contract based on the winning proposal.
Once the contract is awarded, the software development process, based on the signed
contract is begun.
This company has been developing and delivering software for over twenty years
and has used the Waterfall development methodology to create the software. Over the
years, several issues continued to haunt the development process including missed
requirements, late delivery, numerous software bugs, unsatisfactory software
performance, and blown budgets resulting in a less than profitable business.
The organization was dysfunctional as marketing created the requirements with
11. 7
the customer, handing the documents off to engineering and then never saw the results
until the product came out of development. At that point, any changes were at a high cost
to both budget and schedule. Software was tossed over the wall from development to
system test and then tossed back from system test to development. Heated arguments
were common between developers and testers, as the definition of bugs became a
constant battle. Everyone was pointing fingers at everyone else and no one wanted to be
accountable for the final results.
Something had to change, as customers began to go to competitors. New
management was brought in to fix the problems. Both the organization and the
development process needed to be fixed. The impact of the Agile methodology, that has
proven successful in IT software development environments (Turk & Vijayasarathy,
2008), was too alluring to pass up. Agile was chosen as a methodology that not only
changes the way software is developed but puts ownership directly on the team creating
the software. This became the silver bullet the business was hoping would solve their
problems. The only question was, could Agile bring the same benefits from the
Information Technology (IT) world to succeed in a fixed-price contractual software
environment?
12. 8
CHAPTER 2: LITERATURE REVIEW
Although there is a wealth of information on the Internet and in bookstores
regarding the use of Agile software development methodologies on IT projects, few
studies have been performed on the use of Agile for commercial software development,
and even less is published on the application of Agile to fixed-price commercial software
contracts. Much of the published information applying an Agile methodology to fixed-
price contracts is speculation and not based on actual experience.
Contractual Obligations
There are inherent differences between software developed by internal IT
organizations and software intended to meet a predetermined contractual agreement
(Cockburn, 2005). The internal organization is not bound by a contractual obligation that
imposes penalties for not delivering on schedule or for missing the predefined
requirements (Marchenko, 2008). A contractual obligation can impose significant
penalties for delays, and the customer must approve the deliverable, which can mean
rework and added cost due to the interpretation of requirements.
An internal IT organization may miss deliverables or have significant delays but,
in most cases, it does not have a material impact on the organization’s customers or
business reputation. The biggest difference is the ability to work with the internal
customer to change requirements and/or provide incremental deliveries that may not meet
all requirements but to provide additional working functionality. Under a contractual
obligation, the customer expects to get delivery of the entire set of requirements and can
13. 9
impose penalties up to and including cancelling the contract if the commitment is not
fully met.
Fixed-price Software Development Process
Commercial software development has traditionally adopted a waterfall
development methodology that follows a serial sequence of events to deliver a
commercial software product. This methodology was adopted from the other engineering
disciplines, such as construction or hardware design (Northrop, 2004). The fixed-price
commercial software business is predicated on the need to have well-defined
requirements up front, along with a fixed-price for the effort and a schedule for delivery.
The Waterfall method and it is derivatives support the need to develop pricing, scope, and
schedule. The belief has been that by properly estimating the iron triangle, the project
will be a success.
However, there are significant issues with the Waterfall methodology, as stated by
Jim Highsmith and Dr. Sam Bayer (1994):
While waterfall methods brought some stability and order to the chaos of earlier
development efforts, their shortcomings are becoming increasingly apparent and
many software development organizations are lost in the "swamp" of waterfall
development (Bayer & Highsmith, 1994).
In a review of the Waterfall process, an experienced software development
manager believes,
The problem with the waterfall method is its inflexible partitioning of the project
into these distinct stages. Delivered systems are sometimes unusable, as they do
not meet the customer’s real requirements (Sommerville, 1995, 85).
A study of over 250 software development projects, that did not include any Agile
14. 10
projects, demonstrates the failure rate of software projects:
Industry software delivery statistics are quite dismal. Fifty percent of commercial
software products are delivered over schedule, 33 percent are cancelled, and 75
percent are operational failures. Government software delivery statistics are
similar (Jensen, 2003).
The evidence for ongoing issues with software development projects is
overwhelming. In a large project study, The Standish Group showed that 45% of the
requirements were never used (Johnson 2002). So, even when the agreed upon
requirements have been identified up front, the customer may not really need or use the
features in the delivered product.
Figure 1 Software Project Success Rate (Hartman, 2006.)
In a summary of surveys done by The Standish Group (Hartman, 2006), the
number of successful projects has gradually improved but the success rate as of 2004 was
15. 11
still less than 30%. With all the issues and failures of software development projects,
including high visibility failures (Charette, 2005), it becomes obvious why companies are
looking for better methods to create software.
Agile Methods
In the years following the signing of the Agile Manifesto in 2001, several studies
have been done to compare Agile software development methods with the previous
Waterfall methods. In a study performed at Penn State University, two teams were
created, one using the Waterfall method and the other using the Agile methods Extreme
Programming (XP) and Rational Unified Process (RUP). According to the author:
Also, based on this assessment of the traditional Waterfall approach versus the
agile methods RUP / XP discussed in this study, both approaches are used for
particular reasons, the iterative methodologies are usually better for new concepts,
while the Waterfall may be better for modifications to existing systems or
building large scale systems (Waters, 2008).
In a survey of early adopters of Agile (Turk & Vijayasarathy, 2008), using a 7-
point scale, it was found that the top benefit (Figure 2) was that the project better met the
customers’ needs, followed closely by improved quality and increased flexibility in
development. Faster delivery time and lower cost were also cited as significant benefits.
This survey’s respondents were primarily IT organizations, not commercial software
companies. Where flexibility is considered a benefit to an internal IT organization, it
may not be perceived as such by a company that is delivering software to a customer
using pre-defined requirements and fixed estimates.
16. 12
Figure 2 Extent of Benefits (Turk, Vijayasarathy, 2008)
In a case study of the Douglas County, Colorado sheriff’s office, the IT
organization could not get an IT project approved due to cost and schedule using the
traditional Waterfall approach. By changing their methodology to Agile, the IT team
delivered the project in half the time (Fredrick, 2007). The Agile Journal is full of Agile
case studies, which are predominately IT projects, in which the IT organization has
changed from a traditional Waterfall method to an Agile method. The results of
implementing Agile methods for the majority of projects have been very successful. In
his review of Craig Larman's "Agile and Iterative Development", Patrick Wilson-Welsh
noted (Wilson-Walsh, 2003):
Many large and small organizations worldwide are formally adopting agile and
iterative practices. Informally, this experimentation has been going on for
decades. Larman points out that iterative process design goes back more than 30
years to Tom Gilb's Evo method. Even the man whose original article gave
waterfall its momentum, Winston Royce, apparently regretted later having written
it. The late Royce, according to his son, was actually a misunderstood advocate of
iterative and incremental development. The research seems to indicate that a
shrinking percentage of projects are using waterfall "by the book" anymore.
17. 13
If you are an Agile skeptic, this book may rattle your conviction. If you are fence-
sitter, it may convince you. And if you already have Agile fire in the belly, then
certainly this book will stoke that fire. After reading it, I am left wondering how
intelligent, experienced software development management can justify the
continued use of a process that has wasted so much money and caused so much
pain. And I am left with no doubt that agile methods constitute an unstoppable
trend in software project management. There is, indeed, a revolution going on
(Wilson-Walsh, 2003).
Mary Poppendieck believes that Agile is not just for small projects but that the methods
can also be adapted to large projects. She also believes that Agile can be adapted to work
in commercial software environments:
The bottom line is that the problems that used to be addressed by traditional
software processes have changed, and those processes are no longer up to the task
of addressing large project development. Meanwhile, the agile practices being
honed in small projects are just the ones needed in a large project environment
which deals with legacy systems and commercial components. So don’t be
surprised to see agile practices move up-market, as disruptive technologies always
do, and take over the large projects as well (Poppendieck, 2006).
Scrum
Scrum was developed as an Agile methodology for improving the productivity,
quality, and cycle time of IT software development projects. This methodology has been
very successful in IT organizations (Walton, 2004) and, therefore, it is thought to be a
candidate for implementing in a commercial software organization where software
companies expect to see the same benefits.
Scrum is a software development process that implements the Agile methodology
(Schwaber, 2006). It utilizes small teams of 6-10 people that work together on self-
assigned requirements called user stories at the beginning of each Scrum cycle. The user
stories for each Scrum cycle are managed by the product owner and must be completed
18. 14
within the pre-determined time frame of three to four weeks with demonstrable working
code. Scrum uses a requirements process that is flexible and allows for changes at the
end of each three- to four-week deliverable. This allows the customer or product owner
to make changes based on the working code demonstrations.
The Scrum teams become self managing, holding daily meetings, and holding
team members accountable for their progress each day. The Scrum Master facilitates the
meetings which are limited to asking three basic questions.
1. What did you accomplish yesterday?
2. What are you going to accomplish today?
3. What is blocking you?
Team members check in their working code as they complete each user story. Testing is
performed in parallel with development as test scripts are written prior to any code being
developed. The test scripts ensure the code will perform the functions as defined by the
user stories.
The freedom for the developers to manage their time and commitments imparts a
sense of ownership and responsibility. Developers cannot blame management for giving
them unrealistic expectations. They also cannot hide their lack of progress since they are
reporting to their peers. This aspect of Scrum is a powerful tool to improve productivity
if managed properly.
Agile and Fixed-price Software Development
A survey of Microsoft employees was performed in 2006 to determine the extent
of Agile adoption within the Microsoft organization (Begal & Nagappan, 2006). They
19. 15
found that the adoption of Agile practices within Microsoft had a 33% penetration. The
majority of those using Agile were using the Scrum form of Agile. This research was
done in the form of a survey of the developers that included both standardized questions
with a five point scale as well as two open-ended questions regarding the benefits and
issues of adopting Agile. Microsoft is a commercial software company that develops
products for sale to the public but does not necessarily fit the category of performing
fixed-price contractual software development, so though this study is very close to our
target, it does not quite hit the mark.
Artem Marchenko (2008), an Agile expert, has commented on the application of
Agile methods to fixed-price commercial contracts:
While it is possible to utilize fixed-price, fixed scope contracts with the agile
methods, fixed contracts are still discouraged in preference of time & materials or
Pay per Use (Soundair, 2008). The reason is that in most if not all cases software
development is not a mass production, but a new product development
(Marchenko, 2008).
This general concern regarding using Agile for fixed-price commercial contracts is
illustrated in the three-fold aspects of any project, scope, schedule, and cost. Martin
Fowler (2003), one of the original signatories’ of the Agile Manifesto commented:
Many people believe that you can't do a fixed-price contract in an agile project.
Since the whole point of an agile process is that you cannot predict the future, this
isn't an unreasonable supposition. However this doesn't mean you can't come up
with a fixed-price agile contract, what it means is that you can't come up with a
fixed-price and fixed scope contract (Fowler, 2003).
20. 16
The gains of the Agile method with respect to efficiency and cycle time reduction
were too great to ignore. The challenge for the team was to improve upon the software
development process under the conditions of a fixed-price contract. This case study will
examine the benefits and problems that occurred within the development team, and
lessons learned in the attempt to adopt Scrum to deliver commercial software on a fixed-
price contract.
21. CHAPTER 3: ANALYSIS
The Project
Due to the high level of software defects and ongoing customer issues, I was
asked to take on the challenge of fixing our software delivery problems. The
organization consisted of multiple departments and each department was responsible for
only a small portion of the effort on each project. For the project, I was put in charge of
all the departments that were involved in creating and delivering the software
applications to our customer. This included engineering, system test, support, and project
management.
I was then assigned a fixed-price software development project for a government
customer. The customer required significant modifications to our existing product such
that the architecture could no longer sustain more changes without extensive rework.
Also, the existing product was written in an obsolete programming language with a flat-
file database. The decision was made to use the new contract to update the product with
a modern relational database and use a modern programming language.
The customer had been informed of the updates to the product along with the
inclusion of their requirements. Many of the requirements were to be derived from the
existing product, based on the functionality being used by the customer. Rough estimates
were made based on using Agile as the development methodology and the teams previous
experience with the existing application. The team, optimistically, factored in
productivity improvements with the new languages and Agile. A one year schedule for
delivery, and pricing, was agreed upon, then the project team got started.
The Old Ways
17
22. The original organization was divided into developers (engineering), system test,
trainers, product management, and support, with a manager over each department. No
one department owned the deliverable, so much time was spent by the departments
discussing who was responsible for missing customer commitments.
This lack of accountability was highlighted when the various managers were
asked about who was responsible for completing a project. All managers or their
representatives had to sign off on any changes and agree to the impact on schedule and
cost within their department. Adding up all the impacts meant that any request took
weeks to months even for the simplest change.
Developers were given their assignments and told when they had to be completed.
Collaboration among developers was rare since each person had their list of tasks to
complete and did not have time to help each other. The tasks were assigned by the
manager who had little technical understanding of the product. Only one or two senior
architects had the big picture and spent most of their time answering questions as
developers ran into problems or didn’t understand their assigned tasks. Since estimates
were done by the most senior developers, they were always optimistic, and even those
estimates were trimmed by management in order to fit the customer’s budget.
System testing was performed manually with written test scripts based on the
customer requirements document. System test wrote the test scripts as the software was
released from engineering. Software bugs continued to plague the product even though
the system test effort was more than twice the development schedule. Releases to system
test were passed back and forth to and from engineering several times with long lists of
bug fixes. Fixes were made only to discover new problems.
18
23. When the product was finally released to the customer, testing, performed by the
customer, discovered requirements that were missed or not what the customer expected,
and of course, numerous software bugs. As a result the customer became involved in the
break fix cycle and rework of the product, which only added to the development cycle
time and cost.
The New Ways
With the new project came the authority to remake the dysfunctional organization.
The goal was to give ownership to the team responsible for creating the product.
Departments were eliminated and the people in them combined into a single organization.
Roles and individual responsibilities did not change, but each individual was part of the
team that had to create the product. The goal was no longer just counting defects, or
creating requirements, or coding a set of modules but, it was getting the working product
delivered and meeting the customer requirements. Thirty people now reported to a single
manager with individual team leads for each of the smaller teams.
Team members were collocated with each other in larger common areas. Cube
walls were taken down and desks were arranged to facilitate quick and easy access to
teammates. These changes were actually requested by the team members in order to
improve communication and reduce time spent looking for each other to get questions
answered or to discuss problems.
Estimates were made and approved by the team which consisted of everyone
necessary to create and deliver the final product. Team members were held accountable
to each other, their team lead, and the manager whom they shared. The team had their
fate in their own hands. Finally, everyone on the team owned the responsibility to make
the project successful. This responsibility fostered collaboration and communication
19
24. among team members, as now the need to help others was not viewed as a hindrance to
getting one’s own work done but as a necessity to ensure the success of the project.
New Process
Software development outside the IT world has greater visibility and more rigid
requirements that are bound by a legal contract. This means the processes for IT
development need to be reviewed, based on the contractual environment, and may not be
a good fit for commercial software.
Unfortunately, and with a few exceptions, internal IT development is generally
the focus for both research and practice in agile methodologies. Commercial
software companies are on their own in figuring out the best way to take
advantage of recent agile practices. Commercial software can definitely benefit
from agile development, but it will take a lot more effort and understanding to
balance the constraints of commercial software with the tenants of agile
development (Maxwell, 2007).
The software development team reviewed different forms of Agile, such as
Extreme Programming (XP) (Wells, 1999) and Scrum (Schwaber, 1996). The project
team was searching for a methodology to fix the organizational issues and to improve the
productivity and morale of the team. Of specific interest to the team was the availability
of any published research regarding the application of Agile software development
methods within a commercial software organization, and specifically within companies
that are doing fixed-price software development.
As the team reviewed the choices, Scrum was selected as the new development
methodology due to its inclusive team roles, daily meetings, self managing teams, and
product ownership processes. The development team adopted the Agile methodology
called Scrum in the desire to gain the benefits of reduced cycle time and cost reduction
that are claimed by the adopters of the Scrum methodology. The following description is
20
27. having completed working code to demonstrate at the end. During the Sprint, the team
members were required to self report on their daily progress and, when a working feature was
checked into the build, the team member would show completion and move on to the next
feature. This process of taking features off the backlog during the Sprint is called the Sprint
Burndown.
At the scheduled completion of the Sprint, a team meeting was held to have each
team member demonstrate the code they had developed and built into the product. The
meeting gave the team members a sense of accomplishment and the team morale improved
significantly over the previous Waterfall method.
At certain intervals, when a set of features were completed, a customer demonstration
was arranged to show the customer the progress being made and to ensure the requirements
had been implemented correctly.
The Tools
The original code was developed using several coding environments, many of
which are now obsolete. Some developers only knew one or two coding languages, so
could not contribute in areas where another language was used. In the new order, coding
was done using two languages, C++ and C#, Microsoft Visual Studio and using
Microsoft Team Server and Conchango’s Scrum for Team System Template to manage
and automate the build process. All developers were trained in the new languages and
use the Visual Studio environment. When they checked in their completed code, it was
included in the daily automated build.
HP Mercury test tools were purchased to automate the test process. Test scripts
were built by developers in collaboration with testers from the feature requirements. The
product owners reviewed the test scripts to ensure they complied with the customer
23
28. requirements. All this was done prior to any code being developed. The test scripts
described how the end result should work so there was little ambiguity. This enabled the
developers to test their code and compare it to the expected results during development as
opposed to waiting several weeks for a system test. This immediate feedback reduced
cycle time, increased productivity, and improved the quality of the code.
A full regression test was performed at the end of each Sprint. This was only
possible due to the automated test tools. It highlighted areas where new code broke old
code, and developers had to fix any bugs prior to the end of the Sprint. Any code that did
not pass the final regression testing was considered incomplete and the team could not
take it off the backlog. It was rescheduled for the next Sprint.
The Results
The project team adopted Scrum because of the desire to find a better way to
develop software over the previous Waterfall method. Some team members did not buy
into Scrum and were skeptical but, as time went on, most accepted it. The main reason
given was that it gave each individual the ability to have some say in what they did each
day. As a result, the morale of the organization improved. For example, one developer
who had been cycling over the weekend broke both his wrists in a fall and ended up bed
ridden in the hospital. He asked his wife to bring his laptop to the hospital so he could
continue to work. Peer pressure was a tremendous asset in getting marginal contributors
to improve their productivity. The team owned the results and most wanted to succeed.
But, the end goal is to deliver on time, on budget, and meet the requirements.
This did not occur. The team, in the rush to adopt Scrum, did not follow through in
forecasting the progress towards the schedule. In fact, only a rough schedule existed
since it was not clear how many backlog items would be completed for the Sprints. As
24
29. time went on, the teams found increasingly detailed requirements that needed to be added
to the backlog. The product backlog Burn Down did not support the schedule, but the
developers optimistically assumed they could still make the delivery date. Everyone
wanted to succeed and evidence that this was not going to happen was ignored.
Management was given the assurance that the delivery date would be met up until
the last Sprint. Only then did the team acknowledge that they could not deliver on time.
The customer was notified and was not pleased with the delay but gave the team more
time to complete their work. After several more months of work, it was determined to
negotiate an end to the contract as the team was not close to completing the project.
Lessons Learned
The project was a failure in the business sense, and one could suggest that the
project would have been better off staying with the previous Waterfall methodology.
That was not the case here. The amount of working code developed by the team in less
than a year exceeded the efforts of another project team that had invested several years in
a similar product with hundreds of developers. The productivity exceeded anyone’s
expectations. So what went wrong? The original estimates were inaccurate because they
did not have enough detail regarding each requirement. A feature that was estimated to
be a few days took weeks due to the additional requirements discovered when it was
detailed in the user stories.
Once it was discovered that the requirements were not in sufficient detail, effort
was made to go through the entire backlog and add missing requirements. Upon
completion of this task, the backlog almost doubled. This would have been the time to
escalate the discrepancy but the project team believed they could improve their
productivity and take on the additional requirements in the remaining four-week Sprints.
25
30. Also, the team failed to notify the stakeholders about the potential impact of the
additional requirements but continued to optimistically believe the project would be
completed on time.
The Product and Sprint backlogs should have been managed as a measure of the
Sprint teams’ productivity and used to assess the progress toward the goal. One problem
with this approach is that in a typical Scrum project, the requirements are viewed as
flexible and can be pushed out, delayed, or renegotiated with the customer so as to keep
the team on schedule and deliver a working product. In a fixed-price contract, that
flexibility is not available, as the customer has given all their requirements and the
business has agreed to deliver them within a fixed schedule. When delays occur, the
customer can charge penalties and/or cancel the project. This eliminates the flexibility
that Scrum expects, as the end goal is to deliver something that is viewed as good
enough, not necessarily everything the customer wants.
The adoption of Scrum also meant the end of project management which was
viewed as a product of the Waterfall methodology. That was a mistake. No matter what
methodology is adopted, there is a need for someone who is monitoring the progress of
the teams against the schedule, identifying issues that need to be resolved, and reporting
the progress to the stakeholders. The lack of project management allowed the team to
ignore the obvious, that the remaining product backlog could never be completed on
time.
The lack of proper requirements up front, and the resulting impact on the project,
shows the need to continue the BUFR process on fixed-price contracts. As discussed in
Chapter 1, proper estimates cannot be made without detailed requirements. Bad
26
31. estimates cause the business to suffer financial loss and the customer suffers the loss of a
product they need to run their business. Everyone loses. It would be great if all
customers partnered with the business to create software that is good enough and allow
the flexibility to create that on a pay-as-you-go basis but government entities do not have
the political will to take the risks associated with cost-plus or Time-and-Materials (T&M)
contracts. In Scrum, the up-front requirements are only what is needed to get the project
started and provide some overall guidance; they are detailed at the beginning of each
Sprint. This process does not work in a fixed-price contract environment.
Adopting Scrum, overall, was a benefit to the organization as it instilled
discipline, daily Scrum meetings, peer pressure, individual roles rather than
organizations, and individual accountability, that are not seen in a typical software
development environment. The real benefit to the business, was allowing the developers
to manage themselves rather than have someone tell them what to do. This had the
greatest impact on the organization. Combining the different groups, testers, product
managers, and developers into a single team removed the walls, both figuratively and
literally, between team members.
27
32. CHAPTER 4: CONCLUSION
This study reviewed the transition of a project team from utilizing a Waterfall
methodology to Scrum and the results from applying this method to a software
development contract with a fixed scope, fixed schedule, and fixed-price. The thirty
member project team, divided into three Scrum teams, implemented new tools and
processes in an attempt to improve the predictability and productivity of the software
development process. The productivity results were realized, but not predictability. The
project failed to complete within the scheduled time frame and the customer cancelled the
contract.
Agile and Scrum failed to make the project a success. So does that mean Agile
will not work for fixed-price contracts? Was it the methodology that failed? Would
Waterfall have saved the day? In the context of this case study, the implementation of
Agile did not fully replace the processes that are required for a fixed-price contract. The
biggest failure was the lack of project management to oversee the progress of the team
and identify the lack of timely progress. The project team chose to follow classical Agile
methods, which do not utilize project management, but depend on the team to keep track
of their progress. Scrum also does not fully develop detailed requirements until the
beginning of each Sprint. Thus, the big picture is obscured and classical project
management approaches do not work. The task to create each requirement is not defined,
so the team cannot be held to a pre-determined estimate that can be measured. Progress
is measured on the implementation of the features for each Sprint. The remaining
features are not developed, so the amount of time remaining to complete the project
remains somewhat of a mystery.
28
33. Agile does not recognize the concept of a fixed schedule with a fixed scope. It
relies on the flexibility of schedule and requirements to deliver product that works in
short iterations. This approach eliminates the BUFR and BUFD processes and replaces
them with a cursory requirements process that gets more detailed at the beginning of each
Sprint. Without detailed requirements up front, the estimates, required for a fixed-price
contract, are not correct, and there is little chance of the project being completed within
the pre-determined schedule. Scrum does not require, in fact, discourages, the level of
detail that would be necessary to develop a complete project schedule. But, without this
detail, it is impossible to ascertain the status of the project other than to compare the
remaining requirements list against what has been completed. What is not known, is the
effort remaining to complete the requirements. The lack of detail eliminates any level of
predictability that is a necessity for fixed-price contracts.
Recommendations
In the Waterfall days, the project manager was managing a matrix organization,
had a fixed schedule estimated from a well defined set of requirements, and tracked
progress against the schedule to get the product completed. Under Scrum, the project
management was dropped by the project team since the team now owned the schedule,
and this contributed to the failure of the project. The project team did not have a pre-
determined schedule, but only a list of requirements that the team members selected from
at the beginning of each Sprint. The inability to track progress against the fixed delivery
schedule caused the stakeholders to doubt the team’s ability to complete the project, so
when the delivery date was missed, the team’s credibility was damaged and the contract
cancelled. It is recommended that project management be continued in any Agile
implementation. There is a benefit to having an objective observer monitoring and
29
34. tracking the team’s progress and providing feedback to the stakeholders, giving them
confidence in the team’s ability to deliver on time.
Big Up Front Requirements and, to a smaller degree, design is a necessity for
performing any fixed-price contract. The degree of effort up front determines the
accuracy of the estimates and pricing. Typical Agile projects minimize this effort since
the method encourages customer feedback and redefining requirements during the
process. This approach does not work under the conditions of a fixed-price contract and,
therefore, BUFR is required. A scope and schedule are required, and the process chosen
needs to support this effort. The only way to tell if a project is on schedule is to measure
progress against original estimates.
Agile provided an improvement in developer productivity along with the ability to
review completed code in very short increments. This quick feedback to both developers
and customers provides ongoing assurance that work is being completed and also shows
how requirements are being implemented. Any discrepancies in the requirements can be
resolved and the changes implemented within another four-week Sprint as opposed to
waiting until the final product is finished.
The collaboration, peer pressure, and daily Scrum meetings add a high level of
visibility to the developers and holds them accountable to meeting their commitments.
The handoffs between development and test are eliminated, as are the issues with regard
to the testing process since both testers and developers are on the same team and have the
same goals. Developers are engaged in creating the test scripts with the testers before
developing code so there is little ambiguity as to how the final result should perform.
30
35. Limitations
This study was limited to one project and one business unit within a larger
company. The team had practiced Waterfall development up until the adoption of Scrum
and thus had no exposure to Agile until this project. A broader exposure to other Agile
methods may have caused the team to choose a different methodology than Scrum.
Time was another limitation, as the study was completed in 12 weeks. Had more
time been available, further analysis of the team and lessons learned could have been
performed.
The Agile development method chosen by the team was Scrum, which is only one
Agile method. There are several Agile methods, such as Extreme Programming, Lean
Software Development, Feature Driven Development (FDD), and Adaptive Software
Development (ASD), that were not evaluated and may be better alternatives for fixed-
price contracts.
The scope of the study is Agile development within the context of a fixed-price
contract and this is a limitation of this study. Other contractual alternatives to fixed-price
contracts were not examined.
The scope of the case study limited it to a single project which was the first
implementation following the adoption of Scrum. If the team had more experience with
Scrum on more projects, the outcome may have been quite different. Agile itself is agile
and thus capable of adapting to new methods of implementation. A team with more
experience may have foreseen the pitfalls and created methods that would be a better fit
for a fixed-price contract.
31
36. Areas of Future Work
The business unit continues to utilize Scrum to some degree on other projects. A
follow-up study should be performed to determine what aspects of Scrum continue to
provide a benefit to the software development process.
Fixed-price contracts limited this study and had a negative effect on the project.
An area of future study would be to evaluate other contractual methods that would be
more consistent with Agile development methods. Some alternatives would be Time and
Materials (T&M), Cost Plus, or other concepts that allow changes to the requirements,
without contractual obligations, during the development process.
This study only evaluated one Agile development methodology and there are
other Agile methodologies that should be studied that may be a better fit for a fixed-price
contract. Scrum may not be the best methodology for fixed-price contracts, the use of
other Agile or semi-agile methods should be examined to determine if there is a better
methodology for these projects.
Other software development methods, besides Agile, should also be studied.
Software development continues to play an ever increasing role in the development of
products and the creation of new business processes. Finding a methodology that will
facilitate greater software development productivity and predictability would provide
huge benefits to the software industry.
A recurring theme in the fixed-price contract environment is the issue of trust. In
other projects, such as construction, the customer can see the results and the progress
toward the goal. Agile provides this same degree of visibility for software projects by
giving the customer demonstrations of working code at the end of each four-week Sprint.
This builds trust between the customer and the project team as they see the product being
32
37. developed. So, how do you get the typical fixed-price customer to trust that the company
will provide the best solution in the shortest amount of time without a fixed-price
contract? This is an area of further study that could revolutionize the software industry as
customers become more sophisticated and partner with the development business to
create products they need.
Commercial software companies need to develop better contracting vehicles that
create a trust relationship with the customer. Over time, as companies prove their ability
to deliver better products at less cost using Agile software development, customers will
move toward a more flexible method of developing requirements. This will benefit all
parties by allowing developers to efficiently manage their own processes while delivering
high quality products to their customers.
33
38. REFERENCES
Ambler, S. (2003). Something’s Gotta Give. Retrieved from
http://www.ddj.com/architect/184414962?cid=Ambysoft
Beck, K; et al, (2001). Manifesto for Agile Software Development. Retrieved from
http://www.agilemanifesto.org/
Begal, A., & Nagappan, N. (2006). Usage and Perceptions of Agile Software
Development in an Industrial Context: An Exploratory Study. Retrieved from
http://research.microsoft.com/hip/papers/AgileDevAtMS-ESEM07.pdf
Bayer, S. & Highsmith J. (1994). RADical Software Development® 1994.
American
Programmer Magazine, June. 20-26. Retrieved from
http://www.jimhighsmith.com/articles/RAD_article.htm
Cauvin, R. (2005). BUFR. Retrieved from http://cauvin.blogspot.com/2005/09/bufr.html
Charette, R. (2005). Why Software Fails. IEE Spectrum, (September). 1-2. Retrieved
from http://www.spectrum.ieee.org/computing/software/why-software-fails/1
Cockburn, A. (2005). Fixed-Price contracts in an agile organization. Retrieved from
http://codebetter.com/blogs/darrell.norton/archive/2005/04/22/62326.aspx
Fredrick, C. (2007). How Douglas County, CO Cut A Project Timeline In Half.
Retrieved from http://www.agilejournal.com/content/view/276/111/
Fowler, M. (2003). Fixed-price. (July). Retrieved from
http://www.martinfowler.com/bliki/FixedPrice.html
Hanly, S. (2006). Zen and the art of software development. Retrieved from
http://www.bcs.org/server.php?show=ConWebDoc.6631
Hartman, D. (2006). Interview: Jim Johnson of the Standish Group. Retrieved from
http://www.infoq.com/articles/Interview-Johnson-Standish-CHAOS
Jensen, R. (2003). Lessons Learned From Another Failed Software Contract, Crosstalk
Journal, (Sep.) Retrieved from
http://www.stsc.hill.af.mil/crosstalk/2003/09/0309jensen.html
Johnson, J. (2002). Keynote speech at XP 2002, Sardinia, Italy. Retrieved from
http://ciclamino.dibe.unige.it/xp2002/
Larman, C. (2003). Agile and Iterative Development: A Manager's Guide, Addison-
Wesley Professional, 1st
Edition. 27.
34
39. Mar, K. & Schwaber, K. (2002). Scrum With XP. Retrieved from
http://faculty.salisbury.edu/~xswang/Research/Papers/SERelated/scrum/ScrumXP.pdf. 3.
Marchenko, A. (2008). Why no fixed contracts in Agile. Retrieved from
http://agilesoftwaredevelopment.com/blog/artem/why-no-fixed-contracts-agile
Maxwell. (2007) Agile Development For Commercial Software. Retrieved from
http://www.exotribe.com/maxx/2007/03/02/agile-development-for-commercial-software/
Miller, J. (2007). The Shade Tree Developer. Retrieved from
http://codebetter.com/blogs/jeremy.miller/archive/2007/08/31/trying-to-answer-hard-
questions-about-agile-development.aspx
Northrop, R. (2004, March). The Fall of Waterfall. Retrieved from http://intelligent-
enterprise.informationweek.com/info_centers/imperatives/6/showArticle.jhtml?articleID
=17701624
Poppendieck, M. (2006). Is Agile Software Development Sustainable? Retrieved from
http://www.poppendieck.com/sustainable.htm
Rashid, D. (2006). Agile and Design for Lean Six Sigma. Retrieved from
http://www.airacad.com/default.aspx?Page=PaperAgileDFLSSSoft
Royce, W. (1970). Managing the Development of Large Software Systems, Proceedings
of IEEE WESCON, August. 1-9.
Schwaber, K. (1996). SCRUM Development Process. Retrieved from
http://jeffsutherland.com/oopsla/schwapub.pdf
Schwaber, K. (2004). Agile Project Management With Scrum. Microsoft Press, 1st
Edition. 1-32.
Schwaber, K. (2006). Scrum For Team System. Retrieved from
http://www.scrumforteamsystem.com/ProcessGuidance/ProcessGuidance.html
Schwaber, K. & Bird, C. (2009). Scrum For Team System. Retrieved from
http://www.scrumforteamsystem.com/processguidance/v2/Process/Process.aspx
Sommerville, Ian. (1995). Software Engineering. (5th
Ed), Addison-Wesley. 85.
Soundar, V. (2008). Choice of Agile Product Development Model. Retrieved from
http://agilesoftwaredevelopment.com/node/441
35
40. 36
Turk, D., & Vijayasarathy, L. (2008). AGILE SOFTWARE DEVELOPMENT: A
SURVEY OF EARLY ADOPTERS. Journal of Information Technology Management,
ISSN #10421319 A Publication of the Association of Management. 1-9.
Waters, S. (2008). Traditional Software Development versus Agile Development.
Retrieved from http://www.personal.psu.edu/saw283/
Walton, B. (2004). Iterative vs. waterfall software development: Why don't companies
get it?. Retrieved from http://computerworld.com/printthis/2004/0,4814,90325,00.html
Wells, D. (1999). Extreme Programming: A gentle introduction. Retrieved from
http://www.extremeprogramming.org/
Wilson-Welsh, P. (2003). The Waterfall Approach: a Critique. Retrieved from
http://adaptionsoft.com/adios_waterfall.html
Zwicker M. (2007). War Stories - Fighter Jets and Agile Development at Lockheed
Martin. Retrieved from http://www.agilejournal.com/content/view/313/111/
