1. Agile Metrics (by Venkatraman L) 1

2. Agile Metrics (by Venkatraman L) 2
Agile Metrics (by
Venkatraman L)
In Projects & Portfolio
Venkatraman L
Head of Project Management
InMobi
20-Jun-13

3. Agile Metrics (by Venkatraman L) 3
Table of Contents
Introduction.................................................................................................................................... 4
Abstract...................................................................................................................................... 4
Keywords................................................................................................................................... 4
Introduction................................................................................................................................ 4
Context of the project............................................................................................................... 5
Context of the program............................................................................................................ 5
Context about the portfolio of releases................................................................................. 6
Overview of Agile ......................................................................................................................... 6
What is Agile ............................................................................................................................. 6
How is it different from the Traditional (waterfall) model.................................................... 7
So, how does Agile solve for it?............................................................................................. 7
How does the triple constraint stack up in Agile.................................................................. 7
Is Agile applicable on all type of projects? ........................................................................... 8
Managing Projects through Agile............................................................................................... 8
Metrics during the Plan Phase ............................................................................................... 8
The Sprint – What is measured during execution............................................................... 9
Risk & Quality Metrics ........................................................................................................... 10
Managing a Program / Portfolio of Releases......................................................................... 10
The Dynamics of a portfolio.................................................................................................. 10
The Prioritization Phase ........................................................................................................ 10
The Planning & Execution Phase........................................................................................ 11
Scrum-Ban and Scrum-fall.................................................................................................... 11
Conclusion................................................................................................................................... 12
Snapshot.................................................................................................................................. 12
References .......................................................................................................................................................13

4. Agile Metrics (by Venkatraman L) 4
Introduction
Abstract
As the cliché goes “You cannot manage what you don’t measure”, it has been always a debate about how
much to measure and what to manage. Agile Projects have been of late topic of discussion for all
reasons. One of them in particular has been the impact of capturing less metrics for managing and
control. To define what is “less” is the tough part. Tougher is to understand what to measure and why it’s
being measured. Agile does not prescribe that set of things has to be followed. If you do so, you are agile.
If you don’t, you can still be agile. The key is in understanding the problem that the teams are expecting
to solve. The paper discusses the various metrics that can be used in Agile to plan, execute and monitor
the entire project over the lifecycle. There is also a fundamental need to measure some of the metrics
that were traditionally in practice, in order to bring out a change, even in Agile teams. Also, these metrics
and the changes that they facilitate are different at the project, program and portfolio levels. The paper
provides a view of the metrics at project and portfolio levels broken down as Technology & Engineering,
Operations and Business metrics. The paper ends with a detailed matrix of the metrics across the three.
Keywords
Story Points, Velocity, Escaped Defects, Burn-down, Burn-up, Ideal Time, Agile EVM, Cycle Time
Introduction
The paper describes a project scenario, a program and a portfolio of releases and how the agile metrics
are used across the varied type of delivery needs. The key is also to understand how some of the metrics
are critical for the success for a program and the portfolio (as the portfolio is a set of programs, releases
and projects eventually).

5. Agile Metrics (by Venkatraman L) 5
Context of the project
The project that is referred here is both a NPD (new product development) and enhancement of the
existing features of the product. The product is a portal that enables users to watch videos of their choice.
The NPD mentioned above talks about revamping the entire watching experience (the new product of the
same capability was built from scratch and replaced the existing product completely). The team in this
case was a self-sufficient team (front-end, middleware, backend, QA and Operations) and the team size
was more or less intact through he duration of the project. However, the teams were a bit distributed (the
product owner and the operations team were in the US and the rest of the team were in India).
Context of the program
The program here, at a different organization refers to versions of releases that define a program. It is
part of the portfolio as mentioned below but its primary objective is to meet the program goals.

6. Agile Metrics (by Venkatraman L) 6
Context about the portfolio of releases
The portfolio that is being referred is a set of releases (>40) cutting across multiple themes in the
organization. There are around 8 themes and the 40+ releases map to a theme. The key to note is that
each release is “contributed to” by more than one team in the organization. The teams are formed by the
nature of the architecture and they are not cross-functional unified team. The colored lines below shows
how multiple teams (M) contribute to multiple releases (N). The organization terms this as the “Release
Dependency Matrix” and is the key behind the success of the project.
Overview of Agile
What is Agile
Agile is a set of values and guiding principles that guide the teams and organizations to the path of
successful delivery. The focus is on value driven delivery than just the triple constraints of Cost, Time and
Scope (Jim Highsmith).

7. Agile Metrics (by Venkatraman L) 7
How is it different from the Traditional (waterfall) model
So, how does Agile solve for it?
Agile is based on Deming’s PDCA (Plan, Do, Check, Act) and focuses on a concept of iterative and
incremental development and delivers the highest value features sprint over sprint (a time-boxed duration
of work is called a sprint). The customer (or the product owner) focuses on always providing the list of
high priority requirements to the team that the team commits to, sprint over sprint. The customer (in the
role of a Product Owner) is expected to be available all through the project lifecycle so that the teams can
collaborate well and get faster feedback from the customer. The team plans every 10 days on a set of
requirements, (that have been stack ranked by the product owner) discuss and thrash the details, commit
to delivering it within the 10-day cycle (sprint). Ad the end of the sprint, the customer is shown a demo of
how the product or feature is progressing. This helps to provide faster feedback and WYSIWYG (what
you see is what you get).
How does the triple constraint stack up in Agile
In case of the traditional waterfall mode, the scope is provided and the project manager develops a
detailed plan to articulate the cost and the time it takes to develop the scope. When scope changes, the
Waterfall model (Traditional model as it is called) focuses on staged
development. Requirements, high level design, low level designs,
development, unit testing, integrated testing, user acceptance testing and
release. The key is that the customer is not always (some purists say never)
that involved in all the phases of the project. The customer eventually takes
a demo during the final phase of the project and end up realizing that what
he / she wanted is a bit far from what has been implemented. This has been
one of the shortcomings of the traditional waterfall model, which has led to
Agile being one the most opted methodologies of delivery.

8. Agile Metrics (by Venkatraman L) 8
entire process is re-run or at least the additional time and cost is being worked out and re-baselined. In
case of Agile, the triangle is flipped around totally. The available constants are the cost (of the team +
operational), the time (2-4 weeks of sprints) and the variable in this case is the scope. One should not
mis-read the variable beyond the need of delivering “high value” scope and not take it as though scope is
completely open for changes all the time.
Is Agile applicable on all type of projects?
Yes, the reason being that Agile is not a process but a set of values, principles and practices, that, put
together can generate wonderful value for the end customer. In large projects that follow waterfall model,
the various phases within the waterfall can be run through agile sprints. For eg: the requirements and
analysis phase that delivers the high level architecture and design can be run as multiple sprints and
each has a specific goal as the deliverable.
Managing Projects through Agile
Metrics during the Plan Phase
When planning a release for a product having multiple new or enhancements, the entire backlog is
broken down into a set of requirements called the product backlog. The teams work off the product
backlog in determining the set of agreeable high priority requirements and commit to the “sprint
backlog”. The various metrics that are used in this case is the concept of ideal days (the duration that is
required ideally if nothing else was being done by the developer / tester), the story points for each user
story (which is nothing but a relative size estimation of the complexity of the work), the velocity of the
team (the number of story points that the team can comfortably deliver). The number of sprints are

9. Agile Metrics (by Venkatraman L) 9
typically calculated by the formula total sprints = total story points for the release / velocity of the
team. This is also called as the Agile EVM – Earned value management where the projection of timeline
happens through the available metrics. Please note that Agile EVM is an elaborate topic in itself.
The Sprint – What is measured during execution
During the sprint execution (after the sprint starts), the teams use the concept of burn-down (a graph that
plots how much of effort in hours) that is still pending to be completed by the team. This plots the
available capacity vs the pending to be completed and compares it with the available work in hand. The
team and in specific the product owner also starts looking at the burn-up (at sprint level) eventually
cumulating at the release burn-up chart level, which talks about how many story points were delivered by
the team. Jim Highsmith also talks about the value points that get associated with the stories that can
also be calculated to understand how much of business value was delivered against the planned. One of
the other key metric to capture in a team is to understand how much time the team is able to focus on
new features vs fixing existing bugs. This is a key in the utilization of the team’s efforts in delivering true
value to the customer. The teams can also capture the stretch factor (how much was the additional
effort put by the team to meet the sprint commitments). These could be due to the improper estimations,
unexpected events of effort loss etc.

10. Agile Metrics (by Venkatraman L) 10
Risk & Quality Metrics
There is a misconception in the community that Agile does not focus on risks. Agile inherently focuses on
the risks of the project and the portfolio by enabling the teams to call it out almost daily if not immediately
as they occur. Mike Cohn also talks about two methods by which Risks can be factored into determining
the schedules using the velocity of the team. They are called the Risky and Rigorous [2] process of risk
management and leads to a risk-adjusted backlog, which is nothing but the product backlog that is
interspersed with, risk responses.
Similarly, the ones that we can capture are the escaped defects that got out of a release and the
customer caught them. There are enough quality metrics that can be captured in terms of priority,
severity of the defects, defects that got moved from the previous release of the product (deprioritized
releases), the defects that got ingested into the system due to fixing few other releases.
Managing a Program / Portfolio of Releases
The Dynamics of a portfolio
Portfolio as the name suggests is used in organizations when their systems and capacity scales across
the board. Portfolios have themes that in turn can map to one or more initiatives or programs. In a fast
growing nature of the business, the themes in the portfolio might have varied number of programs and
initiatives within them. The key is to focus on what is priority for the business and for the customer.
The Prioritization Phase
During the portfolio prioritization phase, the metrics to look at are the ROI (return on investment), NPV
(Net present value) of each of the projects that are within the portfolio. This can be using the “value
points” [1] suggested by Jim Highsmith. The Cost vs Value analysis would provide the details around
whether a particular project or feature needs to be implemented. Eventually, the initiative that the feature
is part of benefits or loses based on the choices made during the prioritization phase.

11. Agile Metrics (by Venkatraman L) 11
The Planning & Execution Phase
During the planning and execution phase, the key is to pick the set of stories at sprint level (user stories
as termed in Agile) that are of high value point [1]. The value flows all the way from the top at the
portfolio level and the costs flow bottom up leading to a consensus. Pie-charts can detail what the
contribution margin that each of the portfolio themes are providing to the organization as a whole.
The other metrics that are important is the cycle time of the entire program (when one team begins and
the final team ends the release) and how the cycle times can be cut down. The other metrics used are the
“Process Cycle Efficiency” that projects how much waste has been cut down in the system.
Scrum-Ban and Scrum-fall
In order to focus better on the cycle time, large programs do the release plans but execute with the
methodology of Kanban. As already called out in the introduction, a combination of waterfall + sprints +
Kanban during execution of the sprints can be really helpful in very large projects where each phase can
be considered a program by itself with clear goals and metrics to achieve. Using Kanban, the following
metrics are captured – lead-time, cycle time, work time and wait time (idle time between various
states). This has been the key to deliver value, faster.

12. Agile Metrics (by Venkatraman L) 12
Conclusion
Snapshot
Here is a short summary of the various metrics that were captured in the above projects. Marked with (G)
is also for the Program and marked with (F) is for the portfolio also along with program
Tech Metrics Operational Business
Project Test Coverage
Escaped Defects
(G)
In Sprint Defects
Cost of rework
Code Quality (G)
Performance (G)
Defect ingestion
Story point velocity per
sprint
Available capacity
Capacity utilization
Cost of the sprint ($)
Story points accepted
Story points not accepted
New scope added in sprint
Technical Debt added
Agile EVM (G)
Risk Register updates (G)
Stretch factor
Time spent on Bugs vs
feature
Value delivered
(P)
Customer
Satisfaction (P)
Program Cycle Time (P)
Process cycle efficiency (%)
Release Burn up
Scope creep
Contractual
metrics ($)
Portfolio Pie-chart of releases
Health of the portfolio
Prioritization changes
ROI in the
portfolio
Contribution
Margin
As called out earlier, this is just a suggestive minimal list of metrics. Based on the governance and the
type of the organization, there might be more metrics such as time spent vs billed, actual billing cards,
agile contracts etc that might need to be added to this list.

13. Agile Metrics (by Venkatraman L) 13
In short, there are enough and more metrics in Agile development that can enable the teams to make
wise decisions using them. The key is to know “what to measure and why”
References
[1] Highsmith, Jim: Agile Project Management: Creating Innovative Products
[2] Cohn, Mike. Agile Estimating and Planning