Session discussion notes from DAAG 2015, Huntington Beach. These notes were taken from the session "Metrics Patterns" conducted by Reidar Bratvold and Somik Raha

1. Metric Patterns
Community Discussion

2. Cultural Change (as relates
to DQ)
• Definition of Culture
• Culture is the stuff you won’t tolerate (the behaviors you reject)
• Culture is shared values and norms
• Culture is a shared aspiration
• If you don’t act in keeping with the culture, you risk being
ostracized
• You know it is part of your culture when it is okay to call someone
out for not doing it
• Gives the less powerful person “standing”

3. Metrics for Cultural Change (DQ)
• Productive metrics
• What % of our decisions meeting basic requirements of DQ ..
• # of executives or powerful people attributing their success to DQ
• # of business process that conforms to DQ standards
• # of times we anticipated something others consider a surprise
• Rate at which decisions are made (with quality) (all of the above from David Matheson)
• Metric: Your % assessment of quality in a decision, Context: Decision maker and/or project team trying to make
a decision, Story: Their own assessment of their level of DQ is a much better motivator than your view. Thinking
about each element in the chain naturally promotes the next set of actions.
• Counter-productive metrics
• Metric: No of people trained, Context: Cultural Change, Story: Mandated training. The Intel Decision Quality
Program office is considering using # of employees who’ve been through the introductory 3-day DQ course as a
metric measuring of our goal towards changing the culture of Intel. This doesn’t account for clarity within a given
area, how many people are actually applying DQ principles or if those people are even still employed at Intel.
• Metric: # of decision analysts
• Metric: Cost of implementation, Context: Whether to pursue DQ related education, Story: If I can tell you how
much it will cost for this massive company to adopt DQ, rather than discuss the potential value, you will suffer
from sticker shock and not adopt — Jim Driscoll
• Neutral
• Existence of a mandate
• # of instances of push vs pull in terms of DQ use

4. Operational
• Productive Metrics
• US FDA - Time to 1st response for regulatory filings seeking approval and
marketing authorization
• Counterproductive Metrics
• Metric: Count of # of projects passing development milestone each year;
Context: Operational Performance; Problem: Incentivizes quantity over
quality — Jon Mauer
• Metric: Billable hours versus chargeable hours; Problem: Separating these
metrics creates a value gap and reduces the focus on time spent
• Metric: Utilization; Context: Unit Operations; Problem: Utilization as a sole
metric for tracking profit maximization. This metric has no notion of cashflow
included in it. — T J Iezzoni

5. Rankings
• Productive Metrics
• Metric: Rank ordering of employees; Context: Fit of role and responsibility to employee and
opportunities .. (illegible); Story: Employees have a clear understanding of how their
performance compares to general peers - can be motivating to certain personality types — R
L K
• Counterproductive Metrics
• Metric: GPA; Context: Ranking candidates; Problem: GPA is not a good absolute
measurement of how well a candidate will do in our organization. Doesn’t show leadership,
innovation, social skills, etc. — Lake McAfee
• Metric: High School Graduation Rates; Context: Education; Problem: You ask me to
increase high school graduation rates for kids in grade school and evaluate me after 2 years.
But my kids won’t graduate high school for 7-8 years! How can you judge a future affect?
• Metric: Rank of business school in US News Ranking; Context: Measure quality of school to
choose where to apply; Story: Business schools lower number of students admitted to
business MBA programs so average GPA + GMAT test scores are higher. So fewer students
get educated. — Robin Keller

6. Business (Balanced Score
Card)
• Productive
• Metric: Balanced Scorecard; Context: Measuring your
strategy by using BSC; Story: you are measuring your
performance only, if you … move toward achieving your
strategy — Osarver
• Counter-Productive
• Metric: Balanced Scorecard; Context: Rank business
decisions; What: A weighting of four factors - customer
satisfaction+… ; Problem: This often results in “more balance
but less score” i.e. Shareholder value creation; Subjective; Not
quantifiable

7. Risk Management
• Define what we/how we measure risk: Must capture
likelihood/consequences
Mike Fulton
Aircraft
Launch
Reliability
Enrico Manlapig
Std. Dev./
Volatility
Steven Glickman
Open Risks
(count)
Greg Parnel
1-5 Risk
Scale
Green
Useful
|||| |
Yellow
Neutral
III II
Red
Counter-
productive
I I I

8. Risk-Management
• Neutral
• Metric: The number of open risks; Context: How close is the program to
completion? Story: The number of open risks provides focus of what to work
on, and where to put priority. The largest expected risk consequence should be
worked first. — Steve Glickman
• Metric: Launch reliability; Context: Divide/conquer what to choose — Mike
Fulton
• Counter-Productive
• Metric: Volatility (Variance); Context: Non-scaled risk, Working Distributions —
Enrico Manlapig
• Metric: Risk Metrics, Ordinal Risk Matrices; Context: Ordinal risk matrices that
do NOT define the type of risk, e.g. technical, cost or schedule — Greg Parnell

9. Advertising or Personal
• Counterproductive
• Metric: Unique views per webpage; Context: Volume of traffic measures exposure of your content;
Problem: Often if a web page receives high unique viewership, the company believes they created an
effective product. But if the viewers do not act on the information, no value was really created. Org
continues to publish similar content. — Kelly Herwick
• Metric: Eyeball/Impressions/Views; Context: Advertising; Pro: Often measured, though not effectively;
Con: Doesn’t track behavior or measure value of different eyeballs — Elisabeth Browne
• Metric: $ sent to me; Context: Life (Trust me, I’m in advertising) — Mona
• Metric: Clicks on web ad; Context: Advertising; Pro: Easily measured and poss. tied to specific ad
campaign; Con: Doesn’t tell you if sale was made. — Elisabeth
• Metric: Facebook “friends”, LinkedIn connections; Context: Networking; Pro: Easy to Count; Con: Tells
you nothing about quality of connection — Elisabeth
• Productive
• Metric: Sales of Advertised Product; Context: Advertising; Pro: Measures what advertiser is trying to
maximize; Con: May be difficult to tie to /track/measure w.r.t. advertising strategies. — Elisabeth Browne

10. Financial Metrics
• Counterproductive
• Metric: Probability-weighted cashflow; Context: This metric used alone to rank
business decision opportunities; Problem: Used as the only metric does not
allow me to see drivers of positive cashflow, nor the 80% confidence cashflows.
— Ellen
• Metric: R&D as a % of Sales/Funding execution %; Context: Rates as measure
of performance; Problem: Only considering funds execution wrt the
accompanying capability or requirement does not measure operational
success. — Ford
• Neutral
• Metric: Program execution vs planned execution; Context: Comparing plan to
actual achievement levels; Problem: Plan vs actual by program is a part of
project or program management’s basis and can assist in measuring
performance — Ford

11. Financial Metrics
• Counterproductive
• Metric: Nominal NPV; Context: Ranking investment alternatives; Problem: In high-risk/high-uncertainty industries, using
nominal NPV to rank investment alternatives would fail to include risk/uncertainty into the discussion — Michael Lee
• Metric: NPV given success; Context: Go/No Go decisions, deciding how much to bid for an acquisition; Problem: NPV
given success only considers the success case ignoring risk, leading decision makers to get enamored with “what could
be” to an unrealistic extent. This leads to going forward with too-risky projects, and overbidding for assets. - Michael Lee
• Metric: NPV; Context: Measure for economic value added for a major business decision; Story: Forties and Prudhoe
bay late life production kept BP “alive” (shareholder value) in 1990’s — but at investment decision time those barrels
would have been discounted to 0. — Steve Begg
• Metric: NPV; Context: Ranking Business Decisions; Story: Does not incorporate non-financial aspects. NPV on its own
can mislead. Also doesn’t capture uncertainty.
• Productive
• Metric: NPV; Context: Short-term economic benefit if simple decision (decisions where risk-neutrality is reasonable);
Story: economic value of an infill well, better than “barrels added,” accounts for cost, benefit and time value of money —
Steve Begg
• Metric: APV; Adjusted Present Value; Context: Ranking Business Decisions; Story: APV factors in the risk and volatility
of components, understanding all components are not equal. — Josh
• Metric: Risk-adjusted NPV; Context: Ranking investment alternatives; Story: In high-risk/high-uncertainty industries,
using risk-adjusted NPV to rank investment alternatives would include risk/uncertainty into the discussion

12. Productivity
• Productive
• Metric: RA Bang for the Buck — Cumulative Sales over R&D cost to
launch — Zach
• Counterproductive
• Metric: Risk-adjusted CAGR; Context: Ranking pipeline programs;
Problem: Risk adjustment on a single program basis doesn’t always
make sense, since you will never actually get that value. — Zach
• Metric: Number of phase go decisions for a given R&D budget;
Context: Corporate goals; Problem: Using phase go decisions in
corporate goals divers focus from making high-quality decisions. —
Michael Lee

13. Abnormal Accounts (Risk)
• Productive
• Metric: Risk-adjusted productivity and rate of return; Story: Useful for
ranking portfolio programs relative to each other on a dimensionless
basis
• Neutral
• Metric: Abnormal Accounts payable; Context: Accounting;
Comments: a measure of risk, but not all risk — Ford
• Counterproductive
• Metric: Deterministic IRR; Context: Ranking Business Decisions;
Problem: IRR of two deals, alone, as it cannot capture inherent
uncertainty in each project. — Ellen Coopersmith

14. Non-Profit
• Productive
• Metric: % of $ that reaches end user/dedicated to cause; Reason: Non-profits often use metrics
on total $s used or goods purchased/ donated but this doesn’t measure the efficiency of these
non-profits in how they were able to …/donate these goods
• Metric: Increase/Decrease in population affected
• Metric: Feelings/Impact on donors
• Metric: Hopefulness
• Neutral
• Distributed goods and services
• Counterproductive
• Metric: Giving efficiency; How much of a donor’s money goes to impact vs overhead?
• Metric: $’s raised; Problem: Gives no indication as to whether that money is being effectively
utilized to effect change in a given area

15. Environment
• Productive
• Metric: Habitat loss thresholds; Context: Environmental impact assessment; Reason: Has context in which to interpret the
results — Laura Keating
• Neutral
• Metric: Hectares of habitat loss for key indicator wildlife species in an environmental impact assessment; Context:
Environmental impact assessment; Reason: Habitat modeling can be used to inform project-specific impact assessments
for wildlife. The hectares of habitat lost can be difficult to interpret in the absence of regulatory guidance — Laura Keating
• Counterproductive
• Metric: Surface Temperature Change; Context: Climate Change; Problem: Effects of Climate Change may be more
significant in other contexts than surface temperature (e.g. changes in ocean life) — Stephen Leung
• Metric: Red/Green/Yellow Risks; Context: Product Technical Success; Story: Often our company sues colors to articulate
risk, particularly in product development and execution. These do not in any way articulate the probabilities of risks in a
consistently understood way nor the magnitude of impact. —- Jordan Stephens
• Metric: % of habitat loss; Context: Environmental impact assessment (wildlife); Problem: The % of habitat is dependent on
the spatial scale, which is often inconsistent across projects and not always biologically relevant. — Laura Keating
• Metric: Benefit-Cost Analysis; Context: Very long-term projects or phenomena; Problem: Benefit-cost analyses are
shortcuts that can over-simplify a problem, especially when the short-term benefits and costs affect one group and the
long-term benefits/costs affect another. There are numerous examples of positive investments made years ago that would
never have passed a benefit-cost analysis. — Pat Leach

16. Health• Productive
• Severity
• impact of life
• Quality of life
• Survival Rate
• # of ptns
• Metric: Daily weigh measured in AM; Context: Try to lose weight; Story: If you weigh before
eating during day, you can be reminded to control food input or increase exercise — Robin Keller
• Metric: % have PFAC patient and family adv. committee; Context: Healthcare/hospital
administration; Reason: Patient-centered care is important topic - having PFAC to provide input
• Counterproductive
• Unmet Medical Need — Metric: Peak Sales; Problem: Peak sales used as measure because it
contains volume and pricing, both assumed to be important indications of unmet medical need.
But competition marketing, timing are all confounded. Solution: Multi-objective measure of
unmet medical need including volume, severity, amount address by new drug
• Peak Sales
• Wait time for veterans

17. Health
• Counterproductive
• Metric: % patients 50 and older who have had colonoscopy
(“up to date”); Context: Healthcare — primary care provider
“report card”; Problem: There are other means of screening
for colon cancer (e.g. stool card test) that have similar
efficiency but their use is not easy to track so providers will get
dinged if patients choose other options, which prevents patient
choice — Karen Sepucha
• Metric: Total Cholesterol; Context: Assess cardiovascular risk;
Story: A patient can have high total cholesterol due to “good”
cholesterol. The ratio can be very healthy, despite high total
cholesterol — Robin Keller

18. Oil & Gas
• Counterproductive
• Using a single metric can be misleading
• Simple is good but must be a good proxy, usually need more
than one
• Useful
• Metrics must be relevant to the decision makers to the implied
goal the metric is trying to describe
• Neutral
• There is no natural - either helpful or not helpful

19. Oil & Gas
• Productive
• Metric: Energy intensity index; Context: Refinery energy consumption; Reason: Benchmarks refinery energy use vs other
refinery across the globe. Not perfect but better than straight intensity or consumption. — Luke McAfee
• Metric: Capital Efficiency; Context: Ranking and selecting alternatives; Reason: Looking at Exp. NPV per $ invested allows
you to maximize value for a given budget — Frank
• Metric: No. of drillable (=mature and worthwhile) exploration prospects; Reason: Shows progress of maturation work, and
understanding of portfolio.
• Counterproductive
• Metric: Unit cost; Context: Selecting activities to continue or eliminate; Problem: In response to price pressure companies
eliminate activity that increases cost but also reduces revenue so value’s ultimately reduced
• Metric: Rate of return (ROR); Context: Business Ranking Decision; Problem: ROR is used as a measure for ranking
projects and/or establishing an interest in invested potential. It has a weakness because often is not probabilistic and it is
very difficult to calculate it probabilistically; Alternative: Value investment ratio
• Metric: Expected Value (EV); Context: Ranking projects in a portfolio; Problem: By using EV of a discounted cashflow to
rank project investments in a portfolio of opportunities, it biases ranking towards the largest investments which have
positive cashflows and typically have larger EVs, but not necessarily the best investments with the highest rate of return for
scarce investment resources. Risk of EV is also not considered when comparing only EV to rank investments. — Jim Weller,
Endeavor Management
• Metric: Exploration volume targets; Problem: Unproductive focus on volume revisions and share assets as not economic