MaRS Discovery District
Collaboration for Competitive Advantage
The problem with standard social metrics
Dr Gillian Kerr – June 2009

Funders and investors want
standard measures
Increasing numbers of funders and investors are asking for standard
outcome measures and other metrics that can help them decide
which services to support.
Ideally, these measures would be summarized in a report, perhaps
as a supplement to the Annual Report along with the financial
statements, providing an objective and clear picture of achieved
outcomes.
It would then be easy to fund organizations that are doing better,
and move funding away from less successful organizations.
What is wrong with this ideal picture?
2

You can’t compare measures
unless they are comparable
Measures must satisfy several criteria before they can be trusted:
Accuracy
- How well do the measures capture the concept? (construct validity)
- How well were the measures collected and reported? Is the
documentation accurate enough? (reliability)
Relevance
●
Timeliness (are the data current?)
●
Do they measure the right concepts? (related to mission and desired
outcomes)
●
Do they meet the needs of users?
Comparability
-
Do measures use the same definitions and collected the same
way?
- Are they comparable over time and between organizations? 3

Financial statements are
comparable
Let's assume for a moment that audited financial statements are
accurate and relevant reflections of an organization's financial status.
Why would we make that assumption?
●
The format and content of financial statements are based on decades of
development, and each definition has been tested and refined over thousands
of cases.
●
Financial statements have been designed to provide information about an
organization's financial position and performance based on a carefully defined
set of accounting processes that are comparable, reliable and understandable.
●
They must be compliant with Generally Accepted Accounting Principles or
some other recognized professional standard.
●
They are audited by qualified experts who are subject to extensive legal and
professional controls.
Even with all of these controls, it's risky to compare financial statements
directly without understanding the organizational context, and you
need some expertise in order to interpret them correctly. 4

Some assumptions
Let's assume that we didn't have this long history of financial
measures and accounting processes to draw on.
Assume further that funders and investors requested annual
summaries of financial results without specifying adequate
processes.
This is what the process of developing financial statements might look
like....
5

1. Brainstorm some measures
First, let's brainstorm about what financial elements
are important to us.
We can use flipcharts and involve stakeholders to
come up with ideas. Let's start with a list of a few
hundred standard financial elements.
●
How about reserves – are they important?
●
How about expenses?
●
Does anyone want to add something else from the list?
Don't worry about the research on which financial
elements are most important to organizational
health or our own mission; it's too technical and we
6
don't have time to look it up.

2. Define the measures
Once we come up with our brainstormed list of financial
elements, let's decide how we want to define them in our
organization.
One or two sentences should be more than enough – we all
sort of know what we mean.
As long as we use the same general descriptions that are
on the standard list it doesn't really matter how we define
them. For example, we will call someone a 'new client' if
they stop coming for two months and then start again,
because we get more funding for new clients. Other
organizations may define them differently.
Be sure not to pay too much attention to professional
accountants, because they just get upset about trivial 7
details. Besides, we can't afford them.

3. Collect the data
Now let's figure out how to collect the financial data.
Our funders won't allow the costs of professional
bookkeepers or accountants in their funding
applications, so we'll ask a couple of high school
students to add up the numbers as part of their
community internships.
We'll get one of our counsellors to supervise them,
even though she doesn't have any training in
financial measurement.
They are all pretty bright; they'll do fine.
8

4. Analyze the data
When we get the financial numbers, we'll ask a Board
member to glance over them to make sure they are not
too unlikely.
It takes too much time to check every number, so we'll
just make adjustments based on common sense.
If a number looks really unfeasible we can always leave
that item out of the final report.
Again, it's a good idea to keep accountants away from
this process because they will just use unpleasant
technical jargon and generally be negative.
9

5. Report the results
Now we'll summarize all of the numbers in our
annual report.
It's important that the final numbers look better
than last year's, or at least better than the
other organizations we're competing with.
It's not that we would lie about the numbers,
but there is plenty of room for guesstimates.
And it's not as if anyone will check.
10

Comments
Outcome measurement looks like this in some
organizations, including some that may be doing a
decent job in service provision (based on many years
of my own experience working with nonprofit agencies
and funders, including file reviews).
Other organizations spend a significant portion of their
budget on defining, collecting, analyzing and reporting
high quality information.
If you just compare their numbers, you can't tell which
reports are accurate and which are completely
worthless.
In fact, conscientious organizations may be at a
disadvantage with funders because their results may 11
look less impressive.

Conclusion
Standard outcome measures and other social
metrics should not be used to compare
organizational results unless funders ensure
that the measures are valid, and that the data
have been collected, verified and analyzed
correctly.
Otherwise the use of measures will penalize
organizations that use accurate data, and will
lead to cheating.
For more information and related articles, see
the web sites listed on the next page.
12

Dr Gillian Kerr
Consultant
MaRS Discovery District
E gkerr@realworldsystems.net
W www.marsdd.com
W www.realworldsystems.net
13