Susan Hoff-Capstone Thesis Project

NONPROFIT MEASUREMENTS:
THE SEARCH FOR THE MIDDLE GROUND
CAPSTONE THESIS PROJECT
Submitted in partial fulfillment of the requirements for the degree of Master of Science
in Philanthropy and Fundraising from the George H. Heyman, Jr. Center for
Philanthropy and Fundraising, New York University
Professor Marian Stern
Susan Hoff
May 6, 2013

Nonprofit Measurements: The Search for the Middle Ground - Susan Hoff 1 | P a g e
TABLE OF CONTENTS
INTRODUCTION..............................................................................................................................3
DEDICATION ....................................................................................................................................5
ACKNOWLEDGEMENTS ...............................................................................................................5
THESIS STATEMENT ......................................................................................................................7
A BRIEF HISTORY............................................................................................................................7
Philanthropies and the Demand for New Practices ....................................................................8
MEASUREMENT AND EVALUATION ......................................................................................12
CURRENT CHALLENGES.............................................................................................................15
Access to Management Information ..........................................................................................15
Public Perception........................................................................................................................15
Donations as a Means to an End................................................................................................18
Choosing the Right Methodology...............................................................................................19
Is the Demand for Data Limiting Innovation?..........................................................................21
Performance Measurements & Risk Management....................................................................22
Leadership: Are Nonprofits Too Nice? ....................................................................................24
Re-Aligning the Culture of the Industry.....................................................................................25
ARGUMENTS FOR MEASUREMENT.........................................................................................27
Accountability and Transparency...............................................................................................28
Competitive Advantage for Funding ..........................................................................................30
More Effective Marketing and Social Media Strategies.............................................................30
Wealthy Young Donors Demanding Results ............................................................................32
Enabling a Performance Oriented Culture................................................................................33
ARGUMENTS AGAINST MEASUREMENT ..............................................................................34
One Size Does Not Fit All .........................................................................................................34

Foundations Demand Metrics, But Will Not Fund Them.......................................................35
Short Timeframes May Diminish Ambition and Idealism.......................................................37
Improved Reporting Does Not Improve Outcomes.................................................................39
Impact Measurement Is Misunderstood, Is Not New, and Will Continue to Fail in Its
Attempt to Right All Wrongs .....................................................................................................39
WHEN PUSH HAS COME TO SHOVE ......................................................................................41
CURRENT MODELS AND METHODOLOGIES......................................................................41
Theory of Change.......................................................................................................................42
Logic Models...............................................................................................................................46
Your Planned Work...................................................................................................................46
Your Intended Results................................................................................................................46
Dashboards .................................................................................................................................48
DEVELOPING A COMMON LANGUAGE.................................................................................49
BUILD SECTOR KNOWLEDGE..................................................................................................52
PerformWell...............................................................................................................................52
McKinsey & Company’s Social Sector Office: Learning for Social Impact .............................53
RECOMMENDATIONS..................................................................................................................53
Books...........................................................................................................................................53
Reports & Guides........................................................................................................................53
Performance Management Software (Examples) ......................................................................54
WHAT MEASUREMENTS HAVE BROUGHT THE FIELD THUS FAR.............................54
CONCLUSION..................................................................................................................................55
WORKS CITED................................................................................................................................58

INTRODUCTION
Performance and impact measurements are not new to the nonprofit industry.
Interestingly, what I discovered through the process of researching this thesis was that performance
and impact measurements are widely misunderstood. My own knowledge prior to this capstone
project was no more than a basic familiarity with measurements. It came from working in the arts
back in the 1990’s where I raised money through foundation and government grants. Back then
the required measurements focused mainly on inputs and outputs and not so much on outcomes
and impact. Since then the nonprofit sector has seen tremendous growth overall with
approximately $300 billion raised in 2012 alone. So with that growth came the need for better
analytics, better oversight and better reporting. The reduced availability of resources that resulted
from the economic recession put a microscope on outcomes and whether or not funds were being
used effectively. The demand for nonprofits to develop a measurement-driven system increased.
But the financial and educational support for establishing measurement systems did not
match the demand. This has left many nonprofits, large and small, without the wherewithal to
produce the metrics now required by granters and donors. A divide has formed between those
who advocate for performance and impact measurement and those who do not. On the one side,
government agencies press nonprofits to produce more with less, and donors who made their
fortunes in business expect results expressed in terms of a bottom line. On the other side there is a
diverse group of nonprofits including but not limited to those that support the arts and those that
address massive and intractable issues of human concern. They stand in opposition because it is
hard for them to quantify short-term results and difficult to qualify ambiguous outcomes.

I chose this subject because it is passionately debated and I wanted to have a better
understanding of how measurements have evolved and how they will impact the future of the
nonprofit sector. What follows is an unbiased overview of the current landscape that attempts to
bring into focus the principle issues and concludes with some recommendations drawn from the
research. This exploration has provided me with a great deal of insight about our industry and it is
my hope that it will serve others as well.

DEDICATION
To Knut, my compass and to Aidan, my light

ACKNOWLEDGEMENTS
Thank you to the following for their generous participation, which made this thesis possible:
Michael Balin, former President, Edna McConnell Clark Foundation
John Bliss, Principal, NonProfit Solutions, LLC
Scott Cargle, Nonprofit Management Consultant
Mirele Goldsmith, PhD, Founder, Green Strides Consulting
William Josephson, former Assistant Attorney General-in-Charge of the Charities Bureau
Brian Kinsella, CTO and Co-Founder, The Virtue Group
Nello McDaniel, Founder & Director, Arts Action Research
Charlie Murphy, Consultant and Special Advisor, Hudson Valley Pattern for Progress
Barron “Buzz” Tenny, former President, Ford Foundation
Rodney Trapp, Director of Institutional Giving, Dance Theatre of Harlem
Michael M. Weinstein, Senior Vice President, Robin Hood Foundation
SPECIAL ACKNOWLEDGEMENTS
Mrs. Naomi Levine, Executive Director, NYU Heyman Center for Philanthropy & Fundraising
Marian Stern, Adjunct Professor, NYU Heyman Center for Philanthropy & Fundraising
Timothy Higdon, Adjunct Professor, NYU Heyman Center for Philanthropy & Fundraising

THESIS STATEMENT
While critics argue that impact measurements for nonprofit organizations are not adequate
for obtaining a comprehensive view of mission outcomes, realigning an organization’s focus toward
performance measurement is not just a trend for nonprofit scholars and bloggers to pontificate
about. It is a significant movement within this industry and it is here for the long haul. The ways in
which sustainable outcomes are realized in philanthropy are an art because of the many nuanced
influences on effectiveness, even though we might like it to be more of a science. Therein lies a
significant challenge. (Karoff 2012) Philanthropy translates as the “love of mankind”. (The Free
Dictionary) The human element of the equation is not one that can or should be reduced to a
metric. In the business sector there are uniform reporting requirements that the philanthropic
sector does not have. It is the Wild, Wild West out there in terms of the diversity of reporting
methodologies with no central place to call home. This paper will present arguments for and
against using outcome measurements in the nonprofit sector and look to see if there is a middle
ground being articulated amidst the diversity of opinion and practice.
A BRIEF HISTORY
Contrary to what some may think, measurements are not new to the sector; they have been
around for 100 years beginning with John D. Rockefeller and Andrew Carnegie who instituted the
“scientific method” of philanthropy which has grown to become what some now term “the
effectiveness movement”. (Karoff 2012)
The establishment of more formalized measurements began primarily in health and
human services. In 1960’s and 70’s guidelines such as Standards of Accounting and Financial
Reporting for Voluntary Health and Welfare Organizations (National Health Council and

National Assembly for Social Policy and Development 1964) and Accounting and Financial
Reporting (United Way of America 1974) were developed to determine how funds were being
spent. Later in the 70’s the focus on measurements expanded to include what the funds generated
and this was determined within two categories: products delivered and people served.
The late 70’s brought more standards of measurement in terms of quality of service
delivery. There were concerns with issues such as staff qualifications, staff-to-client ratios, specific
service delivery practices, record keeping, confidentiality protections and condition of facilities.
These concerns led to the establishment of accreditation and certification groups such as The
Council on Accreditation of Services for Families and Children.
The 1980’s brought additional concerns from funders that nonprofits were not serving
those who were deemed the neediest. Participant-related measures were established for the
collection and reporting of data such as demographic characteristics. Public accounting firms began
employing key performance indicators for inputs, services, outputs and total costs. Later in the 80’s
participant satisfaction measurements were added to track quality assurance.
By the 1990’s these measurement standards had succeeded in collecting critical
information about the services nonprofits were providing, but what was still lacking was something
to indicate whether or not the participants were better off as a result of the services. This
encouraged the development of outcome measurements to better understand what kind of impact
nonprofits are having, shifting the focus from activities to results.
Philanthropies and the Demand for New Practices
By the end of 2010 there were more than 1.6 million nonprofits registered with the United
States Internal Revenue Service, an increase from a base in the year 2000 of about 1.3 million.

(National Center for Charitable Statistics 2012) While the wealth of the nation increased during
that time, the common wisdom was that nonprofits were in danger of overwhelming available
funding sources. It seemed that there were more and more organizations chasing fewer dollars.
This drove the more energetic organizations to look for ways to present a more efficient and
trustworthy profile. To project efficiency, they looked to their donor base. Much of the funding
they sought would come from people who were successful in business, and it was felt that efficiency
would communicate seriousness of purpose to this demographic. As for trust, a series of highly
publicized scandals over the course of twenty years or more had tarnished the industry in the eyes
of the public.
Two of the highest profile scandals involved two of the most familiar names in American
philanthropy, the United Way of America and the American Red Cross. In the early 1990’s it
came to light that the long-standing president of United Way had embezzled funds to woo and win
a young woman forty-three years his junior. The fact that his actions came to light through the
testimony of one of his mistresses, all of whom were employed by the organization, didn’t make
the revelation sit any better with the public and their elected representatives. The sense that
corruption could invade the philanthropic sector caught on with the public, some of whom were
only too happy to feel vindicated having previously felt coerced by the management in their
workplaces to contribute to the United Way while it in turn wined and dined their senior
management. The scandal involving the American Red Cross was perhaps worse in the eye of the
public, since it involved money collected for victims of the 9/11 attack on the World Trade
Center. Dependent as it is on hundreds of local chapters to collect donations and provide services,
the reputation of the Red Cross is at the mercy of hundreds of local managers. When the Red
Cross headquarters moved 9/11 funds to an account that the local chapters could not access, the

rebellion drew public attention, causing it to be revealed that the financial management practiced
by the Red Cross was extremely ineffective. Funds collected locally for a national cause were not
always being forwarded to Red Cross headquarters. Funds that were forwarded were placed into a
single account that the local chapters could withdraw from largely at their own discretion. And the
accounting for these funds at the local as well as the national level could not be reconciled.
(Broeckling 2010)
It seemed obvious to many that the fault should be laid at the feet of an industry that lacked
oversight and willfully avoided transparency (Broeckling 2010). To regain public trust, many of the
larger foundations began looking for some way to separate themselves from the past and associate
themselves with a better future. It so happened that there was a model they could adopt from a
source very close at hand, their donors. Two industries had exploded in size beginning at almost
the same time. They were the financial services industry and the electronics industry, the latter led
largely by the worldwide adoption of the personal computer. The financial services revolution
began first when toward the end of October 1979 the Federal Reserve Bank of the United States
officially stopped managing the economy by setting interest rates. No longer constrained, interest
rates could now move freely based on supply of and demand for interest-bearing securities, and
with that there sprang up a bond market. Over a very short time, more and more people were
attracted to the banking and financial service industries as a career choice and many of them went
on to make significant amounts of money, enough money to persuade some of them to become
involved in charitable giving and philanthropy. But their donations came with certain expectations.
For one thing, they were all trained in accounting, having been required to study it for a year or
more to qualify for the degrees that were the gateway to the jobs that brought them their wealth.
They also worked in organizations that were subject to oversight by one federal agency or another.

They were not only used to the careful oversight of money, they expected it, and that meant that
they expected it of the nonprofit philanthropic organization that they chose to engage.
The electronic revolution got traction only a couple of years later in 1983 when the first
IBM personal computer went on sale. In the frenzy that followed, new computer makers
proliferated, as did manufacturers of peripherals from printers to special desks for working at your
computer, and the developers and publishers of software to run on the new computers. Skilled
people were drawn to the myriad of new opportunities this represented and many came away
wealthy. Many of them turned around and reinvested their money in the industry, either in their
own startups or in the startups of others. This was the entrepreneurial spirit that became identified
with the period, a time when the venture capitalist was transformed from a small-time risk taker to
a super hero of the financial world and an example to be followed.
These were the people that the nonprofits now wanted to attract, and they realized that they
were more likely to attract them if they were to behave in ways that were familiar to and approved
by these newly wealthy people. It meant they would have to start accounting for the outcomes
produced by their grants, or as their new funders would put it, the return on their investments.
People in the financial services industry were used to marking positions to market every day and
never going home until the books were closed. Owners of technology startups were used to
developing business cases to sell their ideas to investors, and if successful to have the investors
appear regularly for meetings, sometimes weekly, to ask about the use of their funds, the results
achieved, and progress to the established goals. Should the entrepreneur fail in any regard, the
investors might take over the business and leave the entrepreneur unemployed. For nonprofits,
this was not how they had performed in the past and there was more than a learning curve
required, investment was required.

There would have to be changes made to the management structures of the nonprofits
adapting to this new model. There would have to be investment made to track the information
needed to produce the reports that these new funders would require. New structures, new skills,
and new costs were required, all in the name of a new practice: performance and impact
measurement. It seemed to be a brave new world with the potential of rewards for those that
adopted measurement systems. But an obstacle raised itself early on. Nonprofits didn’t function in
the financial services sector and they didn’t participate in the technology sector. For those who did
decide to bring new reporting processes to their organization, the realization soon dawned that they
were not just adopting a new way of functioning they were inventing a new way of functioning
because, after all, for a nonprofit, there were no standard practices for performance and impact
measurement.
MEASUREMENT AND EVALUATION
This paper addresses various attitudes about and approaches to the measurement and
reporting of impacts and outcomes in the nonprofit sector. Measurement in this context is
sometimes referred to as impact assessment and is defined by the International Association for
Impact Assessment.
“Social impact assessment includes the processes of analyzing, monitoring and managing
the intended and unintended social consequences, both positive and negative, of planned
interventions (policies, programs, plans, projects) and any social change processes invoked
by those interventions. Its primary purpose is to bring about a more sustainable and
equitable biophysical and human environment."

To be clear, there is a very real difference between measurement and another term,
performance evaluation, though the two are sometimes used interchangeably. “Conceptually,
evaluation and performance measurement are still being conflated in the minds of nonprofit
boards and managers.” (Saul 2003) Put simply, the measurement of outcomes is an ongoing
management task that tracks the progress of a nonprofit organization toward its goals.
Measurement reports come out at regular intervals, whether annually, quarterly or monthly, and
because of the regularity of production and the consistency of reporting, the results are comparable
from period to period.
The United States Government Accountability Office’s (GOA) 2011 report titled
Performance Measurement and Evaluation: Definitions and Relationships, explains it this way.
“Performance measurement is the ongoing monitoring and reporting of program
accomplishments, particularly progress toward pre-established goals. It is typically conducted by
program or agency management. Performance measures may address the type or level of program
activities conducted (process), the direct products and services delivered by a program (outputs), or
the results of those products and services (outcomes). A “program” may be any activity, project,
function, or policy that has an identifiable purpose or set of objectives. Program evaluations are
individual systematic studies conducted periodically or on an ad hoc basis to assess how well a
program is working. They are often conducted by experts external to the program, either inside or
outside the agency, as well as by program managers. A program evaluation typically examines
achievement of program objectives in the context of other aspects of program performance or in
the context in which it occurs.” (Kingsbury 2011)
This GOA report then goes on to explain the relationship between performance
measurement and program evaluation in terms of differences in focus and use. Different focus

“Performance measurement focuses on whether a program has achieved its objectives, expressed
as measurable performance standards. Program evaluations typically examine a broader range of
information on program performance and its context than is feasible to monitor on an ongoing
basis.” Different use “Both forms of assessment aim to support resource allocation and other
policy decisions to improve service delivery and program effectiveness. But performance
measurement, because of its ongoing nature, can serve as an early warning system to management
and as a vehicle for improving accountability to the public. A program evaluation’s typically more
in-depth examination of program performance and context allows for an overall assessment of
whether the program works and identification of adjustments that may improve its results.”
(Kingsbury 2011)
The debate within the nonprofit industry is that the preference for evaluation has skewed
the field , creating a kind of tunnel vision that focuses on “proving whether a program or initiative
works, rather than on improving programs” (W.K. Kellogg Foundation 2004). Instead, some
practitioners advocate using “performance data as business intelligence in order to systematically
improve results.” (Saul 2003)
Understanding the similarities and differences between measurement and evaluation can
help nonprofits to decide which performance methodologies to apply. They can be selective,
focusing implementation to benchmarks and logic models that will futher the mission and avoid
methods that conceal qualitative results under quantitative excess. By defining a suitable reporting
strategy and following through, they can showcase the impact, accountability, and sustainability of
their programs and advance their relationships with donors and funders alike.

CURRENT CHALLENGES
Access to Management Information
Many nonprofit professionals, board members, and donors are not familiar with
measurements and how to use them. As discussed in the preceding section, many organizations
have not grasped the distinction between impact assessment and performance evaluation. “Despite
all the right intentions, the vast majorities of nonprofits do not have the benefit of good
information and tools to determine where they’re headed, chart a logical course, and course-
correct when they’re off….Only a fortunate few have a reliable way to know whether they’re doing
meaningful, measurable good for those they serve.” (Morino 2011) Without access to the
appropriate management information, some nonprofit organizations basically do without.
Public Perception
Corruption Some in the industry feel that the push for outcome measurements is a judgment cast
upon the sector due to the actions of a few bad apples. New York’s Governor Cuomo has weighed
in on the topic. “Not-for-profits that provide services to the poor and the needy have a special
obligation to the taxpayers that support them. Executives at these not-for-profits should be using
the taxpayer dollars they receive to help New Yorkers, not to line their own pockets.” (Cuomo
2011) Clearly, corruption is not exclusive to any one industry. But the prevailing opinion seems to
be that the nonprofit sector should be held to a higher standard than the for-profit and government
sectors. Perhaps it is because of the special tax treatment afforded nonprofits, or maybe it is the
thought that good deeds should be reward enough in themselves. Either way, impact and
performance measurement is now one of the tools by which some in our society wish to enforce
that standard.

Charity Watchdogs There are a number of charity watchdog organizations that proactively
examine nonprofit organizations, applying their own standards. Some examples are Guidestar.org,
the Better Business Bureau, Charity Watch, GreatNonprofits.org and GiveWell.org. Though
some philanthropists consider their methods controversial, they are making a significant impact on
how the public perceives the level of financial health, accountability and transparency a nonprofit
organization is achieving. Charity Navigator, which was established in 2002 by businessman John
Dugan, is one of the largest online sources for evaluating nonprofit groups. (Strom 2010) In
response to the growing demand for measurements, in January 2013 they released Charity
Navigator 3.0: The Third Dimension of Intelligent Giving. This expanded version of their rating
system includes what they term Results Reporting. “Charity Navigator 3.0 will include a dimension
focused on the quality of reporting of results that charities provide to the public. We believe this
will highlight those charities that are high performing and results oriented, as well as encouraging
other charities to become so.” (Charity Navigator 2013) Their plans to implement this new
evaluation category are as follows:
 For the next several years, our professional analysts will be researching the applicability of
this tool to each of the 34 charity “cause areas” that we evaluate, making appropriate
modifications as needed, and applying the criteria to the 10,000 charities we plan to rate as of
2016.
 To give donors immediate access to our research during this process, we will post our
findings on each charity’s page month by month, as the new data is gathered.
 However, this new analysis will not impact any charity’s star rating until we have gathered
the data for all 10,000. Given the complexity and variety of charities we are evaluating, we will
need this time to continue our research on each of the 34 cause areas to determine how all 5
rating elements apply and what modifications are needed for some cause areas. We also need
this time to compile an ample amount of data to determine the appropriate weighting of this
information in our rating system. (Charity Navigator 2013)

If in fact charity watchdog organizations end up directing donors and funders to certain
organizations and eventually away from others, they then become arbiters in the nonprofit sector
wielding a power based on measurements that are not universally calibrated. Besides, each self-
appointed participant in the oversight space asserts its own standards, methods and point of view.
In the end it is not clear who is watching the watchdogs.
The Double Standard A recent TED Talks video, The Way We Think about Charity is Dead
Wrong, was posted sometime during March 2013 and has gone viral. (1,579,422 views as of May
3, 2013) It features a speech given by activist and fundraiser Dan Pallotta whose goal is to
transform the way society thinks about charity and giving and change. In it he says that “everything
the donating public has been taught about giving is dysfunctional”. He calls out the “double
standard that drives our broken relationship to charities”, the notion that the business sector is
encouraged and then rewarded for spending as much as is needed, no matter the amount, to
achieve the highest level of profit possible. While on the other hand, too many nonprofits are
rewarded only for how “little they spend -- not for what they get done”. The public, he feels, is
confusing “frugality with morality”. There are essentially “two rulebooks”, one applied to for-profit
organizations and the other to nonprofits, and this he claims has kept the nonprofit industry at a
standstill for more than 40 years, limiting its contribution to the GDP and denying it the resources
needed if the industry is to be recognized for the serious role it plays in changing the world.
(Pallotta 2013)
We’ve all been taught that charities should spend as little as possible on overhead things
like fundraising under the theory that, well, the less money you spend on fundraising, the
more money there is available for the cause. Well, that’s true if it’s a depressing world in
which (the) pie cannot be made any bigger. But if it’s a logical world in which investment in
fundraising actually raises more funds and makes the pie bigger, then we have it precisely
backwards, and we should be investing more money, not less, in fundraising, because
fundraising is the one thing that has the potential to multiply the amount of money
available for the cause that we care about so deeply. (Pallotta 2013)

Donations as a Means to an End
Because overhead, that now has to budget for assessment, is not a priority for funders to
support, the majority of nonprofits have to divert funding from other areas in order to produce
information on performance and the impact of programs. For an organization constrained in terms
of the size and makeup of its donor base, the funds would have to come from existing donations,
draining resources and curtailing the organization’s ability to support its mission. “There is no
uniformly accepted way to measure social impact, and no single repository for information about
nonprofit activities and results. This information-poor environment makes it difficult to have
honest conversations about performance, limiting opportunities for learning and improvement.”
(O'Flanagan, Harold and Best 2008) In comparison to high-performing, high-participation markets
such as the stock exchanges, commodity markets, or Amazon.com, the nonprofit marketplace is at
a disadvantage. Unlike the financial and consumer markets, the nonprofit’s commitment is not
directed to market participants who invest or spend money, but to people whose welfare the
organization is dedicated to support. Donors are the equivalent of investors in the other markets
and are therefore expected to share the same focus as the nonprofit organization. To confuse or
conflate the for-profits with nonprofits in this respect is to do a disservice to the nonprofit for
which the money is a means to an end and not an end in itself. It is not clear how many donors
would be content to think that a substantial portion of their gift would be spent on in-depth
quantitative analysis and reporting when a qualitative accounting would be sufficient.
Still, there is a gap between the nonprofit and the for-profit when it comes to accounting for
funds. In order to bridge this gap, the sector needs to find a more efficient way to capture, analyze,
distribute, and use information on social impact and organizational performance instead of hoping

to impress by adopting goals designed to be easily analyzed and reported on. (O'Flanagan, Harold
and Best 2008)
Choosing the Right Methodology
Currently there is a mix of methodologies, analytical models, and industry consultants, each
claiming to offer an advantage in supporting performance measurement. But there are pitfalls
awaiting organizations that feel the need to adopt a solution in a box. A reporting method that
doesn’t complement the organization, its structure and its mission will produce information that is
likely to miss the point at best and divert the organization from its mission at worst. Either way, the
effort will cost resources that might be better spent. “In today's climate of scarce economic
resources, the pressure for nonprofits to show quantifiable results is greater than ever; as a result,
an organization without a strong sense of strategic direction and the internal data to understand its
own strengths and weaknesses can be overly influenced by outside demands for metrics that may
not always be relevant to its ultimate success.” (Hanna 2011)
Ideally, impact assessments and performance measurements should support comparisons
between reporting organizations, contrasting differences in efficiency and operational competence,
and highlighting effectiveness, but within the context of the challenges faced by an organization
given its mission and goals. “A basic problem that plagues philanthropic decisions is that there is
no natural yardstick by which to measure, and therefore compare, different philanthropic
outcomes.” (Weinstein and Bradburd 2013) For Michael Weinstein, an economist and Senior
Vice President at the Robin Hood Foundation, counterfactuals are the most important thing they
do. “Counterfactuals” is a contraction of a philosophical term, “counterfactual theories of
causation”. As explained in the Standford Encyclopedia of Philosophy, “The basic idea of
counterfactual theories of causation is that the meaning of causal claims can be explained in terms

of counterfactual conditionals of the form “If A had not occurred, C would not have occurred”.
(Menzies 2009) Counterfactuals make up one part of the Foundation’s continual effort to develop
more effective ways of reporting on every benefit related to their mission. (Weinstein, Senior Vice
President and Chief Program Officer 2013) In finance terms, counterfactuals are very much like
the practice of calculating incremental ROI, or the value of the next dollar spent, and it suits the
Robin Hood Foundation and the high-net-worth donors it works with.
A funder gives money to a job training group, ten of the trainees get jobs. So how much
good did the funder do? How much good did that training program do? If you answer ten,
you’re measuring it in a very different way from the way Robin Hood goes about its
business. Because we need to ask, how many of those trainees would have gotten jobs even
if they had never entered the program. And that’s what is called a counterfactual. How
many of these trainees would have gotten jobs without our help? And that’s what Robin
Hood’s staff spends a lot of time trying to estimate…We fund programs that cater to New
Yorkers who we think are going to fail without our help.” (M. Weinstein)
The Foundation uses other reporting methods as well. Along with his co-author Ralph M.
Bradburd, Mr. Weinstein recently published a book, The Robin Hood Rules for Smart Giving.
In it he describes the framework he developed for Robin Hood called Relentless Monetization
(RM). The Foundation defines “monetization” as the assignment of dollar values to philanthropic
outcomes, and “relentless” in this context means making assignments even when the benefits
associated with those outcomes are hard to measure and the evidentiary basis for assigning dollar
values to specific outcomes is slim. (Weinstein and Bradburd, The Robin Hood Rules for Smart
Giving 2013) Relentless Monetization methodically applies the workhorse of modern economics,
benefit/cost analysis, to the task of making effective philanthropic decisions.
The power of Relentless Monetization (RM) lies in its consistent and persistent application of
benefit/cost analysis. Implemented carefully, the strategy takes full account of the funder’s
philanthropic mission, the preferences and values of nonprofit actors (funders, donors, policy
makers, academics, service deliverers) and resources. RM also takes full account of the best
available evidence about the impact of philanthropic interventions on the outcomes that are
relevant to the missions of funders and donors. And the strategy does all this in a manner that
leaves a tangible trail of accountability, thereby exposing philanthropic decisions to challenge
and revision. (Weinstein and Bradburd, The Robin Hood Rules for Smart Giving 2013)

This type of in-depth, multi-employee strategy takes an enormous amount of time and
dedicated resources. The Robin Hood Foundation is fortunate to have a board of directors whose
members are successful hedge fund managers and celebrities who supply 100% of the funds
needed for general operating costs. That allows the Foundation to apply 100% of its donor
proceeds directly to the cause of fighting poverty in New York City. But not every organization is
funded like the Robin Hood Foundation. For the rest, challenges remain. Not the least of which is
a shifting consensus within the measurement movement; the strategies favored for measuring
impact keep changing as the market turns from less productive methodologies to new ones. The
Robin Hood Foundation, given its mission and its donor base, can invest in researching and
developing the reporting models that best suit their needs. This is not something that is available to
the majority of organizations in the industry.
Is the Demand for Data Limiting Innovation?
Funders are demanding more and more data. The common frustration among nonprofits is
that even though funders demand more data, they are unwilling to fund or supply the resources
needed to collect that data. William Josephson, retired partner of the New York City law firm
Fried, Frank, Harris, Shriver & Jacobson LLP, and Assistant Attorney General-in-Charge of the
Charities Bureau under former Attorney General Eliot Spitzer, feels that the demands for
measurements have damaged philanthropy’s innovative purpose in such a way that it may no
longer be rectifiable. “Reliable social science research is very expensive and often inconclusive.
Such research also often takes a long time to yield any results. Foundations have not done badly
without such research in the past. Demanding it now, particularly of start-ups, will diminish
innovation.” However, a long-time friend and colleague of Mr. Josephson, Michael Bailin, does
not agree with Mr. Josephson’s criticism. Mr. Bailin is the former President of the Edna

McConnell Clark Foundation and during his time there was responsible for a strategic initiative
that moved the foundation away from traditional foundation practices to a performance-based
approach. In an interview, Mr. Bailin argued that when measurements are designed with the right
focus, they are an invaluable tool for producing “just communications” by which he means
substantive, mission-based reporting on foundation performance. (Bailin 2013) “The approach we
have chosen arose because it addressed a problem that had been plaguing us for some time: We
were committed to changing huge systems with small resources, we believed we were highly likely
to fail at that mission, and we needed a different way of looking at our goals and our relationship
with grantees.” (Bailin, Philanthropy in Practice: Great Expectations Versus Getting the Job Done
2004) In other words, if performance-based reporting will indeed force change within an
organization, use it, and let it direct change to the good.
For all the benefits ascribed to impact assessments and performance measurements, the
demand for data remains suspect even among some funders. Abigail E. Disney, founder and
president of the Daphne Foundation, who supports community-based groups in New York, among
other causes, warned donors against allowing their focus on metrics and measurement to prevent
them from supporting nonprofit leaders who are tackling society’s most intractable issues. “I
wonder if Martin Luther King Jr. showed up at my door today, if any of us would fund him,” she
said. “I think we would have kicked him to the curb in such a hurry.” (Preston, Philanthropy Must
Do More to Influence Policy, Say Government Officials 2013)
Performance Measurements & Risk Management
Lehn M. Benjamin is an assistant professor with the Nonprofit Management Studies Program
in the Department of Public and International Affairs at George Mason University. Her research

examines the intersection between nonprofit organizations, accountability, and democratic
practice. Ms. Benjamin wrote an article advocating that risk management deserves more direct
attention in discussions about nonprofit accountability generally and performance accountability
specifically. The article is based upon a study of performance measurement frameworks in
nonprofit relational work. It concludes that choosing the appropriate method of reporting is
important because an inappropriate reporting method can work at cross purposes with the needs
of the beneficiaries. The organizations in the study raise funds that they in turn distribute to
neighborhood organizations dedicated to community development. The study employs risk
analysis to demonstrate that an inappropriate stress on efficiencies can lead to unintended
consequences “Performance accountability systems require nonprofits to bear more risk for
achieving results. Although a growing body of work has examined nonprofit accountability, less
attention has been given to the concept of risk. (There is) a potential conflict between performance
accountability frameworks and nonprofit work. This conflict can be best understood as one
between managing risk in task-driven relationships, in which relationships are formed simply to
achieve desirable results, and managing risk in developmentally driven relationships, in which
performing a task is intended not only to achieve desirable results but also to build enduring
capacity to take action on common problems.” (Benjamin 2008)
Relationships are the backbone of nonprofit work and vital to long-term sustainability.
Preserving relationships among organizations and with beneficiaries often requires trust, especially
when beneficiaries feel socially marginalized because they may have learned to become distrustful
of anyone promoting change, no matter how constructive. “Public administration scholars have
given much attention to the advantages and disadvantages of ensuring accountability through
hierarchy or by incentives coupled with competition, but perhaps nonprofit organizations require
an accountability approach that recognizes the distinct work they do in building relationships.”

(Benjamin 2008) The accountability model adopted in the study linked levels of funding to the
achievement of certain goals. But threatening to reduce levels of funding if goals were not met,
proved counterproductive. The unintended result was the transfer of risk from the funding
organization to the more dependent and less resilient community organizations, and a reversal of
the traditional responsibilities of the benefactor and the beneficiary.
Leadership: Are Nonprofits Too Nice?
When a nonprofit is faced with the dual challenges of first establishing and then managing
performance and impact measures, the net effect on the culture of an organization that is not
completely committed to this new approach can threaten the success of the measurement plan. As
with all oversight and management, the responsibility for creating and sustaining an effective
performance-driven culture lies with the leadership. “Without a leader, who is committed to
measurement as a top priority, articulates how it enhances impact, and identifies someone within
the organization to lead the charge, organizations will not overcome the natural reluctance among
staff to embrace what seems like such an overwhelming enterprise.” (Forti 2012)
There is a sense that because the nonprofit industry is sacrificing for the common good, it is
unfair to criticize them. Unlike business, with which it is often compared when speaking of
performance and reporting, the nonprofit sector is an area that many choose to be involved in
because the emphasis is on creating a better world, as opposed to accumulating wealth. To some,
criticizing a philanthropic organization by saying it has not achieved its mission effectively is like
judging a nun for not being holy enough. But because of the economic challenges that continue to
exert pressure on funding, many nonprofit professionals feel that without honest conversations
about performance and impact, the industry overall is likely to be diminished in its effectiveness
and blamed for unwillingness to address issues of leadership, board governance and self-regulation.

Vignetta Charles, senior vice president of AIDS United made this point recently in the Chronicle
of Philanthropy. “It’s harder for us to talk about this in philanthropy than in the corporate world
because we’ve had this genteel attitude. We’ve been too nice to each other for too long,” she says.
“It’s especially timely for us to think through the ways we communicate, how transparent we are,
and who needs to potentially fade gracefully into the night.” (Preston, Some Nonprofit Leaders
Ask: Is Philanthropy Killing Itself with Kindness? 2013)
In his book, Leap of Reason: Managing to Outcomes in an Era of Scarcity, Mario Morino
makes a similar point:
In my business life we once brought in a speaker to inspire our team and get everyone on
the same page. He gave great examples of getting folks involved and buying into mission,
the normal song and dance of inspirational speakers. But he wrapped up the session with a
pithy statement that is indelibly etched into my memory: “Catch the vision or catch the
bus!” Harsh? For sure, and it’s unlikely that you’ll use it at your next all-hands meeting. On
point? Very much so. (Morino 2011)
Morino believes that what commonly prompts a shift in favor of metrics is the recognition that
while measurements might be necessary to report results to funders, the greatest power lies in using
them as a tool to improve the nonprofit’s ability to fulfill its mission. But without internal buy-in,
there may not be external acceptance. Identifying what needs to be measured and how to measure
it requires education, internal commitment, financial support, planning, and openness to
experimentation and change. Inaugurating the program and obtaining the commitment of staff and
the support of funders takes leadership.
Re-Aligning the Culture of the Industry
If the natural evolution of the nonprofit sector will lead to universal acceptance of impact
assessment and performance measurement, it is not to be without a struggle. “There’s an ongoing,
pitched battle for the soul of philanthropy.” (Preston, Some Nonprofit Leaders Ask: Is

Philanthropy Killing Itself with Kindness? 2013) And if it is a battle, how will it end? Battles
generally end when someone wins and someone loses, wherein lie the fears of some who oppose
reporting. For others, there is a less traumatic and more pragmatic question, do performance and
impact measurements really lead to success? Which is to say, will they keep an organization
focused on its mission, will they add value by helping an organization function more efficiently, and
will they help to attract and retain a loyal base of donors and funders? As yet, there is no sure
answer to these questions. Evaluator of winners and losers, effective management tool or strain on
resources, impact assessments and performance measurements do represent one thing, though:
change. And if nonprofits and the nonprofit sector do not evolve to accept and embrace this
change, the concern is that they may be forced to instead.
Some nonprofits, those who apply for grants from certain funders or who contract with
government agencies, must already submit to reporting requirements that they would not otherwise
adopt by themselves. But the others, who are the majority, have and do perform their roles and
meet expectations without allocating the resources that reporting would require of them. To do
otherwise might require significant changes in function, and perhaps changes in mission. To do
otherwise might require significant changes in organizational culture. Realigning the culture of an
organization is a tough, uphill climb requiring change that begins at the top, but what we are talking
about probably requires a re-alignment of the culture of an entire industry, and that is an even
more formidable task. Even now, success does not come easily. “What we want when we donate
money is to feel good,” Susan Davis, President of BRAC USA says. “It’s much harder to engage
the public in something more complicated, but the truth is, as our world has become more
interconnected, people are becoming more sophisticated, and what we’re trying to do is invite
people on a learning journey.” (Preston, Some Nonprofit Leaders Ask: Is Philanthropy Killing

Itself with Kindness? 2013) Few will willingly give up the methods that have served them in the past
when there is no assurance of a benefit for the organization or its beneficiaries to do otherwise.
ARGUMENTS FOR MEASUREMENT
According to a 2010 paper by Alnoor Ebrahim and V. Kasturi Rangan of the Social Enterprise
Initiative at the Harvard Business School, the debates on measuring outcomes and impacts are
playing out in three arenas:
 Private foundations aiming to be more strategic about their philanthropy
 U.S. nonprofit organizations in response to pressures from foundations and government
 International development organizations such as bilateral government agencies and
nongovernmental organizations (NGOs) seeking to improve development effectiveness
Economic conditions play a part as well. “The pressures to demonstrate impact are likely to
increase across all of these players in times of economic crisis, as public and private resources
diminish and as competition for existing resources heightens.” (Ebrahim and Rangan 2010) This is
not to suggest that the impulse to adopt performance and impact measurements is simply a
reaction to the worldwide recession that began in 2008 with the collapse of financial markets
following the bursting of the housing bubble.
For one thing, the arguments in support of performance and impact measurements for
nonprofits are well thought out, and bear more of the weight of logic than one would expect from a
faddish reaction to economic stress. Instead, the authors foresee an imminent risk to the sector as
a whole. A risk that if the industry is unable to acknowledge and embrace the importance of
measuring and communicating mission impact, it may see an erosion of public trust and eventual
marginalization as a functioning sector of the economy.

Accountability and Transparency
At its core, the demonstration of accountability in a market participant is a way to establish
trust. For the nonprofit sector of the market, trust plays a more significant role in achieving success
than in business or government since the customers or investors, represented by the donor base
are in some cases not direct recipients of the benefits produced by the organization. Even though
there are situations where a donor is the recipient of benefits, as in hospitals, universities, religious
institutions, to name a few, often the recipients of the benefits are separate from the donor,
separate by location, economic stratum, physical location, and any number of other possible
differences that make it unlikely that the donor will otherwise be able to support them directly.
Accountability can provide the proof needed by the donor to ensure that the donor’s wishes were
fulfilled by the nonprofit organization as intermediary and thus deserve the trust put in the
organization by the donor.
Beyond the nonprofit’s donor base, the public at large expects accountability because
nonprofits are by definition of the IRS tax exempt organizations. It is understood that a tax exempt
status means that every tax-exempt organization is in fact being supported indirectly by the
taxpayers of the nation. Furthermore the tax deductibility of donations to tax-exempt organizations
represents foregone government revenue and further implies that the donors to these organizations
are being supported by the taxpayer as well. When considered in this light, it is not a stretch to see
why people can conclude that nonprofits are working with the public’s money, no matter what the
source of their funding is, whether public or private. “The broader definition of accountability,
including the requirement to produce results, also has become more pronounced with the growing
impact of donors who view their giving as social investment and demand specific evidence of the
impact of their support.” (Worth 2012)

Form 990 The recently revised Form 990 has allowed for more information to be
accessible about the work of nonprofit organizations including specifics concerning governance
practices. Even more recently, President Obama released a budget asking legislators to pass a law
mandating electronic filing for the Form 990. (McRay, White House Advocates for Mandatory
Form 990 E-File 2013) There has been a call for the IRS to disclose more data from Form 990’s
and to do so in a way that makes it easier to search. Tom Pollak, program director of the National
Center of Charitable Statistics at Urban Institute remarked, “We really appreciate the IRS taking
this major step forward to make a lot of data available on a timely basis, and we are very hopeful
that in future years the IRS will be able to expand on what they include in the file.” (McRay 2013)
But the purpose and structure of the Form 990 in its current incarnation has limited value for
measuring organizational performance, and the information provided is not adequate for readers
to have the ability to conceptualize the data. Yet the Form 990 is the only publicly available source
of data for all nonprofits. “In the absence of better information, individual donors, foundations,
advisors, and the news media tend to fixate on operating expenses and fundraising ratios or reports
of financial abuses or scandals at unrelated nonprofits to form their perceptions of nonprofit fiscal
responsibility.” (McKinsey & Company 2008) In a 2005 final report to Congress and the nonprofit
sector, the Panel on the Nonprofit Sector concluded that “The (Form 990 is) not useful as a tool to
communicate complex information about program goals, accomplishments, failures, and changes
that have affected an organization’s overall performance or the performance of a particular
program. Each organization is therefore encouraged, as a recommended practice, to share more
detailed information about its programs through an annual report or other appropriate document
that is available to the public on the same basis as its Form 990. Organizations are also encouraged
to post such information on their websites.” (Strengthening Transparency, Governance, and
Accountability of Charitable Organizations 2005)

If prepared properly, performance and impact measurements can play a significant role in
filling the information gaps in Form 990. They could provide more transparent and mission driven
communications for public viewing that will simultaneously create a road map for accountability
and efficiency. If nonprofits prove unwilling to take this step on their own, it is safe to assume that
given public sentiment, the character and content of Form 990 will change. It may behoove the
nonprofit to ensure that the information they now provide is sufficient to satisfy the burgeoning
watchdog groups, public and private, that will be analyzing and reporting on this information.
Competitive Advantage for Funding
As the nonprofit sector has grown, so has its need for funding. In the not so distant past, a
nonprofit in pursuit of support spent most of its time crafting the perfect “elevator pitch” that
would capture the spirit and essence of its mission as well as the eyes of donors and grantmakers.
Now more than ever, with the state of the economy and the vast numbers of nonprofits in business
today, many organizations whose missions barely differentiate them from other similar
organizations, need to provide data that will allow a nonprofit to obtain singular recognition and
make a positive impression on funders’ minds. “As needs increase and resources dwindle, the
community organization which can demonstrate impact of resources more substantially than an
ally (or competitor?) organization may end up with more resources than those unable to show a
direct impact.” (Boland 2012)
More Effective Marketing and Social Media Strategies
It is believed that communication of successful performance measurements can serve as a
powerful marketing tool. According to the Stanford Social Innovation Review, measurement and
transparency is the number one technology trend to watch in 2013. “What gets measured gets

improved…Nonprofits should not only capture data about their own performance for reflection
internally, but also consider how sharing that data with people who use their services can positively
affect their area of work—whether it’s health care, education, energy, or another issue.” (Tobias
2013) Nonprofits are marketing to more than the individual donors and the foundations they
solicit for funding. They are also marketing themselves to their future board members, major
donors and up-and-coming, self-selected, nonprofit professionals whose financial and
organizational training and sophistication will be accompanied by higher expectations of
professionalism and purpose. In this context, nonprofits with successfully implemented
performance measurement systems will be more attractive to the top talent and the wealthiest
donors. Marketing built on the results of performance measurement can also serve as a tool for
educating the public on public policy agendas. Framed within professionally managed marketing
campaigns, measurements of success are made to order for influencing public perceptions as to
those organizations best prepared to handle public issues efficiently and effectively. (Sawhill and
Williamson 2001)
Social media will become more important to a nonprofit’s ability to market. Though
currently not the most reliable resource for communication, social media is none-the-less a
medium to which people turn when they want information. Performance results are the kind of
hard data that can make a difference in this environment. Here an organization can also engage
those who seek them out and not just as for support but to obtain data in service to their mission
goals. “Social media and that whole realm of crowdsourcing are going to be more and more
helpful in getting data.” (Berger 2012) In the crowded nonprofit sector, a presence in social media
that tonally reflects the organization and quantitatively defines the organization has the potential of
creating an advantage for the organization. A presence that can engage and encourage participation

in this venue, could improve the standing of an organization substantially. In the social media
environment, the nonprofit presence that wants to demonstrate professionalism and sophistication
will need to explain itself with hard data in the form of performance data that reinforces the spirit
and substance of its mission.
Wealthy Young Donors Demanding Results
It is no longer so much about the Baby Boomers as it is about their children. The current
projection for the transfer of wealth that is set to begin in this decade is approximately 41 trillion
dollars, the largest transfer of wealth ever. The difference between the Boomers and their kids is
that the kids are more active in the many aspects of their philanthropy and they have begun
making mega gifts much earlier than their parents. That increase in involvement includes higher
expectations for results. “Affluent young donors say they are more focused than their parents and
grandparents on producing a measurable impact with their giving…What’s more, some of them say
they care more about advancing a cause than helping an institution, the reverse of how they see the
philanthropy of their elders.” (Preston 2013) So as a result of this heightened involvement and
demand for performance and impact measurements, the future of this industry will possibly see
some of the most profound philanthropy that it has seen since Rockefeller and Carnegie. “The
next generations of major philanthropists, who fit into “Gen X” (born 1964-1980) or “Gen
Y/Millennial” (born 1981-2000) generational cohorts, will wield more philanthropic power than
any previous generation. With an unprecedented amount of wealth, these donors hold the future
of philanthropy in their hands, yet, until now, there has been little previous research on the
powerful but very private group of young people who stand to become the major donors of the
future.” (21/64 and the Dorothy A. Johnson Center 2013)

Enabling a Performance Oriented Culture
The absence of a standard accountability system within the industry has let nonprofits delay
taking responsibility for establishing best practices in the management of performance and impact.
When asked in an interview, “Where do you stand on the trend that is mandating impact
measurements for nonprofits?” John Bliss, Principal, NonProfit Solutions, LLC and Board
President, Hudson Highlands Nature Museum, Cornwall NY, simply replied, “If you can’t
measure it, you can’t manage it.” (Bliss 2013) For a nonprofit, performance and impact
measurements present opportunities to establish strategic, mission specific goals while creating a
manageable, rational approach to cultivating and sustaining an internal culture of accountability
and purpose. The culture of an organization informs not only practices and attitudes that frame its
internal life, culture is also outward facing. An organization’s culture is a broad concept, but it is
simultaneously an intimate thing. It is disclosed through every transaction, every exchange,
between a representative of the organization with other representatives, and with every outsider
from prospective donor, to funder, to grantee. To reveal the culture of an organization is ultimately
to reveal the best and the worst of the organization. An organization that does not find the
resources to engage in periodic self-evaluation risks losing touch with itself and its mission, and that
is difficult to conceal. It will be communicated. Self-evaluation can and should take many forms,
but performance management in some form is a necessary introspective element, besides being
one of the best forms of communication to the public. These points can be found in the literature.
The U.S. Office of Personnel Management defines performance management as: the
systematic process by which an agency involves its employees, as individuals and members of a
group, in improving organizational effectiveness in the accomplishment of agency mission and
goals. (Walker Ph.D. and Anderson Moore Ph.D. 2011) So understanding that “Missions are

typically better at providing inspiration than direction” (Bradach, Tierney and Stone 2008), it is the
responsibility of the leaders to embrace a method or system that will enable the organization as a
whole to align itself toward a more unified set of standards for reaching the impact their mission
denotes. “The absence of processes for setting employees’ goals and obtaining feedback, for
instance, disconnects individuals and their performance from the organization’s strategy.”
(Bradach, Tierney and Stone 2008)
ARGUMENTS AGAINST MEASUREMENT
In the Bill Moyers PBS series, The Power of Myth, Joseph Campbell recalls the quote by
Carl Jung, “Religion is a defense against a religious experience.” For Campbell, religion is a
bureaucratic impediment to the “experience of deep mystery that one has to regard as the ultimate
religious experience.” (Campbell 1988) Similarly, the process of establishing measurement metrics
for social impact has been blamed for creating an impediment to innovation. There is a fear of
unmanageable bureaucracy undermining the value of individual aspirations, and taking away the
purpose and meaning of the work of nonprofit organizations. “Too much emphasis on measuring
performance could create a “Dilbert world,” in which the passion and commitment of the
professional staff and volunteers are replaced by caution, even skepticism, which might undermine
the nonprofit culture and its traditional strengths.” (Worth 2012)
One Size Does Not Fit All
One of the main criticisms of performance and impact measurements is that nonprofits do
not focus individually on the same or even similar issues. Their methods are not uniform either.
The story was once told about someone putting three flies in one narrow-necked glass bottle and
three bees in another. The bottles were placed on their sides, uncapped, in a position where

daylight shone through the bottom of the bottle. The three bees bumbled and fumbled at the glass
at the bottom of the bottle. They never found their way out. The three flies, so the story goes, kept
zinging around randomly in their bottle until finally each of the three found the open end and
escaped. (Peters and Waterman 1984) Though it is not a flattering comparison, some think of the
nonprofit sector as acting like the flies, facing intractable problems and somehow finding answers,
each in its own way, by working tirelessly and perhaps seemingly randomly, to find a solution, or to
come up with an expedient solution that serves a need, improves a bad situation, or gives someone
an opportunity to get up and try again tomorrow, without trumpeting that the solution has been
found, the problem has been solved, and from now on everything will be better.
Nor is their uniformity in the demand for reporting from the government or foundations
that are the lifeline for many nonprofits that survive on grants. There is no standard for the
grantees to adhere to, nor is there a standard underlying the requests made by the grantees. As any
nonprofit leader can tell you, she may have to report outcomes in as many different ways as the
number of grants she receives from foundations and government, for there is no generally
accepted, uniform way for such outcomes to be reported. (Schambra 2011)
Foundations Demand Metrics, But Will Not Fund Them
It’s a fact of nonprofit life, overhead is rarely funded. Some foundations will offer grants for
administrative purposes, but providing funding for office work is not generally the kind of ambition
that calls someone to philanthropy, and for most people supporting the sector, analysis and
progress reporting is a kind of office work, and not the good kind. The good kind of office work is
performed in a second-story office in a rundown building in a city in a third-world country where
one or two people supported by concerned funders thousands of miles away minister to the needs

of a population for whom some of the most basic needs to support health and wellbeing can only
be supplied from outside.
One of the reasons that this latter kind of office work is supportable is that it requires
relatively little payment of overhead. In the eyes of the public and many funders, overhead is to be
kept to a minimum. Somehow it is in and of itself a bad thing, even if the downside is evident in
the very country where the philanthropic effort is focused, where there is no such thing as public
health since the government there cannot or will not pay for the overhead it represents.
“Investing in an IT system that can track program results is good; paying excessive rent for
opulent office space is bad. Attempts to limit all overhead blur this distinction and severely
undermine nonprofits’ ability to invest in the people and HR processes necessary to deliver great
results year after year.” (Bradach, Teirney, Stone, Delivering on the Promise of Nonprofits,
Harvard Business Review, December, 2008) It comes back to a lack a standards, a diversity of
expectations, and a diversity of effort as each foundation and nonprofit stakes out an area of
concern and goes about addressing the shortfalls that now call for intervention in its own way based
on its own insights and abilities. For the critics of performance measurement, there are many
things that measurement does not do. For one thing, the publishing and distribution of reports
does not in and of itself eliminate the need for the funder to monitor resources, activities, and
outputs. The results gleaned from a report may show spectacular results and unprecedented
achievement, but is the program measuring and reporting on the right outcomes? Can the
explanations provided in the report explain why the program achieved the level of outcome it did?
Is there any proof offered that the program itself caused the observed outcomes, or did a decade-
long drought suddenly end in a monsoon that not only made farming possible again but washed
away the miscreants who had been living up in the hills and terrorizing the inhabitants below. At

base, the critics ask, can essentially ad hoc reports confirm whether or not the outcome being
described is one in which resources should be invested.
Short Timeframes May Diminish Ambition and Idealism
There is an expression used by business managers, “Nobody wants a story.” In other words
don’t tell me why you didn’t succeed; don’t tell me why you don’t know the answer. If you didn’t
succeed, you have failed. If you don’t have the answer, you have failed. This is the attitude, one
most associated with the commercial business world, which worries some critics of measurement
reporting in the nonprofit world. The fear is that in a short-term reporting cycle, where
measurements are expressed only as success or failure, longer-term commitments will end up
looking like failures in the short-term.
It is a keenly felt concern. “Assessments don’t include the role nonprofits play in averting
problems, such as keeping kids from delinquent crimes that land them in jail or providing
preventive health counseling and nutrition services so adults won’t later need costly medical care.
Nonprofits contribute to a strengthening of civil society and democracy, to increasing civic good
and social capital…Unless the full range of nonprofit purposes is included in assessing the value of
all charitable programs, everything but the most fundamental functions will be discounted and
marginalized…we will turn away from the long-term investments…in favor of a narrowly defined
short-term individual payout.” (Rosenman 2013)
Once organizations begin to manage reporting cycles, commitments to efforts affording
long-term results may no longer seem attractive. In his 1973 book, Management: Tasks,
Responsibilities, Practices, Peter Drucker wrote, “It is meaningless to speak of short-range and
long-range plans. There are plans that lead to action today…and there are plans that talk about
action tomorrow-they are dreams…The essence of planning is to make present decisions with

knowledge of their futurity. It is the futurity that determines the time span, and not vice versa.”
(Drucker 1973) This is the modern business approach that began in the 1950’s and has remained
as a part of our commercial culture. But in this case Mr. Drucker is referring to planning for profit.
Can this blanket approach apply to nonprofits? Can a nonprofit be managed the same way as a for-
profit? It is well known that innovation is the pulse that drives nonprofit work. It takes time and
money to produce results. And sometimes those results are that the plan is not working. As
Thomas Edison said, “I have not failed. I’ve just found 10,000 ways that won’t work.” (Edison) In
many cases, failure in the nonprofit world is not necessarily a bad thing. “It is appropriate for
foundations to risk their funds on new ideas/programs, even with the possibility that some of them
will fail, because it ultimately saves taxpayer dollars, which might otherwise be appropriated to the
same measures that don't work.” (Stern 2013)
Openness and readiness to support ideas from outside an organization may be restricted by
a report driven culture. Nonprofits must be open, open to needs, open to ideas, open to change.
They cannot afford to be closed to the outside. They cannot afford to be exclusively inward
looking. Critics see in the imposition of ill-defined reporting requirements the drawing of a blind,
where the only view into the organization will be through the distorted lenses of measurement
reports. Specific promises will have to be made and then evidence provided that they were
achieved. Because of the risk of the consequences of failure that would accompany the inability to
deliver on a projected outcome, the management of the description of results will have to be
paramount. No information will be permitted to circulate that has not been reviewed, filtered and
packaged in a report. According to critics of reporting, what will follow will be a farcical imitation
of philanthropy as we now know and practice it.

Improved Reporting Does Not Improve Outcomes
Critics of reporting say that most programs do not serve enough individuals to affect
community-wide statistics, and so whether the program can be deemed successful or not can’t be
relegated to a report designed to be reviewed by someone removed from the community being
served. They point to the myriad problems that can consort to create conditions that may not be
resolvable in generations: economic conditions, environmental conditions, demographics, public
policies, and the local culture, its norms and expectations. In spite of these daunting facts,
nonprofits have traditionally taken on the task of trying to alleviate misery and offer hope to the
individual, even if the community itself is too large or two despairing to be helped as a whole. In
this kind of environment, refining the nonprofit’s reporting process will not produce any advantage
or improvement of conditions for the community being served.
Impact Measurement Is Misunderstood, Is Not New, and Will Continue to Fail in Its Attempt to
Right All Wrongs
Impact measurement is not a recent trend the critics tell us. “Lost in today’s measurement
mania…is one disturbing fact: This devotion to measurable outcomes is hardly new. Indeed, it is at
least a century old. More to the point, it has itself apparently had so little measurable impact on the
way we do business that a full century later, we’re talking about measuring impact as if we’ve just
discovered the concept”. (Schambra 2011) And not only are there no standards, but we don’t even
understand what we are talking about when we talk of measurements and reporting on impact.
“Many organizations do not yet grasp the all-important distinction between impact assessment and
performance measurement.” (McGill 2011) In other words, why do we keep trying to implement
something that has never worked and that we don’t understand anyway?

Maybe we are trying to be something we are not. “Impact assessment is not the end goal of
foundation and nonprofit work.” (McGill 2011) Or maybe we are not equipped as a sector to use
the reporting tools employed in other sectors. “All those numbers are being gathered without any
way to make meaningful comparisons among them.” (Schambra 2011) But then the numbers we
do gather are suspect after all. “A bare handful of programs—again, this is a century after we began
counting—can claim to have been scientifically validated according to the “gold standard” of
measurement, using randomized control groups.”(Schambra 2011) But then how could we achieve
the necessary scientific validation of the effort after all? We are at a distinct disadvantage. “Social
investment does not take place in a controlled laboratory setting.” (McGill 2011) For that reason,
“measurement error creeps in everywhere.” (McGill 2011) We can only ever hope to come to
grips with the issue of reporting when we are able to have “a thorough discussion about the
measurement challenges in the field of philanthropy in order to be able to talk meaningfully about
the possibility of "social impact assessment." (McGill 2011) And we will know that day has come
when we can “decide jointly on a simple, coherent, user-friendly system to which we can both pay
attention, which will prevail over bureaucratic inertia and political connections, and which will feed
into a serious body of knowledge.”(Schambra 2011)
Back in 2007, a speech given by the former president of the Ford Foundation, Susan
Beresford addressed what she referred to as the recent popular dichotomy of “old” and “new”
philanthropy. She asked "what truly is new"? Is today's new philanthropy "new" and "better" because
it is strategically aimed at root causes, results oriented, global, influenced by the business model
and driven by donor engagement?… Hardly…"Old" philanthropies have been doing this work for
decades…This approach has been common sense in philanthropy as well as in business for many
years, and is increasingly standard practice.” She warned her fellow industry colleagues to tread

carefully not to “discredit the special capacities the nonprofit sector brings to the table” and that the
business and venture capital approach is “leading some donors to distort philanthropic practices
that served society well”. Ms. Beresford stated that she is a 40 year veteran in the field and that this
notion of “old” versus “new” philanthropy “makes good copy”, but, “it does not fit the reality I
know”. From her point of view, “it has the potential to damage our field”. (Beresford 2007)
WHEN PUSH HAS COME TO SHOVE
Amid this clamor for accountability, transparency and impact it is hard not to be deeply
concerned about how the industry will evolve as the result of the demand for increased
measurements. Many, if not all arguments for and against measurements are valid and should
continue to be thoughtfully considered. Because of the industry’s tremendous growth in the past
15-20 years, it has no choice but to change. Push has come to shove. (Roumain 1947) “The
challenge for leadership and management is to prioritize among competing accountability
demands. This involves deciding both to whom and for what they owe accountability.” (Ebrahim,
The Many Faces of Nonprofit Accountability 2010) How we will make these changes as individual
professionals, organizations and as an industry is as yet unknown.
CURRENT MODELS AND METHODOLOGIES
According to Theodore H. Poister’s book, Measuring Performance in Public and Nonprofit
Organizations, 2003:
A measurement system must be designed to serve the needs of the particular management
process it is intended to support. Performance measurement systems are used to support a variety
of management functions, including the following:
 Monitoring and reporting
 Strategic planning
 Budgeting and financial management

 Program management
 Program evaluation
 Performance management
 Quality improvement, process improvement
 Contract management
 External benchmarking
 Communication with the public
Each of these functions can be carried out in ways that facilitate results oriented
management, and in each case performance measures are critical to provide the feedback
that allows it to focus on results. Yet these are very different, though often complementary,
management actions that serve different purposes. Each of these functions represents a
distinct use of and a specific set of requirements for performance measures. Thus, any
given measurement system needs to be tailored to its purpose and developed very
deliberately to support its intended use. (Poister 2003)
Theory of Change
Theory of Change is a term of art used to describe the use of maps to guide organizations
and communities through a transformation effort, maps that lay out the steps needed to arrive at a
desired outcome. “A theory of change offers a picture of important destinations and guides you on
what to look for on the journey to ensure you are on the right pathway.” (Organized Research
Services 2004) Like flow-charting, theories of change can come in many forms from simple linear
progressions from one point to the next to maps used to coordinate simultaneous and overlapping
processes. Like any good map, it will show you how to reach your destination, and show you how
to get back on your way if you have made a wrong turn or the highway is washed out. Well-
developed theories of change can be used as blueprints for achieving specific results in well-defined
domains and to make a nonprofit’s strategic vision operational. (Morino 2011)
Outcome Map One approach is to develop an outcome map, a visual diagram that
illustrates relationships between initiative strategies and intended results. The results will typically

include outcomes, both short and long term, and reflect changes at individual, organizational,
systems and community levels.
List of Assumptions about Change Maps need keys to interpret them. Geographic maps
may show roads and highways, elevations, ecological information, or population concentrations. A
key helps the reader to understand the markings. An outcome map needs a key as well, and it
comes in the form of a list of assumptions about change that accompanies the map and should
include the “philosophies, principles or values; ways to work together; community context and
other assumptions on which you have based your change effort.” (Organized Research Services
2004)
There is no right or wrong way to draw an outcome map; each map will look different,
depending on the organization’s unique needs and preferences. Within the theory of change, the
short and intermediate term outcomes for separate strategies or programs can be clearly identified.
This is important because many projects must produce change at the individual level if they are to
be able to instigate the change that is their ultimate goal and the outcome map will show the way to
the ultimate goal through these intermediate goals. Labeling these outcomes in the map will not
only bring a focus to evaluation and reporting but will also help show how the shorter term changes
contribute to the long term vision of change as illustrated in the table below.

Impact: Individual and Family Outcome Areas and Sample Outcome Statements
Outcome Area Sample Outcome Statements
Changes in attitudes, e.g.
perceptions and beliefs
Increased view among parents that local schools are positive "hubs" for families.
Increased desire among neighborhood residents to become engaged in community change efforts.
Increased feeling of safety among residents.
Increased desire of parents to create a personal savings plan.
Youth have increased belief that they will have a positive future.
Changes in knowledge Increased knowledge among neighborhood residents of community resources.
Increased knowledge of parents and caregivers about child development milestones.
Changes in awareness Increased awareness among neighborhood residents of a neighborhood's history.
Increased awareness of U.S. Laws regarding acceptable child discipline among immigrants and refugees.
Changes in skills Increased parents' employment skills.
Increased parent ability to locate child care.
Changes in behavior Neighbors more frequently call on one another for assistance.
Increased parent involement in their child's education.
Residents vote more frequently.
Changes in health Decreased blood pressure.
Families have access to a "medical home"
Changes in family stability Families' children attend the same school for all of their elementary grades.
Families maintain a stable residence.
Changes in financial status Increased family income.
Increased family savings.
The final map can incorporate many elements initially developed in table form into a visual
rendering, from short term changes to final outcomes, and including “So That” chains, core
capacities, influences and available leverage. A finished map might follow any flow-charting
protocol, standard or nonstandard, to illustrate the steps and directions the project needs to follow
to arrive at its outcome. The key to the map, the List of Assumptions, can accompany it in tabular
or narrative form.
(Organized Research Services 2004)

Collaborative grantmaking (pooled,
aligned and matching grants)
Coordinating, convening and
networking among
private/public funders
Funding priority issues
Increased ability to
influence community
goals
Increased ability to
achieve better outcomes
for the community
Increased amount
of funding
directed to
priority issues
Increased
networking
dedicated to
priority issues
Increased
stability of
funding and
consistency in
approach to
priority issues
Improved outcomes for
“aligned fund” goals
Improved “outcomes for family,
friends and neighbor care” goals
Improved outcomes for “school
readiness” goals
Improved effectiveness
as grantmakers
All children have an equal opportunity to succeed in school and as adults
Increased
intentionality
in the
approach to
priority
issues
Increased ability to leverage
resources to support
community goals
Increased
clout and
pressure to
tackle
priority
issues
Increased
visibility of
and
community
attention to
priority issues
Increased
willingness
to “tackle”
tough
priority
issues
Example of a completed outcome map
(Organized Research Services 2004)

Logic Models
Logic Models are a “systematic and visual way to present and share your understanding of
the relationships among the resources you have to operate your program, the activities you plan,
and the changes or results you hope to achieve.” (W.K. Kellogg Foundation 2004) Creating a
written logic model of program inputs, activities, outputs, outcomes and impact is a useful process
for thinking through the changes participants experience during and after a program. It also allows
for an analysis of the logic of the “if/then” influences the program intends to set in motion. (Plantz,
Greenway and Hendricks 1997) A logic model employs five basic components designed to
exemplify the connection between your planned work and your intended results.
Your Planned Work The first two components address your planned work, identifying the
resources needed for program implementation and the activities it will pursue.
1. Resources/Inputs include the human, financial, organizational, and community resources a
program has available to direct toward doing the work.
2. Program Activities are what the program does with the resources. Activities are the processes,
tools, events, technology, and actions that are an intentional part of the program implementation.
These interventions are used to bring about the intended program changes or results.
Your Intended Results include all of the program’s desired results: outputs, outcomes, and impact.
3. Outputs are the direct products of program activities and may include types, levels and targets of
services to be delivered by the program.
4. Outcomes are the specific changes in program participants’ behavior, knowledge, skills, status
and level of functioning. Short-term outcomes should be attainable within 1 to 3 years, while

longer-term outcomes should be achievable within a 4 to 6 year timeframe. Longer-term outcomes
should result in impacts that begin to be felt in 7 to 10 years.
5. Impact is the fundamental intended or unintended change occurring in organizations,
communities or systems within 7 to 10 years of the project’s inception. In the current W. K.
Foundation model, grantmaking and evaluation, impact often occurs after the conclusion of project
funding.
The term logic model is frequently used interchangeably with the term “program theory” in the
evaluation field. Logic models are alternatively referred to as theory because they describe how a
program works and to what end. (W.K. Kellogg Foundation 2004)
Sample Logic Model
(Pell Institute n.d.)
)

Dashboards
Performance dashboards are not “one size fits all”. Every organization needs to track metrics
that are relevant to its own mission, activities, output and outcomes. However, effective dashboards
share the following characteristics:
 They contain meaningful, timely, accurate data
 They are rich with information but not overwhelming
 They help nonprofit organizations make short- to medium-term operational decisions with
a clear eye toward improving long-term outcomes
 They flag issues or problems as well as accomplishments. (O'Flanagan, Harold, Best 2008)
The key factor in designing a dashboard is deciding what kind of data to collect and how to use
it. According to Jason Saul, author of Benchmarking for Nonprofits: How to Measure, Manage,
and Improve Performance: “A dashboard could just be a fancy report that says nothing if the data
isn’t meaningful. It’s easy to create a database and plug in numbers. It’s much harder to create
performance metrics that can be measured against desired results and used to make real-time
programmatic, financial, or managerial decisions.” (Saul 2003)
Indiana Museum of Art Dashboard
(Indianapolis Museum of Art)

One thing worth noting in the dashboard illustrated above, it is not an end in itself. Even
though the data shown is summary data, each data item is accompanied by a link that allows the
viewer to drill down to a greater level of detail. That next level might be another dashboard, or it
might be a tabular display of data. While the term dashboard itself might suggest a single layer of
information, in practice it should allow the user/visitor to follow lines of inquiry as desired. In fact,
a well designed dashboard should draw the user to information that the organization wants to
emphasize.
DEVELOPING A COMMON LANGUAGE
In 2011, the Center on Nonprofits and Philanthropy sponsored a symposium that
addressed the barriers to and opportunities for making performance management more common
in the social sector. Mindy Tarlow from the Center for Employment Opportunities addressed the
lack of a common language for measurements:
“Around performance management time and again, you hear (that) it is somehow
antithetical to passion and mission. Or that no one is ever going to understand the impact
of what you do if you talk about numbers. They are only going to understand it if you talk
about stories. And I would submit that that is just not true. You have to really be able to
talk about both….I think it is not so much about numbers; it is about facts. If you actually
just use that word—instead of saying ‘numbers,’ ‘performance,’ ‘outcomes,’ you just used
the word ‘facts’—I think most people would agree that you would rather know facts than
fiction. That you would rather push yourself toward something that you actually know than
something that somebody just told you and so you are running with it. So, I think that is a

language issue that we would be well served to think about.” (Tough Times, Creative
Measures 2011)
In 2006, The Urban Institute and the Center for What Works created a draft taxonomy of
candidate quality indicators for nonprofit outcomes to assist nonprofits in collecting outcome data.
As important as the taxonomy may prove to be, it is instructive to read how the Urban Institute has
gone about assembling it.
While there is no shortage of outcomes and their indicators in some program areas, there
is no centralized grouping of them or assessment of their quality that could serve as a
resource for organizations that wish to develop outcome measurement systems. And
because of the vast range of programs in the voluntary sector, major gaps exist in the
coverage of indicators that have been developed. The taxonomy attempts to provide a way
to help reduce this gap – for those programs for which indicators are not yet available.
We collected information from a wide range of sources, from national nonprofit umbrella
groups in the US, national accreditation agencies in specific fields, and from national
nonprofits with local affiliates. They were assessed with thought about which ones were
useful, relevant, and feasible. We also considered outcome indicators that were seldom
currently used but appear to be very appropriate for inclusion.
The most useful taxonomies tend to reflect the manner in which the sector itself organizes,
collects and reports the information. Although essential taxonomic principles of
comprehensiveness, mutual exclusivity of elements, and logical consistency must be
followed, a grounding is needed in what is actually in use by practitioners and what has
worked for the specific program areas. Thus, testing by stakeholders (including nonprofit

Susan Hoff-Capstone Thesis Project

Susan Hoff-Capstone Thesis Project

Recommended

Recommended

More Related Content

Similar to Susan Hoff-Capstone Thesis Project

Similar to Susan Hoff-Capstone Thesis Project (20)

Susan Hoff-Capstone Thesis Project