1. CSO SI METHODOLOGY REVIEW:
CONCEPTUALIZATION AND
MEASUREMENT
S. Wojciech Sokolowski
Washington DC, June 10, 2015
2. Areas of Examination
• Appropriateness and clarity of concepts
• Validity and reliability of measures
• Scoring of measures
• Weighting of measures
• Periodicity of data collection
• Comparability with other indices and data sources
• Assorted technical issues
3. Overall CSO SI Assessment
Good job, but there is room for improvement:
• Clearer conceptualization of measured dimensions;
• More objective scoring technique that uses Thurstone
scale employing factual True/False statements to
calculate the score; and
• Use communication process based on Delphi method
• Add a “map” of what is measured in target countries;
4. Finding 1: Comprehensive CSO Definition, but
not followed in data collection
The current Index methodology uses an internationally
accepted definition that encompasses a wide variety of
CSOs, including:
• health, educational, and social services providers
• foundations, charities, and international NGOs
• grassroots community organizations
• business, professional and labor organizations,
• cultural and religious organizations,
• nonprofit cooperatives, and
• other membership associations.
5. Civil Society Conceptualizations
NARROW SCOPE:
1. Defined by law (e.g. 501c3,
Registered Charities)
2. Defined by a specific function (e.g.
NGOs, CBOs, foundations)
COMPREHENSIVE SCOPE:
3. NPI/CSO sector (UNSD, CSO SI)
4. Social Economy (EU)
BROAD SCOPE:
5. Third Sector
6. Sphere between households, market and state (CIVICUS)
6. Data collection does not reflect
conceptualization
The existing practices of data collection and assembly
have an unintended consequence of de facto limiting
the scope to much more narrow subsets of CSOs that
vary from country to country, which risks comparing
apples to oranges.
7. Recommendation 1: Map civil society in target
countries
Type of organization Any CSOs of this
type?
(Yes, No)
Approximate size of CSO
in scope (number of
people or organizations)
Information availability
(Good Medium Low)
Associations Yes about 20,000 orgs Medium
Foundations Yes about 500 orgs Good
Cooperatives Yes about 800 orgs Medium
Informal/Community based organizations Yes about 1 million volunteers Low
Educational institutions No
Religious organizations Yes about 50,000 churches Low
Health care organizations No
Social assistance organizations Yes About 3,000 orgs Medium
Youth organizations Yes about 200,000 members Medium
Housing associations No
Water/electricity supply orgs Yes about 200 orgs Good
Environmental organizations Yes
Labor unions Yes about 2 million members Medium
Business/employer associations Yes
Professional associations Yes
Political advocacy/lobbying organizations No
Civil/human rights advocacy organizations Yes
Museums/historical sites/cultural orgs Yes
Micro-finance institutions No
International cooperation/aid orgs Yes
Paramilitary groups No
Other (specify)
L
I
H
8. Finding 2: Inadequate conceptualization of CSO
sustainability
• CSO operating in different environments have
different sustainability models, some emphasizing
government grants and contracts, other market sales,
and still other volunteer mobilization and private
fundraising.
• CSO sustainability is not “one size fits all” but rather
involves the ability of organizational leadership to
develop effective strategies of dealing with
environmental challenges.
• Applying the concept of sustainability to an entire
organizational environment borders on the logical
fallacy of composition.
13. Recommendation 2: Rearrange existing CSO SI
into 3-dimensional framework
Organizational
Environment:
1. Legal
2. Political
3. Social
4.Institutional
Organizational
Inputs:
1. Professional staff
2. Volunteers,
members, supporters
3. Financial
resources
4. Technology and
information
Composite
CSO Index
score
Organizational
Outputs and
Outcomes:
1. Service
2. Advocacy
3. Interest
representation
4. Social inclusion
and cohesion
L/M
I/E
H
14. Findings 3. Problematic Validity of 7 dimensions
Absent a clearly defined “target”- a causal model of
CSO sustainability- validity assessment leads to a
circular tautology. They measure whatever they
capture and they capture whatever they measure.
16. Recommendation 3: More rigorous validity test
Engage a team of experts to test the
validity of the measures of the three
dimensional conceptual framework.
L/M
I/E
L/M
17. Findings 5-7: Problematic scoring technique
• Scoring methodology is the most
frustrating, challenging and controversial
aspect of the Index production.
• Scores are often driving the process of
data evaluation, rather than the other way
around.
• The subjectivity of data collection and
interpretation has been identified as the
most serious challenge in the Index
methodology.
19. Recommendations 4-5: Revise scoring
methodology
• Replace the existing ordinal Likert scale
with an Equal-Appearing Interval Scale.
• Use Delphi method to improve
communication process among
panelists, experts, and EC members.
M
I/E
T
L
I
M
20. Example of Equal-Appearing Interval Scale
Hypothetical indicators of legal environment TRUE FALSE CONF*.
1. The right of association is guaranteed by law or constitution. 5
2. The law limits allowable purposes of CSO beyond general legality, morality or public order. 3
3. Unincorporated or unregistered CSOs are legally permitted to operate. 3
4. The law gives government agencies the authority to dissolve any CSO for any reason. 2
5. The law provides severe penalties for operating unregistered CSOs. 4
6. CSO registration process is burdensome and costly. 1
7. The law does not allow appeals from decisions of administrative bodies/government agencies. 2
8. The law gives government agencies the authority to alter CSO missions or operations. 2
9. Appealing government/administrative decisions is burdensome and costly. 1
10. CSOs are legally allowed to obtain funding from any generally legal source. 1
21. Example of Delphi MethodROUND1
Panelists and
experts
answer
questionnaire
ROUND2
Panelists and
experts
provide and
review
feedback
from each
other, amend
answers if
needed
ROUND3
EC reviews
answers and
provides
feedback;
feedback
internally
reviewed and
coordinated
ROUND4
EC feedback
reviewed by
IPs, panelists
and experts;
answers
amended if
needed
ROUND5
Answers
finalized and
submitted for
scoring
22. Other Recommendations
• Keep weighting all dimensions equally
• Keep annual data collection and
publication
• UNSD and ILO recommend compiling
data on CSOs and volunteering; take
advantage of them when available
• Keep the USAID “brand” – it is a mark
of high quality
L
I
M
23. CSOs in National Accounts Statistics
1.6%
1.9%
2.0%
2.2%
2.3%
2.5%
3.4%
4.5%
4.7%
4.9%
5.2%
5.3%
5.8%
6.6%
6.7%
7.1%
8.1%
4.5%
0.0% 1.0% 2.0% 3.0% 4.0% 5.0% 6.0% 7.0% 8.0% 9.0%
Thailand
Czech Republic
Portugal
Mexico
Kyrgyzstan
Cameroon
Brazil
Norway
France
Australia
Japan
New Zealand
Belgium
United States
Mozambique
Israel
Canada
AVERAGE
Percent of GDP
Source: JHU Center for Civil Society Studies
