IAOS-OECD 2018 conference, 19-21 September 2018, Paris. More information at: http://www.oecd.org/iaos2018/

1. 2018 PARIS21 ANNUAL
MEETINGS
5 APRIL 2018
PROPOSING A USE OF STATISTICS INDICATOR
IN NATIONAL DEVELOPMENT PLANS
International Association for Official Statistics Conference 2018
Paris – September 20 2018
Parallel session on Statistical evidence on policy making

2. 2
History of the indicator
• Introduced in 2010 to assess impact of international statistical capacity
building activities in developing countries, following external evaluation of
P21 in 2009.
• Reports in 2010, 2012, 2014 and 2015. A similar indicator is used to inform
logframes and monitor Global Action Plans.
• In 2015 the manual coding was partially replaced with data mining.

3. 3
Why is it useful?
• Link between statistical development and policy making (impact of P21’s
activity).
• Replicable, low-cost methodology already implemented in other indicators (e.g.
statistical literacy, tweet global summits)
• Extensible to new areas of work for P21: gender, sectoral (e.g. agriculture)
• Popular indicator for partners and contribution to current debate on “quality” of
NDPs (e.g. well-being, SDGs)

4. 4
• Scoring system to measure use of statistics in public policy documents
• Automatised methods (data mining) to produce quantitative metrics on
countries’ official use of statistics
• Different dimensions considered:
• Upstream: how statistics contribute to decision-making
• Downstream: Monitoring and evaluation
• Target audience: policy makers, academics and users
What does it do?

5. 5
Example: Sierra Leone NDP
Food poverty incidence in rural areas fell from 71%
in 2003 to 51% in 2011; in urban areas it declined
even more significantly, from 63.5% to 40.8% in
2011, particularly in the poorest quantiles.

6. 6
Scoring
Component Total score Description
Up
stream
Basic use
Level 1: 10 pts Basic: Introduction of measurable concept
Level 2: 20 pts Diagnosis: depiction of state of measurable concept
Level 3: 30 pts
Analysis: use of statistical methods to describe state and
causes of variation
Disaggregation 20 pts
Frequency of disaggregation keywords across NDP
sectors/topics
Down
stream
Monitoring and evaluation
arrangements
10 pts Frequency of M&E keywords across NDP sectors
Assessment of previous plans 10 pts Analysis undertaken about previous plans

7. 7
Innovations
• Coverage: 102 countries (199 documents)
• Keyword list
• New sectoral classification
• International databases by topic
• Clustering for validation
• Three different levels on sophistication of use  based on Stat. Literacy index
• Sentence level analysis in four dimensions
• New weights, related to depth of analysis for each section

8. 8
Example: Sierra Leone
Food poverty incidence in rural areas fell from 71%
in 2003 to 51% in 2011; in urban areas it declined
even more significantly, from 63.5% to 40.8% in
2011, particularly in the poorest quantiles.
Level 1: indicator
Level 2: indicator + figure
Level 3: indicator + statistical term
Disaggregation: dimensions of disaggregation

9. 9
Summary results
Max: 62Min: 8
Mean
SD: 11 Min Max Mean SD
2000-2008 8 62 38 11
2009-2017 11 62 42 11
PRSP 12 60 40 9
NDP 8 62 40 12

10. 10
Results: average scores
Source: PARIS21 (forthcoming) Proposing a Use of Statistics indicator in National Development Plans.

11. 11
Robustness checks: Sensitivity
Boxplot showing the median rank and the corresponding 25th and 95th percentile
Rank 85:
Weighted
aggregate of
standardized
scores
Rank 31:
Weighted
geometric
aggregate

12. 12
Robustness checks: Uncertainty
0
50
100
150
200
0 50 100 150 200
Excludinglevel2(Rank8)
Original Rank
0
50
100
150
200
0 50 100 150 200
Non-weighted(Rank2)
Original Rank
Rank Country Year
Diff
Rank 2
Diff
Rank 3
Diff
Rank 4
Diff
Rank 5
Diff
Rank 6
Diff
Rank 7
Diff
Rank 8
Diff
Rank 9
Diff Rank
10
Avg
diff.
2 PHL 2017 -2 -3 -5 -2 -1 -4 -1 -4 -1 2
47 BTN 2013 +12 -38 -4 +16 -6 -9 -32 -37 -26 20
92 NGA 2004 -2 -4 -4 +20 +19 -49 -52 -7 20
197 PNG 2004 0 +1 0 +1 +1 +2 +1 0 1
Average difference 8 11 6 30 10 10 12 13 7 12

13. 13
Research and policy in development
Framework
The political context – political
and economic structure, culture,
institutional pressures, incremental
vs radical change etc.
The evidence – credibility,
research approaches and
methodology, communication,
etc
External Influences
Socio-economic and
cultural influences,
donor policies etc
The links between policy
and research communities –
networks, relationships, power,,
trust, knowledge etc.
Source: ODI

14. 14
Proxy indicators
Political context
• CPIA
• WGI
Evidence
• SCI
• Statistical literacy
External Influences
• GDP per capita
• ODA
• HDI
• Adult literacy
Links
• Press freedom
UoS

15. 15
Concurrent validity checks
SCI
(Yr-1)
CPIA
(Yr-1)
WGI: Voice and
Accountability
(Yr-1)
Press Freedom
(Yr-1)
HDI
(Yr-1)
Literacy
(Yr-1)
GDP per capita
(Yr-1)
Official Development
Assistance
(10 prev Yrs)
Total 0.17* 0.15 -0.12 0.19* 0.08 0.09 -0.06 0.09
Africa 0.17 0.27* -0.02 0.08 -0.03 -0.08 -0.18 0.11
Asia 0.06 -0.12 -0.29* 0.33* -0.01 0.03 -0.13 0.10
Oceania,
Europe
and
America
0.23 0.50 0.22 0.05 0.44* 0.63* 0.13 -0.10
PRSP 0.33* 0.23 -0.01 0.13 0.18 0.18 0.10 -0.05
NDP 0.07 0.10 -0.19 0.23* 0.03 0.02 -0.09 0.13
2000-2008 0.20* 0.16 -0.14 0.26* 0.10 0.13 -0.16 0.05
2009-2017 0.09 0.16 -0.08 0.06 -0.05 0.03 -0.02 0.07
Pairwise Pearson correlation between selected indicators and use of statistics for the various sub-populations in the
sample

16. 16
• Complement with in-depth analysis of specific components in
NDPs/PRSPs/Sectoral policies
• Develop UoS indicator for specific sectors (e.g. gender, agriculture, labour)
• Feedback on statistical planning tools (e.g. ADAPT)
• Go beyond policy documents review: strengthen methodology
• Monitoring & Evaluation
• NDPs progress reports
• Feed statistical literacy work stream and PARIS21 Flagship report
Way forward

17. 17
• Is the indicator relevant for a policy maker? How it can be improved?
• What other dimensions to analyse in NDPs (e.g. disaggregation)?
• Limitations (e.g. bias in keywords, no sematic analysis). How to tackle?
Questions for the audience

18. THANK YOU!

19. 19
Towards better use of statistics
CD 4.0: a
demand-driven
approach
Channels: from 
capacity to  use
From  Use to
 Governance

20. 20
Rank 2: Non-weighted aggregate
Rank 3: Weighted aggregate of standardized scores (z scores)
Rank 4: Non-weighted aggregate of z scores
Rank 5: Weighted geometric aggregate (only for documents where all
components are different than zero)
Rank 6: Aggregate excluding level 3, rescaled weights for the rest of the
components
Rank 7: Aggregate of z scores, excluding level 3 and rescaling weights for the
rest of the components
Rank 8: Aggregate excluding level 2, rescaled weights for the rest of the
components
Rank 9: Aggregate of z scores, excluding level 2 and rescaling weights for the
rest of the components
Rank 10: Weighted aggregate of rescaled scores for individual components

21. 21
Minimum Maximum Mean
Standard
deviation
Asia 8 62 41 8
Africa 12 60 39 9
Americas 12 55 38 12
Oceania 8 55 33 15
Europe 16 58 45 14