Speaker: Rachel Philippson, Mott MacDonald Presentation at the Eldis 20th Anniversary event "Learning from 20 years of digital knowledge sharing for global development" held at IDS on Thursday 15 September 2016 and Friday 16 September 2016. A video of this presentation is available at: https://youtu.be/aNboIDhAOjw

1. Evaluation of DFID Online Portals
and Repositories
Eldis 20th Anniversary Workshop
Institute of Development Studies
Brighton
15th September, 2016
Rachel Phillipson
Mott MacDonald Senior Economist &
Evaluation Specialist

2. 5 evaluation objectives
1. To describe user populations and examine how they interact with
online research portals and repositories;
2. To assess the quality and accessibility of online research portals
and repositories and to collate and analyse the available evidence on
their use;
3. To draw out and illustrate ‘plausible pathways’ between portal use
and uptake of evidence in policy and practice;
4. To assess whether the DFID-funded portals and repositories
present value for money, in their own right and in relation to sites not
funded by DFID;
5. To provide recommendations for how the DFID-funded websites
might be improved and better monitored.

3. R4D – DFID Repository (£300,000 p.a)

4. Eldis – A portal for aid information (£400,000 p.a)

5. SciDev.Net – Science & Tech news (£2.2M p.a)

6. A simplified Theory of Change

7. Evidence mapping for the ToC
Theory of Change
hypothesis
Source Summary of conclusions from the
literature
1
Portals should be mobile friendly Sylla et al. (2012)
Starkey (2013)
Global Internet Report (2014)
Debeljak, K. (2010)
Euforic Services (undated)
J.Adams & T. Loach (2015)
Batchelor (2013)
Rapidly increasing use of smart phones to access
internet services and social media; rapid increase in
sharing citations and alerts about research via mobiles in
Europe and USA; rapid catching up in mobile use in
lower income countries, though little evidence of this yet
extending to social media use by them to access or
promote research. Strength of evidence: Medium
2
Content should be prioritised on
search engines
Pew Research (2014)
De Satge (2011)
Intermedia (2010a)
Prakash (2013).
Users of online research, particularly non-academics,
often prefer to search portals using general search
engines (eg Google, Yahoo) rather than dedicated site
search functions. Strength of evidence: Strong
3
Portals should meet region
specific cultural requirements
Chavan et al (2009)
Chavan, (2004)
Bidwell & Winschiers-Theophilus
(2010)
Moalosi et al (2007)
Faiola & Matei (2005)
Perceptions of a system’s usefulness are culture-based;
websites that are tailored for local audiences are
demonstrably more effective in reaching that audience.
Strength of evidence: Strong
4
Portals should be specialised Intermedia (2010b)
Bayliss et al. (2012)
Development policy makers often start a web search with
a known ‘expert’ website (eg World Bank, the Lancet).
Strength of evidence: Weak
5
Portals should be linked to
reputed authors
McKenzie & Özler (2014)
AidDATA (2015)
References, mentions, citations by known sources,
‘influentials’ and ‘elite bloggers’ have demonstrable
impact on attention given to the referenced item. These
results are from Northern users only. Strength of
evidence: Medium

8. Summary of research tools used
 Online survey sent to 10,000+ addressees: 40 questions,
half of which were free text.
 3x Country case studies: Ghana, Tanzania, Nigeria using
(i) 2-part ‘contextual inquiry’ and (ii) research diaries.
 Heuristic evaluation ( independent ‘expert review’) of the
websites’ usability.
 VFM ‘3Es’ review: Eldis and SciDev site visits, web
metrics and financial analysis. Light-touch comparisons
with Pambazuka, Zunia and GSDRC (helpdesk) portals.
Plus...
 ‘Plausible Pathways’ drew on case studies, market
research and portals’ personae and impact studies.

9. Our market research respondents
N=734 from market research including new multilateral/donor (South) category

10. Respondents by organisation type
Based on full sample of 945 completed questionnaires

11. Our case study participants
Male
75%
Female
25%
Gender overall
0 2 4 6 8 10 12
Network (e.g Proff. Assoc.)
Parliament/Politcal Party
Private charitable foundation (e.g. Bill and Melinda
Gates)
Media (e.g. National news paper)
National Civil Society Org (Charity faith-based org.)
Gov't Donor agency (USAID, DFID)
National Gov't(excluding overseas Aid Dept)
Uni Dept/ Res Inst/ Think tanks
Consultancy
Organisation Type

12. Findings

13. Research evidence-seeking online is
frequent, rapid and impatient
 Half of all MR respondents said
they search online for research
evidence for work a few or many
times a day
 Case study participants frequently
referred to lack of time to devote
to searching
 Searching often limited to 1st page
of Google
 Failures, eg of downloading,
searching, broken links, lead to
immediate abandonment

14. Although there are some sceptics,
‘Google is King’…
0.0%
5.0%
10.0%
Eldis SciDev Zunia GSDRC
Percentageof sessionswith internal search - All
traffic data 2015
All Session
New Visitor
Returning Visitor
- More efficient - will cover all portals in one search
- More effective - Case study instances of Google being
better at searching the site than own search function
- Returning users tend to use internal search more
frequently

15. World Bank& UN are the ‘go-to’ sites
for international data & guides

16. Governments’ own websites are popular
• Own government portals frequently used for local stats
• But country variation in perceived reliability of Govt
websites
• Stats are often sought independently of ‘research’

17. A wide range of formats & services are
used when searching for evidence
online

18. Credibility of evidence found online is
assessed initially – and quickly - via
the source
 Nearly two-thirds of respondents use the source of research
evidence to help judge if they should trust it; 20% listed no
other criteria
 Sources can be author, affiliation, host site, respected
referee (personal or journal based)
 DFID, WB, UN are key trusted host sites
 There is a probably a two-stage process: credibility of source
is assessed online; validity of the evidence is assessed in
longer time. But the first is key.

19. DFID is a trusted source and
portals’content is perceived to be high
quality - but not always accessible
 “I used them because I trusted the information posted in these
website” TZ5 diary (Scidev and r4d)
 “That’s very useful … relevant to my work” TZ7
 “I will come back to this website [R4D].... for more information on
DFID projects”. NIG10
 “I’ve never heard of these anywhere. I feel shameful... especially if
you are working on a DFID-related project, you want to be
efficient” GH11
 Becomes surprised that the specific article is not the first title. Also
realises that the specific article automatically opens on a different tab
NIG02

20. Research evidence-seeking is going
mobile
-50.00%
0.00%
50.00%
100.00%
150.00%
200.00%
Eldis SciDev.Net
Desktop
Mobile
Tablet
Year on year % change in sessions by device used
- 40% of MR respondents use a computer, phone and a
tablet to access the internet
- Case study participants moved effortlessly between
devices in search tasks [GH5, TZ5, TZ1]

21. Southern users per se are not different
in their information behaviour
 Biggest differences are between (global) academics/researchers and
Southern civil servants
 This suggests differences may be driven by training, time-
constraints and research needs
 Women (especially Northern) are more likely than men (especially
Southern) to use email newsletters/alerts to keep up to date. This may
also be linked to time constraints.
 Webmetrics suggest Southern users generally spend somewhat
longer on site - but could be due to internet availability problems

22. But other barriers can still be a problem
 Three-quarters of respondents (n=534) reported commonly
experiencing one or more problems when using the internet
to obtain research evidence for work. Of these:
 1 in 3 reported problems relating to paywalls and firewalls
 1 in 4 reported problems with searching the internet
 1 in 5 reported problems with internet access
“Despite having Hinari access, some journals are still not freely available online.
Internet speeds still poor, unreliable and expensive in Malawi. Power often
down when speeds are faster!” [Southern Academic/Researcher in Health]
“For this office I don’t get much journal articles because we don’t subscribe to them ....I am
able to get access to them when I go to the University” …“I don’t get internet in my office
regularly, and even when it is available, it is not always fast”. GH12
 Uses phone and hotspots for internet connection outside Accra GH1

23. Strong evidence of individual and
inter-personal level behaviour
change occurring
Increased supply of evidence is the main reason for perceptions
of own more frequent and easier use of internet research:
 particularly improved availability (better and cheaper internet
access, more free journals)
 Increased ‘discoverability’ (eg Google and Google Scholar
facility) is also commonly quoted
Increased demand from colleagues and organisations is noted:
‘There is more demand for researched-based quality reports and
discussion with colleagues is a necessary activity to improve quality.’
‘[...]now we discuss the references and sometimes can
triangulate data we have found on the web.’

24. And indications of both organisational
and institutional behaviour changes
emerging
Most evidently driven by the demands of (aid) funders:
‘Stronger emphasis from clients on research evidence’
‘[B]ecause there is greater need now to understand the quality of the
evidence, particularly as donors are pushing harder for proof that the
evidence is robust.’
‘More focus on evidence from politicians and universities’
‘It's become more of a topic across the NGO sector, with more awareness
about it and tools for trying to assess quality (i.e. BOND evidence
principles)’

25. ‘Plausible Pathways’ between online
evidence use and uptake are discernible
1. The Personal pathway: self-efficacy
and motivation, credibility and influence
2. The Technocratic pathway: sign-
posting solutions for policy makers and
practitioners
3. The Democratic pathway: informing
opinions, generating engagement

26. 1. The ‘Personal’ pathway – self-efficacy
and motivation, credibility and influence
 “Access to information is empowering ....We used to be able to talk only about Tanzania, but
now we can go to a conference and talk about... many countries” Tanzania Think Tank
 “As long as the data is credible, it’s a good thing and it causes dialogue between people on
techniques as well as data. Tanzania research institute
 “I can download and then when I am on the plane 35,000 ft high, I can read it
offline…”Nigerian consultant
 “Nowadays, no one can deceive me; I am able to cross check information..to see if it is
authentic”. Nigerian NGO
* Individuals have a wide range of internet
skills and degrees of confidence in them
*But all find it saves them time and opens up
sources to them
*They generally therefore feel more effective
influencers

27. 2. The ‘Technocratic’ pathway– sign-
posting solutions for policy makers and
practitioners
 “It’s rare to hear that they [Government] have been reading to
understand the underlying problem.” Tanzanian think-tank
 Politicians want steps for how to change, and unfortunately
researchers are not doing that. Tanzanian NGO
*Policy makers and practitioners are very time poor
[Case studies]
*Evidence needs are pragmatic, not academic –
*search terms are ‘how to...’ ‘manual’[Case studies]
* Donor-funded TA has privileged access and remit to
work with Government

28. 3. The ‘Democratic’ pathway – informing
opinions, generating engagement
 “It’s forcing the credibility issue on us that we must look for the right
information if we really want to debate, otherwise you’d better keep your
mouth shut” Ghana MP.
 “For example, in transport sectors, we wouldn’t know how the
Tanzanian port is regarded, e.g. corruption, speed of handling....’
Tanzanian consultant
 They [politicians] are very strategic, for example looking through the references to check their
credibility. Tanzanian research institute
 It has also made verification of information easier [especially] by accessing
information from reliable local internet sites. Nigerian media professional
* Users are extremely time aware – speed, relevance, real time
feedback
* They seek and check stats/facts - not extended rationales
* Trust in the source is important
* Other media and devices are employed – radio, facebook, mobiles
* Not all directed at policy makers but also at the public
*Uptake is not immediate or direct: ‘They will remember us at
election time… and listen’.

29. Key Lessons
for online research dissemination

30.  Policy actors are characterised by pressure of time; don’t have the luxury to
assess quality in depth when online
 There are concerns about quality of research online but policy makers
are happy to use proxies and short cuts to assess them
 Trusted sources are paramount
 Optimising the discoverability of the portals and their content through general
search engines is key to being found
 Users of all types want access to statistics – local, national and international
 A wide range of formats and media are used by people searching for
research evidence online
 North – South divides between user behaviours are not strongly evident
 Handheld devices should be taken seriously to disseminate research
 Distinguishing between use and uptake, defining uptake as behaviour
change, and thinking in terms of pathways help unravel ‘impact’.