Preparing to produce an online evidence map of empirical social sciences studies about changing climate in South Africa
•Download as PPT, PDF•
1 like•1,075 views
This document discusses preparing an evidence map of social science studies on climate change in South Africa. It notes some anticipated challenges including conceptual diversity in how climate change is understood, the diverse nature of primary research studies, and differing epistemological perspectives. The evidence map aims to visually summarize the available evidence to help identify gaps and priorities for future research. It will involve a systematic search, screening and categorizing included studies, and creating user-friendly visualizations and summaries. The map is intended as a tool to help inform policy and practice around climate change in South Africa.
Recommended
More Related Content
What's hot
What's hot (19)
Viewers also liked
Viewers also liked (20)
Similar to Preparing to produce an online evidence map of empirical social sciences studies about changing climate in South Africa
Similar to Preparing to produce an online evidence map of empirical social sciences studies about changing climate in South Africa (20)
More from Carina van Rooyen
More from Carina van Rooyen (9)
Recently uploaded
Recently uploaded (20)
Preparing to produce an online evidence map of empirical social sciences studies about changing climate in South Africa
- 1. Carina van Rooyen, Natalie Rebelo Da Silva & Ruth Stewart Preparing to produce an online evidence map of empirical social sciences studies about changing climate in South Africa by Ruth Stewart Carina van Rooy Marcel Korth Admire Chereni Natalie Rebelo D Thea de Wet September 2012 Q53 Microfinance cover.indd 10 Presentation at the First International CEE Conference in Stockholm, Sweden on 27-29 August 2016
- 3. Research synthesis is about... Masekela et al 2012 Ibrahim 2009 Masuka et al 2007 Ngema et al 2016 Baloi et al 2014 Gwangwa et al 2013 Nwazas et al 2013 Dube et al 2010 Makeba et al 2007 Kimani 2008 Slidesource:AfricaEvidenceCentre2016
- 4. …different ways to collect & organise research evidence Rapid Evidence Assessments Literature reviews Systematic reviews Evidence maps Slidesource:AfricaEvidenceCentre2016
- 5. What is evidence mapping? Photosource: https://thegoodistablog.files.wordpress.com/2013/10/worldmap-in- grains.jpg
- 6. Map Included evidence Systematic search Critical appraisal Analysis Visualisation Access to user friendly summaries SRs IEs Other 3ie’s evidence- gap map Yes Can do No Yes Yes (SR) Yes Yes Yes Evidence map Yes Yes Yes Yes No Yes No Yes Scoping review (Arksey & O’Malley 2005) Can do Can do Can do May be limited No Yes No No Systematic map Can do Can do Can do Yes Limited Yes No No Policy-relevant evidence map Yes Yes Yes Yes Yes Yes Yes Yes Source: Snilstveit et al 2013:6; Langer et al 2016 Various kinds of maps
- 7. Purposes of evidence maps Slidesource:AfricaEvidenceCentre2016
- 8. Mapping the evidence: why? BIG QUESTIONBIG QUESTION MAP OF THE EVIDENCE MAP OF THE EVIDENCE Interventions in place, but no evidence Interventions in place, but no evidence Consider stopping the intervention Consider stopping the intervention Consider starting the intervention Consider starting the intervention Pilot and researchPilot and research No intervention, good evidence No intervention, good evidence No intervention, no research No intervention, no research Stop doing research on this, summarise it Stop doing research on this, summarise it Saturation – don’t need more evidence Saturation – don’t need more evidence Slidesource:AfricaEvidenceCentre2016
- 9. Main steps in a systematic review 1. Publish title registration and then full protocol, often including pilot searches 2. Full searches 3. Screening 4. Coding and critical appraisal 5. Data extraction of findings 6. Synthesis 7. Writing up full review 9 Slidesource:Stewart2016
- 10. 1. Publish title registration and then full protocol, often including pilot searches Develop the scope (can include published protocol) 2. Full searches 3. Screening 4. Coding and critical appraisal (and categorisation for map) 5. Data extraction of findings 6. Synthesis 7. Writing up full review Visualisation (including user-friendly summaries) Main steps in this evidence map
- 12. Anticipated challenges of EM of social sciences studies on changing climate in South Africa
- 13. Conceptual diversities re climate change Photo source: http://www.mdpi.com/nutrients/nutrients-06-00546/article_deploy/html/images/nutrients-06-00546-g003-1024.png
- 14. Diverse natures of primary research Photosource:http://grainlegumes.cgiar.org/wp- content/uploads/2013/01/13-CIAT-bean-gene-bean-diversity-CIAT- bank-from-Flicker.jpg
- 19. Concluding
- 20. Carina van Rooyen CEE Johannesburg Africa Centre for Evidence University of Johannesburg cvanrooyen@uj.ac.za | @carinavr | www.africaevidencenetwork.org ceejoburg@uj.ac.za | @ceejoburg | www.ceejoburg.com
- 21. References Africa Evidence Centre 2016 Evidence mapping: Exploring capability and use of the map for Human Settlements. Presentation by Laurenz Langer on 23 June Arskey, H. & O’Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8(1):19-32. Armstrong R, Hall, BJ, Doyle J & Waters E 2011 ‘Scoping the scope’ of a Cochrane review. Journal of Public Health, 33(1): 147-150 Bates S, Clapton J & Coren E 2007 Systematic maps to support the evidence base in social care. Evidence and Policy: A Journal of Research, Debate and Practice 3(4): 339-351 Clapton J, Rutter D & Sharif N 2009 SCIE systematic mapping guidance, April 2009 [draft]. Using knowledge in social care, research resource 03. Social Care Institute for Excellence Daudt HML, van Mossel C & Scott SJ 2013 Enhancing the scoping study methodology: A large, inter-professional team’s experience with Arksey and O’Malley’s framework. BMC Medical Research Methodology 13:48 Haddaway NR, Bernes C, Jonsson B & Hedlund K 2016 The benefits of systematic mapping to evidence-based environmental management. Ambio DOI 10.1007/s13280-016-0773-x
- 22. Jones K 2004 Mission drift in qualitative research, or moving toward a systematic review of qualitative studies, moving back to a more systematic narrative review. The Qualitative Report 9(1): 95-112 Mays, Arksey H, O’Malley L, … 2005 Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology: Theory & Practice 8:5 Miake-Lye IM, Hempel S, Shanman R & Shekelle PG 2016 What is an evidence map? A systematic review of published evidence maps and their definitions, methods and products. Systematic Reviews 5: 28 Raymond CM, Fazey I, Reed MS, Stringer LC, Robinson GM & Evely AC 2010 Integrating local and scientific knowledge for environmental management. Journal of environmental management 97(8): 1766- 1777 Snilstveit B, Vojtkova M, Bhavsar A, Gaarde M 2013 Evidence gap maps: A tool for promoting evidence- Informed policy and prioritizing future research. World Bank Policy Research Working Paper 6725. Washington: World Bank Stewart R 2016 Reflecting on range of ways in which we have adapted full systematic review methods to meet demands of different stakeholders. Presentation at the 1st International CEE conference, 25-27 August in Stockholm.
Editor's Notes
- Not an expert in SR methodology; I am an environmental anthropologists interested in the nature of evidence for decision by whom for what. Have been trained on-the-job by Ruth Stewart. My presentation fits with the first leg of the conference focus, namely BETTER EVIDENCE. Background to this proposed map: Being produced in context of a project about diverse knowledge(s) about changing climate in South Africa, that considers both scientific knowledge and local knowledges in a transdisciplinary project aimed local level adaptation & mitigation policies and interventions in a specific locale. Also, as an environmental anthropologist, I am concerned about the lack of engagement by SA Social Anthropology in the research on climate change. In look around, couldn’t find much by other social science fields in SA either. So, want to know: What does published literature tell us about the nature of social sciences research on changing climate in South Africa? Further, in 2012 CEE Joburg undertook a priority setting exercise as part of the inception activities of this CEE centre. We then matched completed SRs with the sectors of expressed concern. One of the sectors where we then couldn’t find a completed SR was climate change.
- Where does systematic mapping fit in the family of research synthesis?
- ....collection and analysis of a body of scientific knowledge. There are many primary studies as there are many crops in the world, primary studies are important, they are the foundation of our knowledge but we can get lost in them easily if we don’t harvest and structure the combined knowledge coming out of these primary studies. Harvesting and structuring, this is what research synthesis is about. Also important that these primary studies be made more accessible and digestible. One use could be for evidence-informed decision-making. Combined with contextual factors, research synthesis nb for policy and practice decisions (Langer et al 2016).
- Research synthesis from primary research can take many forms: REAs: get an answer quickly but might miss detail Lit reviews: often danger that experts cherry pick the studies / knowledge they like most/confirms with their view point SRs a more comprehensive and careful form to review all evidence in a structured and transparent way and not steamroll past details and context Evidence maps
- SR has been adapted in various ways. One of these has been to shift the objective of review: rather than attempt to compare results, review is used to compare methods or theory (Delaney et al 2016:2). EM is “a robust, repeatable and transparent scientific method used to identify, categorise and map available literature relevant to a topic” (CEE Systematic Mapping Methods Group (SMMG). It is about identifying and describing the characteristics of the evidence base. It provides a database of ‘map’ showing where evidence on specific topic can be found, with descriptive information; it is about cataloguing. Typical outputs are a report, a database, and GIS sometimes. It does not set out to provide an answer to a question of impact (thus not providing synthesis of results), and may not always critically appraise the relevant research. Rather answers questions related to the state of evidence itself (Haddaway et al 2016:2). Asks questions about “what research has been undertaken, which study settings have been examined and what methods have been used across an evidence base” (Haddaway et al 2016:1). Thus, gives broad overview of existing evidence.
- “Systematic mapping was first undertaken in the social sciences (Bates et al. 2007; Clapton et al. 2009)” (Haddaway et al 2016:1), since the 2000s. 1st use of SM “in environmental management research was published in 2012 (Randall and James 2012)” (Haddaway et al 2016:2). Various kinds of evidence-synthesis maps have developed: Their methodologies differ across range of dimensions – from aim, scope, type of evidence included, data extraction and analysis (Snilstvei et al 2013:4). And also whether pre-determined framework (deductive) or inductive. For example, in Mays et al’s (2005) definition of scoping studies they are “to map rapidly the key concepts underpinning a research area and the main sources and types of evidence available, and can be undertaken as standalone projects in their own right, especially where an area is complex or has not been reviewed comprehensively before” (2005) (quoted in Daudt et al 2013:2). In 3ie’s maps the focus is on studies on intervention effectiveness (Snilstveit et al 2013).
- Maps usually try to answer broader question of what is out there, rather than providing focused answers on a narrow question. Maps allow for descriptive overview for us to understand the research landscape; what research has been done, on what issues, by whom, with what methods, in which regions. We can see macro patterns, structures, inequalities, saturated silos of research, empty deserts of areas where no research has been conducted. Haddaway et al (2016) refers to knowledge gaps (require more research) and knowledge gluts (full SR is possible). Can also see patterns that promote ‘good practice’ (Haddaway et al 2016:1). But a map also allows you to zoom in areas of interest, so you could have a very detailed map of only the research showing you exactly what knowledge has been produced on a certain question, if it was extracted and coded for. Often, this then is an answer in its own right because you can see that either your question has not been answered or that there is a lot knowledge already, so you don't need to conduct new research here and can just draw from the already produced knowledge.
- Thus EM can be useful for research, policy and practice. EM can be used: “- As a road map of evidence in an ongoing policy debate (descriptive) - As an indication of evidence gap, quality of research, patterns of research (normative/prescriptive) - As a pre-cursor for other research synthesis (SRs, IEs) - As a tool to target commission of new primary research - As a knowledge management tool for internal departmental (policy) resource - As a tool to start a conversation with the research sector” (Langer 2016 doc) Importantly, note that the map does not constitute evidence of effectiveness, since no synthesis has taken place.
- What is the evidence mapping process? Also rigorous & transparent as SRs. EM is broader in scope.
- Set the framework & inclusion criteria: maps can structured around effectiveness, but also “theories of change, barriers and facilitators to change, concepts, and other contextual factors.” (Langer 2016 doc). Framework “defines how you will categorise and structure the research evidence” (Langer 2016 doc). Find the evidence: [“Search techniques adopted in rapid reviews/ evidence assessments might be particularly relevant. This includes for instance only searching English language literature, only covering a limited time period, applying specific search strings if used16, or applying a restriction on publication status (Khangura et al., 2012; The Civil Service, 2008). Moreover, focusing predominantly on key repositories of impact evaluations (such as 3ie’s Database of Impact Evaluations17, J-PAL, the Cochrane Register of Trials), snowballing and citation tracking (Waddington et al., 2012), use of list serves, key databases and websites relevant to the thematic area under investigation are other techniques which can increase the efficiency of the search” (Snitsvleit et al 2013:12).] Categorise it: code according to framework (deductive), or inductive. [(Appraise it & extract findings): “For the critical appraisal, we identified four key domains to interrogate in order to judge the trustworthiness of a piece of evidence. These referred to evidence design, evidence conduct, evidence analysis, and evidence conclusion” (Langer 2016 doc)] Present / visualise it - The final report and the database should be available as an easily usable resource for policy-makers and practitioners to work with (Clapton et al. 2009:12).
- Example of a heat map
- “In fields like climate change research that lack the degree of standardisation of clinical medical research, attempts to use systematic review to synthesise the results of empirical studies can be frustrated by a lack of equivalence between those studies.” (Delaney et al 2016:20). “there are two ways to think about commensuration. First, and most simply, it can be understood as the standardisation of measurement. Second, commensuration refers to the specification of the frame within which measurement of a given object is negotiated.” (Delaney et al 2016:3). Climate change, changing climate, climate variability, climate / weather.
- About their focuses: many social science studies on CC is about people’s perceptions, views, experiences and knowledges. Also power, processes, policies, relations, institutions. Relate to framework, which say about next.
- Most of studies likely local scale contextualised qualitative studies, but how mutually compatible are they, as they are so localised and specific. This makes aggregation and comparison tricky. Complexity and nuances are reduced, but information becomes portable (Delaney et al 2016:3). Being an environmental anthropologists my commitment is to keep with ‘spirit’ of included studies. One of those ‘spirits’ are narrative. Thus descriptive map to show richness of context, and visualisation as portal to richness. Also important to consider level of aggregation of mapping framework – “Too many sub-categories will make your map look empty, while too few categories will lead to large clusters of evidence—both of which are difficult to interpret thereafter” (Langer 2016 doc). [“systematic review of qualitative research is best served by reliance upon qualitative methods themselves” (Jones 2004:95) – such as the use of a narrative inductive interpretive method of analysis – use reflective team to analyse data]
- Get bodies of knowledge to talk to one another [DDG of DPME – Ahmed Vawda] Also, framework not to be forced, but inductively created.
- Not intervention-oriented per se. Map many times seen as not end in itself – or can it be? “Depending on the research objective, evidence maps can either be conducted in the process of developing a full systematic review, or as much operate as a research product in their own right (Gough and Thomas 2012).” (Langer 2016 doc). To generate frequencies, as well as patterns/structures of social sciences research on changing climate. Users as researchers, local level policy-makers, business (farmers, mines), civil society – to be used in manner that want.
- Visualisation to be meaningful and allow for a filtering of evidence that allow for a personalised generation evidence maps. Colleagues warned me about importance of to extract data in a format that is compatible with the software used for the EM” (Langer 2016 doc) They worked with the Department of Human Settlements & DPME - led to development of code to visualise. Use SQL to communicate with database [?].
- Map to contribute to our understanding of the state of the literature Get map on nature of studies, not findings of studies Methods to reflect about methodology – keep research diary, keep minutes of meetings, discussion with contributors re challenges For now not thinking about updating map, just getting it done?