Building the evidential quality        of evaluations      ALNAP March 5, 2013         Tony Redmond
Humanitarian vs Development
Hierarchy of Evidence
Most evidence in thehumanitarian sphere is in theweakest categories or outside   the pyramid altogether
Expert Opinion• Consensus Statements Regarding the  Multidisciplinary Care of Limb Amputation Patients  in Disasters or Hu...
Gathering the Evidence
bias• Incentive• Observer• Systematic (non random) error  – Confidence intervals
RCT               Sibbald B, BMJ 1998;316:201• the most rigorous way of determining  whether a cause-effect relation exist...
Dangerous Data• Wrong inference• Randomisation• Open and transparent
HERR• Medical teams are 200 times as cost  effective as UKISAR• That is they cost less, stay longer, treat  more people an...
sampling
Convenience sampling• Using those who are willing to  volunteer, or cases which are presented to  you as a sample.
Judgement sampling• Based on deliberate choice and excludes  any random process.• Normal application is for small samples ...
Simple random sampling• Ensures every member of the population  has an equal chance of selection.• Produces defensible est...
Cluster samplingUnits in the population can often be found in geographical groups or clusters. Arandom sample of clusters ...
Why is evidence ignored?• Evidence influencing policy• Policy influencing evidence• Weve always done it this way…• ……and i...
Epidemiology/statistics•   Observation•   Pattern Recognition•   Cause and effect or coincidence•   Change of practice
CONFIDENCEINTERVALS (CI)                       MEAN VALUE                               You have a 95%                    ...
assessments•   Early warning•   Risk analysis•   Trending•   Needs assessment
Evidence based medicine• Clinical trials in the field – in an  emergency?• Multiple disciplines
Evaluation• What was done?• What was the reason for doing it?• What did they find?• SO WHAT?
Benefit cost
What you have to evaluate          adapted from DFID WASH Evaluation 2011• Internal validity – was what they did good  eno...
Is good enough, good enough?
Humanitarian• Historically draws on concept of  compassion, individuality and reform  (transformation)• It is evidence and...
Building the evidential quality of evaluations (Tony Redmond, Uni Manchester)
Building the evidential quality of evaluations (Tony Redmond, Uni Manchester)
Building the evidential quality of evaluations (Tony Redmond, Uni Manchester)
Upcoming SlideShare
Loading in …5
×

Building the evidential quality of evaluations (Tony Redmond, Uni Manchester)

404 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
404
On SlideShare
0
From Embeds
0
Number of Embeds
24
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Building the evidential quality of evaluations (Tony Redmond, Uni Manchester)

  1. 1. Building the evidential quality of evaluations ALNAP March 5, 2013 Tony Redmond
  2. 2. Humanitarian vs Development
  3. 3. Hierarchy of Evidence
  4. 4. Most evidence in thehumanitarian sphere is in theweakest categories or outside the pyramid altogether
  5. 5. Expert Opinion• Consensus Statements Regarding the Multidisciplinary Care of Limb Amputation Patients in Disasters or Humanitarian Emergencies: Report of the 2011 Humanitarian Action Summit Surgical Working Group on Amputations Following Disasters or Conflict• Delphi Studiens
  6. 6. Gathering the Evidence
  7. 7. bias• Incentive• Observer• Systematic (non random) error – Confidence intervals
  8. 8. RCT Sibbald B, BMJ 1998;316:201• the most rigorous way of determining whether a cause-effect relation exists between intervention and outcome and• for assessing the cost effectiveness of an intervention.
  9. 9. Dangerous Data• Wrong inference• Randomisation• Open and transparent
  10. 10. HERR• Medical teams are 200 times as cost effective as UKISAR• That is they cost less, stay longer, treat more people and save more lives
  11. 11. sampling
  12. 12. Convenience sampling• Using those who are willing to volunteer, or cases which are presented to you as a sample.
  13. 13. Judgement sampling• Based on deliberate choice and excludes any random process.• Normal application is for small samples from a population that is well understood and there is a clear method for picking the sample.• Is used to provide illustrative examples or case studies.
  14. 14. Simple random sampling• Ensures every member of the population has an equal chance of selection.• Produces defensible estimates of the population and sampling error.• Simple sample design and interpretation.
  15. 15. Cluster samplingUnits in the population can often be found in geographical groups or clusters. Arandom sample of clusters is taken, then all units within those clusters areexamined.• Quicker, easier and cheaper than other forms of random sampling.• Does not require complete population information.• Useful for face-to-face interviews.• Works best when each cluster can be regarded as a microcosm of the population.• Larger sampling error than other forms of random sampling.• If clusters are not small it can become expensive.• A larger sample size may beneeded to compensate forgreater sampling error.
  16. 16. Why is evidence ignored?• Evidence influencing policy• Policy influencing evidence• Weve always done it this way…• ……and it’s done no harm (regardless of the evidence)• Maybe the evidence was always there….
  17. 17. Epidemiology/statistics• Observation• Pattern Recognition• Cause and effect or coincidence• Change of practice
  18. 18. CONFIDENCEINTERVALS (CI) MEAN VALUE You have a 95% confidence that the TRUE VALUE should fall between the CI if the study is repeated multiple times in a similar population 95% CONFIDENCE INTERVAL
  19. 19. assessments• Early warning• Risk analysis• Trending• Needs assessment
  20. 20. Evidence based medicine• Clinical trials in the field – in an emergency?• Multiple disciplines
  21. 21. Evaluation• What was done?• What was the reason for doing it?• What did they find?• SO WHAT?
  22. 22. Benefit cost
  23. 23. What you have to evaluate adapted from DFID WASH Evaluation 2011• Internal validity – was what they did good enough?• External validity – relevance to you?• Can you implement it ?• Sustainability?• Wider impact?• COST
  24. 24. Is good enough, good enough?
  25. 25. Humanitarian• Historically draws on concept of compassion, individuality and reform (transformation)• It is evidence and evaluation that will help us reform and lead us to prevention

×