Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Data extraction/coding and database structure in meta-analysis

Systematic review and meta-analysis

Source & acknowledgement

https://www.terripigott.com/meta-analysis
https://mason.gmu.edu/~dwilsonb/ma.html
https://www.campbellcollaboration.org/research-resources/training-courses.html
https://handbook-5-1.cochrane.org/
https://www.youtube.com/user/Biostat100
https://www.youtube.com/user/collaborationtube
https://www.meta-analysis.com/pages/videotutorials.php?cart=BWJJ3522355

  • Login to see the comments

  • Be the first to like this

Data extraction/coding and database structure in meta-analysis

  1. 1. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Data extraction/coding & database structure Dr. S. A. Rizwan M.D., Public Health Specialist & Lecturer, Saudi Board of Preventive Medicine – Riyadh, Ministry of Health, KSA 24.11.2019
  2. 2. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Outline • Data extraction/coding – Levels of Study Coding • Study eligibility screening • Study content coding • Effect size coding • Data structure • Common mistakes 24.11.2019
  3. 3. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course DATA EXTRACTION/CODING 24.11.2019
  4. 4. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Why do study coding? • Provide an accounting of the research included in your review. – Also helps identify what’s missing. • Identify the characteristics of the interventions, subjects, and methods in the research. • Assuming different primary studies produce different results, study coding allows you to identify variables that might explain those differences. 24.11.2019
  5. 5. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Development of coding protocol • Coding protocol: essential feature of meta-analysis • Goal: transparent and replicable – description of studies – extraction of findings 24.11.2019
  6. 6. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Topics for coding • Eligibility criteria and screening form • Development of coding protocol • Hierarchical nature of data • Assessing reliability of coding • Training of coders 24.11.2019
  7. 7. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Describing the literature Table 1: Characteristics of Included Studies Study Intervention Personnel Design Outcomes % Male Wilson, 2013 CBT Teachers RCT Aggression 56% Lipsey, 2009 Behavioral Psychologists QED Suspensions Fighting 75% Tanner-Smith, 1999 CBT Psychologists RCT Conduct disorder 89% Allen, 1978 Anger Mgmt. Teachers Cluster RCT Aggression 45% Jones, 2000 Site A Anger Mgmt. Teachers QED Conduct disorder 46% Site B Anger Mgmt. Teachers QED Conduct disorder 82% Site C Anger Mgmt. Teachers QED Conduct disorder 100% 24.11.2019
  8. 8. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Describing the literature Table 1: Characteristics of Included Studies Study Characteristic k % Publication Year 1980s 5 16.7 1990s 10 33.3 2000s 15 50.0 Method of Assignment Random (indiv.) 2 6.7 Random (cluster) 8 26.7 Non-random 20 66.7 mean sd Duration of treatment (weeks) 23.5 10.2 mean range Age of intervention participants 12.3 8-13 24.11.2019
  9. 9. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Study characteristics as moderators Meta-regression Model b se 95% CI Methodological Characteristics Random assignment (1=yes) -.03 .15 -.33, .27 Matched groups design (1=yes) -.01 .15 -.31, .29 Program Characteristics Implementation quality (1-3) .30* .06 .18, .41 Classroom program (1=yes) .45* .12 .20, .69 24.11.2019
  10. 10. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Eligibility criteria • The PICOS framework: • Population/Participants (problems/conditions) • Interventions • Comparison group (e.g., absolute vs. relative effects, counterfactual conditions) • Outcomes (primary and secondary outcomes, acceptable outcome measures) • Study Design • Geographic area, time, language, other criteria 24.11.2019
  11. 11. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Abstract level relevance screening • Does the document look like it might be relevant? – Based on reading titles and abstracts • If yes, retrieve full text of article – Exclude obviously irrelevant studies, but don’t assume the title and abstract are going to give reliable information on study design, outcomes, subject characteristics, or even interventions • When in doubt, double code – At least two trained raters working independently 24.11.2019
  12. 12. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Abstract screening example 24.11.2019
  13. 13. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Full-text eligibility screening • Develop an eligibility screening form with criteria. • Before screening, link together all reports from the same study. • Complete form for all studies retrieved as potentially eligible. • Modify criteria after examining sample of studies (controversial). • Double-code eligibility. • Maintain database on results for each study screened. 24.11.2019
  14. 14. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Eligibility database example 24.11.2019
  15. 15. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Eligibility screening results • Once screening of all relevant full-text reports is complete, you will have: – An accounting of the ineligible studies and the reasons for their ineligibility. • Campbell and Cochrane reviews often include a table of ineligible studies as an appendix. – A set of studies eligible for coding. 24.11.2019
  16. 16. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Study content and effect size coding • Develop a coding manual that includes: – Setting, study context, authors, publication date & type – Methods and method quality – Program/intervention – Participants/clients/sample – Outcomes – Findings, effect sizes • Coding manual should have a “paper” version and an electronic version. 24.11.2019
  17. 17. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Sources for coding items • Other systematic reviews – Campbell reviews often include coding manuals as appendices in protocols and reviews. – Useful for generic coding items (e.g., research design, risk of bias) • The literature – Start with the assumption that you will find variability in treatment effects across the studies in your review. – What does the literature tell you about the plausible sources of that variability? • The literature reviewed at the beginning of a primary study often provides clues. • Theory of change for the intervention can also be useful. 24.11.2019
  18. 18. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Study context coding • Setting, study context, authors, publication date and type, etc. – Multiple publications; “study” vs. “report” – Geographical/national setting; language – Publication type and publication bias – Publication date vs. study date – Research, demonstration, practice studies 24.11.2019
  19. 19. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Publication type • Type of publication. If you are using more than one type of publication to code your study, choose the publication that supplies the effect size data. – Book – Journal article – Book chapter (in an edited book) – Thesis or dissertation – Technical report – Conference paper – Other – Cannot tell 24.11.2019
  20. 20. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Study method coding • Risk of Bias • Variety of options for coding study methods – Cochrane risk of bias framework – GRADE system – Method quality checklists – Direct coding of methodological characteristics 24.11.2019
  21. 21. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Cochrane risk of bias framework • Focus on identifying potential sources of bias: – Selection bias - Systematic differences between groups at baseline (allocation concealment and sequence generation) – Performance bias - Something other than the intervention affects groups differently, e.g., contamination (blinding of participants) – Attrition bias - Participant loss affects initial group comparability – Detection bias - Method of outcome assessment affects group comparisons (blinding of data collectors) – Reporting bias - Selective reporting of outcomes • Coded as low risk, high risk, unclear risk 24.11.2019
  22. 22. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Modifications of Cochrane framework • Quasi-experimental studies may need additional coding with regard to selection bias, e.g., potential third variables that might account for results when groups were not randomized. • Some risk of bias items don’t make sense in social science research – Blinding of study participants to what condition they’re in • Code performance bias directly, e.g., was contamination evident? Were there refusals? – Blinding of data collectors 24.11.2019
  23. 23. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Method quality checklists • Method quality ratings (or not) • More than 200 scales and checklists available, few are appropriate for systematic reviews • Overall study quality scores have questionable reliability and validity (Jüni et al., 2001) – Conflate different methodological issues with other study features, which may have different impacts on reliability/validity – Preferable to examine potential influence of key components of methodological quality individually • Weighting results by study quality scores is not advised! 24.11.2019
  24. 24. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Direct method coding • Methods: Basic research design – Nature of assignment to conditions – Attrition, crossovers, dropouts, other changes to assignment – Nature of control condition – Multiple intervention and/or control groups • Design quality dimensions – Initial & final comparability of groups (pretest effect sizes) – Treatment-control contrast • Treatment contamination • Blinding 24.11.2019
  25. 25. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Direct method coding • Methods: Other aspects – Issues depend on specific research area – Procedural, e.g., • monitoring of implementation, fidelity • credentials, training of data collectors – Statistical, e.g., • statistical controls for group differences • handling of missing data 24.11.2019
  26. 26. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Sample coding item • Assignment of participants • Unit of group assignment. The unit on which assignment to groups was based. – individual (i.e., some children assigned to treatment group, some to comparison group) – group (i.e., whole classrooms, schools, therapy groups, sites, residential facilities assigned to treatment and comparison groups) – program area, regions, school districts, counties, etc. (i.e., region assigned as an intact unit) – cannot tell 24.11.2019
  27. 27. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Sample coding item • Method of group assignment. • Random or near--random: – randomly after matching, yoking, stratification, blocking, etc. – randomly without matching, etc. – regression discontinuity design: – wait list control or other quasi--random procedure • Non-random, but matched: – matched ONLY on pretest measures of some or all variables used later as outcome measures. – matched on demographic characteristics. 24.11.2019
  28. 28. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course GRADE system for method quality • Quality of evidence across studies included in a systematic review • Outcome-specific • Considers: sparse data, consistency/inconsistency of results across trials, study designs, reporting bias, possible influence of confounding variables 24.11.2019
  29. 29. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course What to do with quality ratings? • Select a method of assessing and coding methodological quality or design your own items. • Code all the studies with whatever method you select. • Examine the influence of the methodological characteristics on effect sizes at the analysis stage. – Use the method variables as moderators. • Summarize the quality of evidence (e.g., GRADE). • The important thing is to USE the information about quality to inform your conclusions! 24.11.2019
  30. 30. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Intervention coding • General program type (mutually exclusive or overlapping?) • Specific program elements (present/absent) • Any treatment received by the comparison group • Treatment implementation issues • Integrity, implementation fidelity – Amount, length, frequency, “dose” – Goal is to differentiate across studies • What does the literature tell you about what’s likely to be important? 24.11.2019
  31. 31. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Intervention coding example 24.11.2019
  32. 32. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Participant coding • Participants/clients/sample – Data are at aggregate level (you’re coding the characteristics of the entire sample, not a single individual) – Mean age, age range – Gender mix – Racial/ethnic mix – Risk, severity – Restrictiveness; special groups (e.g., clinical) 24.11.2019
  33. 33. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Study outcome coding • Outcome measures – Construct measured – Measure or operationalization used – Source of information, informant – Composite or single indicator (item) – Scale: dichotomous, count, discrete ordinal, continuous – Reliability and validity – Time of measurement (e.g., relative to treatment) 24.11.2019
  34. 34. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Procedural details • Double coding: • Cohen’s kappa • Agreement on key decisions – Study inclusion/exclusion, key characteristics, risk of bias, coding of results • Pilot-test and refine coding manual 24.11.2019
  35. 35. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Creating your coding manual • Write up a detailed document that includes all the coding items. – Organize the manual hierarchically • One section for study-level, one for group-level, one for dependent variables, one for effect sizes – Include coding manual in Campbell Collaboration protocol • Translate the coding manual into “forms” to use for actual coding – Paper forms, spreadsheets – Databases 24.11.2019
  36. 36. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Sample coding database 24.11.2019
  37. 37. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Sample coding database 24.11.2019
  38. 38. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Summary of data items 24.11.2019
  39. 39. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course DATA STRUCTURE 24.11.2019
  40. 40. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Structuring your data • Hierarchical structure of meta-analytic data – Multiple outcomes within studies – Multiple measurement points within outcomes – Multiple subsamples within studies – Multiple effect sizes within studies • You need to design coding scheme and eventual analytic datasets in a way that handles the hierarchical nature of your data. 24.11.2019
  41. 41. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Development of coding protocol • Iterative nature of development • Structuring data – Data hierarchical (findings within studies) – Coding protocol needs to allow for this complexity – Analysis of effect sizes needs to respect this structure – Flat-file (example) – Relational hierarchical file (example) 24.11.2019
  42. 42. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Example of data sheets 24.11.2019
  43. 43. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Example of data sheets 24.11.2019
  44. 44. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Example of a flat file ID Paradigm ES1 DV1 ES2 DV2 ES3 DV3 ES4 DV4 22 2 0.77 3 23 2 0.77 3 31 1 -0.1 5 -0.05 5 -0.2 11 36 2 0.94 3 40 1 0.96 11 82 1 0.29 11 185 1 0.65 5 0.58 5 0.48 5 0.068 5 186 1 0.83 5 204 2 0.88 3 229 2 0.97 3 246 2 0.91 3 274 2 0.86 3 -0.31 3 0.79 3 1.17 3 295 2 7.03 3 6.46 3 . 3 0.57 . 626 1 0.87 3 -0.04 3 0.1 3 0.9 3 1366 2 0.5 3 Note that there is only one record (row) per study Multiple ESs handled by having multiple variables, one for each potential ES. 24.11.2019
  45. 45. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Example of a hierarchical structure ID PubYear MeanAge TxStyle 100 92 15.5 2 7049 82 14.5 1 Outcome ID ESNum Type TxN CgN ES 100 1 1 24 24 -0.39 100 2 1 24 24 0 100 3 1 24 24 0.09 100 4 1 24 24 -1.05 100 5 1 24 24 -0.44 7049 1 2 30 30 0.34 7049 2 4 30 30 0.78 7049 3 1 30 30 0 Note that a single record in the file above is “related” to five records in the file to the right Study Level Data File Effect Size Level Data File 24.11.2019
  46. 46. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Example of complex multiple file data structure ID PubYear MeanAge TxStyle 100 92 15.5 2 7049 82 14.5 1 Study Level Data File Outcome Level Data File ID OutNum Constrct Scale 100 1 2 1 100 2 6 1 100 3 4 2 7049 1 2 4 7049 2 6 3 ID OutNum ESNum Months TxN CgN ES 100 1 1 0 24 24 -0.39 100 1 2 6 22 22 0 100 2 3 0 24 24 0.09 100 2 4 6 22 22 -1.05 100 3 5 0 24 24 -0.44 100 3 6 6 22 21 0.34 7049 1 2 0 30 30 0.78 7049 1 6 12 29 28 0.78 7049 2 2 0 30 30 0 Effect Size Level Data File Note that study 100 has 2 records in the outcomes data file and 6 outcomes in the effect size data file, 2 for each outcome measured at different points in time (Months) 24.11.2019
  47. 47. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Adv. & disadv. of multiple flat files data structure • Advantages – Can “grow” to any number of ESs – Reduces coding task (faster coding) – Simplifies data cleanup – Smaller data files to manipulate • Disadvantages – Complex to implement – Data must be manipulated prior to analysis – Must be able to select a single ES per study for any analysis • When to use – Large number of ESs per study are possible 24.11.2019
  48. 48. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Concept of “Working” analysis files Study Data File Outcome Data File ES Data File Composite Data File create composite data file select subset of ESs of interest to current analysis, e.g., a specific outcome at posttest verify that there is only a single ES per study yes Working Analysis File Permanent Data Files Average ESs, further select based explicit criteria, or select randomly no 24.11.2019
  49. 49. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Example: SPSS ES data file 24.11.2019
  50. 50. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Example: data file 24.11.2019
  51. 51. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Example: creating subset for analysis 24.11.2019
  52. 52. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Example: Final working file for a single analysis 24.11.2019
  53. 53. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course What about sub-samples? • What if you are interested in coding ES separately for different sub- samples, such as, boys and girls, or high-risk and low-risk youth, etc? – Just say “no”! • Often not enough of such data for meaningful analysis • Complicates coding and data structure – Well, if you must, plan your data structure carefully • Include a full sample effect size for each dependent measure of interest • Place sub-sample in a separate data file or use some other method to reliable determine ES that are statistically dependent 24.11.2019
  54. 54. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Coding mechanics • Paper Coding – include data file variable names on coding form – all data along left or right margin eases data entry • Coding into a spreadsheet • Coding directly into a computer database 24.11.2019
  55. 55. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Coding directly in computer database • Advantages – Avoids additional step of transferring data from paper to computer – Easy access to data for data cleanup – Data base can perform calculations during coding process (e.g., calculation of effect sizes) – Faster coding • Disadvantages – Can be time consuming to set up • the bigger the meta-analysis the bigger the payoff – Requires a higher level of computer skill 24.11.2019
  56. 56. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Example of database with forms 24.11.2019
  57. 57. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Assessing reliability of coding • Inter-rater reliability and double coding • Intra-rater reliability 24.11.2019
  58. 58. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Training coders • Regular meetings (develops normative understandings) • Annotate coding manual • “Specialist” coders 24.11.2019
  59. 59. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Software • Archie • Covidence • EPPI reviewer • MS Excel • MS Access 24.11.2019
  60. 60. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Common mistakes in coding • Over-coding—Trying to extract more detail than routinely reported • Too many subjective coding items • Coding two reports from the same study as two different studies • Coder drift • Failure to ask questions 24.11.2019
  61. 61. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Common mistakes in data structure • Not understanding or planning the analysis prior to coding • Underestimating time, effort, and technical/statistical demands • Using a spreadsheet for managing a large review • Variable names not on coding forms • Not breaking apart difficult judgments 24.11.2019
  62. 62. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Take home messages • Remember the purposes of coding: – To describe the methods, interventions, participants, outcomes, and findings common in the literature you’re reviewing. – To identify gaps in the literature. • What’s NOT covered • What’s NOT reported – To explain observed variation in treatment effects. 24.11.2019
  63. 63. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 03/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course THANK YOU Kindly email your queries to sarizwan1986@outlook.com 24.11.2019

×