How to calculate Cohen's kappa for full-text articles in a systematic review

1. How to calculate Cohen's kappa for full text articles in a systematic review
Nay Aung, BDS, PhD
Cohen's kappa is a statistical measure of inter-rater agreement for categorical items, and it is
commonly used in systematic reviews to assess the reliability of coding or categorization between
different reviewers. If you want to calculate Cohen's kappa for full-text articles in a systematic review,
follow these steps:
1. Define Categories or Codes:
 Determine the categories or codes that reviewers will use to classify information in the full-
text articles. These categories should be mutually exclusive and exhaustive.
2. Select a Sample:
 Randomly select a sample of full-text articles from your systematic review. It's not always
feasible to review all articles, so a representative sample is often used.
3. Choose Reviewers:
 Select the reviewers who will independently code or classify the articles based on the
predefined categories. Ideally, you should have at least two reviewers for reliability
assessment.
4. Coding Process:
 Have the reviewers independently code the selected articles according to the predefined
categories. Make sure they are blind to each other's coding.

2. 5. Create a Contingency Table:
 After coding is complete, create a contingency table that shows the agreement and
disagreement between the two reviewers. The table should have rows and columns for each
category.
 Reviewer 1: Category A Reviewer 1: Category B ... Total
Reviewer 2: A
Reviewer 2: B
...
Total

3. 6. Interpret the Result:
 Interpret the kappa value based on guidelines. Generally, a kappa value above 0.6 is
considered substantial agreement.
Repeat these steps for additional samples or articles if needed. It's crucial to assess inter-rater
agreement early in the review process and address discrepancies or provide clarification to reviewers
to improve reliability.

4. General guide on how to approach coding for a systematic review and meta-
analysis:
Step 1: Define Data Extraction Variables
1. Define Variables:
 Identify the key variables you need to extract from each full-text article. These
could include study characteristics, participant demographics, intervention
details, outcomes, and statistical data.
Step 2: Develop a Data Extraction Form
2. Create a Data Extraction Form:
 Design a structured data extraction form using a spreadsheet tool (e.g., Excel,
Google Sheets) or a specialized tool like REDCap or DistillerSR.
 Include columns for each variable, such as study title, author, publication year,
study design, sample size, intervention details, outcome measures, effect sizes,
etc.
Step 3: Train Reviewers
3. Training Reviewers:
 If multiple reviewers are involved, ensure they are trained on the data extraction
process and are familiar with the form.
 Establish clear guidelines on how to extract information consistently.
Step 4: Extract Data
4. Data Extraction:
 Reviewers independently extract data from each full-text article using the data
extraction form.
 Ensure that the extraction is thorough and accurate, capturing all relevant details.
Step 5: Resolve Discrepancies
5. Resolve Discrepancies:
 If there are discrepancies between reviewers, establish a process for resolving
them. This may involve discussion, consensus building, or consulting a third
reviewer.

5. Step 6: Code Data for Meta-Analysis
6. Code for Meta-Analysis:
 If your meta-analysis involves effect sizes, ensure that you code the effect sizes
accurately. Common effect sizes include mean differences, risk ratios, odds ratios,
correlation coefficients, etc.
 Follow established guidelines for coding and calculating effect sizes, considering
the nature of your data (continuous, binary, correlation, etc.).
Step 7: Document Methods
7. Document Methods:
 Clearly document the methods used for data extraction and coding in your
systematic review protocol.
 Specify how missing or unclear data were handled.
Step 8: Data Verification
8. Data Verification:
 Conduct a verification process to ensure the accuracy of the extracted data. This
may involve having a second reviewer check a subset of the data for consistency.
Step 9: Data Synthesis
9. Data Synthesis:
 Use the extracted and coded data to perform a narrative synthesis or quantitative
synthesis (meta-analysis) as per the objectives of your systematic review.
Step 10: Quality Assessment
10. Quality Assessment:
 If applicable, assess the quality of included studies using established tools. This
information may be used to inform sensitivity analyses.
Step 11: Reporting
11. Reporting:
 Clearly report the details of your data extraction and coding process in the
methods section of your systematic review and meta-analysis.

6. Always refer to the specific guidelines provided by the organization or journal
publishing the systematic review for any additional requirements or recommendations.
A reference can be excluded as soon as it fails to meet one of
the criterion, for example:
1. Study was not a randomized trial ⛔
2. Study had the wrong comparator ⛔
3. Study had the wrong patient population ⛔
References
https://www.covidence.org/blog/how-to-quickly-complete-full-text-screening-in-a-systematic-
review/?fbclid=IwAR0fCx24sNcYOkq4I17PD_Mw7ZcoR2fEllQNz_nUcCtP3byajwth-ckbtFE
https://stats.stackexchange.com/questions/574094/calculating-cohens-kappa-in-spss-for-a-systematic-
review