Validation studies are essential to accurately assess the sensitivity, specificity, and predictive values of point prevalence surveys (PPS) of healthcare-associated infections (HAI). Previous validation studies of PPS have shown varied results, underscoring the need for formal evaluations. Without validation, true HAI prevalence is unknown and differences between locations cannot be properly investigated. International organizations can help support national validation efforts to improve HAI surveillance.

1. The case for PPS validation
Professor Jacqui Reilly
Health Protection Scotland and Glasgow Caledonian University
UK

2. Introduction to validation
• Validation studies are essential to assess the sensitivity
(Se), specificity (Sp), positive predictive value (PPV) and
negative predictive value (NPV) of PPS.
• Reproducibility (or reliability) is also an important concept
for ensuring data is robust.
• Validation studies are rarely published, there are a variety
of approaches described:
Less than half of the prevalence surveys published to date included an evaluation
of the data.
Of those that included either a validation or an IRR study, the results were varied
substantially
Underscoring the need for a formal evaluation to add confidence to the
interpretation of the data.
Ref: Llata E, Gaynes RP, Fridkin S. Measuring the scope and magnitude of hospital-associated infection
in the United States: the value of prevalence surveys. Clin Infect Dis 2009 05/15;48(10):1434-1440.

3. Why does it matter?
 3Cs: Consistency, comparisons and
confidence
 Low sensitivity (false negatives, or underreporting) of HAIs is a
frequently encountered problem in HAI surveillance systems.
 Low specificity (false positives, or over reporting) is usually less of a
problem
 Both may be related to one or more of following factors:
Difficulty in confirming the case definition of an infection if signs
and symptoms were not well documented in the patient’s records
If diagnostic tests included in the case definition of a particular HAI
type were not done
Non compliance with the definition of the key term ‘healthcare-
associated’: even if the case definition of an infection is matched
due to cultural or financial/ political incentives and disincentives
at a hospital or country level

4. Prevalence validation

5. The case for validation in multi-country
studies
 In order to investigate
variation between
countries the first
question to ask is: is it
the data?
 Validity
 Reliability
Ref: European Centre for Disease Prevention and Control. Point prevalence survey of healthcare
associated infections and antimicrobial use in European acute care hospitals. Stockholm: ECDC; 2013.

6. Validation results for PPS 2012
Ref: European Centre for Disease Prevention and Control. Point prevalence survey of healthcare
associated infections and antimicrobial use in European acute care hospitals. Stockholm: ECDC; 2013.

7. Interpretation
 The large differences observed between HAI prevalence
across Member States are in reality smaller
 The overall weighted HAI prevalence of 5.7% is likely to be a
slight underestimate.
.

8. International PPS validation
 Indonesia:
– Significant differences were noted between the teams of data collectors
in terms of:
completeness of data,
identifying patients who had undergone surgery in the previous month,
and most importantly in the number of HAI detected.
SSI, UTI and septicaemia (p=0.01).
(κ) did not exceed 0.60 and 0.59 respectively for any infection type.
– Ascertainment was affected by underreporting in medical records, the
retrospective nature of data collection for validation purposes, and
suboptimal adherence to the PPS protocol and case definitions.
Ref: Duerink DO, Roeshadi D, Wahjono H, Lestari ES, Hadi U, Wille JC, et al. Surveillance of healthcare-
associated infections in Indonesian hospitals. J Hosp Infect 2006 02;62(2):219-229

9. Challenges with undertaking
validation remain….
 No internationally agreed standard/ protocol?
– ECDC now have a validation protocol for use in EU for
national validation
 Practical issues?
– Time/ funding
– Identifying an external gold standard /independent of the
primary data collection

10. National and international validation
 National validation:
– Required for interpretation of HAI prevalence + burden
estimates
– At the same time as the primary PPS
– Recommended: re-examine 750 patients in 25 hospitals
– Minimum: 250 patients in 5 hospitals
– Support contracts with ECDC (10 000 EUR / country),
budget to be spread over (at least) 2 years (2016-2017)
 International validation:
– Validation of national validation teams (VT)
– Accompany national VTs in 1-2 hospitals/country
– Who?: Part of HAI-Net support call for tender (published
soon): contractors + ECDC experts

11. How can ECDC help?
 Contracts for validation- financial support, feedback of
national validation results
 Contracts for International validation

12. Summary
 Validation is a key component of surveillance for
comparisons, consistency and confidence
 Without it we do not know the true prevalence of HAI
 Without it we cannot investigate reasons for variation
in HAI prevalence between hospitals and/ or countries
 Knowing the true burden makes the case for infection
prevention and control measures and enables
improvement in HAI