Presentation from the 3rd Joint Meeting of the Antimicrobial Resistance and Healthcare-Associated Infections (ARHAI) Networks, organised by the European Centre of Disease Prevention and Control - Stockholm, 11-13 February 2015

1. The case for validation in ICU surveillance
Professor Jacqui Reilly
Health Protection Scotland
UK

2. Why does it matter?
 3Cs: Consistency, comparisons and
confidence
 Low sensitivity (false negatives, or underreporting) of HAIs is a
frequently encountered problem in HAI surveillance systems.
 Low specificity (false positives, or over reporting) is usually less of a
problem
 Both may be related to one or more of following factors:
• Difficulty in confirming the case definition of an infection if signs
and symptoms were not well documented in the patient’s records
• If diagnostic tests included in the case definition of a particular
HAI type were not done
• Non compliance with the definition of the key term ‘healthcare-
associated’: even if the case definition of an infection is matched
due to cultural or financial/ political incentives and disincentives
at a hospital or country level

3. The case for validation in multi-country ICU
surveillance
In order to investigate variation between countries the first
question to ask is: is it the data? Validity? Reliability?
Reference:
European Centre for Disease Prevention and Control. Annual Epidemiological Report 2013.
Reporting on 2011 surveillance data and 2012 epidemic intelligence data. Stockholm: ECDC; 2014

4. ICU HAI surveillance validation

5. Validation findings by HAI type
Reference:
http://www.cdph.ca.gov/programs/hai/Documents/BuildingConfidenceInReportedHAIDataSuccessAndChallengesF
romState-basedValidationEffortsInCAandBeyond102012.pdf

6. Reasons for errors in reporting
Reference:
http://www.cdph.ca.gov/programs/hai/Documents/BuildingConfidenceInReportedHAIDataSuccessA
ndChallengesFromState-basedValidationEffortsInCAandBeyond102012.pdf

7. Validity of automated surveillance-ICU
 Manual ward surveillance
(MS) and electronic
surveillance (ES) were
performed
 ES was found to be more
effective than MS

8. Validity of denominator data
 1988 ICU patient charts from 23 hospitals reviewed by
DPH external team
 74% of hospitals collected data manually
 Over reporting of 300 PD and 200 CLD
 PD manual collection methods were more accurate than
electronic methods (P < .01)
 For central line days, there was no significant difference in
collection method (P > .05)

9. Other potential reasons for errors.....
– national targets with financial penalties
– the fear of creating a negative image of clinical areas
or hospitals
– lack of diagnostic testing and strict case definitions in
the protocol
– the consequences of these “underreporting is
probable,” “there will be less cases” and “the most
common consequence is that some HAI will not have
met the criteria”
Ref:
Price L et al (2014) A Cross-Sectional Survey of the acceptability of data collection processes for
validation of an European Point Prevalence Study of Healthcare-Associated Infections and
Antimicrobial Use (ECDC Pilot study of PPS validation)
.

10. Summary
 Validation is a key component of surveillance for
comparisons, consistency and confidence
 Without it we do not know the true incidence of HAI in
ICU
 Without it we cannot investigate reasons for variation
in HAI incidence between hospitals and/ or countries
 Knowing the true incidence of HAI makes the case for
infection prevention and control measures and enables
improvement in ICU

11. Key issues for discussion today

12. European surveillance of Healthcare-Associated
Infections in intensive care units (HAI-Net ICU):
Validation of ICU surveillance data
Carl Suetens
Surveillance and Response Support Unit
European Centre for Disease Prevention and Control

13. HAI validation in ECDC PPS
0
10
20
30
40
50
60
70
80
90
100
0 2 4 6 8 10
Primary PPS HAI%
Validation-Sensitivity(%)

14. Validation in surveillance vs PPS
• Validation is crucial for reliable burden estimates and
interpretation of inter-country variations
• Unlike validation of PPS data, validation of surveillance data
needs to be performed after the primary surveillance
(retrospective surveillance). (hospital staff to prepare patient
files of the selected surveillance period).
• Blind data collection: the validation team member(s) is/are
not allowed looking at the primary ICU surveillance forms
during the data collection (except for identifying the patient
number in the primary surveillance database).

15. Selection
• Selection of intensive care units: Validated ICUs should be selected
randomly from the list of ICUs participating to the primary ICU
surveillance, using systematic sampling after sorting the ICU list by
number of patients included in the surveillance. For each selected
hospital, select the next one as reserve hospital. Should be proportional
to N of pts in surveillance
 Selection of ICUs: include all ICUs included in the surveillance
 Selection of surveillance period: depends on the number of patients
included per surveillance period; from 2009 to 2011, an ICU contributed
on average 155 patients (median 126 patients) per surveillance-year and
21 patients (median 18 patients) per surveillance-month.
 Selection of patients:
– include all patients staying more than 2 days in the selected ICUs, at
least until the required number of validation records per hospital is
obtained.
– Random selection of patients (only possible if standard protocol is
followed)/select all HAI pos – Random selection of HAI negatives

16. Variation of the 95% confidence interval around a
sensitivity of 80% according to the number of
patients included in the validation sample
40%
50%
60%
70%
80%
90%
100%
250 500 750 1000 1250 1500 1750 2000
Sensitivity
Se
LL (Pr 7%)
UL (Pr 7%)
LL (Pr 2%)
UL (Pr 2%)

17. Validation of HAI-Net ICU data
 Ideally: 750 patients, 30 ICUs (or all if less)
 Pragmatic solution:
– Min 250 patients, 5 ICUs
 Support contracts with ECDC (PPS: 10 000 EUR per contract)
 Interrater reliability of national validation team members
 Minimum data:
– Validation of Infection data
– Additional validation data: validation method, primary
patient ID, reason for discordance (if any)
– Optional:
Denominator data (exhaustiveness)

18. Data forms: infection data
Patient Counter: Date of admission in ICU: ___ / ___ / _____
Age in years: ____ yrs Gender: M F UNK Date of ICU discharge: ___ / ___ / _____
Patient ICU outcome: O discharged alive O death, HAI definitely contributed to death
O death, HAI possibly contributed to death O death, no relation to HAI O death, relationship to HAI unknown
Case definition code
Relevant device in
situ before onset*
Date of onset**
BSI: source of BSI***
Micro-organism 1
Micro-organism 2
Micro-organism 3
*** C-CVC, C-PER, C-ART, S-PUL, S-UTI, S-DIG, S-SSI, S-SST, S-OTH, UNK
European Surveillance of ICU-acquired infections
HAI and AMR form, light protocol
*relevant device use (intubation for PN, CVC for BSI, urinary catheter for UTI) in 48 hours before onset of infection (even
intermittent use), 7 days for UTI **Only for infections not present/active at admission
MO-codeMO-code MO-code
___ / ___ / ______
ICU-acquired infections
HAI 1 HAI 2 HAI 3
O Yes O No
O Unknown
O Yes O No
O Unknown
___ / ___ / ______
O Yes O No
O Unknown
___ / ___ / ______

19. Additional validation data
ICU level: validation survey date, protocol primary surveillance, method for
selection of patients
note: if the primary surveillance is LIGHT, all patients should be selected
Patient level:
Primary Patient Counter (in primary surveillance)
VT results checked with primary PPS results after data collection?
O No O Yes
IF YES:
Discordant results discussed: O No O Yes O NA
VT decision changed: O No O Yes O NA
Reasons for discordance / VT comments for this patient/HAI: (Free text)

20. HAI-Net ICU validation protocol part of
Helicswin.Net

21. Discussion…
 National experiences
 Feasibility of validation and financial support
 Validation of HAI data, including mortality data

22. Strategies for extension of HAI
surveillance
 “Extension”:
– Increase N of participating countries
– Increase N of participating ICUs
– Increase duration of surveillance in ICUs
– Validation of ICU surveillance
 Tools:
– Light protocol
– Free hospital software (Helicswin.Net)
– Infection prevention indicators increase added value
– Financial support for validation