Talk by: Felix Joussein Icinga is used to monitor and alert in case of temperature deviations in a GxP (pharmaceutical industry) regulated environment. The holy grale in a laboratpry environment are the study samples (animal but also human). So regulators (EMA, FDA,..) around the world want to ensure, that the sample-stability is at it’s best. To ensure this, depending on the known stability, certain temperature ranges in cooling-units are defined and have to be trackable thourghout the whole lifecycle of such a sample.

1. Monitoring Cooling units
in a GxP regulated
pharmaceutical environment
Welcome to Icinga Camp Milan
Oct. 17, 2023

2. Monitoring Cooling units
in a GxP regulated
pharmaceutical environment
About me!
like “Jalousie” and “Cousin”
Felix Joussein

3. Monitoring Cooling units
in a GxP regulated
pharmaceutical environment
● What is GxP? (1)
● Good x Practices
● x = Laboratory | Clinical | Manufactoring
● Basic meaning?
● What am I doing?
What is the intended result of my process?
Which risks are involved?
● Where am I doing it?
Is the used equipment and it’s consumables suitable for my process?
Lifecycle based approach to all equipment and consumables
● Why am I doing it?
Is my process transparent and documented in a comprehensive manner?
Have my documents a lifecycle policy applied with periodic review?

4. Monitoring Cooling units
in a GxP regulated
pharmaceutical environment
● What is GxP? (2)
● Who checks the compliance?
● Your local compliance authority
AGES, BFR, AIFA,...
● EMA (European Medicine Agency, EU)
● FDA (Food and Drug Administration, US)
● And many more local and international GxP regulatory offices
● Purpose of those regulations
● Global comparability
● Human wealth
● Animal wealth
● Cost efficiency

5. Monitoring Cooling units
in a GxP regulated
pharmaceutical environment
● GxP Requirements and considerations (1)
● defined temperature set-points
depending the stability of cooling goods
● defined deviation-ranges around the set-point
compressor-based cooling units oscillate to keep a defined temperature

6. Monitoring Cooling units
in a GxP regulated
pharmaceutical environment
● GxP Requirements and considerations (2)
● Cooling unit type
● Depending on top- or front loader, compartments and other, thaw and freeze
curves must be considered to find an acceptable time frame for loading and
unloading cooling goods.
● Ensure data integrity and completenes
● Open systems: Recorded data must be available and readable for 25 or more
years
● Alerting for record gaps must be in place

7. Monitoring Cooling units
in a GxP regulated
pharmaceutical environment
● Commercial Infrastructure involved
● Datalogger
● blackbox with incomprehensive internal monitoring and alerting options but
accessible via Modbus over tcp (this is the way to Icinga2)
● Software to the Datalogger
● polling server retrieves data from the datalogger and pushes it to an MS
Access Database
● used to draw graphs and export data in CSV

8. Monitoring Cooling units
in a GxP regulated
pharmaceutical environment
● Icinga2: monitoring the temperature (1)
Monitoring temperature deviations is like monitoring a ping’s rtt
● Modbus Script
● Bash polling script, that retrieves every 5 Minutes (via cron) all available
channel names with temperature value and writes its output to /tmp
● Icinga2 check
● each Cooling unit is created as a host-object
● a Bash Check-command script will query the result file of the polling script
● hosts- and service template ease the creation of such host-objects
● templates use the defined command configuration that offers configuration for
set temperature, warning and critical treshholds (both to hot or to cold)
● Default settings of host- and service variables can be overwritten in the
host object.

9. Monitoring Cooling units
in a GxP regulated
pharmaceutical environment
● Icinga2: monitoring the temperature(2)
● Alerting
● A check is performed every 5 minutes
● On Warn | Crit it tries 5 more times before executing a notification script
● Re-check every 30 minutes
● Notifications contain the current-, set-, warn- and critical values
● Notifications are sent to
● the cooling unit logbook, which is implemented with Znuny (OTRS fork)
● Via SMS (text message on cell) to alarm responsible staff
● Visualization
● Using Graphite, fancy temperature curves are drawn

10. Monitoring Cooling units
in a GxP regulated
pharmaceutical environment
● Icinga2: monitoring data completeness (1)
● Data source
● Data recorder creates a temperature record every 5 minutes
● Data recorder’s rawdata-format temperproof, encrypted and proprietary
● Import via data recorder compendium software to MS Access database
● One way sync with commercial tool to MSSQL on a daily bases
● Icinga2 check
● A service-template is added to the cooling unit host’s object
● A Command configuration provides via variables for the host-service the
cooling unit’s channel name within the proprietary database structure
● A php check command (running as CLI) script executes the query against the
database and returns a check result for Icinga2.

11. Monitoring Cooling units
in a GxP regulated
pharmaceutical environment
● Icinga2: monitoring data completeness (2)
● Acceptance criteria for “completeness”
● > 263 / 288 datapoints per day
● Alerting
● A check is performed every 5 minutes
● Notifications output contains a list of missing days (if any) as well as
the check-period in days.
● Notifications are sent to the cooling unit logbook

12. Monitoring Cooling units
in a GxP regulated
pharmaceutical environment
● Conclusion(s):
● Passing End-User acceptance-test
● Icingaweb2’s UI is suitable and well accepted by end-users (not only for
admins)
● Accepted by regulatory agencies
● Regulatory officers are not familiar with “non-industry-standard”-Systems,
and yet approve this system as a valid solution to ensure sample-stability
and data integrity

13. Monitoring Cooling units
in a GxP regulated
pharmaceutical environment
● Outlook / enhancements:
● Temperature-Rawdata acquisition
● Skipping the various database import/sync operations feeding the modbus-
→
data directly to a well organized SQL DB
● Telemetry processing
● Feeding data to ELK-Stack for enhanced reporting possibilities

14. Monitoring Cooling units
in a GxP regulated
pharmaceutical environment
Thank you for your attention!
Any Questions?