Software for Computer Vision is often used in embedded systems to support automatic workflow and to providesafety critical functionality. This presentation will show how one of the world-leading companies for in vitro diagnostic (IVD) solutions, bioMérieux, applied Computer Vision in laboratory instruments and transitioned from using black-box and integrated devices to designing more distributed hardware/software solutions. Additionally the presentation will provide a case study on how Altran succeeded in defining the tool chain underpinning the entire development, from Computer Vision algorithms design to embedded firmware deployment and testing. Architectural flexibility and a robust design control approach were fundamental drivers for the project and will facilitate meeting safety and regulatory requirements in bioMérieux’s diagnostics solutions.

1. A Computer Vision Application for In Vitro Diagnostics Devices
Andrea Carignano, Giacomo Donzelli

2. SAFETY IN IVD DEVICES

3. “The concept of Safety considers the notion of Basic Safety as the freedom from unacceptable risk directly caused by physical hazards when the equipment is used under normal and single fault conditions as well as the notion of Essential Performances”
(IEC 60513:1994—Fundamental aspects of safety standards for medical electrical equipment).
“Essential Performances” are “…..accuracy, precision and stability for their intended use, based on appropriate scientific and technical methods. In particular……… sensitivity, specificity, trueness, repeatability, reproducibility…”
(GHTF/SG1/N41R9:2005-Essential Principles of Safety and Performance of Medical Devices)
“Medical devices should be designed and manufactured in such a way that……they will not compromise the clinical condition or the safety of Patients, or the safety and health of Users or, where applicable, other persons (i.e. Environment)”
(GHTF/SG1/N41R9:2005- Essential Principles of Safety and Performance of Medical Devices)
Safety Principles in Medical Devices

4. Safety in Medical Devices is :
Basic Safety for Users and Environment and Essential Performances provided to Patients over time
Safety in Medical Devices, General Definition
Essential Performances
Basic Safety

5. Based upon Risk Management (ISO 14971) Safety in IVD is :
Basic Safety and Analytical Results (i.e. Essential Performances) provided within the required time (urgencies, turnaround time /drugs treatment, etc.)
Safety in In Vitro Diagnostics (IVD) Devices
Essential Performances (i.e. Results in time)
Basic Safety for Users and Patients’ Samples (i.e. Serum, Whole Blood, Paediatric Specimens, etc.)

6. ADDRESSING SAFETY IN DESIGN

7. Make or Buy vs. Architectural Options
Integrated Distributed
Buy
Ownership of design usually facilitates Product certification
Product Certification may become a challenge when Safety is addressed by integrated design and without disclosure for uncertified OTS components
Make
Additionally, distributed architectures usually ease the certification roadmap and allow for reuse, design margins and flexibility

8. Dedicated Plastic disposables are detected and recognized on board the instrument to prevent reagents from mismatching (false results) and to assure system productivity (before the required time to result expires)
Safety Through Computer Vision, an Integrated Solution
Integrated device embedding safety critical functions and including:
›Camera board
»CMOS camera
»Lighting system
›Interface Board hosting drivers, embedded OS, firmware and algorithms
»Micro Controller (Freescale MCIMX27VOP4)
»Memory Asset (Flash, Ram, E2Rom)
»Non-standard interface with the Master board (custom protocol)

9. Safety Through Computer Vision, a Distributed Design Option
Master Board (TI DM3730)
A commercial OTS USB camera used only for basic functionalities (capturing images) and
a master board embedding every safety critical function, and based upon a certified
RTOS and firmware
›Portability, Expandability, Scalability are additional and positive side effects
USB

10. HW/FW
ARCHITECTURE

11. Encapsulating Safety Critical Functionalities
›Once a use case is activated, the ARM main routine (VAS) calls the computer vision library (image processing/algorithms) on the DSP Main Routine
›Well Defined Boundaries/Interfaces (BSP/DSP Link) , “Split & Conquer” approach
›Encapsulating safety critical routines (computation engine)
Ethernet
CAN
USB BSP
DSP Link
ARM/RT
Green Hills INTEGRITY RTOS
DSP/SYSBIOS
ARM
Main Routine (VAS)
DSP Main Routine
Image Processing Libraries
Use Cases Algorithm
Certified
RTOS
Multithread
Services
OS Single Thread Services
Safety Critical Routines
TI DM3730 DUAL CORE
HW/camera

12. Verification Strategy, from Imaging to Use Cases algorithms
Reuse/
Effort
For Verification &Validation
(ISO 13485)
HW camera ARM/Certified OS DSP Link DSP/OS UC Algorithms
DSP Link
Products with Basic Use
Certified Products (procured)
Certified Products (reused)
Products to Certify

13. AN IVD CASE STUDY (MOLECULAR BIOLOGY)

14. An IVD distributed Computer Vision System able to….
“To Count the amount
of plastic tips in the
box and assure system
productivity ”
“To Detect the presence/absence of
liquid in the wells and prevent from
sample pollution, splashing, overflow
and sample wasting”
Use Case 1 Use Case 2

15. PROJECT’S CONSTRAINTS

16. Design and Tools Chain
›The computer vision algorithms are:
»Developed as Matlab source code
»Converted from floating point to fixed point (Matlab Fixed Point Designer)
»Translated into C language (Matlab Coder)
»Compiled
»Loaded onto the TI DSP
»Tested on board a fixture emulating the real application

17. Project’s Tasks
Localize the available tips
Measure liquid volume
Use computer vision algorithms that process images acquired by a low cost 2D camera and bar code reader
Use Case 2
Use Case 1

18. ALGORITHMS ADDRESSING USE CASES

19. General Workflow
›Calibration (reference image)
»The region of interest is
localized using image edges
»Knowledge of 3D models allows
for perspective correction
»Equalization, Morphological
operations and contrast
enhancement are applied to
rectify images
Edge detection for skew angle estimation
Cover holes localization for skew angle estimation
Region of interest
Interpolation points to rectify image regions inside the slots
Features arrays (small square images)
4229
2697
1197
276
76
0
4229
2697
1197
276
76
0
Use Case 1 Use Case 2
»Operation (generic image)
»Same processing as calibration image
»The rectified operative image is compared to reference and a
correlation analysis is carried out
»Use case-specific criteria are applied

20. Use Case 1
Perspective undistorsion
Illumination correction
Features arrays (small square images)
Operation image
Features arrays (small square images)
Rectified
Generic
features for
either tips
or holes
Same processing
Rectified
Reference
features for
tips
Tips present

21. Use Case 1
Perspective undistorsion
Illumination correction
Features arrays (small square images)
Operation image
Features arrays (Correlation coefficients small square images)
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Correlation coefficients
between features (all tips
vs. either tips or holes)
Tips present
Tips
identified

22. Use Case 2
Average calibration image
Cover holes localization for skew angle estimation
Interpolation points to rectify image regions inside the slots
4229
2697
1197
276
76
0
Operation image
Same processing
Rectified
operative
image
Rectified
reference
image

23. Use Case 2
Average calibration image
Cover holes localization for skew angle estimation
Interpolation points to rectify image regions inside the slots
Operation image
4229
276
76
0
Reference
(empty)
Operative
(filled)
Volume (ul)
4229
2697
1197
276
76
0 Estimated volume
Visual Artifacts between operative and reference
images can stem from both liquid and glares

24. Results
Test on 41 images
›82 total wells
›11 different volumes
Use Case 2
Use Case 1
Real volume (μl)
0
100
200
300
400
500
600
700
0
50
100
150
200
250
400
500
1400
1600
1800
2000
2200
2400
2600
2800
3000
3200
3400
1500
2000
3000
Measured volume (μl)
672
Holes
Tips
Actual
Holes
Tips
Prediction
624
0
0
Test on 37 images
›672 Holes
›624 Tips

25. CONCLUSIONS

26. Computer Vision and IVD Critical Safety Systems
›We addressed safety in IVD system by using Computer Vision technologies and meeting safety requirements
›A distributed architectural option allowed for product certification, leveraged reuse and optimized verification effort.
›A “Split and Conquer” approach will ease product evolution, upgrades and roadmap for further certifications and ISO13485 compliance
›Computer vision algorithms , in the presented application, offered also some key technical and business benefits:
»Automatic control over the final device giving correct operations
»A single control system and configuration for different tasks
»Use of commercial low cost camera and illumination system
»Robust solution in presence of uneven illumination and glares