Principles of Software
Verification and Validation
for Medical Imaging
Twin Spin
October 7, 2010
07-OCT-2010
Topics
2
 Short history of medical imaging
 How SW “Changed the Picture”
 Challenges and Solutions to SoftwareVerificat...
History
07-OCT-2010 3
 Röntgen discovers X-rays in 1885,
receives Nobel Prize in 1901.
 Rapid research and discovery
lea...
12/10/2008 4
Boom in x-ray-associated marketing.
Medical Imaging Today
Multiple Modalities
 Electron microscopy
 Radiographic – standard x-rays, fluoroscopy
 MRI
 Nucl...
Medical Imaging Today
607-OCT-2010
Demonstration
 2D images on e-film portable reader, courtesy my CT
scan.
 3D, 4D Images from Osirix open source reader.
...
For purposes of understanding these principles, this talk
will focus on CT technology.
07-OCT-2010 8
How Software “Changed the Picture”
Yesterday:
07-OCT-2010 9
Hardware
config and
control
Hardcopy
output (film)
Manual
arch...
Today:
07-OCT-2010 10
Software config and
control console
Software patient
positioning control
Digital
raw data
Dedicated ...
The essential question:
Can I trust these pictures?
07-OCT-2010 11
To help answer the question, we ask
two more questions:
Q: What is essential in the image?
A: This is validation. “What is...
The essential question changes:
From:
Q: Can I trust these pictures?
To:
Q: How much can I trust these
pictures?
07-OCT-20...
Answering Question 1: An Art Lesson
Visual artists have known that images are an
interpretation of reality and exploit tha...
How to create trust in a subjective
process?
A: Experience
Through certification, advanced education, on-the-
job training...
Product Validation
 Validation involves the customer or it isn’t validation.
 Validation of images by medical imaging pr...
Product Validation Approaches
 Internal panel of expert employees – training team,
field application teams.
 Internal pa...
07-OCT-2010 18
User error
Modality quantization,
noise error
Reconstruction
algorithmic error
“false images”
Data transmis...
Error in the dataflow.
Type of Error Frequency Root source of error
User error at console Potentially often Human factors ...
The technical effort simplifies to reducing
the impact of undetected functional error
on the image to be less than or equa...
The four areas for an image to be
trustworthy from the user perspective:
 Image orientation
 Image fidelity
 Measuremen...
Workstation error in the end image.
Area of concern Root source of error
Image orientation Functional error processing DIC...
How to verify Image Orientation
07-OCT-2010 23
Scan multiple times
Manufacture an object of known
dimension, HU values, or...
Phantom manufacturers
 http://www.universalmedicalinc.com/diagnostic-
imaging/imaging-quality-control/phantoms/ct-phantom...
How to verify Image Fidelity
07-OCT-2010 25
Use reference datasets
• Phantoms
• Synthetic data
Check the result against re...
How to verify Measurements
07-OCT-2010 26
Use reference datasets
• Phantoms
• Synthetic data
Check the result against refe...
How to verify Data Integrity
07-OCT-2010 27
Use reference datasets
• Phantoms
• Synthetic data
•Large, small,
•Multiple mo...
Other things that affect
trustworthiness Reliability
 Stability
 Security
 Installation & Upgrades
 System integratio...
In other words….
Very few of the verification factors for trustworthiness
are medical imaging-specific. Most are core
soft...
Example of SW Engineering lapse.
GE Healthcare, August 2008.
Optovue, 2010
12/10/2008 30
Risk-based verification
 Everything can’t be tested
 Organizations inevitably take a risk-based approach
whether they kn...
Risk-based verification derived from
Business Needs/Risks
All risks are secondary to patient risk. Recall vs. patch
07-OCT...
Summary
 The key to medical imaging is trustworthiness of the
image – even with known error sources.
 Verifying medical ...
Biography
Alex Dietz has been applying product quality
verification principles for 20+ years in
telecommunications, data t...
References
History
 NakedToThe Bone: Medical Imaging InTheTwentieth Century by Bettyanne Kevles
 http://en.wikipedia.org...
Upcoming SlideShare
Loading in …5
×

Principles of Software Verification and Validation

1,775 views

Published on

Published in: Health & Medicine, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,775
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
37
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Principles of Software Verification and Validation

  1. 1. Principles of Software Verification and Validation for Medical Imaging Twin Spin October 7, 2010 07-OCT-2010
  2. 2. Topics 2  Short history of medical imaging  How SW “Changed the Picture”  Challenges and Solutions to SoftwareVerification  4 Areas of image trustworthiness  Importance of SW Engineering Principles  Product SoftwareValidation  Summary 07-OCT-2010
  3. 3. History 07-OCT-2010 3  Röntgen discovers X-rays in 1885, receives Nobel Prize in 1901.  Rapid research and discovery leading to working prototype x-ray imaging system by Edison in 1901  Other modalities follow rapid development in 20th century.
  4. 4. 12/10/2008 4 Boom in x-ray-associated marketing.
  5. 5. Medical Imaging Today Multiple Modalities  Electron microscopy  Radiographic – standard x-rays, fluoroscopy  MRI  Nuclear medicine – PET, gamma  Thermography  Tomography – CT scans  Ultrasound  Photoacoustic imaging - lasers + ultrasound. 12/10/2008 5
  6. 6. Medical Imaging Today 607-OCT-2010
  7. 7. Demonstration  2D images on e-film portable reader, courtesy my CT scan.  3D, 4D Images from Osirix open source reader. Courtesy 12/10/2008 7
  8. 8. For purposes of understanding these principles, this talk will focus on CT technology. 07-OCT-2010 8
  9. 9. How Software “Changed the Picture” Yesterday: 07-OCT-2010 9 Hardware config and control Hardcopy output (film) Manual archiving Limited review Hardware positioning
  10. 10. Today: 07-OCT-2010 10 Software config and control console Software patient positioning control Digital raw data Dedicated digital signal processing Transmission over enterprise network DICOM data output Archival File Server (PACS) PACS reading station Advanced visualization workstation
  11. 11. The essential question: Can I trust these pictures? 07-OCT-2010 11
  12. 12. To help answer the question, we ask two more questions: Q: What is essential in the image? A: This is validation. “What is the right information?” Q: How much error does the essential information contain? A: This is verification. “How do I insure the information is as error-free as possible?” 07-OCT-2010 12
  13. 13. The essential question changes: From: Q: Can I trust these pictures? To: Q: How much can I trust these pictures? 07-OCT-2010 13
  14. 14. Answering Question 1: An Art Lesson Visual artists have known that images are an interpretation of reality and exploit that fact to convey essential messages. Medical imaging is an interpretation of reality too, intentionally distorted through reconstruction algorithms to convey essential diagnostic information (i.e. right information). 07-OCT-2010 14 Interpretation involves subjectivity. Boy or girl?
  15. 15. How to create trust in a subjective process? A: Experience Through certification, advanced education, on-the- job training, etc. 07-OCT-2010 15
  16. 16. Product Validation  Validation involves the customer or it isn’t validation.  Validation of images by medical imaging professionals is absolutely vital: radiologists, CT/MR technologists, etc.  Validation can be done at nearly all steps in the development.  Beta field phase absolutely essential.  Some issues can only be validated  Human factors: presentation of on-screen information, segmentation preferences, image fidelity preferences.  Visualization of small vessels or structures whose phantoms are too costly. 12/10/2008 16
  17. 17. Product Validation Approaches  Internal panel of expert employees – training team, field application teams.  Internal panel of expert consultants – medical advisory board, focus groups.  Luminary sites willing to partner in product development.  Industry-acknowledged body of knowledge. Walter Reed colon datasets, Stanford bake-off. 12/10/2008 17
  18. 18. 07-OCT-2010 18 User error Modality quantization, noise error Reconstruction algorithmic error “false images” Data transmission protocol error Database error User error, visual errors 3D processing, user, & algorithmic errors Answering Question 2: Dataflow error
  19. 19. Error in the dataflow. Type of Error Frequency Root source of error User error at console Potentially often Human factors design, functional flaw. Quantization and noise Every scan Law of nature. Reconstruction error Every scan Law of nature, functional flaw Data transmission error Rare and getting rarer TCP/IP stack, DICOM stack, functional flaw. Database error Rare and getting rarer Configuration error, functional flaw. PACS user error Potentially often Human factors design, image processing, functional flaw. Viz workstation error Potentially often Human factors design, image processing, functional flaw. 07-OCT-2010 19
  20. 20. The technical effort simplifies to reducing the impact of undetected functional error on the image to be less than or equal to the impact of inherent error on the image SO THAT the images and data derived from the images are trustworthy. Any “questionable” image abnormalities visual artifacts, or other deficiencies are due to laws of physics inherent in the modality. 07-OCT-2010 20
  21. 21. The four areas for an image to be trustworthy from the user perspective:  Image orientation  Image fidelity  Measurements  Data integrity 07-OCT-2010 21
  22. 22. Workstation error in the end image. Area of concern Root source of error Image orientation Functional error processing DICOM header and placing orientation markers on screen. Manifested by rotation of image by 90 degree increments on screen. Image fidelity Functional error introduced by incorrect graphic engine processing and algorithm application. Manifested by obscured anatomy, rendering, incorrect segmentation. Measurements Functional error processing scaling factor from DICOM header or error in measurement algorithms. Manifested by rulers wrong by large fixed multiples, measurement of same anatomical structures changing as code changes. Data Integrity Failure in data transfer, processing of incoming DICOM data, cross-check of header information, management of database, volume. Manifested by missing or repeated areas in the 2D and 3D images. 07-OCT-2010 22
  23. 23. How to verify Image Orientation 07-OCT-2010 23 Scan multiple times Manufacture an object of known dimension, HU values, orientation Check the result The reference library of orientation datasets is ideal for regression automation.
  24. 24. Phantom manufacturers  http://www.universalmedicalinc.com/diagnostic- imaging/imaging-quality-control/phantoms/ct-phantom  http://www.phantomlab.com/rsvp_head.html  http://www.cirsinc.com 12/10/2008 24
  25. 25. How to verify Image Fidelity 07-OCT-2010 25 Use reference datasets • Phantoms • Synthetic data Check the result against reference image using image pixel checker Problem: subtle differences due to video driver revs cause false failures. **Error below threshold of human eye** Tools must not flag errors that are not noticeable (or relevant) to the user. The sample image must be transferred through a daisy chain of imaging workstations to simulate the enterprise environment. Reprocessing the image can, in rare cases, lead to image degradation. User acceptance panels are another tactic – often for selecting default color tables. DICOM xfer
  26. 26. How to verify Measurements 07-OCT-2010 26 Use reference datasets • Phantoms • Synthetic data Check the result against reference measurement DICOM xfer In some cases it is necessary to manually complete a typical patient “workup” or workflow, transfer the patient record to a second workstation, and verify the measurements maintain consistency in data transfer & processing. All manual measurements are, by their nature, subject to human error. Define +- bounds
  27. 27. How to verify Data Integrity 07-OCT-2010 27 Use reference datasets • Phantoms • Synthetic data •Large, small, •Multiple modalities •Load testing •Negative testing Check the results at the database, not the user interface. We assume the user interface & visualization do not corrupt the data (it is prudent to verify this assumption if using this strategy). DICOM xfer DICOM Database Data transfer testing is executed in isolation and in concert with orientation, fidelity, and measurement verification.
  28. 28. Other things that affect trustworthiness Reliability  Stability  Security  Installation & Upgrades  System integration  Localization  Licensing  Performance  Manufacturing & distribution  Etc. 07-OCT-2010 28
  29. 29. In other words…. Very few of the verification factors for trustworthiness are medical imaging-specific. Most are core software engineering principles and good quality engineering.  Good requirements development and management  Good code development and management  Good test case development and management 07-OCT-2010 29 Following good (not even best) practices automatically generates artifacts that audit agencies look for as proof of regulatory compliance.
  30. 30. Example of SW Engineering lapse. GE Healthcare, August 2008. Optovue, 2010 12/10/2008 30
  31. 31. Risk-based verification  Everything can’t be tested  Organizations inevitably take a risk-based approach whether they know it or not.  “Good enough” by instinct to begin  “Good enough” by systematic classification by end.  Cross departmental effort to classify risks (patient risks, business risks)  Living document of decision-making as-you-go.  Complies with regulatory risk assessment deliverables.  High business value. 07-OCT-2010 31
  32. 32. Risk-based verification derived from Business Needs/Risks All risks are secondary to patient risk. Recall vs. patch 07-OCT-2010 32 Business Need: (product will not harm patient) Project Risk Assessment (schedule, market, patient hazards, etc) Design Mitigation Risk-based system testing Unit testingBusiness Need: ( ) Business Need ( )
  33. 33. Summary  The key to medical imaging is trustworthiness of the image – even with known error sources.  Verifying medical imaging software is an engineering task approaching image orientation, image fidelity, measurements, and data integrity  Validating imaging software is a clinical user task overlapping with human factors and artistic questions of interpretation and presentation.  All other challenges are familiar core software engineering problems. 07-OCT-2010 33
  34. 34. Biography Alex Dietz has been applying product quality verification principles for 20+ years in telecommunications, data transmission, and medical imaging. He currently manages the Software Verification and Validation team for the EnSite cardiac mapping system at St. Jude Medical. He has spoken locally and nationally, most recently at the Software Design for Medical Devices conference in San Diego. adietz@sjm.com 07-OCT-2010 34
  35. 35. References History  NakedToThe Bone: Medical Imaging InTheTwentieth Century by Bettyanne Kevles  http://en.wikipedia.org/wiki/X-ray#History Phantom Manufacturers  http://www.phantomlab.com/  http://www.universalmedicalinc.com  http://www.cirsinc.com 07-OCT-2010 35

×