SlideShare a Scribd company logo
1 of 26
Download to read offline
Technical Validation through Automated Testing
Computational Phenotyping and Software Engineering
S36
Martin Chapman
King’s College London
#IS24
AMIA 2024 Informatics Summit | amia.org 1
1. Background
AMIA 2024 Informatics Summit | amia.org 2
Automated testing
Floor 1 Æ ¥
r
AMIA 2024 Informatics Summit | amia.org 3
Automated testing
Floor 1 Æ ¥
r
Floor 2 Æ ¥
AMIA 2024 Informatics Summit | amia.org 3
Automated testing
Floor 1 Æ ¥
r
Floor 2 Æ ¥
Floor 3 Æ ¥
W Æ q
AMIA 2024 Informatics Summit | amia.org 3
Automated testing
Floor 1 Æ ¥
r
Floor 2 Æ ¥
Floor 3 Æ ¥
W Æ q
W Æ q
AMIA 2024 Informatics Summit | amia.org 3
Automated testing
Floor 1 Æ ¥
r
Floor 2 Æ ¥
Floor 3 Æ ¥
https://github.com/sindresorhus/got
AMIA 2024 Informatics Summit | amia.org 3
Unit tests vs. integration tests
W Æ q
W Æ q
Floor 1 Æ ¥
Floor 2 Æ ¥
Floor 3 Æ ¥
r
AMIA 2024 Informatics Summit | amia.org 4
Unit tests vs. integration tests
W
Æ
q
W
Æ
q
Floor
1
Æ
¥
Floor
2
Æ
¥
Floor
3
Æ
¥
r
AMIA 2024 Informatics Summit | amia.org 4
Unit tests vs. integration tests
W
Æ
q
W
Æ
q
Floor
1
Æ
¥
Floor
2
Æ
¥
Floor
3
Æ
¥
r
Æ q
AMIA 2024 Informatics Summit | amia.org 4
Unit tests vs. integration tests
W
Æ
q
W
Æ
q
Floor
1
Æ
¥
Floor
2
Æ
¥
Floor
3
Æ
¥
r
Æ q
https://github.com/mattermost/mattermost
AMIA 2024 Informatics Summit | amia.org 4
Automated testing and Health Informatics
https://github.com/OHDSI/CommonDataModel
AMIA 2024 Informatics Summit | amia.org 5
2. Automated testing and phenotyping
AMIA 2024 Informatics Summit | amia.org 6
The phenotype development lifecycle
Health informatics applications, libraries and packages that are tested are more likely to operate
correctly. We want the same to be true of phenotypes.
There are several places in the phenotype development lifecycle where we can apply testing tools:
Author
1. Modelling
languages
2. Multi-
dimensional
descriptions
4. Modular
relationships
Refine
3. Versioning and
provenance
13. Comprehensive
metadata
Implement
5. Implementation
details in model
6. Prog. language
agnostic
7. Multiple data
formats/standards
Validate
8. Defined
validation process
9. Automate
multiple validation
techniques
10. Enable
feedback
Publish
11. Expose
standard API
12. Advanced
search
capabilities
AMIA 2024 Informatics Summit | amia.org 7
Running example: CQL phenotypes
Pascal S. Brandt, Jennifer A. Pacheco, and Luke V. Rasmussen. “Development of a repository of
computable phenotype definitions using the clinical quality language”. In: JAMIA Open (2021)
AMIA 2024 Informatics Summit | amia.org 8
1. Testing during phenotype authoring
We’ve seen that there is a necessary difference between a phenotype definition (abstract logic;
generalisable) and a computable phenotype (programming code; specific).
AMIA 2024 Informatics Summit | amia.org 9
1. Testing during phenotype authoring
Phenotype
definition:
Computable
phenotype:
AMIA 2024 Informatics Summit | amia.org 9
1. Testing during phenotype authoring
We’ve seen that there is a necessary difference between a phenotype definition (abstract logic;
generalisable) and a computable phenotype (programming code; specific).
This does not, however, mean that phenotype definitions cannot be computable. If they are, we
can apply testing tools to this abstract logic as another measure of correctness. In our running
example, a CQL phenotype definition can itself be tested using the CQL Testing Framework
(CTF) and YAML-based test definitions.
AMIA 2024 Informatics Summit | amia.org 9
1. Testing during phenotype authoring
Phenotype
definition:
Computable
phenotype:
AMIA 2024 Informatics Summit | amia.org 9
1. Testing during phenotype authoring
We’ve seen that there is a necessary difference between a phenotype definition (abstract logic;
generalisable) and a computable phenotype (programming code; specific).
This does not, however, mean that phenotype definitions cannot be computable. If they are, we
can apply testing tools to this abstract logic as another measure of correctness. In our running
example, a CQL phenotype definition can itself be tested using the CQL Testing Framework
(CTF) and YAML-based test definitions.
The application of unit tests during phenotype authoring is likely to be particularly effective, as
phenotype definitions are often designed as distinct units of logic.
AMIA 2024 Informatics Summit | amia.org 9
1. Testing during phenotype authoring
https://github.com/PheMA/phekb-phenotypes
AMIA 2024 Informatics Summit | amia.org 9
2. Testing during phenotype implementation
Phenotype
definition:
Computable
phenotype:
AMIA 2024 Informatics Summit | amia.org 10
2. Testing during phenotype implementation
Testing tools can be applied during phenotype
implementation in much the same way testing
tools are applied during general software de-
velopment.
For example, CQL connects with the popular
Mocha Javascript testing framework. At this
level, the tests we see are more akin to inte-
gration tests.
https://github.com/cqframework/cql-execution
AMIA 2024 Informatics Summit | amia.org 10
3. Testing during phenotype validation
Recall that the reason we are motivated to apply (automated) testing tools to the development of
phenotypes is to ensure they are operating correctly.
Phenotype validation has the same goal: to determine whether a given phenotype can correctly
identify which individuals exhibit the true phenotype and which do not.
Therefore, testing can be applied and used as an additional form of validation. We term this
technical validation. Practically, technical validation is likely to manifest as writing and
executing test cases for existing phenotypes.
,
Technical validation and phenotype validation are still very distinct processes.
AMIA 2024 Informatics Summit | amia.org 11
Summary
• Automated testing tools, which include both unit and integration testing, are commonplace in
software development, including software developed in the Health Informatics domain.
• However, they are relatively underused in the development of phenotypes.
• We identify huge potential for automated testing tools to be applied to phenotype authoring
(running testing tools against computable definitions), phenotype implementation (to test
whole definitions) and phenotype validation (as an additional way to assess the correctness
of a definition).
AMIA 2024 Informatics Summit | amia.org 12
Thank you!
AMIA 2024 Informatics Summit | amia.org 13

More Related Content

Similar to Technical Validation through Automated Testing

Verification of the protection services in antivirus systems by using nusmv m...
Verification of the protection services in antivirus systems by using nusmv m...Verification of the protection services in antivirus systems by using nusmv m...
Verification of the protection services in antivirus systems by using nusmv m...ijfcstjournal
 
M2CAT: Extracting reproducible simulation studies from model repositories usi...
M2CAT: Extracting reproducible simulation studies from model repositories usi...M2CAT: Extracting reproducible simulation studies from model repositories usi...
M2CAT: Extracting reproducible simulation studies from model repositories usi...Martin Scharm
 
st-notes-13-26-software-testing-is-the-act-of-examining-the-artifacts-and-the...
st-notes-13-26-software-testing-is-the-act-of-examining-the-artifacts-and-the...st-notes-13-26-software-testing-is-the-act-of-examining-the-artifacts-and-the...
st-notes-13-26-software-testing-is-the-act-of-examining-the-artifacts-and-the...mwpeexdvjgtqujwhog
 
Diving into the World of Test Automation The Approach and the Technologies
Diving into the World of Test Automation The Approach and the TechnologiesDiving into the World of Test Automation The Approach and the Technologies
Diving into the World of Test Automation The Approach and the TechnologiesQASymphony
 
An Essential Guide to Effective Test Automation Leveraging Open Source
An Essential Guide to Effective Test Automation Leveraging Open SourceAn Essential Guide to Effective Test Automation Leveraging Open Source
An Essential Guide to Effective Test Automation Leveraging Open SourceRapidValue
 
Testing parallel programs
Testing parallel programsTesting parallel programs
Testing parallel programsPVS-Studio
 
Data Science for Smart Manufacturing
Data Science for Smart ManufacturingData Science for Smart Manufacturing
Data Science for Smart ManufacturingCarlo Torniai
 
Replication and Benchmarking in Software Analytics
Replication and Benchmarking in Software AnalyticsReplication and Benchmarking in Software Analytics
Replication and Benchmarking in Software AnalyticsUniversity of Zurich
 
Evaluating Model Testing and Model Checking for Finding Requirements Violatio...
Evaluating Model Testing and Model Checking for Finding Requirements Violatio...Evaluating Model Testing and Model Checking for Finding Requirements Violatio...
Evaluating Model Testing and Model Checking for Finding Requirements Violatio...Lionel Briand
 
Speeding-up Software Testing With Computational Intelligence
Speeding-up Software Testing With Computational IntelligenceSpeeding-up Software Testing With Computational Intelligence
Speeding-up Software Testing With Computational IntelligenceAnnibale Panichella
 
Parasoft .TEST, Write better C# Code Using Data Flow Analysis
Parasoft .TEST, Write better C# Code Using  Data Flow Analysis Parasoft .TEST, Write better C# Code Using  Data Flow Analysis
Parasoft .TEST, Write better C# Code Using Data Flow Analysis Engineering Software Lab
 
Comparative Study on Different Mobile Application Frameworks
Comparative Study on Different Mobile Application FrameworksComparative Study on Different Mobile Application Frameworks
Comparative Study on Different Mobile Application FrameworksIRJET Journal
 
Top 5 Code Coverage Tools in DevOps
Top 5 Code Coverage Tools in DevOpsTop 5 Code Coverage Tools in DevOps
Top 5 Code Coverage Tools in DevOpsscmGalaxy Inc
 
How to Guarantee Continuous Value from your Test Automation
How to Guarantee Continuous Value from your Test AutomationHow to Guarantee Continuous Value from your Test Automation
How to Guarantee Continuous Value from your Test AutomationPerfecto by Perforce
 
Welcome to International Journal of Engineering Research and Development (IJERD)
Welcome to International Journal of Engineering Research and Development (IJERD)Welcome to International Journal of Engineering Research and Development (IJERD)
Welcome to International Journal of Engineering Research and Development (IJERD)IJERD Editor
 
Machine Learning Meets Quantitative Planning: Enabling Self-Adaptation in Aut...
Machine Learning Meets Quantitative Planning: Enabling Self-Adaptation in Aut...Machine Learning Meets Quantitative Planning: Enabling Self-Adaptation in Aut...
Machine Learning Meets Quantitative Planning: Enabling Self-Adaptation in Aut...Pooyan Jamshidi
 
CS640_Thesis_AbulAalaAlamsBari13250849
CS640_Thesis_AbulAalaAlamsBari13250849CS640_Thesis_AbulAalaAlamsBari13250849
CS640_Thesis_AbulAalaAlamsBari13250849Almas Bari
 
Automated requirements based testing for ISO 26262
Automated requirements based testing for ISO 26262 Automated requirements based testing for ISO 26262
Automated requirements based testing for ISO 26262 QA Systems
 
How to get the most out of your CI/CD workflow using automated testing - Sauc...
How to get the most out of your CI/CD workflow using automated testing - Sauc...How to get the most out of your CI/CD workflow using automated testing - Sauc...
How to get the most out of your CI/CD workflow using automated testing - Sauc...twaintaylorb2b
 

Similar to Technical Validation through Automated Testing (20)

Verification of the protection services in antivirus systems by using nusmv m...
Verification of the protection services in antivirus systems by using nusmv m...Verification of the protection services in antivirus systems by using nusmv m...
Verification of the protection services in antivirus systems by using nusmv m...
 
M2CAT: Extracting reproducible simulation studies from model repositories usi...
M2CAT: Extracting reproducible simulation studies from model repositories usi...M2CAT: Extracting reproducible simulation studies from model repositories usi...
M2CAT: Extracting reproducible simulation studies from model repositories usi...
 
st-notes-13-26-software-testing-is-the-act-of-examining-the-artifacts-and-the...
st-notes-13-26-software-testing-is-the-act-of-examining-the-artifacts-and-the...st-notes-13-26-software-testing-is-the-act-of-examining-the-artifacts-and-the...
st-notes-13-26-software-testing-is-the-act-of-examining-the-artifacts-and-the...
 
50120140502017
5012014050201750120140502017
50120140502017
 
Diving into the World of Test Automation The Approach and the Technologies
Diving into the World of Test Automation The Approach and the TechnologiesDiving into the World of Test Automation The Approach and the Technologies
Diving into the World of Test Automation The Approach and the Technologies
 
An Essential Guide to Effective Test Automation Leveraging Open Source
An Essential Guide to Effective Test Automation Leveraging Open SourceAn Essential Guide to Effective Test Automation Leveraging Open Source
An Essential Guide to Effective Test Automation Leveraging Open Source
 
Testing parallel programs
Testing parallel programsTesting parallel programs
Testing parallel programs
 
Data Science for Smart Manufacturing
Data Science for Smart ManufacturingData Science for Smart Manufacturing
Data Science for Smart Manufacturing
 
Replication and Benchmarking in Software Analytics
Replication and Benchmarking in Software AnalyticsReplication and Benchmarking in Software Analytics
Replication and Benchmarking in Software Analytics
 
Evaluating Model Testing and Model Checking for Finding Requirements Violatio...
Evaluating Model Testing and Model Checking for Finding Requirements Violatio...Evaluating Model Testing and Model Checking for Finding Requirements Violatio...
Evaluating Model Testing and Model Checking for Finding Requirements Violatio...
 
Speeding-up Software Testing With Computational Intelligence
Speeding-up Software Testing With Computational IntelligenceSpeeding-up Software Testing With Computational Intelligence
Speeding-up Software Testing With Computational Intelligence
 
Parasoft .TEST, Write better C# Code Using Data Flow Analysis
Parasoft .TEST, Write better C# Code Using  Data Flow Analysis Parasoft .TEST, Write better C# Code Using  Data Flow Analysis
Parasoft .TEST, Write better C# Code Using Data Flow Analysis
 
Comparative Study on Different Mobile Application Frameworks
Comparative Study on Different Mobile Application FrameworksComparative Study on Different Mobile Application Frameworks
Comparative Study on Different Mobile Application Frameworks
 
Top 5 Code Coverage Tools in DevOps
Top 5 Code Coverage Tools in DevOpsTop 5 Code Coverage Tools in DevOps
Top 5 Code Coverage Tools in DevOps
 
How to Guarantee Continuous Value from your Test Automation
How to Guarantee Continuous Value from your Test AutomationHow to Guarantee Continuous Value from your Test Automation
How to Guarantee Continuous Value from your Test Automation
 
Welcome to International Journal of Engineering Research and Development (IJERD)
Welcome to International Journal of Engineering Research and Development (IJERD)Welcome to International Journal of Engineering Research and Development (IJERD)
Welcome to International Journal of Engineering Research and Development (IJERD)
 
Machine Learning Meets Quantitative Planning: Enabling Self-Adaptation in Aut...
Machine Learning Meets Quantitative Planning: Enabling Self-Adaptation in Aut...Machine Learning Meets Quantitative Planning: Enabling Self-Adaptation in Aut...
Machine Learning Meets Quantitative Planning: Enabling Self-Adaptation in Aut...
 
CS640_Thesis_AbulAalaAlamsBari13250849
CS640_Thesis_AbulAalaAlamsBari13250849CS640_Thesis_AbulAalaAlamsBari13250849
CS640_Thesis_AbulAalaAlamsBari13250849
 
Automated requirements based testing for ISO 26262
Automated requirements based testing for ISO 26262 Automated requirements based testing for ISO 26262
Automated requirements based testing for ISO 26262
 
How to get the most out of your CI/CD workflow using automated testing - Sauc...
How to get the most out of your CI/CD workflow using automated testing - Sauc...How to get the most out of your CI/CD workflow using automated testing - Sauc...
How to get the most out of your CI/CD workflow using automated testing - Sauc...
 

More from Martin Chapman

Principles of Health Informatics: Artificial intelligence and machine learning
Principles of Health Informatics: Artificial intelligence and machine learningPrinciples of Health Informatics: Artificial intelligence and machine learning
Principles of Health Informatics: Artificial intelligence and machine learningMartin Chapman
 
Principles of Health Informatics: Clinical decision support systems
Principles of Health Informatics: Clinical decision support systemsPrinciples of Health Informatics: Clinical decision support systems
Principles of Health Informatics: Clinical decision support systemsMartin Chapman
 
Mechanisms for Integrating Real Data into Search Game Simulations: An Applica...
Mechanisms for Integrating Real Data into Search Game Simulations: An Applica...Mechanisms for Integrating Real Data into Search Game Simulations: An Applica...
Mechanisms for Integrating Real Data into Search Game Simulations: An Applica...Martin Chapman
 
Scalable architectures for phenotype libraries
Scalable architectures for phenotype librariesScalable architectures for phenotype libraries
Scalable architectures for phenotype librariesMartin Chapman
 
Using AI to understand how preventative interventions can improve the health ...
Using AI to understand how preventative interventions can improve the health ...Using AI to understand how preventative interventions can improve the health ...
Using AI to understand how preventative interventions can improve the health ...Martin Chapman
 
Using AI to autonomously identify diseases within groups of patients
Using AI to autonomously identify diseases within groups of patientsUsing AI to autonomously identify diseases within groups of patients
Using AI to autonomously identify diseases within groups of patientsMartin Chapman
 
Using AI to understand how preventative interventions can improve the health ...
Using AI to understand how preventative interventions can improve the health ...Using AI to understand how preventative interventions can improve the health ...
Using AI to understand how preventative interventions can improve the health ...Martin Chapman
 
Principles of Health Informatics: Evaluating medical software
Principles of Health Informatics: Evaluating medical softwarePrinciples of Health Informatics: Evaluating medical software
Principles of Health Informatics: Evaluating medical softwareMartin Chapman
 
Principles of Health Informatics: Usability of medical software
Principles of Health Informatics: Usability of medical softwarePrinciples of Health Informatics: Usability of medical software
Principles of Health Informatics: Usability of medical softwareMartin Chapman
 
Principles of Health Informatics: Social networks, telehealth, and mobile health
Principles of Health Informatics: Social networks, telehealth, and mobile healthPrinciples of Health Informatics: Social networks, telehealth, and mobile health
Principles of Health Informatics: Social networks, telehealth, and mobile healthMartin Chapman
 
Principles of Health Informatics: Communication systems in healthcare
Principles of Health Informatics: Communication systems in healthcarePrinciples of Health Informatics: Communication systems in healthcare
Principles of Health Informatics: Communication systems in healthcareMartin Chapman
 
Principles of Health Informatics: Terminologies and classification systems
Principles of Health Informatics: Terminologies and classification systemsPrinciples of Health Informatics: Terminologies and classification systems
Principles of Health Informatics: Terminologies and classification systemsMartin Chapman
 
Principles of Health Informatics: Representing medical knowledge
Principles of Health Informatics: Representing medical knowledgePrinciples of Health Informatics: Representing medical knowledge
Principles of Health Informatics: Representing medical knowledgeMartin Chapman
 
Principles of Health Informatics: Informatics skills - searching and making d...
Principles of Health Informatics: Informatics skills - searching and making d...Principles of Health Informatics: Informatics skills - searching and making d...
Principles of Health Informatics: Informatics skills - searching and making d...Martin Chapman
 
Principles of Health Informatics: Informatics skills - communicating, structu...
Principles of Health Informatics: Informatics skills - communicating, structu...Principles of Health Informatics: Informatics skills - communicating, structu...
Principles of Health Informatics: Informatics skills - communicating, structu...Martin Chapman
 
Principles of Health Informatics: Models, information, and information systems
Principles of Health Informatics: Models, information, and information systemsPrinciples of Health Informatics: Models, information, and information systems
Principles of Health Informatics: Models, information, and information systemsMartin Chapman
 
Using AI to understand how preventative interventions can improve the health ...
Using AI to understand how preventative interventions can improve the health ...Using AI to understand how preventative interventions can improve the health ...
Using AI to understand how preventative interventions can improve the health ...Martin Chapman
 
Using Microservices to Design Patient-facing Research Software
Using Microservices to Design Patient-facing Research SoftwareUsing Microservices to Design Patient-facing Research Software
Using Microservices to Design Patient-facing Research SoftwareMartin Chapman
 
Using CWL to support EHR-based phenotyping
Using CWL to support EHR-based phenotypingUsing CWL to support EHR-based phenotyping
Using CWL to support EHR-based phenotypingMartin Chapman
 
Phenoflow: An Architecture for Computable Phenotypes
Phenoflow: An Architecture for Computable PhenotypesPhenoflow: An Architecture for Computable Phenotypes
Phenoflow: An Architecture for Computable PhenotypesMartin Chapman
 

More from Martin Chapman (20)

Principles of Health Informatics: Artificial intelligence and machine learning
Principles of Health Informatics: Artificial intelligence and machine learningPrinciples of Health Informatics: Artificial intelligence and machine learning
Principles of Health Informatics: Artificial intelligence and machine learning
 
Principles of Health Informatics: Clinical decision support systems
Principles of Health Informatics: Clinical decision support systemsPrinciples of Health Informatics: Clinical decision support systems
Principles of Health Informatics: Clinical decision support systems
 
Mechanisms for Integrating Real Data into Search Game Simulations: An Applica...
Mechanisms for Integrating Real Data into Search Game Simulations: An Applica...Mechanisms for Integrating Real Data into Search Game Simulations: An Applica...
Mechanisms for Integrating Real Data into Search Game Simulations: An Applica...
 
Scalable architectures for phenotype libraries
Scalable architectures for phenotype librariesScalable architectures for phenotype libraries
Scalable architectures for phenotype libraries
 
Using AI to understand how preventative interventions can improve the health ...
Using AI to understand how preventative interventions can improve the health ...Using AI to understand how preventative interventions can improve the health ...
Using AI to understand how preventative interventions can improve the health ...
 
Using AI to autonomously identify diseases within groups of patients
Using AI to autonomously identify diseases within groups of patientsUsing AI to autonomously identify diseases within groups of patients
Using AI to autonomously identify diseases within groups of patients
 
Using AI to understand how preventative interventions can improve the health ...
Using AI to understand how preventative interventions can improve the health ...Using AI to understand how preventative interventions can improve the health ...
Using AI to understand how preventative interventions can improve the health ...
 
Principles of Health Informatics: Evaluating medical software
Principles of Health Informatics: Evaluating medical softwarePrinciples of Health Informatics: Evaluating medical software
Principles of Health Informatics: Evaluating medical software
 
Principles of Health Informatics: Usability of medical software
Principles of Health Informatics: Usability of medical softwarePrinciples of Health Informatics: Usability of medical software
Principles of Health Informatics: Usability of medical software
 
Principles of Health Informatics: Social networks, telehealth, and mobile health
Principles of Health Informatics: Social networks, telehealth, and mobile healthPrinciples of Health Informatics: Social networks, telehealth, and mobile health
Principles of Health Informatics: Social networks, telehealth, and mobile health
 
Principles of Health Informatics: Communication systems in healthcare
Principles of Health Informatics: Communication systems in healthcarePrinciples of Health Informatics: Communication systems in healthcare
Principles of Health Informatics: Communication systems in healthcare
 
Principles of Health Informatics: Terminologies and classification systems
Principles of Health Informatics: Terminologies and classification systemsPrinciples of Health Informatics: Terminologies and classification systems
Principles of Health Informatics: Terminologies and classification systems
 
Principles of Health Informatics: Representing medical knowledge
Principles of Health Informatics: Representing medical knowledgePrinciples of Health Informatics: Representing medical knowledge
Principles of Health Informatics: Representing medical knowledge
 
Principles of Health Informatics: Informatics skills - searching and making d...
Principles of Health Informatics: Informatics skills - searching and making d...Principles of Health Informatics: Informatics skills - searching and making d...
Principles of Health Informatics: Informatics skills - searching and making d...
 
Principles of Health Informatics: Informatics skills - communicating, structu...
Principles of Health Informatics: Informatics skills - communicating, structu...Principles of Health Informatics: Informatics skills - communicating, structu...
Principles of Health Informatics: Informatics skills - communicating, structu...
 
Principles of Health Informatics: Models, information, and information systems
Principles of Health Informatics: Models, information, and information systemsPrinciples of Health Informatics: Models, information, and information systems
Principles of Health Informatics: Models, information, and information systems
 
Using AI to understand how preventative interventions can improve the health ...
Using AI to understand how preventative interventions can improve the health ...Using AI to understand how preventative interventions can improve the health ...
Using AI to understand how preventative interventions can improve the health ...
 
Using Microservices to Design Patient-facing Research Software
Using Microservices to Design Patient-facing Research SoftwareUsing Microservices to Design Patient-facing Research Software
Using Microservices to Design Patient-facing Research Software
 
Using CWL to support EHR-based phenotyping
Using CWL to support EHR-based phenotypingUsing CWL to support EHR-based phenotyping
Using CWL to support EHR-based phenotyping
 
Phenoflow: An Architecture for Computable Phenotypes
Phenoflow: An Architecture for Computable PhenotypesPhenoflow: An Architecture for Computable Phenotypes
Phenoflow: An Architecture for Computable Phenotypes
 

Recently uploaded

How to Make a Duplicate of Your Odoo 17 Database
How to Make a Duplicate of Your Odoo 17 DatabaseHow to Make a Duplicate of Your Odoo 17 Database
How to Make a Duplicate of Your Odoo 17 DatabaseCeline George
 
Beauty Amidst the Bytes_ Unearthing Unexpected Advantages of the Digital Wast...
Beauty Amidst the Bytes_ Unearthing Unexpected Advantages of the Digital Wast...Beauty Amidst the Bytes_ Unearthing Unexpected Advantages of the Digital Wast...
Beauty Amidst the Bytes_ Unearthing Unexpected Advantages of the Digital Wast...DhatriParmar
 
MS4 level being good citizen -imperative- (1) (1).pdf
MS4 level   being good citizen -imperative- (1) (1).pdfMS4 level   being good citizen -imperative- (1) (1).pdf
MS4 level being good citizen -imperative- (1) (1).pdfMr Bounab Samir
 
Objectives n learning outcoms - MD 20240404.pptx
Objectives n learning outcoms - MD 20240404.pptxObjectives n learning outcoms - MD 20240404.pptx
Objectives n learning outcoms - MD 20240404.pptxMadhavi Dharankar
 
ESP 4-EDITED.pdfmmcncncncmcmmnmnmncnmncmnnjvnnv
ESP 4-EDITED.pdfmmcncncncmcmmnmnmncnmncmnnjvnnvESP 4-EDITED.pdfmmcncncncmcmmnmnmncnmncmnnjvnnv
ESP 4-EDITED.pdfmmcncncncmcmmnmnmncnmncmnnjvnnvRicaMaeCastro1
 
31 ĐỀ THI THỬ VÀO LỚP 10 - TIẾNG ANH - FORM MỚI 2025 - 40 CÂU HỎI - BÙI VĂN V...
31 ĐỀ THI THỬ VÀO LỚP 10 - TIẾNG ANH - FORM MỚI 2025 - 40 CÂU HỎI - BÙI VĂN V...31 ĐỀ THI THỬ VÀO LỚP 10 - TIẾNG ANH - FORM MỚI 2025 - 40 CÂU HỎI - BÙI VĂN V...
31 ĐỀ THI THỬ VÀO LỚP 10 - TIẾNG ANH - FORM MỚI 2025 - 40 CÂU HỎI - BÙI VĂN V...Nguyen Thanh Tu Collection
 
BIOCHEMISTRY-CARBOHYDRATE METABOLISM CHAPTER 2.pptx
BIOCHEMISTRY-CARBOHYDRATE METABOLISM CHAPTER 2.pptxBIOCHEMISTRY-CARBOHYDRATE METABOLISM CHAPTER 2.pptx
BIOCHEMISTRY-CARBOHYDRATE METABOLISM CHAPTER 2.pptxSayali Powar
 
Q-Factor HISPOL Quiz-6th April 2024, Quiz Club NITW
Q-Factor HISPOL Quiz-6th April 2024, Quiz Club NITWQ-Factor HISPOL Quiz-6th April 2024, Quiz Club NITW
Q-Factor HISPOL Quiz-6th April 2024, Quiz Club NITWQuiz Club NITW
 
4.9.24 Social Capital and Social Exclusion.pptx
4.9.24 Social Capital and Social Exclusion.pptx4.9.24 Social Capital and Social Exclusion.pptx
4.9.24 Social Capital and Social Exclusion.pptxmary850239
 
Integumentary System SMP B. Pharm Sem I.ppt
Integumentary System SMP B. Pharm Sem I.pptIntegumentary System SMP B. Pharm Sem I.ppt
Integumentary System SMP B. Pharm Sem I.pptshraddhaparab530
 
Decoding the Tweet _ Practical Criticism in the Age of Hashtag.pptx
Decoding the Tweet _ Practical Criticism in the Age of Hashtag.pptxDecoding the Tweet _ Practical Criticism in the Age of Hashtag.pptx
Decoding the Tweet _ Practical Criticism in the Age of Hashtag.pptxDhatriParmar
 
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...Nguyen Thanh Tu Collection
 
Oppenheimer Film Discussion for Philosophy and Film
Oppenheimer Film Discussion for Philosophy and FilmOppenheimer Film Discussion for Philosophy and Film
Oppenheimer Film Discussion for Philosophy and FilmStan Meyer
 
Tree View Decoration Attribute in the Odoo 17
Tree View Decoration Attribute in the Odoo 17Tree View Decoration Attribute in the Odoo 17
Tree View Decoration Attribute in the Odoo 17Celine George
 
6 ways Samsung’s Interactive Display powered by Android changes the classroom
6 ways Samsung’s Interactive Display powered by Android changes the classroom6 ways Samsung’s Interactive Display powered by Android changes the classroom
6 ways Samsung’s Interactive Display powered by Android changes the classroomSamsung Business USA
 
ARTERIAL BLOOD GAS ANALYSIS........pptx
ARTERIAL BLOOD  GAS ANALYSIS........pptxARTERIAL BLOOD  GAS ANALYSIS........pptx
ARTERIAL BLOOD GAS ANALYSIS........pptxAneriPatwari
 

Recently uploaded (20)

How to Make a Duplicate of Your Odoo 17 Database
How to Make a Duplicate of Your Odoo 17 DatabaseHow to Make a Duplicate of Your Odoo 17 Database
How to Make a Duplicate of Your Odoo 17 Database
 
Beauty Amidst the Bytes_ Unearthing Unexpected Advantages of the Digital Wast...
Beauty Amidst the Bytes_ Unearthing Unexpected Advantages of the Digital Wast...Beauty Amidst the Bytes_ Unearthing Unexpected Advantages of the Digital Wast...
Beauty Amidst the Bytes_ Unearthing Unexpected Advantages of the Digital Wast...
 
MS4 level being good citizen -imperative- (1) (1).pdf
MS4 level   being good citizen -imperative- (1) (1).pdfMS4 level   being good citizen -imperative- (1) (1).pdf
MS4 level being good citizen -imperative- (1) (1).pdf
 
Faculty Profile prashantha K EEE dept Sri Sairam college of Engineering
Faculty Profile prashantha K EEE dept Sri Sairam college of EngineeringFaculty Profile prashantha K EEE dept Sri Sairam college of Engineering
Faculty Profile prashantha K EEE dept Sri Sairam college of Engineering
 
Objectives n learning outcoms - MD 20240404.pptx
Objectives n learning outcoms - MD 20240404.pptxObjectives n learning outcoms - MD 20240404.pptx
Objectives n learning outcoms - MD 20240404.pptx
 
ESP 4-EDITED.pdfmmcncncncmcmmnmnmncnmncmnnjvnnv
ESP 4-EDITED.pdfmmcncncncmcmmnmnmncnmncmnnjvnnvESP 4-EDITED.pdfmmcncncncmcmmnmnmncnmncmnnjvnnv
ESP 4-EDITED.pdfmmcncncncmcmmnmnmncnmncmnnjvnnv
 
31 ĐỀ THI THỬ VÀO LỚP 10 - TIẾNG ANH - FORM MỚI 2025 - 40 CÂU HỎI - BÙI VĂN V...
31 ĐỀ THI THỬ VÀO LỚP 10 - TIẾNG ANH - FORM MỚI 2025 - 40 CÂU HỎI - BÙI VĂN V...31 ĐỀ THI THỬ VÀO LỚP 10 - TIẾNG ANH - FORM MỚI 2025 - 40 CÂU HỎI - BÙI VĂN V...
31 ĐỀ THI THỬ VÀO LỚP 10 - TIẾNG ANH - FORM MỚI 2025 - 40 CÂU HỎI - BÙI VĂN V...
 
BIOCHEMISTRY-CARBOHYDRATE METABOLISM CHAPTER 2.pptx
BIOCHEMISTRY-CARBOHYDRATE METABOLISM CHAPTER 2.pptxBIOCHEMISTRY-CARBOHYDRATE METABOLISM CHAPTER 2.pptx
BIOCHEMISTRY-CARBOHYDRATE METABOLISM CHAPTER 2.pptx
 
Q-Factor HISPOL Quiz-6th April 2024, Quiz Club NITW
Q-Factor HISPOL Quiz-6th April 2024, Quiz Club NITWQ-Factor HISPOL Quiz-6th April 2024, Quiz Club NITW
Q-Factor HISPOL Quiz-6th April 2024, Quiz Club NITW
 
4.9.24 Social Capital and Social Exclusion.pptx
4.9.24 Social Capital and Social Exclusion.pptx4.9.24 Social Capital and Social Exclusion.pptx
4.9.24 Social Capital and Social Exclusion.pptx
 
prashanth updated resume 2024 for Teaching Profession
prashanth updated resume 2024 for Teaching Professionprashanth updated resume 2024 for Teaching Profession
prashanth updated resume 2024 for Teaching Profession
 
Mattingly "AI & Prompt Design" - Introduction to Machine Learning"
Mattingly "AI & Prompt Design" - Introduction to Machine Learning"Mattingly "AI & Prompt Design" - Introduction to Machine Learning"
Mattingly "AI & Prompt Design" - Introduction to Machine Learning"
 
Integumentary System SMP B. Pharm Sem I.ppt
Integumentary System SMP B. Pharm Sem I.pptIntegumentary System SMP B. Pharm Sem I.ppt
Integumentary System SMP B. Pharm Sem I.ppt
 
Plagiarism,forms,understand about plagiarism,avoid plagiarism,key significanc...
Plagiarism,forms,understand about plagiarism,avoid plagiarism,key significanc...Plagiarism,forms,understand about plagiarism,avoid plagiarism,key significanc...
Plagiarism,forms,understand about plagiarism,avoid plagiarism,key significanc...
 
Decoding the Tweet _ Practical Criticism in the Age of Hashtag.pptx
Decoding the Tweet _ Practical Criticism in the Age of Hashtag.pptxDecoding the Tweet _ Practical Criticism in the Age of Hashtag.pptx
Decoding the Tweet _ Practical Criticism in the Age of Hashtag.pptx
 
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
 
Oppenheimer Film Discussion for Philosophy and Film
Oppenheimer Film Discussion for Philosophy and FilmOppenheimer Film Discussion for Philosophy and Film
Oppenheimer Film Discussion for Philosophy and Film
 
Tree View Decoration Attribute in the Odoo 17
Tree View Decoration Attribute in the Odoo 17Tree View Decoration Attribute in the Odoo 17
Tree View Decoration Attribute in the Odoo 17
 
6 ways Samsung’s Interactive Display powered by Android changes the classroom
6 ways Samsung’s Interactive Display powered by Android changes the classroom6 ways Samsung’s Interactive Display powered by Android changes the classroom
6 ways Samsung’s Interactive Display powered by Android changes the classroom
 
ARTERIAL BLOOD GAS ANALYSIS........pptx
ARTERIAL BLOOD  GAS ANALYSIS........pptxARTERIAL BLOOD  GAS ANALYSIS........pptx
ARTERIAL BLOOD GAS ANALYSIS........pptx
 

Technical Validation through Automated Testing

  • 1. Technical Validation through Automated Testing Computational Phenotyping and Software Engineering S36 Martin Chapman King’s College London #IS24 AMIA 2024 Informatics Summit | amia.org 1
  • 2. 1. Background AMIA 2024 Informatics Summit | amia.org 2
  • 3. Automated testing Floor 1 Æ ¥ r AMIA 2024 Informatics Summit | amia.org 3
  • 4. Automated testing Floor 1 Æ ¥ r Floor 2 Æ ¥ AMIA 2024 Informatics Summit | amia.org 3
  • 5. Automated testing Floor 1 Æ ¥ r Floor 2 Æ ¥ Floor 3 Æ ¥ W Æ q AMIA 2024 Informatics Summit | amia.org 3
  • 6. Automated testing Floor 1 Æ ¥ r Floor 2 Æ ¥ Floor 3 Æ ¥ W Æ q W Æ q AMIA 2024 Informatics Summit | amia.org 3
  • 7. Automated testing Floor 1 Æ ¥ r Floor 2 Æ ¥ Floor 3 Æ ¥ https://github.com/sindresorhus/got AMIA 2024 Informatics Summit | amia.org 3
  • 8. Unit tests vs. integration tests W Æ q W Æ q Floor 1 Æ ¥ Floor 2 Æ ¥ Floor 3 Æ ¥ r AMIA 2024 Informatics Summit | amia.org 4
  • 9. Unit tests vs. integration tests W Æ q W Æ q Floor 1 Æ ¥ Floor 2 Æ ¥ Floor 3 Æ ¥ r AMIA 2024 Informatics Summit | amia.org 4
  • 10. Unit tests vs. integration tests W Æ q W Æ q Floor 1 Æ ¥ Floor 2 Æ ¥ Floor 3 Æ ¥ r Æ q AMIA 2024 Informatics Summit | amia.org 4
  • 11. Unit tests vs. integration tests W Æ q W Æ q Floor 1 Æ ¥ Floor 2 Æ ¥ Floor 3 Æ ¥ r Æ q https://github.com/mattermost/mattermost AMIA 2024 Informatics Summit | amia.org 4
  • 12. Automated testing and Health Informatics https://github.com/OHDSI/CommonDataModel AMIA 2024 Informatics Summit | amia.org 5
  • 13. 2. Automated testing and phenotyping AMIA 2024 Informatics Summit | amia.org 6
  • 14. The phenotype development lifecycle Health informatics applications, libraries and packages that are tested are more likely to operate correctly. We want the same to be true of phenotypes. There are several places in the phenotype development lifecycle where we can apply testing tools: Author 1. Modelling languages 2. Multi- dimensional descriptions 4. Modular relationships Refine 3. Versioning and provenance 13. Comprehensive metadata Implement 5. Implementation details in model 6. Prog. language agnostic 7. Multiple data formats/standards Validate 8. Defined validation process 9. Automate multiple validation techniques 10. Enable feedback Publish 11. Expose standard API 12. Advanced search capabilities AMIA 2024 Informatics Summit | amia.org 7
  • 15. Running example: CQL phenotypes Pascal S. Brandt, Jennifer A. Pacheco, and Luke V. Rasmussen. “Development of a repository of computable phenotype definitions using the clinical quality language”. In: JAMIA Open (2021) AMIA 2024 Informatics Summit | amia.org 8
  • 16. 1. Testing during phenotype authoring We’ve seen that there is a necessary difference between a phenotype definition (abstract logic; generalisable) and a computable phenotype (programming code; specific). AMIA 2024 Informatics Summit | amia.org 9
  • 17. 1. Testing during phenotype authoring Phenotype definition: Computable phenotype: AMIA 2024 Informatics Summit | amia.org 9
  • 18. 1. Testing during phenotype authoring We’ve seen that there is a necessary difference between a phenotype definition (abstract logic; generalisable) and a computable phenotype (programming code; specific). This does not, however, mean that phenotype definitions cannot be computable. If they are, we can apply testing tools to this abstract logic as another measure of correctness. In our running example, a CQL phenotype definition can itself be tested using the CQL Testing Framework (CTF) and YAML-based test definitions. AMIA 2024 Informatics Summit | amia.org 9
  • 19. 1. Testing during phenotype authoring Phenotype definition: Computable phenotype: AMIA 2024 Informatics Summit | amia.org 9
  • 20. 1. Testing during phenotype authoring We’ve seen that there is a necessary difference between a phenotype definition (abstract logic; generalisable) and a computable phenotype (programming code; specific). This does not, however, mean that phenotype definitions cannot be computable. If they are, we can apply testing tools to this abstract logic as another measure of correctness. In our running example, a CQL phenotype definition can itself be tested using the CQL Testing Framework (CTF) and YAML-based test definitions. The application of unit tests during phenotype authoring is likely to be particularly effective, as phenotype definitions are often designed as distinct units of logic. AMIA 2024 Informatics Summit | amia.org 9
  • 21. 1. Testing during phenotype authoring https://github.com/PheMA/phekb-phenotypes AMIA 2024 Informatics Summit | amia.org 9
  • 22. 2. Testing during phenotype implementation Phenotype definition: Computable phenotype: AMIA 2024 Informatics Summit | amia.org 10
  • 23. 2. Testing during phenotype implementation Testing tools can be applied during phenotype implementation in much the same way testing tools are applied during general software de- velopment. For example, CQL connects with the popular Mocha Javascript testing framework. At this level, the tests we see are more akin to inte- gration tests. https://github.com/cqframework/cql-execution AMIA 2024 Informatics Summit | amia.org 10
  • 24. 3. Testing during phenotype validation Recall that the reason we are motivated to apply (automated) testing tools to the development of phenotypes is to ensure they are operating correctly. Phenotype validation has the same goal: to determine whether a given phenotype can correctly identify which individuals exhibit the true phenotype and which do not. Therefore, testing can be applied and used as an additional form of validation. We term this technical validation. Practically, technical validation is likely to manifest as writing and executing test cases for existing phenotypes. , Technical validation and phenotype validation are still very distinct processes. AMIA 2024 Informatics Summit | amia.org 11
  • 25. Summary • Automated testing tools, which include both unit and integration testing, are commonplace in software development, including software developed in the Health Informatics domain. • However, they are relatively underused in the development of phenotypes. • We identify huge potential for automated testing tools to be applied to phenotype authoring (running testing tools against computable definitions), phenotype implementation (to test whole definitions) and phenotype validation (as an additional way to assess the correctness of a definition). AMIA 2024 Informatics Summit | amia.org 12
  • 26. Thank you! AMIA 2024 Informatics Summit | amia.org 13