1. Technical Validation through Automated Testing
Computational Phenotyping and Software Engineering
S36
Martin Chapman
King’s College London
#IS24
AMIA 2024 Informatics Summit | amia.org 1
14. The phenotype development lifecycle
Health informatics applications, libraries and packages that are tested are more likely to operate
correctly. We want the same to be true of phenotypes.
There are several places in the phenotype development lifecycle where we can apply testing tools:
Author
1. Modelling
languages
2. Multi-
dimensional
descriptions
4. Modular
relationships
Refine
3. Versioning and
provenance
13. Comprehensive
metadata
Implement
5. Implementation
details in model
6. Prog. language
agnostic
7. Multiple data
formats/standards
Validate
8. Defined
validation process
9. Automate
multiple validation
techniques
10. Enable
feedback
Publish
11. Expose
standard API
12. Advanced
search
capabilities
AMIA 2024 Informatics Summit | amia.org 7
15. Running example: CQL phenotypes
Pascal S. Brandt, Jennifer A. Pacheco, and Luke V. Rasmussen. “Development of a repository of
computable phenotype definitions using the clinical quality language”. In: JAMIA Open (2021)
AMIA 2024 Informatics Summit | amia.org 8
16. 1. Testing during phenotype authoring
We’ve seen that there is a necessary difference between a phenotype definition (abstract logic;
generalisable) and a computable phenotype (programming code; specific).
AMIA 2024 Informatics Summit | amia.org 9
18. 1. Testing during phenotype authoring
We’ve seen that there is a necessary difference between a phenotype definition (abstract logic;
generalisable) and a computable phenotype (programming code; specific).
This does not, however, mean that phenotype definitions cannot be computable. If they are, we
can apply testing tools to this abstract logic as another measure of correctness. In our running
example, a CQL phenotype definition can itself be tested using the CQL Testing Framework
(CTF) and YAML-based test definitions.
AMIA 2024 Informatics Summit | amia.org 9
20. 1. Testing during phenotype authoring
We’ve seen that there is a necessary difference between a phenotype definition (abstract logic;
generalisable) and a computable phenotype (programming code; specific).
This does not, however, mean that phenotype definitions cannot be computable. If they are, we
can apply testing tools to this abstract logic as another measure of correctness. In our running
example, a CQL phenotype definition can itself be tested using the CQL Testing Framework
(CTF) and YAML-based test definitions.
The application of unit tests during phenotype authoring is likely to be particularly effective, as
phenotype definitions are often designed as distinct units of logic.
AMIA 2024 Informatics Summit | amia.org 9
23. 2. Testing during phenotype implementation
Testing tools can be applied during phenotype
implementation in much the same way testing
tools are applied during general software de-
velopment.
For example, CQL connects with the popular
Mocha Javascript testing framework. At this
level, the tests we see are more akin to inte-
gration tests.
https://github.com/cqframework/cql-execution
AMIA 2024 Informatics Summit | amia.org 10
24. 3. Testing during phenotype validation
Recall that the reason we are motivated to apply (automated) testing tools to the development of
phenotypes is to ensure they are operating correctly.
Phenotype validation has the same goal: to determine whether a given phenotype can correctly
identify which individuals exhibit the true phenotype and which do not.
Therefore, testing can be applied and used as an additional form of validation. We term this
technical validation. Practically, technical validation is likely to manifest as writing and
executing test cases for existing phenotypes.
,
Technical validation and phenotype validation are still very distinct processes.
AMIA 2024 Informatics Summit | amia.org 11
25. Summary
• Automated testing tools, which include both unit and integration testing, are commonplace in
software development, including software developed in the Health Informatics domain.
• However, they are relatively underused in the development of phenotypes.
• We identify huge potential for automated testing tools to be applied to phenotype authoring
(running testing tools against computable definitions), phenotype implementation (to test
whole definitions) and phenotype validation (as an additional way to assess the correctness
of a definition).
AMIA 2024 Informatics Summit | amia.org 12