SigOpt Research Engineer Michael McCourt and DarwinAI CTO Alexander Wong explain how they used SigOpt and hyperparameter optimization to successfully improve accuracy of detecting COVID-19 cases from chest X-Rays, using the COVID-Net model and the COVIDx open dataset.

1. Detecting COVID Cases with Deep
Learning
Tuesday, June 9, 2020

2. Introducing our panelists
Dr. Alexander Wong
Chief Scientist, DarwinAI
Canada Research Chair, University of
Waterloo
Dr. Michael McCourt
Head of Research, SigOpt

3. SigOpt. Confidential.
Agenda
Problem: Overview of the problem that led to COVID-Net
Design: Trade-offs and choices in COVID-Net development
Tuning: Setup and results of the tuning process
Impact: Real-world and research impact of this project

4. SigOpt. Confidential.
Problem
Overview of the problem that led to COVID-Net

5. Problem: Detecting COVID-19 with chest x-rays
Pneumonia
COVID-19
Source: COVID-Net preprint paper on Arxiv, https://arxiv.org/pdf/2003.09871.pdf

6. SigOpt. Confidential.
Design
Summary of trade-offs and choices in the model development process

7. How it works: Quick introduction to GenSynth
Source: DarwinAI, http://www.darwinai.com

8. Data
Source: COVID-Net preprint paper on Arxiv, https://arxiv.org/pdf/2003.09871.pdf

9. Model
Source: COVID-Net preprint paper on Arxiv, https://arxiv.org/pdf/2003.09871.pdf

10. Results
Source: COVID-Net preprint paper on Arxiv, https://arxiv.org/pdf/2003.09871.pdf

11. Explainability-Driven Analysis
Source: COVID-Net preprint paper on Arxiv, https://arxiv.org/pdf/2003.09871.pdf

12. SigOpt. Confidential.
Tuning
Discussion on the hyperparameter tuning and evaluation process

13. How it works: Quick introduction to SigOpt

14. Data Augmentation
There are only 183 total COVID-19 examples (as of mid-April), but 14000
non-COVID examples.
● To help treat this imbalance, we add a fixed fraction of COVID-19 examples
to each batch during stochastic gradient descent.
● Data augmentation is a necessary element to produce more COVID-19
“knowledge” for our network to learn.
○ Zooming
○ Translation
○ Rotation
○ Brightness
○ Trimming
Source: SigOpt Blog Post, https://sigopt.com/blog/parametrizing-data-augmentation-in-covid-net-development/

15. Tuning COVID-Net
Olivia and Mike met with Linda to learn about the design of COVID-Net.
● In particular, we wanted to learn how parameters of the training process
and data augmentation were chosen.
● The only work that had been done thus far was a manual search.
○ Linda confirmed that the hyperparameters are important, but did not
thoroughly explore the situation.
● We identified 7 parameters to tune in a multimetric experiment.
Data augmentation - 4 parameters
Loss function - 1 parameter
Training process - 2 parameters
Source: SigOpt Blog Post, https://sigopt.com/blog/parametrizing-data-augmentation-in-covid-net-development/

16. Results
Doctors have said that high sensitivity is
preferable to high PPV -- this has the effect of
minimizing the false negatives.
Source: SigOpt Blog Post, https://sigopt.com/blog/parametrizing-data-augmentation-in-covid-net-development/

17. Analysis
Our real value in this project is being able to produce
insights about the relevance of certain parameters to
the success of the model building process.

18. SigOpt. Confidential.
Impact
Discussion on the real-world and research impact of this work

19. SigOpt. Confidential.
Explore the tuning
experiment in SigOpt’s
dashboard
Find the public
SigOpt experiment here
Use SigOpt at no cost in
your academic research
sigopt.com/edu
Contribute labeled x-ray
data to COVID-Net
https://figure1.typeform.com/to/lLrHwv
Read about other exciting
DarwinAI projects
DarwinAI Success Stories

20. Thank you