The document describes a study using explainable boosting machines (EBMs) to model variant calling accuracy as a function of genomic context. The goals are to understand sequencing errors to enable more precise benchmarking and to predict which variant types and contexts a variant caller may miss. The EBMs are trained on true and false variant calls compared to ground truth data. The models show genomic features like homopolymer length that increase the likelihood of incorrect variant calling between PCR-free and PCR-plus sequencing. The models also predict variants likely to be missed in comparisons between variant calling pipelines and the ground truth data.

1. Model results: SNVs and INDELs
• Despite advances in sequencing technology, correctly identifying complex variation in the
human genome remains challenging
• Challenging variants have general characteristics, but we lack a data-driven model to
quantifiably link these characteristics with likelihood of successful identification
• We are using explainable boosting machines (EBMs) to model variant calling accuracy as a
function of genomic context, with specific goals to:
• understand sequencing errors to enable more precise stratifications when benchmarking
• predict which types of variants and genome contexts a variant caller might miss
Introduction
Generating Features from Genomic Regions
StratoMod: Using Machine Learning Models to Understand
Errors in Human Genomic Variant Calling
N. Dwarshuis1
, J. Wagner1
, N. Olson1
, J. McDaniel1
, P. Tonner2
, F Sedlazeck3
, J.M. Zook1
1) Material Measurement Laboratory, National Institute of Standards and Technology, Gaithersburg, MD
2) Information Technology Laboratory, National Institute of Standards and Technology, Gaithersburg, MD
3) Human Genome Sequencing Center, Baylor College of Medicine, Houston, TX, USA
Analysis Pipeline Overview
Genome in a Bottle
Consortium
Explainable Boosting Machines
Transparency
Accuracy
𝑔( 𝑦 )= 𝑓 1 (𝑥1 )+ 𝑓 2 (𝑥2)+…+ 𝑓 1, 2 (𝑥1 , 𝑥2 )+…
g: logit link function
fk: decision tree with feature k
EBM: GAM with
interaction terms
 Flexibility of blackbox models
(random forest, XGBoost)
 Interpretability of simpler
models
 Efficiently fits 2nd
order terms
Nori et al, arXiv. 2019
Lou et al. SIGKDD Proc. 2012
github.com/interpretml/interpret
region y
1 TP
2 TP
3 FP
4 TP
…
region REF ALT
0 A AAT
1 C G
2 G GG
3 CG G
…
region REF ALT
0 A AAT
1 C G
2 CG G
…
region y x1 x2 x3
1 TP
2 TP
3 FP
4 TP
…
region x1 x2 x3
1
2
3
4
…
Benchmark variants
(ground truth)
Comparison variants
Compare
variants
Variant labels
w/ regions
Join
features
and labels
by region
Genomic
Region Features
region x1 x2 x3
1
2
3
4
…
region x1 x2 x3
1
2
3
4
…
Fit EBM on
y vs x’s
Interpretation
Labels
TP: variant in both benchmark and comparison
FP: variant only in comparison (error)
Model Fit
Shows (transparently) the genomic contexts that
are likely to lead to incorrect variant calling
Feature Class Description
Segmental Duplications Highly similar sequences >1000b repeated throughout genome
Tandem Repeats Short repeated patterns (eg TATATA)
Transposable Elements Sequences that copy themselves to other locations
Other features included:
In general, variants in highly repetitive regions are hard to identify correctly
Example: imperfect homopolymers – regions (almost
entirely) of just one base repeated
AAAAAAAAAAAAA
GGGGGGGGGCGGGGGGGGGGGG
CCCCCCCCCCCCCCCCCCC
Use length of
these regions
as a feature
Funding: NIST Use-Inspired AI Program (FARSAIT)
BioRxiv preprint:
https://doi.org/10.1101/2023.01.20.524401
Contact: njd2@nist.gov
• Parameters: 19 main effects, 18
interactions (all features crossed with
PCR-free vs PCR-plus)
• Trained on false positives (FPs) in
unfiltered DeepVariant candidate
calls vs true positives (TPs)
• FP errors are overall less likely for PCR-Free
• The length at which FP error rate exceeds
baseline (dotted line) is longer in PCR-Free
• FP errors are equally likely1
in both technologies
• FP errors are likely in A/T and G/C
homopolymers >15 and any length respectively
INDELs
SNVs
Use case 1: assessing FP errors in PCR-free/plus Illumina sequencing methods
• Parameters: 17 main effects, 16
interactions with vcf input file to
compare pipelines
• Modeled missed/incorrectly
filtered variants by training on
false negatives (FNs) in final
DeepVariant callsets vs TPs
Use case 2: assessing FN error likelihood in HiFi and Illumina analysis pipelines
1. “score” on each y-axis is in logistic space, and each plot is the sum of the indicated feature, its interaction with VCF input, and the VCF input main effect
2. Difficult to map regions defined from GIAB v3.0 stratifications
INDELs
SNVs
Performance was consistent across
genomic samples
• DeepVariant/Benchmark had more overlap than
StratoMod/BenchMark
• StratoMod/DeepVariant had significant non-
benchmark overlap
• Most ClinVar variants that were predicted to
be missed were in hard-to-map regions
• This has an additional advantage of also
identifying difficult variants in large INDELs
and tandem repeats
Element shows advantage for longer AT
homopolymers
Linear Model y = B1x1 + B2x2 + ...
Generalized
Linear Model
g(y) = B1x1 + B2x2 + ...
Additive Model y = f1(x1) + f2(x2) + ...
Generalized
Additive Model
g(y) = f1(x1) + f2(x2) + ...
Full Complexity
Model
y = f(x1, x2, ...)
Shared (p(FN) > 0.9)
Non-Shared(p(FN) > 0.9)