This document summarizes an evaluation of structural variant calls made by Pacific Biosciences (pbsv) compared to the Genome in a Bottle (GIAB) v0.5.0 call set for the HG002 sample. The evaluation found that the GIAB call set meets a high standard for specificity, with most putative false positives from pbsv found to be actually true by PacBio read evidence. However, the GIAB call set's sensitivity does not yet meet this same standard, with many putative false negatives from pbsv supported by PacBio reads. Areas of lower sensitivity were extended tandem repeats and biallelic loci. Next steps include understanding why some pbsv calls were not included in GIAB, and

1. For Research Use Only. Not for use in diagnostics procedures. © Copyright 2018 by Pacific Biosciences of California, Inc. All rights reserved.
Pacific Biosciences, the Pacific Biosciences logo, PacBio, SMRT, SMRTbell, Iso-Seq, and Sequel are trademarks of Pacific Biosciences. All other trademarks are the sole property of their respective owners.
Comparison of the GIAB v0.5.0 structural
variant callset to one “decent” caller
Aaron Wenger 2018-01-25

2. HG002 VARIANT CALLS
PASS variant calls with |SVLEN| ≥50 bp where HG002 GT has an ALT allele
0
500
1,000
1,500
2,000
2,500
3,000
3,500
variantcalls
variant size (bp)
0
50
100
150
200
250
300
350
400
450
500
variantcalls
variant size (bp)
Deletions (7,887) Insertions (11,112)

3. QUALITY STANDARD FOR A GIAB BENCHMARK SET
most putative false
negatives are truly
false negatives
GIAB benchmark call set
most putative false
positives are truly
false positives
Zook Standard When the GIAB benchmark and call
set differ, the GIAB benchmark is usually correct.

4. COMPARISON OF GIAB V0.5.0 AND PBSV (SURVIVOR +/- 500BP)
PASS variant calls with |SVLEN| ≥50 bp where HG002 GT has an ALT allele compared against pbsv calls using SURVIVOR +/- 500 bp
GIAB v0.5.0 pbsv
15,033 4,3721,577
SURVIVOR merges nearby variant calls, which reduces the total counts

5. COMPARISON OF GIAB V0.5.0 AND PBSV
comparison with SURVIVOR 5 +/-500 bp; variants <1kb
Deletions Insertions
both
v0.5.0
only
(put. FN)
pbsv
only
(put. FP)

6. COMPARISON OF GIAB V0.5.0 AND PBSV
comparison with SURVIVOR 5 +/-500 bp; variants ≥1kb
Deletions Insertions
both
v0.5.0
only
(put. FN)
pbsv
only
(put. FP)

7. COMPARISON OF GIAB V0.5.0 AND PBSV (SURVIVOR +/- 500BP)
PASS variant calls with |SVLEN| ≥50 bp where HG002 GT has an ALT allele compared against pbsv calls using SURVIVOR +/- 500 bp
GIAB v0.5.0 pbsv
15,033 4,3721,577
SURVIVOR merges nearby variant calls, which reduces the total counts
putative false positives
(sample 20)

8. PUTATIVE FALSE POSITIVE PacBio reads support
heterozygous deletion
pbsv is correct

9. PUTATIVE FALSE POSITIVE pbsv is correct PacBio reads support
heterozygous insertion
(& deletion)

10. COMPARISON OF GIAB V0.5.0 AND PBSV (SURVIVOR +/- 500BP)
PASS variant calls with |SVLEN| ≥50 bp where HG002 GT has an ALT allele compared against pbsv calls using SURVIVOR +/- 500 bp
GIAB v0.5.0 pbsv
15,033 4,3721,577
SURVIVOR merges nearby variant calls, which reduces the total counts
putative false negatives
(sample 20)

11. PUTATIVE FALSE NEGATIVE GIAB is correct
duplicate call
PacBio reads support a
deletion but not
definitively the 54bp one

12. PUTATIVE FALSE NEGATIVE pbsv is correct PacBio reads support
an insertion not a
deletion

13. COMPARISON OF GIAB V0.5.0 AND PBSV (SURVIVOR +/- 500BP)
GIAB v0.5.0 pbsv
15,033 4,3721,577
0% 50% 100%
GIAB correct pbsv correct
0% 50% 100%
GIAB correct pbsv correct
Putative false negatives Putative false positives
Most putative false negatives are false negatives...
But most putative false positives are true positives.

14. DISAGREEMENTS ARE ENRICHED IN EXTENDED TANDEM
REPEATS AND BIALLELIC VARIANT LOCI
GIAB v0.5.0 pbsv
15,033 4,3721,577
0%
25%
50%
75%
100%
Tandem repeat ≥500 bp
0%
25%
50%
75%
100%
Nearby call within 2kb

15. SUMMARY
Evaluation
• GIAB v0.5.0 callset is excellent
• Specificity meets the Zook Standard
• Sensitivity does not yet meet the Zook Standard
• Two challenges to increased sensitivity
1. extended short tandem repeats
2. biallelic loci
Next Steps
• Trace “pbsv correct” calls to understand why they do not
make the final call set
• Manually curate differences between v0.5.0 and a few other
“decent” callers from multiple technologies