1. Standard
Performance
Metrics
for
Gene
Expression
Experiments
with
the
“erccdashboard”
Sarah
Munro

2. Method
ValidaOon
with
erccdashboard
R
package
Feature
A_1
A_2
A_3
B_1
B_2
B_3
T1
1
5
4
10
0
2
3
T2
200
204
199
101
97
103
T3
142
153
147
5
149
130
155
ERCC-­‐0001
5
8
10
20
23
19
0
…
−5
Log2 Normalized ERCC Counts
Ratio
4:1
1:1
1:1.5
1:2
Sample
CTL
MET
erccdashboard Package Vignette
Sarah A. Munro
−10
May 4, 2014
This vignette describes the use of the erccdashboard R package to analyze External RNA Control Con-sortium
−10 −5 0 5 10
Log2 ERCC Spike Amount(attomol nt/μg total RNA)
(ERCC) spike-in control ratio mixtures in gene expression experiments. If you use this package for
method validation of your gene expression experiments please cite our publication:
4
Please cite our paper when you use the erccdashboard
package for analysis. This is a placeholder citation,
because our manuscript is still under review.
3
Munro SA, Lund S, Pine PS, Binder H, Clevert D,
Conesa A, Dopazo J, Fasold M, Hochreiter S, Hong H,
Jafari N, Kreil DP, °A
2
, Aabaj PP, Liao Y, Lin S, Meehan
J, Mason CE, Santoyo J, Setterquist RA, Shi L, Shi
W, Smyth GK, Stralis-Pavese N, Su Z, Tong W, Wang
C, Wang J, Xu J, Ye Z, Yang Y, Yu Y, Salit M (Under
Review, 2014). Assessing Technical Performance in
Gene Expression Experiments with External Spike-in
RNA Control Ratio Mixtures.
A BibTeX entry for LaTeX users is
@Article{,
1.00
0.75
0.50
0.25
1
0
−1
1e+00
1e−01
1e−02
1e−03
1e−04
title = {Assessing Technical Performance in Gene Expression Experiments with External Spike-in RNA Control author = {Munro SA and Lund S and Pine PS and Binder H and Clevert D and Conesa A and Dopazo J and Fasold journal = {Under Review},
volume = {0},
pages = {0},
year = {2014},
}
−2
−3
Munro SA, Lund S, Pine PS, Binder H, Clevert D, Conesa A, Dopazo J, Fasold M, Hochreiter S, Hong H,
Jafari N, Kreil DP, ˘0141abaj PP, Li S, Liao Y, Lin S, Meehan J, Mason CE, Santoyo J, Setterquist RA, Shi
L, Shi W, Smyth GK, Stralis-Pavese N, Su Z, Tong W, Wang C, Wang J, Xu J, Ye Z, Yang Y, Yu Y, Salit
0.00
TPR
log(rm)
Weighted Mean
− 0.07014
(+/−) Weighted Standard Error
0.1495
−4
−10 −5 0 5 10
Log2 Average of Normalized Counts
Log2 Ratio of Normalized Counts
Ratio
4:1
1:1
1:1.5
1:2
1e−05
DE Test P−values
Ratio
4:1:1:• Open-­‐source
R
package
– erccdashboard
• Assess
technical
performance
of
a
gene
expression
experiment
• Compare
results
– Within
a
single
laboratory
– Between
laboratories

3. Method
ValidaOon
with
erccdashboard
R
package
• Open-­‐source
R
package
– erccdashboard
• Assess
technical
performance
of
a
gene
expression
experiment
• Compare
results
– Within
a
single
laboratory
– Between
laboratories

4. ERCC
RaOo
Mixture
Design

5. Example
Experiments
for
erccdashboard
Method
ValidaOon
Universal
Human
Reference
RNA
(UHRR)
vs.
Human
Brain
Reference
RNA
(HBRR)
Library
PreparaOon
Replicates
Microarrays
RNA-­‐Seq

6. Dynamic
Range
4
2
0
−2
Microarray
RNA-­‐Seq
−5 0 5 10
Log2 ERCC Spike Amount(attomol nt/μg total RNA)
Log2 Normalized Fluorescent Intensity
Sample
HBRR
UHRR
Ratio
4:1
1:1
1:1.5
1:2
10
5
0
−5
−10
−10 −5 0 5 10
Log2 ERCC Spike Amount(attomol nt/μg total RNA)
Log2 Normalized ERCC Counts Sample
HBRR
UHRR
Ratio
4:1
1:1
1:1.5
1:2

7. DiagnosOc
Performance
Ratio
4:1
1:1.5
1:2
AUC
0.933
0.929
0.917
Detected
14
14
16
Spiked
23
23
23
1.00
0.75
0.50
0.25
0.00
0.00 0.25 0.50 0.75 1.00
FPR
TPR
Ratio
4:1
1:1.5
1:2
Ratio
4:1
1:1.5
1:2
AUC
0.966
0.826
0.894
Detected
22
22
23
Spiked
23
23
23
1.00
0.75
0.50
0.25
0.00
0.00 0.25 0.50 0.75 1.00
FPR
TPR
Ratio
4:1
1:1.5
1:2
Microarray
RNA-­‐Seq

8. LODR:
Limit
of
DetecOon
of
RaOos
1e+00
1e−01
1e−02
1e−03
1e−04
1e−05
1e−08
1e−10
1e−12
100 1000 10000
Average Fluorescence Intensity
DE Test P−values
Ratio
4:1
1:1
1:1.5
1:2
Ratio
4:1
1:1.5
1:2
LODR Estimate
<150
200
150
90% CI Lower Bound
<150
<160
<130
90% CI Upper Bound
<150
210
160
1e+00
1e−01
1e−02
1e−03
1e−04
1e−05
1e−08
1e−10
1e−12
1e+01 1e+03 1e+05
Average Counts
DE Test P−values
Ratio
4:1
1:1
1:1.5
1:2
Ratio
4:1
1:1.5
1:2
LODR Estimate
21
170
37
90% CI Lower Bound
5.9
72
22
90% CI Upper Bound
27
210
40
Microarray
RNA-­‐Seq

9. RaOo
DetecOon,
Bias,
and
Variability
log(rm)
Weighted Mean
0.207
(+/−) Weighted Standard Error
0.09446
4
3
2
1
0
−1
−2
−3
−4
−2 0 2 4
Log2 Average of Normalized Intensity
Log2 Ratio of Normalized Intensity
Ratio
4:1
1:1
1:1.5
1:2
log(rm)
Weighted Mean
0.2836
(+/−) Weighted Standard Error
0.03764
4
3
2
1
0
−1
−2
−3
−4
−10 −5 0 5 10
Log2 Average of Normalized Counts
Log2 Ratio of Normalized Counts
Ratio
4:1
1:1
1:1.5
1:2
Microarray
RNA-­‐Seq

10. SAME
PLATFORM,
DIFFERENT
LABORATORIES

11. 10
5
0
−5
−10
0 5 10 15 20
Log2 ERCC Spike Amount(attomol nt/μg total RNA)
Log2 Normalized ERCC Counts
Sample
HBRR
UHRR
Ratio
4:1
1:1
1:1.5
1:2
Ratio
4:1
1:1.5
1:2
AUC
0.935
0.849
0.870
Detected
22
23
23
Spiked
23
23
23
1.00
0.75
0.50
0.25
0.00
0.00 0.25 0.50 0.75 1.00
FPR
TPR
Ratio
4:1
1:1.5
1:2
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
● ● ●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
log(rm)
Weighted Mean
0.08596
(+/−) Weighted Standard Error
0.09636
4
3
2
1
0
−1
−2
−3
−4
−10 −5 0 5 10
Log2 Average of Normalized Counts
Log2 Ratio of Normalized Counts
Ratio
4:1
1:1
1:1.5
1:2
●
● ●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
● ●
●
● ● ● ●
●
●
●
● ● ●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
1e+00
1e−01
1e−02
1e−03
1e−04
1e−05
1e−08
1e−10
1e−12
1e+01 1e+03 1e+05
Average Counts
DE Test P−values
Ratio
●
●
●
●
4:1
1:1
1:1.5
1:2
Ratio
4:1
1:1.5
1:2
LODR Estimate
26
130
47
90% CI Lower Bound
9.8
43
26
90% CI Upper Bound
40
160
61
Lab
8

12. 10
5
0
−5
−10
0 5 10 15 20
Log2 ERCC Spike Amount(attomol nt/μg total RNA)
Log2 Normalized ERCC Counts
Sample
HBRR
UHRR
Ratio
4:1
1:1
1:1.5
1:2
Ratio
4:1
1:1.5
1:2
AUC
0.822
0.648
0.842
Detected
23
23
22
Spiked
23
23
23
1.00
0.75
0.50
0.25
0.00
0.00 0.25 0.50 0.75 1.00
FPR
TPR
Ratio
4:1
1:1.5
1:2
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
log(rm)
Weighted Mean
0.2441
(+/−) Weighted Standard Error
0.265
4
3
2
1
0
−1
−2
−3
−4
−10 −5 0 5 10
Log2 Average of Normalized Counts
Log2 Ratio of Normalized Counts
Ratio
4:1
1:1
1:1.5
1:2
●
●
●
●
●
● ●
● ●
●
●
●●
●
●
●
●
●
●
●
●
● ● ●
● ●
●
●
●
●
●
●
●
● ●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
● ●
● ●
●
●
●
●
●
●
1e+00
1e−01
1e−02
1e−03
1e−04
1e−05
1e−08
1e−10
1e−12
1e+01 1e+03 1e+05
Average Counts
DE Test P−values
Ratio
●
●
●
●
4:1
1:1
1:1.5
1:2
Ratio
4:1
1:1.5
1:2
LODR Estimate
29
Inf
65
90% CI Lower Bound
12
NA
20
90% CI Upper Bound
39
NA
230
Lab
7

13. Interlab
study
–
FDA-­‐led
SEQC
Project
and
the
ABRF
RNA-­‐Seq
study
-­‐
Single-­‐round,
three
different
measurement
processes
INTERLABORATORY
STUDY
RATIO
PERFORMANCE
MEASURES

14. AUC
Diagnos/c
performance
LODR
Limit
of
Detec/on
of
Ra/os
mRNA
FracOon
RaOo
Bias
RaOo
Variability

15. Acknowledgements
• NIST
– Marc
Salit
– Jenny
McDaniel
– Maghias
Roesslein
– Steve
Lund
– P.
Scog
Pine
– JusOn
Zook
– David
Duewer
– Margaret
Kline
• Members
of
the
ERCC
• FDA
SEQC
Study
ParOcipants
and
co-­‐
authors
For
informaOon
on
the
erccdashboard
sojware
and
ERCC
2.0
contact:
sarah.munro@nist.gov