Comedy of Errors of
Analytical Proportions
Court Sandau, PhD, PChem
Presented at CLRA AGM 2008 on
August 1st, 2008
A word from our sponsors…
www.chemistry-matters.com 2
Consulting Fees: $2,000
www.chemistry-matters.com 3
Contractor Disbursements: $5,000
www.chemistry-matters.com 4
Spending $7,000 for nothing: Priceless
www.chemistry-matters.com 5
There are some things money can’t buy,
for everything else there’s
www.chemistry-matters.com 6
Changing QA-QC in the Industry
• Educational Presentations/Seminars
• Offering Data Quality Services
– Data Quality Manage...
Pre-amble: Setting the scene
www.chemistry-matters.com 8
Who is protecting your liability?
0
20
40
60
80
100
LAB A LAB B LAB C
SoilConcentration(mg/kg)
0
1
2
3
4
5
EnvironmentalLi...
Cost of Environmental Data Quality
• $$$ for further investigation
• $$$ for further cleanup
• $$$ for potential lawsuit –...
The Foundation of All Environmental
Investigations
Litigation
Remediation &
Risk
Assessment
Site Investigation
Phase I, II...
Levels of QA-QC
Litigation
Liability Protection
Quantitative
Qualitative
- Most expensive
- Highest level of QA-QC
- Less ...
Data Flow for Environmental
Investigations
Data Generation and Review
QAPP Analytical requirements sent to lab
Quality Ass...
Data Quality and the Value-Chain
Model
• Planning
• Sampling
• Storage/Shipping
• Analyses/Methodology
• Laboratory Qualit...
Tiered Approach to Data Quality
Validation
- Recalculations
- Complete data package review
- Errors measured
Preliminary D...
What is QA-QC?
www.chemistry-matters.com 16
Definitions
• Data quality Summary of all the features that
make the values appropriate for the intended use
• Quality ass...
QA-QC
• QA – is the “ISO process”
• QA documents:
– how samples taken
– how they are handled
– how they are to be analyzed...
When to use Tier 3?
• Large scale site assessment and remediation
• Closure sampling (future liability)
• Long term monito...
The Secret
www.chemistry-matters.com 20
• Most companies employ ineffective QC programs
Field → Lab → Extraction → Analysis → Results
First
Run
= Discrete Sample ...
The Batch Concept (cont’d)
Field → Lab → Extraction → Analysis → Results
First
Run
= Discrete Sample = Duplicate Sample = ...
When a number is not a
number…
www.chemistry-matters.com 23
When is a number not a number?
ERROR
EXTRACTION PURIFICATION CALIBRATION
UNCERTAINTY
SHIPPING &
TRANSPORT
SEASONAL
VARIATI...
Error Propagation
ERROR
Addition/Subtraction
Multiplication/Division
www.chemistry-matters.com 25
What is Error?
• Latin for wandering or straying
QC HIGH (1-NAP)
1000
1200
1400
1600
1800
2000
2200
2400
2600
2800
3000
pa...
Accuracy and Precision
Not accurate, not precise
Precise
Accurate
www.chemistry-matters.com 27
To close to see…
www.chemistry-matters.com 28
All Results for F3 in Testing Program
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
1 6 11 16 21 26 31 36
F3-mg/kg
...
3500 4500 5500 6500 7500 8500 9500
0
1
2
3
4
5
6
7
8
Noofobservations
x
-1σ +1σ
+2σ-2σ
-3σ +3σ
Mean = 6627
SD =1142
RSD = ...
Need to know uncertainty of the
measurement
x x
Concentration Concentration
RegulatoryGuidance
RegulatoryGuidance
Uncertai...
QA-QC Statement
• For scientists, without knowing a measurement’s
uncertainty, the measurement is worthless
Concentration
...
Liability – who is responsible?
www.chemistry-matters.com 33
Responsibilities
Consultant
• Sampling
procedures
• Sampling design
• Extra QA-QC
(blanks, QA-QC
samples)
• Testing labora...
Economic Reality
Laboratory
• Faster
• Cheaper
• 100s samples
per day
• Margin is low
so only
volume
increases
revenue
Con...
Measuring and Reducing Error
(Laboratory Control)
www.chemistry-matters.com 36
Quality is measurable
• Produced several reference materials to evaluate
soil data quality
• Measured and evaluated variab...
Current Myths
• All labs are equal?
TRUE FALSE
S1 S2 S3 S4 S5 S6 S7 S8 S9 S10S11
2,000
6,000
10,000
8,000
4,590
(mg/kg)
F3...
F1 Data - Interlaboratory
0
20
40
60
80
100
120
140
All Labs Lab A Lab B Lab C Lab D Lab E Lab F Lab G Lab H Lab I
mg/kg
M...
Barium Interlaboratory
mg/kg
0
50
100
150
200
250
300
All Labs Lab A Lab B Lab C Lab D Lab E Lab F Lab G Lab H Lab I
Mean ...
Current Myths
• All labs are equal
TRUE FALSE
X
2,000
6,000
10,000
(mg/kg)
S1 S2 S3 S4 S5 S6 S7 S8 S9S10S11
8,000
4,590
F3...
Current Myths
• All labs are equal
• The same lab
will give me the
same day to day
results
TRUE FALSE
X
www.chemistry-matt...
Day to Day Variability for Two Labs
• Considerable daily variation in results for the
same sample when analyzed on differe...
Reruns and RPD Issues
Reported
Value
<10
<0.05
<0.05
329
35
94
Target
Value
1466
10.4
10.4
47
8
21
Rerun
Value
1420
10
11....
Current Myths
• All labs are equal
• The same lab will
give me the same
day to day results
• Using the same
lab for the sa...
0
5
10
15
Concentration(mg/kg)
April 2007 September 2007
Mean = 10.0
SD = 2.5
Mean = 3.0
SD = 0.2
Mean = 10
Mean = 3
Time
...
Current Myths
• All labs are equal
• The same lab will
give me the same
day to day results
• Using the same
lab for the sa...
Current Myths
• Labs are
infallible
TRUE FALSE
www.chemistry-matters.com 53
Laboratory Changes
Barium
Concentration
Nickel
Concentration
Zinc
1 2 3 4 5 6 7 8
Sample
Concentration
www.chemistry-matte...
Catching Errors
• Recent data submitted with RM
RM
RM
• Results reported
• Only caught because we knew which was
the RM
ww...
F3 RM Values
0
500
1000
1500
2000
Time
Mean
+1SD
+2SD
+3SD
-3SD
-1SD
-2SD
2067 mg/kg
853 mg/kg
RSD = 14%
n = 54
Concentrat...
F3 RM Values
0
500
1000
1500
2000
Time
Mean
+1SD
+2SD
+3SD
-3SD
-1SD
-2SD
2067 mg/kg
853 mg/kg
Concentration(mg/kg)
RSD = ...
Naphthalene RM Values
0.0
5.0
10.0
15.0
20.0
25.0
30.0
Time
Mean
+1SD
+2SD
+3SD
-3SD
-1SD
-2SD
19.5 mg/kg
2.1 mg/kg
RSD = ...
Nickel RM Values
0.0
10.0
20.0
30.0
40.0
Time
Mean
-2SD
+1SD
+2SD
-3SD
D
-1SD
+3SD
3.3 mg/kg
12.9 mg/kg
RSD = 20%
n = 52
C...
Moisture RM Values
0
10
20
30
40
50
Time
Mean
-3SD
+3SD
Z-score = +94.5
8.0%
3.8 %
RSD = 12%
n = 12
%Moisture
Soil Samplin...
Current Myths
• Labs are infallible
TRUE FALSE
X
www.chemistry-matters.com
Using Data Quality
www.chemistry-matters.com 57
Actually Using Data Quality
1. Laboratory performance, value and
accountability
2. Increased confidence in results (and
de...
Quality is measurable
Uncertainty is manageable
Change is inevitable.
www.chemistry-matters.com 59
Questions?
Contact Info:
Chemistry Matters
Court Sandau
csandau@chemistry-matters.com
www.chemistry-matters.com
Twitter – ...
Upcoming SlideShare
Loading in …5
×

How to control variability in environmental data

497 views

Published on

This presentation provides a tongue in cheek look at environmental data quality issues faced by clients and consultants alike. The presentation is not meant to blame laboratories for data issues, but to point out that errors happen and that it is the consultant that is responsible for catching those errors. Working together with the laboratories and developing a system to catch random errors that occur will increase data confidence and reliability of decisions being made for a particular site or case.

Published in: Environment, Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
497
On SlideShare
0
From Embeds
0
Number of Embeds
20
Actions
Shares
0
Downloads
2
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

How to control variability in environmental data

  1. 1. Comedy of Errors of Analytical Proportions Court Sandau, PhD, PChem Presented at CLRA AGM 2008 on August 1st, 2008
  2. 2. A word from our sponsors… www.chemistry-matters.com 2
  3. 3. Consulting Fees: $2,000 www.chemistry-matters.com 3
  4. 4. Contractor Disbursements: $5,000 www.chemistry-matters.com 4
  5. 5. Spending $7,000 for nothing: Priceless www.chemistry-matters.com 5
  6. 6. There are some things money can’t buy, for everything else there’s www.chemistry-matters.com 6
  7. 7. Changing QA-QC in the Industry • Educational Presentations/Seminars • Offering Data Quality Services – Data Quality Management Systems – Data Validation Services – Laboratory Testing and Evaluation – Customized Training Seminars: • Data Quality Basics • Tiered Approach for Data Quality Review • Basic Chemistry of Sampling www.chemistry-matters.com 7
  8. 8. Pre-amble: Setting the scene www.chemistry-matters.com 8
  9. 9. Who is protecting your liability? 0 20 40 60 80 100 LAB A LAB B LAB C SoilConcentration(mg/kg) 0 1 2 3 4 5 EnvironmentalLiability ($$inmillions) F1 Hydrocarbons www.chemistry-matters.com 9
  10. 10. Cost of Environmental Data Quality • $$$ for further investigation • $$$ for further cleanup • $$$ for potential lawsuit – especially if health implicated www.chemistry-matters.com 10
  11. 11. The Foundation of All Environmental Investigations Litigation Remediation & Risk Assessment Site Investigation Phase I, II, III Data Quality Precision, Accuracy, Reproducibility, Comparability, Representiveness ProgressionofInvestigation www.chemistry-matters.com 11
  12. 12. Levels of QA-QC Litigation Liability Protection Quantitative Qualitative - Most expensive - Highest level of QA-QC - Less limitations for use of data - Easily defensible - EPA methods (minimum requirements) - CAEAL Accreditation - Least expensive - Only basic QA-QC - Limitations of data www.chemistry-matters.com 12
  13. 13. Data Flow for Environmental Investigations Data Generation and Review QAPP Analytical requirements sent to lab Quality Assurance Project Plan (QAPP) Requirements Identified External Parties Validation of the Laboratory Analytical Data (Not the Lab) Project Scope Defined Field Activities Laboratory Analysis Data Quality Assessment Data Usability Data Repository e.g., Database (General or Project) Data may support future Project needs Samples + COC info Analytical Data Package Validated Analytical Data Data on Sample Location Field Logs Information Project Decision(s) + Supporting Data 1 2 3 4 56 78 Each investigation is its own scientific experiment… www.chemistry-matters.com 13
  14. 14. Data Quality and the Value-Chain Model • Planning • Sampling • Storage/Shipping • Analyses/Methodology • Laboratory Quality Controls • External Quality Controls • Verification • Validation www.chemistry-matters.com 14
  15. 15. Tiered Approach to Data Quality Validation - Recalculations - Complete data package review - Errors measured Preliminary Data Review - Compliance - Surficial review The “iso” process of data quality review - Completeness - All basics are in order - Information received Tier I – Process Review Tier 2 – Error Identification Tier 3 – Uncertainty Management www.chemistry-matters.com 17
  16. 16. What is QA-QC? www.chemistry-matters.com 16
  17. 17. Definitions • Data quality Summary of all the features that make the values appropriate for the intended use • Quality assurance The assembly of all planned and systematic actions necessary to provide adequate confidence that a product, process or service will satisfy given quality requirements • Quality control The operational techniques and activities that are used to satisfy quality requirements www.chemistry-matters.com 17
  18. 18. QA-QC • QA – is the “ISO process” • QA documents: – how samples taken – how they are handled – how they are to be analyzed – Which method and what criteria is to be met – Completed before the field (e.g. QAPP) • QC – ensures the data is valid • QA-QC is the last line of defense for liability protection www.chemistry-matters.com 18
  19. 19. When to use Tier 3? • Large scale site assessment and remediation • Closure sampling (future liability) • Long term monitoring (trend analysis) • Health risk assessments • Contentious sites • Litigious matters (potential) www.chemistry-matters.com 19
  20. 20. The Secret www.chemistry-matters.com 20
  21. 21. • Most companies employ ineffective QC programs Field → Lab → Extraction → Analysis → Results First Run = Discrete Sample = Duplicate Sample Basis of Data Validation Program (DVP) The Batch Concept Rerun www.chemistry-matters.com 21
  22. 22. The Batch Concept (cont’d) Field → Lab → Extraction → Analysis → Results First Run = Discrete Sample = Duplicate Sample = RM • All samples MUST be analyzed together • A batch is considered a group of samples where all are treated equal by the laboratory • Each sample set includes external QA-QC samples DVP Rerun www.chemistry-matters.com 22
  23. 23. When a number is not a number… www.chemistry-matters.com 23
  24. 24. When is a number not a number? ERROR EXTRACTION PURIFICATION CALIBRATION UNCERTAINTY SHIPPING & TRANSPORT SEASONAL VARIATIONSSAMPLING www.chemistry-matters.com 24
  25. 25. Error Propagation ERROR Addition/Subtraction Multiplication/Division www.chemistry-matters.com 25
  26. 26. What is Error? • Latin for wandering or straying QC HIGH (1-NAP) 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000 pa04106 pa04109 pa04112 pa04128 pa04131 pa04134 pa04219 pa04222 pa04238 pa04241 pa04244 pa04309 pa04312 pa04315 pa04262 pa04265 pa04328 pa04331 pa04334 pa04357 pa04360 pa04363 Concentration(pptr) CV = 7%, n=62 www.chemistry-matters.com 28
  27. 27. Accuracy and Precision Not accurate, not precise Precise Accurate www.chemistry-matters.com 27
  28. 28. To close to see… www.chemistry-matters.com 28
  29. 29. All Results for F3 in Testing Program 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 1 6 11 16 21 26 31 36 F3-mg/kg 6760 4450 9070 www.chemistry-matters.com 29
  30. 30. 3500 4500 5500 6500 7500 8500 9500 0 1 2 3 4 5 6 7 8 Noofobservations x -1σ +1σ +2σ-2σ -3σ +3σ Mean = 6627 SD =1142 RSD = 17% Data Distribution www.chemistry-matters.com 30
  31. 31. Need to know uncertainty of the measurement x x Concentration Concentration RegulatoryGuidance RegulatoryGuidance Uncertainty can be visualized with a normal distribution curve www.chemistry-matters.com 31
  32. 32. QA-QC Statement • For scientists, without knowing a measurement’s uncertainty, the measurement is worthless Concentration Measurements Guideline DL Same result – different uncertainty Are you above or below guidance? What are the implications on the long term liability for the site? www.chemistry-matters.com 32
  33. 33. Liability – who is responsible? www.chemistry-matters.com 33
  34. 34. Responsibilities Consultant • Sampling procedures • Sampling design • Extra QA-QC (blanks, QA-QC samples) • Testing laboratory (blind duplicates) • Reviewing laboratory data • Data interpretation • Evaluating data quality and doing something about it Laboratory • Appropriate methodology • Appropriate certification • Follow laboratory QA-QC • Get the analysis right Client • Understanding the issues • Footing the bill • Choosing the right consultant (and sometimes the laboratory) • Allowing the proper QA-QC to be collected • Maintain/Eliminate Liability Regulators • Setting the QA- QC guidance • Staying up to date (e.g. EPA) • Setting the standards for data quality • Enforcing non- compliance www.chemistry-matters.com 34
  35. 35. Economic Reality Laboratory • Faster • Cheaper • 100s samples per day • Margin is low so only volume increases revenue Consultant • Faster • Cheaper • Multi-task • Least amount of work possible • Margins vary – usually dependent on volume Client • Lowest price (budget or purchasing dept.) • Many sites • Leave it to the trusted consultant • Best quality with lowest margin • Value of long term liability? www.chemistry-matters.com 35
  36. 36. Measuring and Reducing Error (Laboratory Control) www.chemistry-matters.com 36
  37. 37. Quality is measurable • Produced several reference materials to evaluate soil data quality • Measured and evaluated variability in the routine laboratory parameters • Many years worth of monitoring and evaluation • Myth busters… www.chemistry-matters.com 37
  38. 38. Current Myths • All labs are equal? TRUE FALSE S1 S2 S3 S4 S5 S6 S7 S8 S9 S10S11 2,000 6,000 10,000 8,000 4,590 (mg/kg) F3 Results www.chemistry-matters.com 42
  39. 39. F1 Data - Interlaboratory 0 20 40 60 80 100 120 140 All Labs Lab A Lab B Lab C Lab D Lab E Lab F Lab G Lab H Lab I mg/kg Mean Value Maximum Value Minimum Value 95% CI www.chemistry-matters.com 39
  40. 40. Barium Interlaboratory mg/kg 0 50 100 150 200 250 300 All Labs Lab A Lab B Lab C Lab D Lab E Lab F Lab G Lab H Lab I Mean Value Maximum Value Minimum Value 95% CI www.chemistry-matters.com 40
  41. 41. Current Myths • All labs are equal TRUE FALSE X 2,000 6,000 10,000 (mg/kg) S1 S2 S3 S4 S5 S6 S7 S8 S9S10S11 8,000 4,590 F3 Results www.chemistry-matters.com 45
  42. 42. Current Myths • All labs are equal • The same lab will give me the same day to day results TRUE FALSE X www.chemistry-matters.com 46
  43. 43. Day to Day Variability for Two Labs • Considerable daily variation in results for the same sample when analyzed on different days Interday Variability in Measured Concentrations 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 Day 1 Day 2 Day 7 TolueneConc.(mg/kg) Lab A Lab B www.chemistry-matters.com 43
  44. 44. Reruns and RPD Issues Reported Value <10 <0.05 <0.05 329 35 94 Target Value 1466 10.4 10.4 47 8 21 Rerun Value 1420 10 11.6 111 16 45 %RPD 197% 198% 198% 99% 75% 71% Example Date Reported Parameter 1 Jul-07 F3 2 Aug-07 Naphthalene 3 Sep-07 Naphthalene 4 Sep-07 Barium 5 Sep-07 Nickel 6 Sep-07 Zinc www.chemistry-matters.com 44
  45. 45. Current Myths • All labs are equal • The same lab will give me the same day to day results • Using the same lab for the same site over time will ensure data integrity for that site TRUE FALSE X X www.chemistry-matters.com 49
  46. 46. 0 5 10 15 Concentration(mg/kg) April 2007 September 2007 Mean = 10.0 SD = 2.5 Mean = 3.0 SD = 0.2 Mean = 10 Mean = 3 Time Results for Naphthalene Data Validation Program www.chemistry-matters.com 46
  47. 47. Current Myths • All labs are equal • The same lab will give me the same day to day results • Using the same lab for the same site over time will ensure data integrity for that site TRUE FALSE X X X www.chemistry-matters.com 52
  48. 48. Current Myths • Labs are infallible TRUE FALSE www.chemistry-matters.com 53
  49. 49. Laboratory Changes Barium Concentration Nickel Concentration Zinc 1 2 3 4 5 6 7 8 Sample Concentration www.chemistry-matters.com 54
  50. 50. Catching Errors • Recent data submitted with RM RM RM • Results reported • Only caught because we knew which was the RM www.chemistry-matters.com 50
  51. 51. F3 RM Values 0 500 1000 1500 2000 Time Mean +1SD +2SD +3SD -3SD -1SD -2SD 2067 mg/kg 853 mg/kg RSD = 14% n = 54 Concentration(mg/kg) Soil Sampling CLPT Program www.chemistry-matters.com 51 CLPT – Continuous Laboratory Proficiency Testing
  52. 52. F3 RM Values 0 500 1000 1500 2000 Time Mean +1SD +2SD +3SD -3SD -1SD -2SD 2067 mg/kg 853 mg/kg Concentration(mg/kg) RSD = 14% n = 54 Soil Sampling CLPT Program www.chemistry-matters.com 52
  53. 53. Naphthalene RM Values 0.0 5.0 10.0 15.0 20.0 25.0 30.0 Time Mean +1SD +2SD +3SD -3SD -1SD -2SD 19.5 mg/kg 2.1 mg/kg RSD = 27% n = 44 Concentration(mg/kg) Soil Sampling CLPT Program www.chemistry-matters.com 53
  54. 54. Nickel RM Values 0.0 10.0 20.0 30.0 40.0 Time Mean -2SD +1SD +2SD -3SD D -1SD +3SD 3.3 mg/kg 12.9 mg/kg RSD = 20% n = 52 Concentration(mg/kg) Soil Sampling CLPT Program www.chemistry-matters.com 54
  55. 55. Moisture RM Values 0 10 20 30 40 50 Time Mean -3SD +3SD Z-score = +94.5 8.0% 3.8 % RSD = 12% n = 12 %Moisture Soil Sampling CLPT Program www.chemistry-matters.com 55
  56. 56. Current Myths • Labs are infallible TRUE FALSE X www.chemistry-matters.com
  57. 57. Using Data Quality www.chemistry-matters.com 57
  58. 58. Actually Using Data Quality 1. Laboratory performance, value and accountability 2. Increased confidence in results (and decisions) 3. Long term liability – you know you have the right answer 4. Proactive decision making – using uncertainty to help make decisions www.chemistry-matters.com 58
  59. 59. Quality is measurable Uncertainty is manageable Change is inevitable. www.chemistry-matters.com 59
  60. 60. Questions? Contact Info: Chemistry Matters Court Sandau csandau@chemistry-matters.com www.chemistry-matters.com Twitter – EnviroCertainty

×