Data Lost in Translation - Skewed Date Collection Prejudices Outcome
•
1 like•195 views
Presented at the 2017 RTM Conference: common sampling practices skew environmental data understating or prejudicing the contaminant mass, nature & extent and remediation program. Unskewing such data can be predictive to better match site "true" site conditions and reduce "surprises", especially environmental insurance claims.
Recommended
More Related Content
Recently uploaded
Recently uploaded (20)
Featured
Featured (20)
Data Lost in Translation - Skewed Date Collection Prejudices Outcome
- 1. Grand Rapids, MI Joseph Berlin, PE, QC, CP Data Lost in Translation – Data Variance, Environmental Forensics and Sampling Practices Presented to:
- 2. Speaker Background – Joseph Berlin, PE, QC, CP 2 Mr. Berlin’s work focuses on the application of scientific and forensic principles to better understand and mitigate environmental contamination and the associated financial implications. Mr. Berlin’s practice focuses on the development and implementation of forensic environmental investigations, including source and origin, migration scenarios, data validity determinations and environmental liability and reserve estimates. Mr. Berlin is a registered Professional Engineer in several states, a Certified Professional (Ohio VAP#351) and a member of the National Academy of Forensic Engineers. Mr. Berlin has BS and MS degrees in Civil Engineering, MBA, plus further graduate work in industrial hygiene and accounting. Mr. Berlin’s career spans over 25 years and has run from DOD “secret” projects to cases with Fortune 500 oil companies to insurance claims for mom-and-pop gas stations.
- 3. Soil, groundwater and vapor sampling may not reflect site conditions…Guess work or Predictable? Let’s take a look at a simple example 3
- 4. 4 Prejudiced data defines outcome Method “B” Prejudiced Method Variance on THE Method OR Method “A” Proper Method
- 5. 5
- 6. What if… 6 Sample results were actually prejudiced to reflect only 5% to 50% of the actual impact The volume of impacted material and contaminant mass were 2 to 10 times greater than estimated
- 7. Data dazed and confused 7 Since I only really care about my bottom line why does this matter?
- 8. Risk Management – Data Review • Assessment of prior data • Validation of Areas of Concern • Exclusions/Carve Outs • Deductible Escrow Size/Structure • Voluntary Program Impact • Remediation Method/Structure/Time • Remediation Cost Estimates 8 Defer no time, delays have dangerous ends. - William Shakespeare
- 9. How are Datasets Prejudiced • Sample collection skewed to “clean” areas • Non-preserved samples (5035 use) • Field sample handling • Use of Method “B” sample • Many others 9
- 10. Original Case - Revisited 10 • Tanker spill along a highway • adjacent to a Great Lake • Loss of over 4,000 gallons of gasoline • Residential area • Shallow groundwater (8’ deep) • Closest home - 100 feet from spill area • Samples collected during source removal • Samples split: Methods “A” and “B” • Modeled data for “A” and “B” samples Included in full version (see last slide to obtain full presentation)
- 11. Original Case - Revisited 11 • Tanker spill along a highway adjacent to a Great Lake • Loss of over 4,000 gallons of gasoline • Residential area • Shallow groundwater (8’ deep) • Closest home - 100 feet from spill area • Samples collected during source removal • Samples split: Methods “A” and “B” • Modeled data for “A” and “B” samples Included in full version (see last slide to obtain full presentation)
- 12. Original Example – Data Variance 12 Method Benzene (ug/kg) Total Contaminant Mass (TCM) Vapor Threshold Exceeded? A 0 to 81,000 7,200 lbs. Yes B 0 to 1,500 86 lbs. No Data Parsing- Method “A”/Method “B”: • Factor (Total Contaminant Mass): 2-6 • Factor (Benzene): 5-54 Included in full version (see last slide to obtain full presentation)
- 13. 13 Method “B” Sample Method “A” Sample Laboratory Chromatograms (For a single data point in population) Note: 1. Lighter ends not present in Method “B” sample 2. These are the same sample locations only the collection method varied Included in full version (see last slide to obtain full presentation)
- 14. 14 Method “B” sampling Leads to wrong/inefficient CSM & remedy NOT Method “A” Actual CSM Method “B” Skewed CSM Conceptual Site Models
- 15. Preference Skews Timeline and Cost 15 Method Volume of Impact (CY) Remedial Comments Projected Cost ($) A Over 3,000 In-situ treatment likely Large source area > $250,000 B Less than 300 Spot removal < $25,000 • Method “A” v. “B” – significant variance • Method “B” – WRONG results/plan Included in full version
- 16. How might I know? Some indicators… • What was clean is now dirty • What was dirty is now clean • A small issue becomes bigger • A big issue becomes smaller • Result variance over time? Firm? Field crew? 16 “Commonly, the variance is best explained by sample collection methods.”
- 17. 17 • Understand risk of data prejudice • Become aware of skewing methods • Request certification & reliance of reports • Consider use of normalized CSM • Call “Skewed” consultants to task In case I care…how do I protect myself? “You know my method. It is founded upon the observation of trifles.” ― Sir Arthur Conan Doyle
- 18. In Closing • Consider the reliability/validity of ALL VOC data • Do not underestimate the importance of methods defining outcome (the trifles of data collection) • Consider terms to include select documentation (SOPs, chromatograms, training logs) • Ensure contractual Reliance of Consultant work • Consider use of normalized data CSM/RCE 18 Included in full version (see last slide to obtain full presentation)
- 19. Follow Up If you wish to have the complete version of this presentation or have any questions or comments please contact Rich Spehar, PE, Joseph “Joe” Berlin, PE, EP, CP or Marty Janowiak (bldi@bldi.com) at our main office at 616-459-3737. Also refer to www.ohioenvironmentalblog.com or www.michiganenvironmentalblog.com 19 Copyright, BLDI, Inc. 2015, all rights reserved.
Editor's Notes
- Thank you Jeff. Good afternoon. My name is Joe Berlin. I am a professional engineer in several states, member of National Academy of Forensic Engineers. Our work includes transactional due diligence, investigation/remediation and forensic engagements.
- A large part of my project portfolio includes agency negotiations, program development and forensic reviews. I’m a data guy. When I review data I look at the data and assess if it makes sense. Was is possibly skewed towards an opinion or outcome This presentation builds on a research program we have been undertaking for the past two years The goal, at the end, is for us to understand and appreciate that datasets, especially soil data, are often skewed to knowingly and unknowingly There are methods to assess such data and, especially related to environmental risk assessment, redevelopment and insurance, how can we better assess data.
- Although many understand or assume that all sampling is conducted the same or has little variance by the end of this presentation I hope you understand that such assumptions are limited at best and the significant impact one field method can have on an environmental program. For example, during our research we found several large national firms use Method B as standard practice. SO what many would say? Well, let’s look at some actual data. Today, we are only going to discuss soil sampling practices, a little history behind current methods and how that influences financial models. There is an inherent assumption in most environmental investigations that unanticipated outcomes may be encountered. Although sometimes true, such negative outcomes are much more predictable than many think. Consider, in many environmental Conceptual Site Models there is an inherent probability of “negative unanticipated” outcomes built into the model. What if that probability (i.e. risk) can be better understood and predicted?
- Ok, if we are willing can we do a simple show of hands for those of us upon receiving a data package questioned the sample results and, thereby, the entire environmental program? How about having “unexpected” or “unforeseen” environmental surprises? These are generally negative surprises aren’t they. How many know what 5035 field preservation is? Thank you. SO it seems that many of us have had such an experience. The model shown is REAL data. These are split samples (only the locations with detectable concentrations are shown here for simplicity). We are looking at the same site with only one difference…the method of collection of the samples. Dataset or Method “B” was collected per a common but WRONG industry practice by sampling from the field screening (headspace sample) Now we understand that many would say, “Hey I want the Method “B” sample results. However, it’s not like the contamination magically disappeared. Voila! Gone. ” The problem is that the Method B results often grossly underestimate the volume, concentration and location of contamination. As an engineer I tend to look at the work in numbers and grids. If my number of affected grids is 10% of reality and the values in the grids are certainly less than 50% of reality I know I have a serious data discrepancy. In our world that means money, time and risk not included in the model (we call it the Conceptual Site Model or CSM). Using common models there are relatively simple methods to correct or enhance the CSM. For example, during our research we found several large national firms use Method B as standard practice. SO what many would say? Well, let’s look at some actual data. Today, we are only going to discuss soil sampling practices, a little history behind current methods and how that influences financial models. There is an inherent assumption in most environmental investigations that unanticipated outcomes may be encountered. Although sometimes true, such negative outcomes are much more predictable than many think. Consider, in many environmental Conceptual Site Models there is an inherent probability of “negative unanticipated” outcomes built into the model. What if that probability (i.e. risk) can be better understood and predicted?
- So…we now come to examples of bad situations caused by contamination not previously assessed, missed or avoided. How does this happen? Mistake? OR…prejudicial data collection methods. Were these events truly unavoidable? We are really talking about prejudicial data collection methods. Whether a lot of consultants know it or not…they collect samples in a manner to prejudice the results And ALWAYS to minimize the issue. Here’s what we know: Vapor concentrations in soil are temporal and can vary significantly over a year, month, week or even day Initial VI screening often uses soil results as a baseline or starting point In fact, PM is now defaulting to soil v. vapor because of the stricter vapor standards. Everything isn’t just about VAPOR today, but vapor does drive the preponderance of environmental mitigation programs today. Scary Things Headlines of bad things Delays in development Increase Costs Stigma Dramatic adverse impact on ROI
- I submit that after we finish this discussion that you may be able to identify a project or two where you thought something was odd (sample results didn’t make sense). Submit that a lot of consultants collect soil samples, we’ll get to vapor and groundwater another time, that skews results and are likely unaware that they inherently do so. IN fact, many regulators do not understand it or care (Ohio EPA). What happens when I run into more “bad stuff” that could have been found earlier? Would I have walked away? Written different terms? Filed suit? We will submit that in many such cases of finding much greater impact than estimated is based on insufficient data parsing (excluding certain data or using select multipliers) and improperly collected samples. If you are the lender, developer or underwriter ensure you agreement allows you to request any and all documentation. Why? Because you want this information to ensure their CSM is accurate. Ok, prior to the deal start by asking for: SOPS staff resumes who were on-site collecting data AND especially the CSM as this should tell you have they looked at the data. Later, if there is an you can also ask for: field notes, field logs, Laboratory chromatograms This initial scan of information can often identify the reason(s) for the difference in impacted media volume, concentration, location and contaminant mass.
- Point: the various data inputs, along with Environmental Sample data, is assumed to be reliable and collected in a consistent manner. If the Environmental data is collected to skew as “Clean” how does that influence the Financial Model? It certainly affects the exposure assessment. Success factors and points over view: Differing points of view and data qualification Developer Lender Regulator/Agency Seller Consultant A Consultant B Risk Managers Underwriter And Workout and Claims
- Forensic methods, including Data Normalization, can be used to better assess the various risk management issues prior to, during or on the back end of environmental programs. We often use data normalization which should probably be defined as it can be a powerful tool. Data is grouped based on a number of factors (date collected, field preservation or not, lab hold time, consultant group, media) Based on the data group the data can then be normalized (or corrected) based on data group and factors likely influencing the output The range of normalization factors can be input to the model to provide various outputs depending on the engagement For example: if we are looking at benzene as the cleanup driver, we might look at sample results from 1998 and utilize a range of factors of 4 to 10 in developing a more rational CSM. Remember, there may be other factors, including is the newest data skewed as well. So, if you better understood the data (normalized if you will) you could better structure the development, cost items out, provide a more accurate (not blue sky) timeline Money is one thing but time, time, is something we can never get more of.
- Many of us here have questioned or question datasets based on various, often, gut feelings. The prejudicial data, especially old data, may not necessarily have been done on purpose. But assessing that data today without normalizing the data provides an inaccurate model. Ultimately a disciplined approach to data parsing for input to a CSM can better align data with outcomes. The real world impact of environmental problems Delays Increased cost Environmental agreements/indemnities Stigma Since I only really care about my bottom line why does this matter? Unforeseen Costs and Delays…or Not? “You know my method. It is founded upon the observation of trifles.” ― Arthur Conan Doyle, The Boscombe Valley Mystery
- You can see the description. Soil conditions: a very nice medium sand The initial source removal started within 12 hours of the accident The uniformity of the soil and application/spill of gasoline provided very good comparisons for the Method A/B co-located samples
- You can see the description. Soil conditions: a very nice medium sand The initial source removal started within 12 hours of the accident The uniformity of the soil and application/spill of gasoline provided very good comparisons for the Method A/B co-located samples
- The results speak for themselves. The data for this case is consistent with data from the dozen or sites for which we have collected data on Method “A” v. Method “B” Let’s remember this isn’t a choice of which data to select. The conditions are the same regardless of which sample set used. One set (Method “A”) better reflects the conditions. The other set (Method “B”), although possibly preferred for transactional purposes, will have issues generally only exposed well after the transaction when the risk/cost had already been transferred to purchaser, developer, lender, insurer and tenants. Notes: Remember these are “split” co-located samples.
- You can see the description. Soil conditions: a very nice medium sand The uniformity of the soil and application/spill of gasoline provided very good comparisons for the Method A/B co-located samples Comparing the chromatograms and by extension the CSMs (on the next slide) between Method A v. Method B and, ultimately, the remedy is many cases
- Ok, if we are willing can we do a simple show of hands for those of us upon receiving a data package questioned the sample results and, thereby, the entire environmental program? How about having “unexpected” or “unforeseen” environmental surprises? These are generally negative surprises aren’t they. How many know what 5035 field preservation is? Thank you. SO it seems that many of us have had such an experience. The model shown is REAL data. These are split samples (only the locations with detectable concentrations are shown here for simplicity). We are looking at the same site with only one difference…the method of collection of the samples. Dataset or Method “B” was collected per a common but WRONG industry practice by sampling from the field screening (headspace sample) Now we understand that many would say, “Hey I want the Method “B” sample results. However, it’s not like the contamination magically disappeared. Voila! Gone. ” The problem is that the Method B results often grossly underestimate the volume, concentration and location of contamination. As an engineer I tend to look at the work in numbers and grids. If my number of affected grids is 10% of reality and the values in the grids are certainly less than 50% of reality I know I have a serious data discrepancy. In our world that means money, time and risk not included in the model (we call it the Conceptual Site Model or CSM). Using common models there are relatively simple methods to correct or enhance the CSM. For example, during our research we found several large national firms use Method B as standard practice. SO what many would say? Well, let’s look at some actual data. Today, we are only going to discuss soil sampling practices, a little history behind current methods and how that influences financial models. There is an inherent assumption in most environmental investigations that unanticipated outcomes may be encountered. Although sometimes true, such negative outcomes are much more predictable than many think. Consider, in many environmental Conceptual Site Models there is an inherent probability of “negative unanticipated” outcomes built into the model. What if that probability (i.e. risk) can be better understood and predicted?
- Some consultants/firms are well known for giving “better” (lower) results. Today, especially with vapor intrusion, a developer, risk manager or underwriter, takes on greater downside risk with the use of “prejudicial” data. Since a case can be made even more strongly today that a person should have known about such “prejudicial” data, how can you say you didn’t know. Areas of contamination are documented using sampling…however, the sampling specifics can dictate the results Does the data drive the results? Consider? (see: single dataset table below)
- You are often provided reports (Phase II ESAs, etc.) so doing your own Phase II or investigation may not be feasible. However, you can request specific information to plug holes in the CSM. I was asked about Forensic engagements and cost, especially to assess a CSM or RCE. Get the basic information during underwriting/risk review If there is a real concern (e.g. blown RCE budget) then get the second set of information, as available We have another set of questions to identify transactional and consultant grouping With this information the data prejudice, gaps and a simple updated, more rational, data-driven CSM can be developed Cost: less than $5,000 Time: in as little as 2 weeks Consider pre-transaction Baselining Environmental forensic lab work as low as $1,000 More commonly add: $5,000 to $10,000 Time: usually 4 weeks
- Get the basic information during underwriting/risk review process. Many of us have people/firms we prefer to work or associate with. If this type of information is important, and you want reliable data, you can do some simple things to increase the reliability. In the engagement or underwriting process provide, much like lenders, that under separate cover, obtain some non-standard information With this information the data prejudice, gaps and a simple updated, more rational, data-driven CSM can be developed