This document discusses strategies for measuring the impact of the National Environmental Education Foundation's (NEEF) programs. It identifies existing datasets NEEF could leverage and new data collection methods. Some key points discussed include: - Tracking grant applications, recipients, and feedback to assess impact on teachers and students - Combining NEEF program data with external environmental datasets - Standardizing common data collection across all NEEF programs - Leveraging tools like surveys, Google Analytics, and social media to track engagement and behavior change - Considering best practices for statistical validity like reducing omitted variables and measurement errors The document provides examples of specific data tracking components and collection methods for different NEEF programs to help evaluate

1. Data Without Borders

2.  By 2022, 300 million Americans actively use
environmental knowledge to ensure the well
being of earth and its people.
 Measurement:
◦ The baseline environment knowledge as the starting
point to reach 300 million Americans
◦ How to count 300 million Americans as actively using
environmental knowledge and what is the definition of
“actively using”?
◦ Individual behavior change based on increased
environmental knowledge

3.  Existing Datasets
◦ Is there any existing data indicative of some of these actions?
◦ Project - Search for these, list their sources, description of one row of the data.
 Advertising Research ( to understand how to measure 300 million
Americans reach)
◦ How do advertising agencies measure reach? Print ads / media ads. 1st order /
2nd order reach.
◦ How do advertising agencies measure action? Who actually bought something
they wouldn't have before because of an ad?
◦ How do we collect digital information that might answer these questions?
Facebook / Google ad models?
 Behavioral Analysis
◦ Were people more likely to do something after working with NEEF than before?
◦ How do you know? (Baselines, control groups, A/B testing)
◦ How do you measure reach vs. actual action?
 Standardize Data Collection
◦ What data or approaches are common across all programs?
 Technology for Data Collection
◦ Are there digital metrics that are correlated with (or directly indicative of)
activity?
◦ Google Analytics, Twitter campaigns, social media and social networks

4.  Leverage existing data sets of NEEF to
increase measurement metrics
◦ Identify enhancements to current data collection
methods
◦ Identify new data collection methods for NEEF programs
 Identify external data sets to help with
impact measurement
◦ Combine external data with internal data sets to
create measurable variables

5.  NEEF’s existing data sets
◦ We identified datasets that NEEF already has and may be able to investigate further.
◦ NEEF keeps records of grant recipients. These may contain insights regarding how
grants turned out.
◦ Google Analytics already collects some information that you may not have noticed.
In particular, it already tracks clicks on links to external websites, so you can see
whether people went to other sites after seeing the NEEF site.
◦ NEEF uses ConstantContact for its newsletter. ConstantContact may already provide
analytics about the readership of the newsletter.
 External data sets
◦ We found several large surveys that gather data on activities related to NEEF's
mission. NEEF may be able to use these as measures of relevant human
behavior in studies of the effect of their campaign.
◦ Also, there are a lot of environmental data on brownfields or air quality. These
may not be very helpful for measuring how human behavior changes, but they
may help NEEF to target their campaigns.
 Behavioral Analysis
◦ Determining whether NEEF campaigns had an impact can be easier if studies
are designed in a particular way. We developed some suggestions of statistical
concerns that NEEF might want to consider.

6. External Data Measurement Source
Air Quality, Brownfields Dataverse
Public transportation use, Car Desired outcomes Transportation Census Bureau
pooling rate, etc (Carpooling rate available in
ACS Table S0802)
Texas Transportation Institute
American Public
Transportation Association
Public Land Visits Visits by state, park, etc. NPS
Analyze the visits by
comparing before/after/on
Public Lands Day
Water conservation, Water EPA
diversion (recycling) (finest scale= Census Regions)
Environmental Pew, Roper
Attitudes/Knowledge (all at national scale)
American time use American Time Use Survey
Farmers Market USDA

7.  Goal: Measure quality of care improvements through advanced environmental health
knowledge. Focus on Institution level decision making (Kaiser, Mayo etc.)- pediatric
medicine and nursing.
 Actors: Primary Care physicians in the health system under measurement
 Future State Process:
 Primary Care --> Environment History + Clinical Data--> Environment Knowledge->
Selective Interventions-> Improve disease prognosis/clinical outcome
 Implementation Plan:
 1. Partner with Integrated Delivery Systems:
Kaiser, Intermountain, Geisinger, Mayo, Group Health or HHS Beacon community.
 http://healthit.hhs.gov/portal/server.pt?open=512&objID=1805&parentname=Commun
ityPage&parentid=2&mode=2&cached=true
 2. Environment History to become part of digitized medical record 3. Track the clinical
interventions based on environmental history information 4. Environmental Health
knowledge through awareness and continuing medical education program. 5. Measure
intervention data to evaluate effectiveness of the education programs.
 Incentives for Health Systems: The cost for environmentally attributable childhood
disease is $54.9 billion. This strategy is less expensive than current alternative.

8.  Things To Think About When Collecting Data (Reference: Stock and Watson 2003)
◦ The end goal is to isolate the causal relationship of interest.
◦ In this case how NEEF participation changes behaviors in the “Primary Audience – Target
Professionals” and the “Secondary Audiences” – students, viewers, volunteers, and patients.
◦ The idea is to make this equation as statistically valid as possible. We can look at validity in a
few ways.
 Omitted Variable Bias
◦ This is a fancy way of saying don’t leave out factors that affect “behavior change” that is in any
way related to “NEEF Participation”.
 Inappropriate Sample Selection
◦ This can happen if the sample is chosen such that it is too small or it does not truly represent
the population you want.
 Errors in Variables
◦ This happens if there are errors in the way you measure your variable. Some examples of how
this can happen:
◦ Survey asks teachers to recall how often they have used NEEF materials in the past year. They
guess.
◦ The person plugging in data was half asleep and starts incorrectly entering data into the
database.
 Incorrect Specification of the Functional Form
◦ A technical way of saying, if the true relationship is in some form, (say a nonlinear relationship
between “Years of participation in NEEF” and say “number of site visits”) but our model is
linear, that would threaten our estimators in the equation above.

9. NEEF Program Tracking Components Data Inference from collected data
Programs Components Collection
Method
1. Track applications Assesses Teachers impact in
-- Where applications are coming from? classroom and potential
-- Applications NOT received from certain influence in the community
regions
2.Teachers Information (school, city,State, Combine external data from EPA
Classroo Grants to
grade taught) Survey about environmental conditions
m Earth teachers
3.Curriculum developed (pertaining to of a region and number of
environment) applications received (or not
--Course ratings received) from that region -> To
4.Number of students in teacher's class determine where to target NEEF's
5. Teachers Feedback/Comments campaign efforts
Assesses grant awarded students
impact in their grade and
1.Track applications
potential influence in the
-- Where applications are coming from?
community
-- Applications NOT received from certain
regions
Planet Grants to Combine external data from EPA
Student Information (school, city, Survey
Connect students about environmental conditions
State,grade)
of a region and number of
2.Project Details
applications received (or not
3.Internship Details
received) from that region -> To
4. Students Feedback/Comments
determine where to target NEEF's
campaign efforts
1. Student's performance - Quiz score
Level of student's engagement
Planet 2.Performance trends over time Google
Quizzes and number of interested
Connect 3. Number of students taking the quiz Analytics
students
4. Number of students taken quiz over time

10. NEEF Program Tracking Components Data Inference from collected data
Programs Components Collection
Method
1. Meteorologists Information
-- Station
-- Location - city, state Google
-- Twitter tags Analytics
-- Time of broadcast
2. Broadcast Reach Estimation Survey
Assesses meteorologists potential
Website & 3. Website Traffic
Earth Gauge influence in the
Meteorologist 4.Meterologist Feedback - thoughts on Custom
community/region
new program directions and products built
5. Track meteorologists tweets related to software to
NEEF track tweets
6.Scan transcripts of local TV stations to NodeXL
determine if meteorologist used NEEF
materials
Google
1. EE week Partners Information EE week participation and
National Analytics
2.Website Traffic participants' demographics
Environment Website &
3.Participant Location Check -Ins (Four
Conservation Event Survey
square) Participants interest in specific
Week
4. Check-Ins Analytics ( Four square) environment topics
Four Square
National
Media Attendee Information Webinar Non-profits that work with public
Public Lands
Webinar Pre and post webinar questionnaire Polls lands
Day

11. NEEF Program Tracking Components Data Inference from collected data
Programs Components Collection
Method
National National Public Lands Day
Volunteer Survey
Public Lands Event Survey participation information and
Program Partner Survey
Day interest
Track the recipients of newsletters
(HTML) To understand how people are
Google
All Programs Newsletters --inject tracker into HTML newsletter being directed to NEEF's websites
Analytics
--track if people are clicking on NEEF and if newsletters are being read
websites from emails or other websites
To understand what information
Search keywords for Google Trends people are looking for
Visitor Online
-- to track what people are searching in Google -- are they interested in EE week/
All Programs Search
google outside of NEEF's websites Analytics National Public Land's day
Behavior
-- when do they start looking for
the information
Use the survey comments or any field
with free form text to quickly identify
Wordle
most commented topic areas - Create Quick and easy to use to visually
All Programs Survey Word Cloud analyze comments from survey
Google
results
Refine
Cleanup comments or free form text in
survey using Google Refine

13.  DC Data Dive Wiki
 Internal Validity
 Data Management Cheat Sheet